Re: Regarding deployment of siddhi application in cluster

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: Regarding deployment of siddhi application in cluster

Chiran Fernando
Adding dev

On Mon, Jun 25, 2018 at 12:33 PM Chiran Fernando <[hidden email]> wrote:
Hi Babravahan,

I configured your deployment with one manager and two resource nodes and deployed specific siddhi file as in [1]. It was deployed as expected in a single resource node.

Just clarifying few details.

In distributed siddhi application, an execution group is a single unit of execution. For each execution group, a specified number of parallel Siddhi application instances are created. This is done via the @dist annotation. As per your query,  no specific number of parallel instances are specified for group1, and therefore, only one instance is created for it at runtime by default. Therefore it will only deployed in a single resource node.

`@App:name("siddhi-app-name-1")
  @App:description("rule for siddhi-app-name-1")

  @source(
       type="kafka",topic.list="input-topic",
       group.id="group1",threading.option="single.thread",
       bootstrap.servers="<list of kafka brokers>",
       @map(type="json"))
  define stream inputstream (tag_id string, tag_type int, timestamp long, value double, quality int, datatype string);

  @sink(
      type="kafka",
      topic="output-topic",
      bootstrap.servers="<list of kafka brokers>",
      @map(type="json"))
  Define stream outputstream (message string);

  @dist(execGroup="group1")
  from inputstream[name=="penta"]
  select tag_id as message
  insert into outputstream;`

Furthermore, WSO2 Stream Processor v4.0.0 uses round-robin to allocate each partial siddhi app.
(From v4.2.0 it supports resource aware allocation). 

Regarding your second question, no configurations needs to be done wrt kafka. If you use different group_ids to subscribe to the same topic, each consumer will receive all the messages.

Please verify the kafka server and client versions.  Refer [2]



On Sun, Jun 24, 2018 at 8:32 PM Babravahan Babravahan <[hidden email]> wrote:
wanted to add, I am using wso2 SP version 4.0.0.

On Sun, Jun 24, 2018 at 6:45 PM, Babravahan Babravahan <[hidden email]> wrote:
Hi,

I have some issues regarding deploying the siddhi application on cluster:

I am planning to support more that 200 rules , each rule having multiple queries in it.
so i had setup a fully distributed siddhi SP cluster with one manager and two workers (in future i want it to be expandable to more workers if number of rules increase). since rule are all unique i am pushing them in batch of 50 each to manager node.

I am facing some issue when i do this.

1.rules pushed can be seen in one worker node's siddhi-files folder but no rule seems to appear in the other worker node's siddhi-files folder.(even though the manager node log seems to show the deployment in it)
2. since all  siddhi rule pushed, is registered to listen to same topic with only their group id being different . I observe that when in push events via this topic only few of rules get executed and output can be seen.

So I need to know is there any thing else not mentioned in https://docs.wso2.com/display/SP400/Fully+Distributed+Deployment that i need to done, w.r.t kafka being the  input and output stream client.

I have asked a question in stackoverflow as well:


Thanks and regards,
babravahan





--
Chiran Fernando
Software Engineer | WSO2

Mobile : +94716007376




--
Chiran Fernando
Software Engineer | WSO2

Mobile : +94716007376



_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev