DAS 3.1.0 minimum HA setup fails to persist half of the events

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

DAS 3.1.0 minimum HA setup fails to persist half of the events

Gayan Yalpathwala
Hi team,

Please note that all analytics datasources are pointed to an HBase cluster. I am publishing a large number of events to both the nodes (client-side load balanced) from an ESB proxy service using PublishEvent mediator. Here, I observed on DAS end, exactly half of the events are not getting persisted and below error is thrown multiple times on DAS master node.

TID: [-1234] [] [2017-06-14 09:32:49,673] ERROR {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  Unable to process events for tenant :-1234 on stream:/Receiver/PageVisitsReceiver {org.wso2.carbon.event.processor.manager.core.internal.EventHandler}
java.lang.NullPointerException
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher.process(QueueInputEventDispatcher.java:102)
at org.wso2.carbon.event.processor.manager.core.internal.EventHandler$1.receive(EventHandler.java:161)
at org.wso2.carbon.event.processor.manager.commons.transport.server.TCPEventServer$ServerWorker$ListenerProcessor.run(TCPEventServer.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Any thoughts on this behavior?

Thanks,

--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: +94 71 868 2704
US: +1 650 695 2324


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: DAS 3.1.0 minimum HA setup fails to persist half of the events

Gimantha Bandara
Hi Gayan,
Is HA enabled in event-processor.xml? Also is pageVisitsReceiver receiver deployed in both servers?
On Wed, Jun 14, 2017 at 4:34 PM Gayan Yalpathwala <[hidden email]> wrote:
Hi team,

Please note that all analytics datasources are pointed to an HBase cluster. I am publishing a large number of events to both the nodes (client-side load balanced) from an ESB proxy service using PublishEvent mediator. Here, I observed on DAS end, exactly half of the events are not getting persisted and below error is thrown multiple times on DAS master node.

TID: [-1234] [] [2017-06-14 09:32:49,673] ERROR {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  Unable to process events for tenant :-1234 on stream:/Receiver/PageVisitsReceiver {org.wso2.carbon.event.processor.manager.core.internal.EventHandler}
java.lang.NullPointerException
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher.process(QueueInputEventDispatcher.java:102)
at org.wso2.carbon.event.processor.manager.core.internal.EventHandler$1.receive(EventHandler.java:161)
at org.wso2.carbon.event.processor.manager.commons.transport.server.TCPEventServer$ServerWorker$ListenerProcessor.run(TCPEventServer.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Any thoughts on this behavior?

Thanks,

--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: +94 71 868 2704
US: +1 650 695 2324

--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: DAS 3.1.0 minimum HA setup fails to persist half of the events

Gayan Yalpathwala
Hi Gimantha,

HA is enabled in event-processor.xml correctly and all artifacts are deployed on both servers. This happens the other way round too when the inactive node is made the master.
I had a debug session with Tishan and identified that the below error is logged recursively on the inactive node every time.

java.util.NoSuchElementException
at java.util.AbstractQueue.remove(AbstractQueue.java:117)
at org.wso2.carbon.event.publisher.core.internal.BlockingEventQueue.remove(BlockingEventQueue.java:144)
at org.wso2.carbon.event.publisher.core.internal.EventPublisher.sendEvent(EventPublisher.java:246)
at org.wso2.carbon.event.publisher.core.internal.EventPublisher.onEvent(EventPublisher.java:294)
at org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:157)
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher$QueueInputEventDispatcherWorker.run(QueueInputEventDispatcher.java:142)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Thanks,

On Wed, Jun 14, 2017 at 7:13 PM, Gimantha Bandara <[hidden email]> wrote:
Hi Gayan,
Is HA enabled in event-processor.xml? Also is pageVisitsReceiver receiver deployed in both servers?
On Wed, Jun 14, 2017 at 4:34 PM Gayan Yalpathwala <[hidden email]> wrote:
Hi team,

Please note that all analytics datasources are pointed to an HBase cluster. I am publishing a large number of events to both the nodes (client-side load balanced) from an ESB proxy service using PublishEvent mediator. Here, I observed on DAS end, exactly half of the events are not getting persisted and below error is thrown multiple times on DAS master node.

TID: [-1234] [] [2017-06-14 09:32:49,673] ERROR {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  Unable to process events for tenant :-1234 on stream:/Receiver/PageVisitsReceiver {org.wso2.carbon.event.processor.manager.core.internal.EventHandler}
java.lang.NullPointerException
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher.process(QueueInputEventDispatcher.java:102)
at org.wso2.carbon.event.processor.manager.core.internal.EventHandler$1.receive(EventHandler.java:161)
at org.wso2.carbon.event.processor.manager.commons.transport.server.TCPEventServer$ServerWorker$ListenerProcessor.run(TCPEventServer.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Any thoughts on this behavior?

Thanks,

--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: <a href="tel:071%20868%202704" value="+94718682704" target="_blank">+94 71 868 2704
US: <a href="tel:+1%20650-695-2324" value="+16506952324" target="_blank">+1 650 695 2324

--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : <a href="tel:071%20496%201919" value="+94714961919" target="_blank">+94714961919



--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: +94 71 868 2704
US: +1 650 695 2324


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Loading...