DAS 3.1.0 minimum HA setup fails to persist half of the events

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

DAS 3.1.0 minimum HA setup fails to persist half of the events

Gayan Yalpathwala
Hi team,

Please note that all analytics datasources are pointed to an HBase cluster. I am publishing a large number of events to both the nodes (client-side load balanced) from an ESB proxy service using PublishEvent mediator. Here, I observed on DAS end, exactly half of the events are not getting persisted and below error is thrown multiple times on DAS master node.

TID: [-1234] [] [2017-06-14 09:32:49,673] ERROR {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  Unable to process events for tenant :-1234 on stream:/Receiver/PageVisitsReceiver {org.wso2.carbon.event.processor.manager.core.internal.EventHandler}
java.lang.NullPointerException
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher.process(QueueInputEventDispatcher.java:102)
at org.wso2.carbon.event.processor.manager.core.internal.EventHandler$1.receive(EventHandler.java:161)
at org.wso2.carbon.event.processor.manager.commons.transport.server.TCPEventServer$ServerWorker$ListenerProcessor.run(TCPEventServer.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Any thoughts on this behavior?

Thanks,

--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: +94 71 868 2704
US: +1 650 695 2324


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: DAS 3.1.0 minimum HA setup fails to persist half of the events

Gimantha Bandara
Hi Gayan,
Is HA enabled in event-processor.xml? Also is pageVisitsReceiver receiver deployed in both servers?
On Wed, Jun 14, 2017 at 4:34 PM Gayan Yalpathwala <[hidden email]> wrote:
Hi team,

Please note that all analytics datasources are pointed to an HBase cluster. I am publishing a large number of events to both the nodes (client-side load balanced) from an ESB proxy service using PublishEvent mediator. Here, I observed on DAS end, exactly half of the events are not getting persisted and below error is thrown multiple times on DAS master node.

TID: [-1234] [] [2017-06-14 09:32:49,673] ERROR {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  Unable to process events for tenant :-1234 on stream:/Receiver/PageVisitsReceiver {org.wso2.carbon.event.processor.manager.core.internal.EventHandler}
java.lang.NullPointerException
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher.process(QueueInputEventDispatcher.java:102)
at org.wso2.carbon.event.processor.manager.core.internal.EventHandler$1.receive(EventHandler.java:161)
at org.wso2.carbon.event.processor.manager.commons.transport.server.TCPEventServer$ServerWorker$ListenerProcessor.run(TCPEventServer.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Any thoughts on this behavior?

Thanks,

--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: +94 71 868 2704
US: +1 650 695 2324

--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: DAS 3.1.0 minimum HA setup fails to persist half of the events

Gayan Yalpathwala
Hi Gimantha,

HA is enabled in event-processor.xml correctly and all artifacts are deployed on both servers. This happens the other way round too when the inactive node is made the master.
I had a debug session with Tishan and identified that the below error is logged recursively on the inactive node every time.

java.util.NoSuchElementException
at java.util.AbstractQueue.remove(AbstractQueue.java:117)
at org.wso2.carbon.event.publisher.core.internal.BlockingEventQueue.remove(BlockingEventQueue.java:144)
at org.wso2.carbon.event.publisher.core.internal.EventPublisher.sendEvent(EventPublisher.java:246)
at org.wso2.carbon.event.publisher.core.internal.EventPublisher.onEvent(EventPublisher.java:294)
at org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:157)
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher$QueueInputEventDispatcherWorker.run(QueueInputEventDispatcher.java:142)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Thanks,

On Wed, Jun 14, 2017 at 7:13 PM, Gimantha Bandara <[hidden email]> wrote:
Hi Gayan,
Is HA enabled in event-processor.xml? Also is pageVisitsReceiver receiver deployed in both servers?
On Wed, Jun 14, 2017 at 4:34 PM Gayan Yalpathwala <[hidden email]> wrote:
Hi team,

Please note that all analytics datasources are pointed to an HBase cluster. I am publishing a large number of events to both the nodes (client-side load balanced) from an ESB proxy service using PublishEvent mediator. Here, I observed on DAS end, exactly half of the events are not getting persisted and below error is thrown multiple times on DAS master node.

TID: [-1234] [] [2017-06-14 09:32:49,673] ERROR {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  Unable to process events for tenant :-1234 on stream:/Receiver/PageVisitsReceiver {org.wso2.carbon.event.processor.manager.core.internal.EventHandler}
java.lang.NullPointerException
at org.wso2.carbon.event.receiver.core.internal.management.QueueInputEventDispatcher.process(QueueInputEventDispatcher.java:102)
at org.wso2.carbon.event.processor.manager.core.internal.EventHandler$1.receive(EventHandler.java:161)
at org.wso2.carbon.event.processor.manager.commons.transport.server.TCPEventServer$ServerWorker$ListenerProcessor.run(TCPEventServer.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Any thoughts on this behavior?

Thanks,

--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: <a href="tel:071%20868%202704" value="+94718682704" target="_blank">+94 71 868 2704
US: <a href="tel:+1%20650-695-2324" value="+16506952324" target="_blank">+1 650 695 2324

--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : <a href="tel:071%20496%201919" value="+94714961919" target="_blank">+94714961919



--
Gayan Kaushalya Yalpathwala
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

LK: +94 71 868 2704
US: +1 650 695 2324


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev