[MB] ValueEvent exception occurred on disruptor for MQTT

[Ayyoob Hamza]

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Ayyoob Hamza]

Please note this works after restarting the server.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

[sumedha rubasinghe]

MB guys,

You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

--

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Pamod Sylvester]

We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging

 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,

Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:

MB guys,

You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

--

--

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Ayyoob Hamza]

Hi Pamod,

I'll enable those logs and will do another run, But it might take couple of weeks to reproduce this.

Thanks,

Ayyoob

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:

We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging

 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,

Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:

MB guys,

You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

--

--

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

[sumedha rubasinghe]

Pamod,

What is the most longest running MB deployment we have?

Is this case, we have had 8 devices pushing data to MB several time within a minute and a DAS listener picking them up. We had this setup running for more than 1 month.

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:

We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging

 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,

Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:

MB guys,

You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

--

--

--

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Pamod Sylvester]

Hi Sumedha,

With the amount of information available it would be hard to conclude where the error occurs. 

Also it would be ideal if there's a way to get disruptor to pin-point the exact location of the Nullpointer when it occurs in the stack trace in it's log exception handler. Will update this thread if i find any. 

Let's also make sure to monitor the packets exchanged between the clients since this issue occurs during the protocol handling and converting the bytes[] to a message, so that we could identify whether it's due to a result of malformed packet etc just to make sure.  

@Pumudu please add your thoughts. Also could you share the details of the long running tests we did (the type of tests we carried out etc)

Thanks,

Pamod

On Thu, Apr 27, 2017 at 9:51 PM, Sumedha Rubasinghe <[hidden email]> wrote:

Pamod,

What is the most longest running MB deployment we have?

Is this case, we have had 8 devices pushing data to MB several time within a minute and a DAS listener picking them up. We had this setup running for more than 1 month.

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:

We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging

 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,

Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:

MB guys,

You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

--

--

--

--

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Pumudu Ruhunage]

Hi Sumedha,

As pamod mention this is a generic exception[1][2][3] which requires more info to identify the exact cause. 

We had long running test(mixed QOS test) for around 5 days(120h+) and during that we didn't observe this particular issue as well. 

Further, we have done several comparisons tests[4] and added few integration mqtt load tests[5] to verify mqtt stability.

[1] https://wso2.org/jira/browse/MB-1211

[2] https://wso2.org/jira/browse/MB-1837

[3] https://wso2.org/jira/browse/MB-1328

[4] https://docs.google.com/a/wso2.com/spreadsheets/d/1C50DqgBqXB8_-SND_YJpmXE-ArhxN8Fbg6-K0kQAZZc/edit?usp=sharing

[5] https://github.com/wso2/product-mb/tree/v3.1.0/modules/integration/tests-integration/tests-mqtt/src/test/java/org/wso2/mb/integration/tests/mqtt/load

Thanks,

On Thu, Apr 27, 2017 at 11:45 PM, Pamod Sylvester <[hidden email]> wrote:

Hi Sumedha,

With the amount of information available it would be hard to conclude where the error occurs. 

Also it would be ideal if there's a way to get disruptor to pin-point the exact location of the Nullpointer when it occurs in the stack trace in it's log exception handler. Will update this thread if i find any. 

Let's also make sure to monitor the packets exchanged between the clients since this issue occurs during the protocol handling and converting the bytes[] to a message, so that we could identify whether it's due to a result of malformed packet etc just to make sure.  

@Pumudu please add your thoughts. Also could you share the details of the long running tests we did (the type of tests we carried out etc)

Thanks,

Pamod

On Thu, Apr 27, 2017 at 9:51 PM, Sumedha Rubasinghe <[hidden email]> wrote:

Pamod,

What is the most longest running MB deployment we have?

Is this case, we have had 8 devices pushing data to MB several time within a minute and a DAS listener picking them up. We had this setup running for more than 1 month.

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:

We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging

 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,

Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:

MB guys,

You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:

Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}

java.lang.NullPointerException

Thanks,

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

--

--

--

--

--

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev