[MB] ValueEvent exception occurred on disruptor for MQTT

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[MB] ValueEvent exception occurred on disruptor for MQTT

Ayyoob Hamza
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [MB] ValueEvent exception occurred on disruptor for MQTT

Ayyoob Hamza
Please note this works after restarting the server.

Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [MB] ValueEvent exception occurred on disruptor for MQTT

sumedha rubasinghe
In reply to this post by Ayyoob Hamza
MB guys,
You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev




--
/sumedha
m: +94 773017743
b :  bit.ly/sumedha

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [MB] ValueEvent exception occurred on disruptor for MQTT

Pamod Sylvester
We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,
Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:
MB guys,
You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev




--
/sumedha
m: <a href="tel:077%20301%207743" value="+94773017743" target="_blank">+94 773017743
b :  bit.ly/sumedha



--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: +94 77 7779495

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [MB] ValueEvent exception occurred on disruptor for MQTT

Ayyoob Hamza
Hi Pamod,

I'll enable those logs and will do another run, But it might take couple of weeks to reproduce this.

Thanks,
Ayyoob

Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:
We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,
Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:
MB guys,
You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev




--
/sumedha
m: <a href="tel:077%20301%207743" value="+94773017743" target="_blank">+94 773017743
b :  bit.ly/sumedha



--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:+94%2077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [MB] ValueEvent exception occurred on disruptor for MQTT

sumedha rubasinghe
In reply to this post by Pamod Sylvester
Pamod,
What is the most longest running MB deployment we have?

Is this case, we have had 8 devices pushing data to MB several time within a minute and a DAS listener picking them up. We had this setup running for more than 1 month.

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:
We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,
Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:
MB guys,
You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev




--
/sumedha
m: <a href="tel:077%20301%207743" value="+94773017743" target="_blank">+94 773017743
b :  bit.ly/sumedha



--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
/sumedha
m: +94 773017743
b :  bit.ly/sumedha

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [MB] ValueEvent exception occurred on disruptor for MQTT

Pamod Sylvester
Hi Sumedha,

With the amount of information available it would be hard to conclude where the error occurs. 

Also it would be ideal if there's a way to get disruptor to pin-point the exact location of the Nullpointer when it occurs in the stack trace in it's log exception handler. Will update this thread if i find any. 

Let's also make sure to monitor the packets exchanged between the clients since this issue occurs during the protocol handling and converting the bytes[] to a message, so that we could identify whether it's due to a result of malformed packet etc just to make sure.  

@Pumudu please add your thoughts. Also could you share the details of the long running tests we did (the type of tests we carried out etc)

Thanks,
Pamod

On Thu, Apr 27, 2017 at 9:51 PM, Sumedha Rubasinghe <[hidden email]> wrote:
Pamod,
What is the most longest running MB deployment we have?

Is this case, we have had 8 devices pushing data to MB several time within a minute and a DAS listener picking them up. We had this setup running for more than 1 month.

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:
We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,
Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:
MB guys,
You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev




--
/sumedha
m: <a href="tel:077%20301%207743" value="+94773017743" target="_blank">+94 773017743
b :  bit.ly/sumedha



--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
/sumedha
m: <a href="tel:077%20301%207743" value="+94773017743" target="_blank">+94 773017743
b :  bit.ly/sumedha



--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: +94 77 7779495

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [MB] ValueEvent exception occurred on disruptor for MQTT

Pumudu Ruhunage
Hi Sumedha,

As pamod mention this is a generic exception[1][2][3] which requires more info to identify the exact cause. 
We had long running test(mixed QOS test) for around 5 days(120h+) and during that we didn't observe this particular issue as well
Further, we have done several comparisons tests[4] and added few integration mqtt load tests[5] to verify mqtt stability.

On Thu, Apr 27, 2017 at 11:45 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Sumedha,

With the amount of information available it would be hard to conclude where the error occurs. 

Also it would be ideal if there's a way to get disruptor to pin-point the exact location of the Nullpointer when it occurs in the stack trace in it's log exception handler. Will update this thread if i find any. 

Let's also make sure to monitor the packets exchanged between the clients since this issue occurs during the protocol handling and converting the bytes[] to a message, so that we could identify whether it's due to a result of malformed packet etc just to make sure.  

@Pumudu please add your thoughts. Also could you share the details of the long running tests we did (the type of tests we carried out etc)

Thanks,
Pamod

On Thu, Apr 27, 2017 at 9:51 PM, Sumedha Rubasinghe <[hidden email]> wrote:
Pamod,
What is the most longest running MB deployment we have?

Is this case, we have had 8 devices pushing data to MB several time within a minute and a DAS listener picking them up. We had this setup running for more than 1 month.

On Thu, Apr 27, 2017 at 8:18 PM, Pamod Sylvester <[hidden email]> wrote:
We've not come across this issue before. 

it seems, this has occurred in one of the disruptor handlers in Moquette. Shall we enable debug/tracing logs in Moquette side to further drill down the issue,

The following logs will be useful,  

 1. org.dna.mqtt.moquette.messaging.spi.impl.SimpleMessaging
 2. org.dna.mqtt.moquette.messaging.spi.impl.ProtocolProcessor

Thanks,
Pamod

On Thu, Apr 27, 2017 at 6:59 PM, Sumedha Rubasinghe <[hidden email]> wrote:
MB guys,
You can consider this as a real production issue. We have been running the server for few weeks now.

On Thu, Apr 27, 2017 at 4:22 PM, Ayyoob Hamza <[hidden email]> wrote:
Hi All,

We did a long running test for a couple of weeks and in MB we see following logs getting printed continuously and devices are failing to publish data. Is this a known issue?

TID: [] [] [2017-04-27 16:03:21,390] ERROR {org.dna.mqtt.wso2.MQTTLogExceptionHandler} -  ValueEvent exception occurred on disruptor. {org.dna.mqtt.wso2.MQTTLogExceptionHandler}
java.lang.NullPointerException


Thanks,
Ayyoob Hamza
Senior Software Engineer
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:%2B94%2077%207779495" value="+94777779495" style="color:rgb(17,85,204)" target="_blank">+94 77 1681010

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev




--
/sumedha
m: <a href="tel:077%20301%207743" value="+94773017743" target="_blank">+94 773017743
b :  bit.ly/sumedha



--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
/sumedha
m: <a href="tel:077%20301%207743" value="+94773017743" target="_blank">+94 773017743
b :  bit.ly/sumedha



--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:+94%2077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Pumudu Ruhunage
Software Engineer | WSO2 Inc

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev