analytics worker overloads CPU after 3 days

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

analytics worker overloads CPU after 3 days

Bernard Paris
Hi,

using wso2 EI 6.4 Analytics worker:  after about 3 days of running well,  the worker process raise almost 400% CPU and the server doesn't respond to requests anymore.  I need to kill it then re-run.

Since I'm testing this 6.4 SP software I always had this behavior: it never never rans more than 3 days ;-(  
I though DB connexion problems could explain this, but no, we do not have DB problems since we use mysql.
  



What could explain this ? … purging process ?
Bernard

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: analytics worker overloads CPU after 3 days

Niveathika Rajendran
Hi Bernard,

We haven't come up against this issue, in our testing. 

Could you give the following information to narrow down the issue,
1. Is the minimum requirement for SP[1] available in the system?
2. How much TPS are you seeing from EI server? 
3. How much data is currently stored in the DB as of now?


Best Regards,
Niveathika Rajendran,
Senior Software Engineer.
Mobile : +94 077 903 7536





On Thu, Feb 28, 2019 at 2:05 PM Bernard Paris <[hidden email]> wrote:
Hi,

using wso2 EI 6.4 Analytics worker:  after about 3 days of running well,  the worker process raise almost 400% CPU and the server doesn't respond to requests anymore.  I need to kill it then re-run.

Since I'm testing this 6.4 SP software I always had this behavior: it never never rans more than 3 days ;-(  
I though DB connexion problems could explain this, but no, we do not have DB problems since we use mysql.
  



What could explain this ? … purging process ?
Bernard
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: analytics worker overloads CPU after 3 days

Bernard Paris
Hello,

This is java version
java version "1.8.0_192-ea"
Java(TM) SE Runtime Environment (build 1.8.0_192-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b02, mixed mode)

This morning, back from the week-end and  as usual after some 3 days, CPU load was 400%.  The JVM rans with options  -Xms512M -Xmx4G
I just restart it with options  -Xms2G -Xmx2G (oracle recommands to make both values equals)

When the problem arrives, there is a big  file "heap-dump.hprof" in the log directory.   A the moment the file was created, no errors in carbon.log.

Our system (EI+Analytics) waits for this problem to be solved in order to go into production mode. So now we are still testing and there are very poor datas/trafic to this servers:  all mySQL DBs are 32,1 Mb, a cron process is querying the EI server every hours with some tens requests…

Thanks for you help,
Bernard






Le 2 mars 2019 à 11:14, Niveathika Rajendran <[hidden email]> a écrit :

Hi Bernard,

We haven't come up against this issue, in our testing. 

Could you give the following information to narrow down the issue,
1. Is the minimum requirement for SP[1] available in the system?
2. How much TPS are you seeing from EI server? 
3. How much data is currently stored in the DB as of now?


Best Regards,
Niveathika Rajendran,
Senior Software Engineer.
Mobile : +94 077 903 7536





On Thu, Feb 28, 2019 at 2:05 PM Bernard Paris <[hidden email]> wrote:
Hi,

using wso2 EI 6.4 Analytics worker:  after about 3 days of running well,  the worker process raise almost 400% CPU and the server doesn't respond to requests anymore.  I need to kill it then re-run.

Since I'm testing this 6.4 SP software I always had this behavior: it never never rans more than 3 days ;-(  
I though DB connexion problems could explain this, but no, we do not have DB problems since we use mysql.
  

<PastedGraphic-2.tiff>


What could explain this ? … purging process ?
Bernard
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: analytics worker overloads CPU after 3 days

Rushmin Fernando
Hi Bernard,

Were you able to resolve your issue? If yes, please ignore the diagnosis steps below.

When there is a high CPU usage, 

1) The thread break down of the CPU usage of that process should be taken
2) A thread dump of the JVM should be taken

Then you can identify the thread which is taking the CPU

By reading your descriptions it seems there is a memory related issue in the related WSO2 server.

[1] has a good explanation on this topic.

Specifically the 'Java heap space' might be a reason for this behaviour.

When the allocated memory for the JVM (-XmX) is not enough for the application to run, the garbage collector frequently tries to free up the memory. This process is CPU intensive, thus causes high CPU usage.

The application can reach the max allocated heap size due to ...

1) The allocated max heap is not enough for the intended operation of the application. In this case, the needed memory should be planned and allocated.

2) There is a memory leak in the application. 

Both of the reasons can be identified by analyzing the heap dump. A heap dump is auto-generated when there a heap issue. This is the 'heap-dump.hprof' file you observed.

Best Regards,
Rushmin




On Mon, Mar 4, 2019 at 1:59 PM Bernard Paris <[hidden email]> wrote:
Hello,

This is java version
java version "1.8.0_192-ea"
Java(TM) SE Runtime Environment (build 1.8.0_192-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b02, mixed mode)

This morning, back from the week-end and  as usual after some 3 days, CPU load was 400%.  The JVM rans with options  -Xms512M -Xmx4G
I just restart it with options  -Xms2G -Xmx2G (oracle recommands to make both values equals)

When the problem arrives, there is a big  file "heap-dump.hprof" in the log directory.   A the moment the file was created, no errors in carbon.log.

Our system (EI+Analytics) waits for this problem to be solved in order to go into production mode. So now we are still testing and there are very poor datas/trafic to this servers:  all mySQL DBs are 32,1 Mb, a cron process is querying the EI server every hours with some tens requests…

Thanks for you help,
Bernard






Le 2 mars 2019 à 11:14, Niveathika Rajendran <[hidden email]> a écrit :

Hi Bernard,

We haven't come up against this issue, in our testing. 

Could you give the following information to narrow down the issue,
1. Is the minimum requirement for SP[1] available in the system?
2. How much TPS are you seeing from EI server? 
3. How much data is currently stored in the DB as of now?


Best Regards,
Niveathika Rajendran,
Senior Software Engineer.
Mobile : +94 077 903 7536





On Thu, Feb 28, 2019 at 2:05 PM Bernard Paris <[hidden email]> wrote:
Hi,

using wso2 EI 6.4 Analytics worker:  after about 3 days of running well,  the worker process raise almost 400% CPU and the server doesn't respond to requests anymore.  I need to kill it then re-run.

Since I'm testing this 6.4 SP software I always had this behavior: it never never rans more than 3 days ;-(  
I though DB connexion problems could explain this, but no, we do not have DB problems since we use mysql.
  

<PastedGraphic-2.tiff>


What could explain this ? … purging process ?
Bernard
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


--
Best Regards

Rushmin Fernando
Technical Lead

WSO2 Inc. - Lean . Enterprise . Middleware 

mobile : +94775615183



_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: analytics worker overloads CPU after 3 days

Bernard Paris
Hello,

As on every Mondays, analytics worker runs to  almost 400% CPU load.
The process was launched with -Xms6G -Xmx6G, anyway I found this kind repeated error in my logs

java.lang.OutOfMemoryError: GC overhead limit exceeded

meaning to me the are probably some memory leaks issues with this application, don't you think so ?

By the way, I can watch when the java process is running in "normal condition"  and responding ok, that the permanent CPU load is almost 100%.  That seems to be very big for a server which does almost nothing at all  (practically no data sent to it).

Regards,
Bernard



Le 6 mars 2019 à 12:11, Rushmin Fernando <[hidden email]> a écrit :

Hi Bernard,

Were you able to resolve your issue? If yes, please ignore the diagnosis steps below.

When there is a high CPU usage, 

1) The thread break down of the CPU usage of that process should be taken
2) A thread dump of the JVM should be taken

Then you can identify the thread which is taking the CPU

By reading your descriptions it seems there is a memory related issue in the related WSO2 server.

[1] has a good explanation on this topic.

Specifically the 'Java heap space' might be a reason for this behaviour.

When the allocated memory for the JVM (-XmX) is not enough for the application to run, the garbage collector frequently tries to free up the memory. This process is CPU intensive, thus causes high CPU usage.

The application can reach the max allocated heap size due to ...

1) The allocated max heap is not enough for the intended operation of the application. In this case, the needed memory should be planned and allocated.

2) There is a memory leak in the application. 

Both of the reasons can be identified by analyzing the heap dump. A heap dump is auto-generated when there a heap issue. This is the 'heap-dump.hprof' file you observed.

Best Regards,
Rushmin




On Mon, Mar 4, 2019 at 1:59 PM Bernard Paris <[hidden email]> wrote:
Hello,

This is java version
java version "1.8.0_192-ea"
Java(TM) SE Runtime Environment (build 1.8.0_192-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b02, mixed mode)

This morning, back from the week-end and  as usual after some 3 days, CPU load was 400%.  The JVM rans with options  -Xms512M -Xmx4G
I just restart it with options  -Xms2G -Xmx2G (oracle recommands to make both values equals)

When the problem arrives, there is a big  file "heap-dump.hprof" in the log directory.   A the moment the file was created, no errors in carbon.log.

Our system (EI+Analytics) waits for this problem to be solved in order to go into production mode. So now we are still testing and there are very poor datas/trafic to this servers:  all mySQL DBs are 32,1 Mb, a cron process is querying the EI server every hours with some tens requests…

Thanks for you help,
Bernard






Le 2 mars 2019 à 11:14, Niveathika Rajendran <[hidden email]> a écrit :

Hi Bernard,

We haven't come up against this issue, in our testing. 

Could you give the following information to narrow down the issue,
1. Is the minimum requirement for SP[1] available in the system?
2. How much TPS are you seeing from EI server? 
3. How much data is currently stored in the DB as of now?


Best Regards,
Niveathika Rajendran,
Senior Software Engineer.
Mobile : +94 077 903 7536





On Thu, Feb 28, 2019 at 2:05 PM Bernard Paris <[hidden email]> wrote:
Hi,

using wso2 EI 6.4 Analytics worker:  after about 3 days of running well,  the worker process raise almost 400% CPU and the server doesn't respond to requests anymore.  I need to kill it then re-run.

Since I'm testing this 6.4 SP software I always had this behavior: it never never rans more than 3 days ;-(  
I though DB connexion problems could explain this, but no, we do not have DB problems since we use mysql.
  

<PastedGraphic-2.tiff>


What could explain this ? … purging process ?
Bernard
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


--
Best Regards

Rushmin Fernando
Technical Lead

WSO2 Inc. - Lean . Enterprise . Middleware 

mobile : +94775615183




_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: analytics worker overloads CPU after 3 days

Bernard Paris
In reply to this post by Rushmin Fernando
Hi again,

in deed:
- after analytics worker restart the CPU load is very low —> ok
- when I send lot of requests to EI the CPU load from analytics worker grows  to about 100% CPU -> ok
- when the trafic to EI  stops, the worker  CPU load stays around 90-100%, il does decrease nor comes back to low level.

Regards,
Bernard



Le 6 mars 2019 à 12:11, Rushmin Fernando <[hidden email]> a écrit :

Hi Bernard,

Were you able to resolve your issue? If yes, please ignore the diagnosis steps below.

When there is a high CPU usage, 

1) The thread break down of the CPU usage of that process should be taken
2) A thread dump of the JVM should be taken

Then you can identify the thread which is taking the CPU

By reading your descriptions it seems there is a memory related issue in the related WSO2 server.

[1] has a good explanation on this topic.

Specifically the 'Java heap space' might be a reason for this behaviour.

When the allocated memory for the JVM (-XmX) is not enough for the application to run, the garbage collector frequently tries to free up the memory. This process is CPU intensive, thus causes high CPU usage.

The application can reach the max allocated heap size due to ...

1) The allocated max heap is not enough for the intended operation of the application. In this case, the needed memory should be planned and allocated.

2) There is a memory leak in the application. 

Both of the reasons can be identified by analyzing the heap dump. A heap dump is auto-generated when there a heap issue. This is the 'heap-dump.hprof' file you observed.

Best Regards,
Rushmin




On Mon, Mar 4, 2019 at 1:59 PM Bernard Paris <[hidden email]> wrote:
Hello,

This is java version
java version "1.8.0_192-ea"
Java(TM) SE Runtime Environment (build 1.8.0_192-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b02, mixed mode)

This morning, back from the week-end and  as usual after some 3 days, CPU load was 400%.  The JVM rans with options  -Xms512M -Xmx4G
I just restart it with options  -Xms2G -Xmx2G (oracle recommands to make both values equals)

When the problem arrives, there is a big  file "heap-dump.hprof" in the log directory.   A the moment the file was created, no errors in carbon.log.

Our system (EI+Analytics) waits for this problem to be solved in order to go into production mode. So now we are still testing and there are very poor datas/trafic to this servers:  all mySQL DBs are 32,1 Mb, a cron process is querying the EI server every hours with some tens requests…

Thanks for you help,
Bernard






Le 2 mars 2019 à 11:14, Niveathika Rajendran <[hidden email]> a écrit :

Hi Bernard,

We haven't come up against this issue, in our testing. 

Could you give the following information to narrow down the issue,
1. Is the minimum requirement for SP[1] available in the system?
2. How much TPS are you seeing from EI server? 
3. How much data is currently stored in the DB as of now?


Best Regards,
Niveathika Rajendran,
Senior Software Engineer.
Mobile : +94 077 903 7536





On Thu, Feb 28, 2019 at 2:05 PM Bernard Paris <[hidden email]> wrote:
Hi,

using wso2 EI 6.4 Analytics worker:  after about 3 days of running well,  the worker process raise almost 400% CPU and the server doesn't respond to requests anymore.  I need to kill it then re-run.

Since I'm testing this 6.4 SP software I always had this behavior: it never never rans more than 3 days ;-(  
I though DB connexion problems could explain this, but no, we do not have DB problems since we use mysql.
  

<PastedGraphic-2.tiff>


What could explain this ? … purging process ?
Bernard
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


--
Best Regards

Rushmin Fernando
Technical Lead

WSO2 Inc. - Lean . Enterprise . Middleware 

mobile : +94775615183




_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev