thrift.transport.TIOStreamTransport- Error closing output stream in SP analytics worker

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

thrift.transport.TIOStreamTransport- Error closing output stream in SP analytics worker

Bernard Paris
Hi devs,

we are evaluating wso2-EI 6.4 in pre-prod environment, running EI with analytics worker.

I followed the guide at
to configure postgres datasources in deployment.yaml, for all dbs: 
METRICS_DB
CARBON_DB
PERMISSIONS_DB
EI_ANALYTICS_DB

Library is postgresql-42.2.2.jar

The system works fine for some days but after a while the analytics worker stops responding.
I need to manually  kill the (worker) process because "carbon.sh stop" doesn't do the stuff, then restart the worker… that will be ok for 3 or 4  days.

In comparison to datasources of wso2-EI in master-datasources.xml, db connector attributes are the same, except an extra one "testOnBorrow" set to true.
Unfortunately this attribute is not allowed in deployment.yaml.

Any help should be appreciate to get the system stable.

[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,289]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
        at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) 
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
        at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
        at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:17,423]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:22,204] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        ... 14 more

[2019-01-24 01:55:53,456]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:31,190]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[2019-01-24 01:56:09,847]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:58,072]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

java.net.SocketTimeoutException with SP analytics worker & postgres

Bernard Paris
Hello,

since this problem is still alive, could anyone give us an idea of what exactly are  
           idleTimeout
and
            validationTimeout

parameters we are to define in deployment.yaml worker file, or what are their matching name in TOMCAT connection JDBC pool ?

Thanks
Bernard


[2019-02-01 02:45:23,770]  WARN {} - IOException occurred while connecting to pgsql.hifolks.be:5440 java.net.SocketTimeoutException
       at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:589)
       at org.postgresql.core.PGStream.<init>(PGStream.java:69)
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
       at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
       at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
       at org.postgresql.Driver.makeConnection(Driver.java:452)
       at org.postgresql.Driver.connect(Driver.java:254)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
       at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
       at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
       at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)

[2019-02-01 02:45:31,458]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
       at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
       at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
       at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
       at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
       at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
       at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
       at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
       at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
       at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)

[2019-02-01 02:45:31,459] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
       at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
       at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
       at org.postgresql.Driver.makeConnection(Driver.java:452)
       at org.postgresql.Driver.connect(Driver.java:254)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
       at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
       at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
       at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException
       at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:589)
       at org.postgresql.core.PGStream.<init>(PGStream.java:69)
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
       ... 14 more





Le 28 janv. 2019 à 10:15, Bernard Paris <[hidden email]> a écrit :

Hi devs,

we are evaluating wso2-EI 6.4 in pre-prod environment, running EI with analytics worker.

I followed the guide at
to configure postgres datasources in deployment.yaml, for all dbs: 
METRICS_DB
CARBON_DB
PERMISSIONS_DB
EI_ANALYTICS_DB

Library is postgresql-42.2.2.jar

The system works fine for some days but after a while the analytics worker stops responding.
I need to manually  kill the (worker) process because "carbon.sh stop" doesn't do the stuff, then restart the worker… that will be ok for 3 or 4  days.

In comparison to datasources of wso2-EI in master-datasources.xml, db connector attributes are the same, except an extra one "testOnBorrow" set to true.
Unfortunately this attribute is not allowed in deployment.yaml.

Any help should be appreciate to get the system stable.

[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,289]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
        at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) 
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
        at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
        at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:17,423]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:22,204] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        ... 14 more

[2019-01-24 01:55:53,456]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:31,190]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[2019-01-24 01:56:09,847]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:58,072]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: java.net.SocketTimeoutException with SP analytics worker & postgres

Dilini Muthumala
Hi Bernard,

Those are Hikari Connection Pool configurations. You can find documentation on each of these parameters in 'Configurations' section in [1]. I have quoted 'idleTimeout' and 'validationTimeout' parameter descriptions below in this email for your convenience. 

As for the issue you are getting, it would be better if you could share the complete log file. However from the looks of it, it seems like a connection problem. Please see [2].

Thanks,
Dilini


HikariCP Parameter descriptions:

idleTimeout
This property controls the maximum amount of time that a connection is allowed to sit idle in the pool. Whether a connection is retired as idle or not is subject to a maximum variation of +30 seconds, and average variation of +15 seconds. A connection will never be retired as idle before this timeout. A value of 0 means that idle connections are never removed from the pool. Default: 600000 (10 minutes)
 

validationTimeout
This property controls the maximum amount of time that a connection will be tested for aliveness. This value must be less than the connectionTimeout. The lowest accepted validation timeout is 1000ms (1 second). Default: 5000

On Wed, Feb 6, 2019 at 3:37 PM Bernard Paris <[hidden email]> wrote:
Hello,

since this problem is still alive, could anyone give us an idea of what exactly are  
           idleTimeout
and
            validationTimeout

parameters we are to define in deployment.yaml worker file, or what are their matching name in TOMCAT connection JDBC pool ?

Thanks
Bernard


[2019-02-01 02:45:23,770]  WARN {} - IOException occurred while connecting to pgsql.hifolks.be:5440 java.net.SocketTimeoutException
       at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:589)
       at org.postgresql.core.PGStream.<init>(PGStream.java:69)
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
       at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
       at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
       at org.postgresql.Driver.makeConnection(Driver.java:452)
       at org.postgresql.Driver.connect(Driver.java:254)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
       at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
       at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
       at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)

[2019-02-01 02:45:31,458]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
       at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
       at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
       at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
       at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
       at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
       at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
       at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
       at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
       at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)

[2019-02-01 02:45:31,459] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
       at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
       at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
       at org.postgresql.Driver.makeConnection(Driver.java:452)
       at org.postgresql.Driver.connect(Driver.java:254)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
       at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
       at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
       at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException
       at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:589)
       at org.postgresql.core.PGStream.<init>(PGStream.java:69)
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
       ... 14 more





Le 28 janv. 2019 à 10:15, Bernard Paris <[hidden email]> a écrit :

Hi devs,

we are evaluating wso2-EI 6.4 in pre-prod environment, running EI with analytics worker.

I followed the guide at
to configure postgres datasources in deployment.yaml, for all dbs: 
METRICS_DB
CARBON_DB
PERMISSIONS_DB
EI_ANALYTICS_DB

Library is postgresql-42.2.2.jar

The system works fine for some days but after a while the analytics worker stops responding.
I need to manually  kill the (worker) process because "carbon.sh stop" doesn't do the stuff, then restart the worker… that will be ok for 3 or 4  days.

In comparison to datasources of wso2-EI in master-datasources.xml, db connector attributes are the same, except an extra one "testOnBorrow" set to true.
Unfortunately this attribute is not allowed in deployment.yaml.

Any help should be appreciate to get the system stable.

[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,289]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
        at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) 
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
        at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
        at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:17,423]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:22,204] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        ... 14 more

[2019-01-24 01:55:53,456]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:31,190]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[2019-01-24 01:56:09,847]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:58,072]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: java.net.SocketTimeoutException with SP analytics worker & postgres

Bernard Paris
Hello,

nothing to do with this, Analytics worker again at 400% CPU this morning, I had to kill -9 the process then restart.  It never ran more than 3 days ;-(.

First I have 

[2019-02-13 23:53:12,795]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
        at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
        at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
        at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


in the logs (often during the night —> purging ?), then 

[2019-02-14 00:01:46,401]  WARN {} - IOException occurred while connecting to pgsql.uclouvain.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[2019-02-14 00:01:48,660]  WARN {} - IOException occurred while connecting to pgsql.uclouvain.be:5440 java.net.SocketTimeoutException
        at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.postgresql.core.PGStream.<init>(PGStream.java:69)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[2019-02-14 00:05:25,337] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException


My datasources config for all 4 DBs:

WSO2EI_ANALYTICS_WKR_PERMISSIONS_DB 
WSO2EI_ANALYTICS_WKR_EI_ANALYTICS_DB
WSO2EI_ANALYTICS_METRICS_DB
WSO2EI_ANALYTICS_CARBON_DB


      definition:
        configuration:
            jdbcUrl: 'jdbc:<a href="postgresql://pgsql.domain:5440/WSO2EI_ANALYTICS_WKR_CARBON_DB'" class="">postgresql://pgsql.domain:5440/WSO2EI_ANALYTICS_WKR_CARBON_DB'
            username: user
            password: secret
            driverClassName: org.postgresql.Driver
            maxPoolSize: 20
            idleTimeout: 20000
            connectionTestQuery: SELECT 1
            validationTimeout: 5000
            isAutoCommit: false
        type: RDBMS
      description: "The datasource used for registry and user manager"
      name: WSO2_CARBON_DB



We plan to switch to mySQL


Bernard






Le 7 févr. 2019 à 10:55, Dilini Muthumala <[hidden email]> a écrit :

Hi Bernard,

Those are Hikari Connection Pool configurations. You can find documentation on each of these parameters in 'Configurations' section in [1]. I have quoted 'idleTimeout' and 'validationTimeout' parameter descriptions below in this email for your convenience. 

As for the issue you are getting, it would be better if you could share the complete log file. However from the looks of it, it seems like a connection problem. Please see [2].

Thanks,
Dilini


HikariCP Parameter descriptions:

idleTimeout
This property controls the maximum amount of time that a connection is allowed to sit idle in the pool. Whether a connection is retired as idle or not is subject to a maximum variation of +30 seconds, and average variation of +15 seconds. A connection will never be retired as idle before this timeout. A value of 0 means that idle connections are never removed from the pool. Default: 600000 (10 minutes)
 

validationTimeout
This property controls the maximum amount of time that a connection will be tested for aliveness. This value must be less than the connectionTimeout. The lowest accepted validation timeout is 1000ms (1 second). Default: 5000

On Wed, Feb 6, 2019 at 3:37 PM Bernard Paris <[hidden email]> wrote:
Hello,

since this problem is still alive, could anyone give us an idea of what exactly are  
           idleTimeout
and
            validationTimeout

parameters we are to define in deployment.yaml worker file, or what are their matching name in TOMCAT connection JDBC pool ?

Thanks
Bernard


[2019-02-01 02:45:23,770]  WARN {} - IOException occurred while connecting to pgsql.hifolks.be:5440 java.net.SocketTimeoutException
       at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:589)
       at org.postgresql.core.PGStream.<init>(PGStream.java:69)
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
       at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
       at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
       at org.postgresql.Driver.makeConnection(Driver.java:452)
       at org.postgresql.Driver.connect(Driver.java:254)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
       at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
       at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
       at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)

[2019-02-01 02:45:31,458]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
       at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
       at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
       at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
       at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
       at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
       at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
       at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
       at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
       at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)

[2019-02-01 02:45:31,459] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
       at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
       at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
       at org.postgresql.Driver.makeConnection(Driver.java:452)
       at org.postgresql.Driver.connect(Driver.java:254)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
       at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
       at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
       at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException
       at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:589)
       at org.postgresql.core.PGStream.<init>(PGStream.java:69)
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
       ... 14 more





Le 28 janv. 2019 à 10:15, Bernard Paris <[hidden email]> a écrit :

Hi devs,

we are evaluating wso2-EI 6.4 in pre-prod environment, running EI with analytics worker.

I followed the guide at
to configure postgres datasources in deployment.yaml, for all dbs: 
METRICS_DB
CARBON_DB
PERMISSIONS_DB
EI_ANALYTICS_DB

Library is postgresql-42.2.2.jar

The system works fine for some days but after a while the analytics worker stops responding.
I need to manually  kill the (worker) process because "carbon.sh stop" doesn't do the stuff, then restart the worker… that will be ok for 3 or 4  days.

In comparison to datasources of wso2-EI in master-datasources.xml, db connector attributes are the same, except an extra one "testOnBorrow" set to true.
Unfortunately this attribute is not allowed in deployment.yaml.

Any help should be appreciate to get the system stable.

[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,291]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:48:52,289]  WARN {org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream. java.net.SocketException: Socket is closed
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
        at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) 
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
        at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
        at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:17,423]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:22,204] ERROR {org.postgresql.Driver} - Connection error: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        ... 14 more

[2019-01-24 01:55:53,456]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:31,190]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[2019-01-24 01:56:09,847]  INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2019-01-24 01:55:58,072]  WARN {} - IOException occurred while connecting to pgsql.server.be:5440 java.io.EOFException
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
        at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
        at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev