Benchmarks for Object Pool Implementations

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Benchmarks for Object Pool Implementations

Isuru Perera
Hi

I wrote $subject using JMH and I committed the code to my GitHub repo [1].

Currently API Manager uses a Stack Object Pool for key validation clients and we noticed that there are lot contentions when trying to use the pool. Therefore we thought of writing a benchmark to compare the performance with other Object Pool implementations.

Following are the results after running the benchmark with 100 threads. The command I used is "java -jar target/objectpools-0.0.1-SNAPSHOT.jar -t 100 -f 3 -wi 5 -i 10"

# Run complete. Total time: 00:12:28

Benchmark                                               (poolSize)   Mode  Cnt          Score          Error  Units
CommonsPool2GenericObjectPoolBenchmark.useObject               100  thrpt   30    2470759.779 ±   204309.885  ops/s
CommonsPool2SoftReferenceObjectPoolBenchmark.useObject         100  thrpt   30    1305244.745 ±    57823.313  ops/s
CommonsPoolGenericObjectPoolBenchmark.useObject                100  thrpt   30    2277105.922 ±    40957.282  ops/s
CommonsPoolSoftReferenceObjectPoolBenchmark.useObject          100  thrpt   30   11163239.940 ±   449147.426  ops/s
FastObjectPoolBenchmark.useObject                              100  thrpt   30   30164548.113 ±   984247.958  ops/s
FuriousObjectPoolBenchmark.useObject                           100  thrpt   30   14279182.642 ±   115644.976  ops/s
StackObjectPoolBenchmark.useObject                             100  thrpt   30    5464597.797 ±   135726.389  ops/s
StormpotBlazePoolBenchmark.useObject                           100  thrpt   30   77752385.827 ±  3015457.000  ops/s
StormpotQueuePoolBenchmark.useObject                           100  thrpt   30    7670529.272 ±   283981.943  ops/s
TestObjectBenchmark.expensiveObjectCreate                      N/A  thrpt   30     101870.001 ±     2872.885  ops/s
TestObjectBenchmark.simpleObjectCreate                         N/A  thrpt   30  917400188.941 ± 20593977.714  ops/s
ViburObjectPoolBenchmark.useObject                             100  thrpt   30    5924114.964 ±   281625.445  ops/s



For this benchmark, Stormpot Blaze Pool [2] has the highest throughput and it performs 14 times better than the Stack Object Pool, which is used in API Manager.

Stormpot is also Apache licensed.

When I was writing the benchmarks, I found out that Stormpot author also has written similar benchmarks [3]. But I continued with my project and benchmark code has more implementations. I also use latest versions of all object pool implementations.

I'm writing a blog about this and planning to get results for different scenarios. I need to analyze latency results as well. I'll share those results in this mail thread.

Thanks!

Best Regards,

[1] https://github.com/chrishantha/microbenchmarks/tree/master/objectpools
[2] https://github.com/chrisvest/stormpot
[3] https://github.com/chrisvest/object-pool-benchmarks

--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks for Object Pool Implementations

Sanjeewa Malalgoda
Thanks for bring this up isuru. Stack object pool is critical component for API Manager.
We are using it to do two most critical operations (key validation and throttle data process and publish) in our gateway. Also we are using it for both blocking and non blocking operations. So improving this component will surely help us. Please keep us update with your findings. When we implement key validation for next version of API Manager we can add this improvement.

Thanks,
sanjeewa.

On Fri, Nov 11, 2016 at 3:55 PM, Isuru Perera <[hidden email]> wrote:
Hi

I wrote $subject using JMH and I committed the code to my GitHub repo [1].

Currently API Manager uses a Stack Object Pool for key validation clients and we noticed that there are lot contentions when trying to use the pool. Therefore we thought of writing a benchmark to compare the performance with other Object Pool implementations.

Following are the results after running the benchmark with 100 threads. The command I used is "java -jar target/objectpools-0.0.1-SNAPSHOT.jar -t 100 -f 3 -wi 5 -i 10"

# Run complete. Total time: 00:12:28

Benchmark                                               (poolSize)   Mode  Cnt          Score          Error  Units
CommonsPool2GenericObjectPoolBenchmark.useObject               100  thrpt   30    2470759.779 ±   204309.885  ops/s
CommonsPool2SoftReferenceObjectPoolBenchmark.useObject         100  thrpt   30    1305244.745 ±    57823.313  ops/s
CommonsPoolGenericObjectPoolBenchmark.useObject                100  thrpt   30    2277105.922 ±    40957.282  ops/s
CommonsPoolSoftReferenceObjectPoolBenchmark.useObject          100  thrpt   30   11163239.940 ±   449147.426  ops/s
FastObjectPoolBenchmark.useObject                              100  thrpt   30   30164548.113 ±   984247.958  ops/s
FuriousObjectPoolBenchmark.useObject                           100  thrpt   30   14279182.642 ±   115644.976  ops/s
StackObjectPoolBenchmark.useObject                             100  thrpt   30    5464597.797 ±   135726.389  ops/s
StormpotBlazePoolBenchmark.useObject                           100  thrpt   30   77752385.827 ±  3015457.000  ops/s
StormpotQueuePoolBenchmark.useObject                           100  thrpt   30    7670529.272 ±   283981.943  ops/s
TestObjectBenchmark.expensiveObjectCreate                      N/A  thrpt   30     101870.001 ±     2872.885  ops/s
TestObjectBenchmark.simpleObjectCreate                         N/A  thrpt   30  917400188.941 ± 20593977.714  ops/s
ViburObjectPoolBenchmark.useObject                             100  thrpt   30    5924114.964 ±   281625.445  ops/s



For this benchmark, Stormpot Blaze Pool [2] has the highest throughput and it performs 14 times better than the Stack Object Pool, which is used in API Manager.

Stormpot is also Apache licensed.

When I was writing the benchmarks, I found out that Stormpot author also has written similar benchmarks [3]. But I continued with my project and benchmark code has more implementations. I also use latest versions of all object pool implementations.

I'm writing a blog about this and planning to get results for different scenarios. I need to analyze latency results as well. I'll share those results in this mail thread.

Thanks!

Best Regards,

[1] https://github.com/chrishantha/microbenchmarks/tree/master/objectpools
[2] https://github.com/chrisvest/stormpot
[3] https://github.com/chrisvest/object-pool-benchmarks

--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha



--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : +94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks for Object Pool Implementations

Malith Jayasinghe
Hi Isuru,

Interesting results and findings and it would be interesting to see how these values vary under different conditions (e.g. number of threads). I notice that the error is quite high for Stormpot Blaze Pool (which has the highest throughput) . Any thoughts on how to interpret the "error" here ? (for example, does it reflect the variability of the throughput). When we bench mark the latency we want to see the min, max and latency percentiles etc. Is there a way to get this using JMH? 

On Fri, Nov 11, 2016 at 7:43 PM, Sanjeewa Malalgoda <[hidden email]> wrote:
Thanks for bring this up isuru. Stack object pool is critical component for API Manager.
We are using it to do two most critical operations (key validation and throttle data process and publish) in our gateway. Also we are using it for both blocking and non blocking operations. So improving this component will surely help us. Please keep us update with your findings. When we implement key validation for next version of API Manager we can add this improvement.

Thanks,
sanjeewa.

On Fri, Nov 11, 2016 at 3:55 PM, Isuru Perera <[hidden email]> wrote:
Hi

I wrote $subject using JMH and I committed the code to my GitHub repo [1].

Currently API Manager uses a Stack Object Pool for key validation clients and we noticed that there are lot contentions when trying to use the pool. Therefore we thought of writing a benchmark to compare the performance with other Object Pool implementations.

Following are the results after running the benchmark with 100 threads. The command I used is "java -jar target/objectpools-0.0.1-SNAPSHOT.jar -t 100 -f 3 -wi 5 -i 10"

# Run complete. Total time: 00:12:28

Benchmark                                               (poolSize)   Mode  Cnt          Score          Error  Units
CommonsPool2GenericObjectPoolBenchmark.useObject               100  thrpt   30    2470759.779 ±   204309.885  ops/s
CommonsPool2SoftReferenceObjectPoolBenchmark.useObject         100  thrpt   30    1305244.745 ±    57823.313  ops/s
CommonsPoolGenericObjectPoolBenchmark.useObject                100  thrpt   30    2277105.922 ±    40957.282  ops/s
CommonsPoolSoftReferenceObjectPoolBenchmark.useObject          100  thrpt   30   11163239.940 ±   449147.426  ops/s
FastObjectPoolBenchmark.useObject                              100  thrpt   30   30164548.113 ±   984247.958  ops/s
FuriousObjectPoolBenchmark.useObject                           100  thrpt   30   14279182.642 ±   115644.976  ops/s
StackObjectPoolBenchmark.useObject                             100  thrpt   30    5464597.797 ±   135726.389  ops/s
StormpotBlazePoolBenchmark.useObject                           100  thrpt   30   77752385.827 ±  3015457.000  ops/s
StormpotQueuePoolBenchmark.useObject                           100  thrpt   30    7670529.272 ±   283981.943  ops/s
TestObjectBenchmark.expensiveObjectCreate                      N/A  thrpt   30     101870.001 ±     2872.885  ops/s
TestObjectBenchmark.simpleObjectCreate                         N/A  thrpt   30  917400188.941 ± 20593977.714  ops/s
ViburObjectPoolBenchmark.useObject                             100  thrpt   30    5924114.964 ±   281625.445  ops/s



For this benchmark, Stormpot Blaze Pool [2] has the highest throughput and it performs 14 times better than the Stack Object Pool, which is used in API Manager.

Stormpot is also Apache licensed.

When I was writing the benchmarks, I found out that Stormpot author also has written similar benchmarks [3]. But I continued with my project and benchmark code has more implementations. I also use latest versions of all object pool implementations.

I'm writing a blog about this and planning to get results for different scenarios. I need to analyze latency results as well. I'll share those results in this mail thread.

Thanks!

Best Regards,

[1] https://github.com/chrishantha/microbenchmarks/tree/master/objectpools
[2] https://github.com/chrisvest/stormpot
[3] https://github.com/chrisvest/object-pool-benchmarks

--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha



--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:%2B94713068779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malith Jayasinghe 

WSO2, Inc. (http://wso2.com)
Email   : [hidden email]
Mobile : 0770704040
Lean . Enterprise . Middleware

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks for Object Pool Implementations

Isuru Perera
Hi Malith,

I'll analyze results with different number of threads.

The "Error" is related to "Score". What you have thought is correct. It's shows the variability of the throughput. I don't know how the error is calculated, but I'll check how that is calculated.

In JMH, we can have warm-up iterations and measurement iterations. We can also specify how many forks we need. In the benchmark results I have specified, 3 forks, 5 warm-up iterations and 10 measurement iterations. The results for warm-up iterations are ignored. That's why you see 30 as the count ("Cnt"). i.e. 3 forks x 10 measurement iterations.

We can run this benchmark in "SampleTime/sample" mode and then we can get latency results.

I'm working on getting more results now. I'll share when it's ready.

Thanks!

Best Regards,

On Fri, Nov 11, 2016 at 9:16 PM, Malith Jayasinghe <[hidden email]> wrote:
Hi Isuru,

Interesting results and findings and it would be interesting to see how these values vary under different conditions (e.g. number of threads). I notice that the error is quite high for Stormpot Blaze Pool (which has the highest throughput) . Any thoughts on how to interpret the "error" here ? (for example, does it reflect the variability of the throughput). When we bench mark the latency we want to see the min, max and latency percentiles etc. Is there a way to get this using JMH? 

On Fri, Nov 11, 2016 at 7:43 PM, Sanjeewa Malalgoda <[hidden email]> wrote:
Thanks for bring this up isuru. Stack object pool is critical component for API Manager.
We are using it to do two most critical operations (key validation and throttle data process and publish) in our gateway. Also we are using it for both blocking and non blocking operations. So improving this component will surely help us. Please keep us update with your findings. When we implement key validation for next version of API Manager we can add this improvement.

Thanks,
sanjeewa.

On Fri, Nov 11, 2016 at 3:55 PM, Isuru Perera <[hidden email]> wrote:
Hi

I wrote $subject using JMH and I committed the code to my GitHub repo [1].

Currently API Manager uses a Stack Object Pool for key validation clients and we noticed that there are lot contentions when trying to use the pool. Therefore we thought of writing a benchmark to compare the performance with other Object Pool implementations.

Following are the results after running the benchmark with 100 threads. The command I used is "java -jar target/objectpools-0.0.1-SNAPSHOT.jar -t 100 -f 3 -wi 5 -i 10"

# Run complete. Total time: 00:12:28

Benchmark                                               (poolSize)   Mode  Cnt          Score          Error  Units
CommonsPool2GenericObjectPoolBenchmark.useObject               100  thrpt   30    2470759.779 ±   204309.885  ops/s
CommonsPool2SoftReferenceObjectPoolBenchmark.useObject         100  thrpt   30    1305244.745 ±    57823.313  ops/s
CommonsPoolGenericObjectPoolBenchmark.useObject                100  thrpt   30    2277105.922 ±    40957.282  ops/s
CommonsPoolSoftReferenceObjectPoolBenchmark.useObject          100  thrpt   30   11163239.940 ±   449147.426  ops/s
FastObjectPoolBenchmark.useObject                              100  thrpt   30   30164548.113 ±   984247.958  ops/s
FuriousObjectPoolBenchmark.useObject                           100  thrpt   30   14279182.642 ±   115644.976  ops/s
StackObjectPoolBenchmark.useObject                             100  thrpt   30    5464597.797 ±   135726.389  ops/s
StormpotBlazePoolBenchmark.useObject                           100  thrpt   30   77752385.827 ±  3015457.000  ops/s
StormpotQueuePoolBenchmark.useObject                           100  thrpt   30    7670529.272 ±   283981.943  ops/s
TestObjectBenchmark.expensiveObjectCreate                      N/A  thrpt   30     101870.001 ±     2872.885  ops/s
TestObjectBenchmark.simpleObjectCreate                         N/A  thrpt   30  917400188.941 ± 20593977.714  ops/s
ViburObjectPoolBenchmark.useObject                             100  thrpt   30    5924114.964 ±   281625.445  ops/s



For this benchmark, Stormpot Blaze Pool [2] has the highest throughput and it performs 14 times better than the Stack Object Pool, which is used in API Manager.

Stormpot is also Apache licensed.

When I was writing the benchmarks, I found out that Stormpot author also has written similar benchmarks [3]. But I continued with my project and benchmark code has more implementations. I also use latest versions of all object pool implementations.

I'm writing a blog about this and planning to get results for different scenarios. I need to analyze latency results as well. I'll share those results in this mail thread.

Thanks!

Best Regards,

[1] https://github.com/chrishantha/microbenchmarks/tree/master/objectpools
[2] https://github.com/chrisvest/stormpot
[3] https://github.com/chrisvest/object-pool-benchmarks

--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha



--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:%2B94713068779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malith Jayasinghe 

WSO2, Inc. (http://wso2.com)
Email   : [hidden email]
Mobile : <a href="tel:0770704040" value="+94770704040" target="_blank">0770704040
Lean . Enterprise . Middleware



--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks for Object Pool Implementations

Isuru Perera
Even though I started writing the benchmarks in last year, I couldn't visualize the results and demonstrate the exact problem we faced with WSO2 API Manager. I started learning Python few months back and I could finally write the Python scripts to visualize results in CSV files. I could prove the issue with Stack Object Pool by using the GC profiler in JMH and visualizing the GC allocation rates.

Please let me know if you have any feedback on this.

Thank you.

On Tue, Nov 15, 2016 at 9:57 AM, Isuru Perera <[hidden email]> wrote:
Hi Malith,

I'll analyze results with different number of threads.

The "Error" is related to "Score". What you have thought is correct. It's shows the variability of the throughput. I don't know how the error is calculated, but I'll check how that is calculated.

In JMH, we can have warm-up iterations and measurement iterations. We can also specify how many forks we need. In the benchmark results I have specified, 3 forks, 5 warm-up iterations and 10 measurement iterations. The results for warm-up iterations are ignored. That's why you see 30 as the count ("Cnt"). i.e. 3 forks x 10 measurement iterations.

We can run this benchmark in "SampleTime/sample" mode and then we can get latency results.

I'm working on getting more results now. I'll share when it's ready.

Thanks!

Best Regards,

On Fri, Nov 11, 2016 at 9:16 PM, Malith Jayasinghe <[hidden email]> wrote:
Hi Isuru,

Interesting results and findings and it would be interesting to see how these values vary under different conditions (e.g. number of threads). I notice that the error is quite high for Stormpot Blaze Pool (which has the highest throughput) . Any thoughts on how to interpret the "error" here ? (for example, does it reflect the variability of the throughput). When we bench mark the latency we want to see the min, max and latency percentiles etc. Is there a way to get this using JMH? 

On Fri, Nov 11, 2016 at 7:43 PM, Sanjeewa Malalgoda <[hidden email]> wrote:
Thanks for bring this up isuru. Stack object pool is critical component for API Manager.
We are using it to do two most critical operations (key validation and throttle data process and publish) in our gateway. Also we are using it for both blocking and non blocking operations. So improving this component will surely help us. Please keep us update with your findings. When we implement key validation for next version of API Manager we can add this improvement.

Thanks,
sanjeewa.

On Fri, Nov 11, 2016 at 3:55 PM, Isuru Perera <[hidden email]> wrote:
Hi

I wrote $subject using JMH and I committed the code to my GitHub repo [1].

Currently API Manager uses a Stack Object Pool for key validation clients and we noticed that there are lot contentions when trying to use the pool. Therefore we thought of writing a benchmark to compare the performance with other Object Pool implementations.

Following are the results after running the benchmark with 100 threads. The command I used is "java -jar target/objectpools-0.0.1-SNAPSHOT.jar -t 100 -f 3 -wi 5 -i 10"

# Run complete. Total time: 00:12:28

Benchmark                                               (poolSize)   Mode  Cnt          Score          Error  Units
CommonsPool2GenericObjectPoolBenchmark.useObject               100  thrpt   30    2470759.779 ±   204309.885  ops/s
CommonsPool2SoftReferenceObjectPoolBenchmark.useObject         100  thrpt   30    1305244.745 ±    57823.313  ops/s
CommonsPoolGenericObjectPoolBenchmark.useObject                100  thrpt   30    2277105.922 ±    40957.282  ops/s
CommonsPoolSoftReferenceObjectPoolBenchmark.useObject          100  thrpt   30   11163239.940 ±   449147.426  ops/s
FastObjectPoolBenchmark.useObject                              100  thrpt   30   30164548.113 ±   984247.958  ops/s
FuriousObjectPoolBenchmark.useObject                           100  thrpt   30   14279182.642 ±   115644.976  ops/s
StackObjectPoolBenchmark.useObject                             100  thrpt   30    5464597.797 ±   135726.389  ops/s
StormpotBlazePoolBenchmark.useObject                           100  thrpt   30   77752385.827 ±  3015457.000  ops/s
StormpotQueuePoolBenchmark.useObject                           100  thrpt   30    7670529.272 ±   283981.943  ops/s
TestObjectBenchmark.expensiveObjectCreate                      N/A  thrpt   30     101870.001 ±     2872.885  ops/s
TestObjectBenchmark.simpleObjectCreate                         N/A  thrpt   30  917400188.941 ± 20593977.714  ops/s
ViburObjectPoolBenchmark.useObject                             100  thrpt   30    5924114.964 ±   281625.445  ops/s



For this benchmark, Stormpot Blaze Pool [2] has the highest throughput and it performs 14 times better than the Stack Object Pool, which is used in API Manager.

Stormpot is also Apache licensed.

When I was writing the benchmarks, I found out that Stormpot author also has written similar benchmarks [3]. But I continued with my project and benchmark code has more implementations. I also use latest versions of all object pool implementations.

I'm writing a blog about this and planning to get results for different scenarios. I need to analyze latency results as well. I'll share those results in this mail thread.

Thanks!

Best Regards,

[1] https://github.com/chrishantha/microbenchmarks/tree/master/objectpools
[2] https://github.com/chrisvest/stormpot
[3] https://github.com/chrisvest/object-pool-benchmarks

--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha



--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:%2B94713068779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malith Jayasinghe 

WSO2, Inc. (http://wso2.com)
Email   : [hidden email]
Mobile : <a href="tel:0770704040" value="+94770704040" target="_blank">0770704040
Lean . Enterprise . Middleware



--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha



--
Isuru Perera
Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture