Hi,
I have few suggestions/changes on the performance testing framework.
After thorough inspection of good performance testing tools I finally
decided that httperf suits our requirements well.
Although apache bench and java bench do well for simple scenarios the
results from them are not as reliable and as
robust as with httperf.
I would like to add detail justifying my selection.
1. httperf is widely accepted as robust and reliable performance testing
tools. Comparatively ab and java bench are just basic tools
2. Computing only one throughput estimate for the entire test may give
way to outliers to skew the whole result. httperf avoid this
by sampling the reply throughput once every five seconds. This gives
enough samples to compute sufficient confidence intervals.
apache bench or java bench compute throughput estimate for the
entire test.
3. We can configure timeout for httperf. This has several advantages.
For example, suppose that each process can have at most 2000 tcp
connection open at any given time. If the connection request rate
from the tool is 100 per second then the test could last at most
20 seconds which is not sufficient for a long running test.(Of
course we can increase the file descriptors per process)
We cannot configure timeouts for ab. I faced this problem when
performance
testing ESB/Java with ab once, because most of the tests time out
for large messages and I could not compare with the ESB/C.
At that time I could successfully perform the tests using httperf.
With my experience with it, I understood that it could be fine tuned
for our testing requirments.
4. httperf support options to select smaller limits for socket receive
and send buffers. This could be useful in memory constraint environments.
This is but few facts about httperf superior design. There are many
more facts are considered in designing httperf like slow system calls,
process scheduling granularity etc. Besides httperf support
configuration files, csv, tsv output files, tools for content
generation, tools for
GnuPlot , Excel graph generation.
The reason to point out so many details is that I understand there is a
strong preference in attendees to use java bench.
My other suggestion is that, is it necessary to keep the results in a
database?. Can't we retain the output in csv, or tsv files, versioned
with the
test product verison?
Thanks,
Damitha
--
__________________________________________________________________
Damitha Kumarage
Technical Lead; WSO2 Inc.
"Oxygenating the Web Service Platform; "
http://www.wso2.com/blog: "
http://damithakumarage.wordpress.com/__________________________________________________________________
Locked
