WSO2 performance testing framework, some changes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

WSO2 performance testing framework, some changes

Damitha Kumarage-4
Hi,
I have few suggestions/changes on the performance testing framework.

After thorough inspection of good performance testing tools I finally
decided that httperf suits our requirements well.
Although apache bench and java bench do well for simple scenarios the
results from them are not as reliable and as
robust as with httperf.
I would like to add detail justifying my selection.

1. httperf is widely accepted as robust and reliable performance testing
tools. Comparatively ab and java bench are just basic tools

2. Computing only one throughput estimate for the entire test may give
way to outliers to skew the whole result. httperf avoid this
    by sampling the reply throughput once every five seconds. This gives
enough samples to compute sufficient confidence intervals.
    apache bench or java bench compute throughput estimate for the
entire test.

3. We can configure timeout for httperf. This has several advantages.
For example, suppose that  each process can have at most 2000 tcp
    connection open at any given time. If the connection request rate
from the tool is 100 per second then the test could last at most
    20 seconds which is not sufficient for a long running test.(Of
course we can increase the file descriptors per process)
    We cannot configure timeouts for ab. I faced this problem when
performance
    testing ESB/Java with ab once, because most of the tests time out
for large messages and I could not compare with the ESB/C.
    At that time I could successfully perform the tests using httperf.
With my experience with it, I understood that it could be fine tuned
    for our testing requirments.

4. httperf support options to select smaller limits for socket receive
and send buffers. This could be useful in memory constraint environments.

   This is but few facts about httperf superior design. There are many
more facts are considered in designing httperf like slow system calls,
   process scheduling granularity etc. Besides httperf support
configuration files, csv, tsv output files, tools for content
generation, tools for
   GnuPlot , Excel graph generation.


The reason to point out so many details is that I understand there is a
strong preference in attendees to use java bench.

My other suggestion is that, is it necessary to keep the results in a
database?. Can't we retain the output in csv, or tsv files, versioned
with the
test product verison?

Thanks,
Damitha
--
__________________________________________________________________

Damitha Kumarage
Technical Lead; WSO2 Inc.
"Oxygenating the Web Service Platform; " http://www.wso2.com/

blog: " http://damithakumarage.wordpress.com/
__________________________________________________________________



Reply | Threaded
Open this post in threaded view
|

WSO2 performance testing framework, some changes

Srinath Perera-3
> 1. httperf is widely accepted as robust and reliable performance testing
> tools. Comparatively ab and java bench are just basic tools

What is the current condition of the tool, does hp labs still support
it? Is it written in C?
>
> 2. Computing only one throughput estimate for the entire test may give
> way to outliers to skew the whole result. httperf avoid this
> ? ?by sampling the reply throughput once every five seconds. This gives
> enough samples to compute sufficient confidence intervals.
> ? ?apache bench or java bench compute throughput estimate for the
> entire test.
+1 this is a very good feature

Does that work well for all our scenarios? e.g. making each message different.

Thanks
Srinath

>
> 3. We can configure timeout for httperf. This has several advantages.
> For example, suppose that ?each process can have at most 2000 tcp
> ? ?connection open at any given time. If the connection request rate
> from the tool is 100 per second then the test could last at most
> ? ?20 seconds which is not sufficient for a long running test.(Of
> course we can increase the file descriptors per process)
> ? ?We cannot configure timeouts for ab. I faced this problem when
> performance
> ? ?testing ESB/Java with ab once, because most of the tests time out
> for large messages and I could not compare with the ESB/C.
> ? ?At that time I could successfully perform the tests using httperf.
> With my experience with it, I understood that it could be fine tuned
> ? ?for our testing requirments.
>
> 4. httperf support options to select smaller limits for socket receive
> and send buffers. This could be useful in memory constraint environments.
>
> ? This is but few facts about httperf superior design. There are many
> more facts are considered in designing httperf like slow system calls,
> ? process scheduling granularity etc. Besides httperf support
> configuration files, csv, tsv output files, tools for content
> generation, tools for
> ? GnuPlot , Excel graph generation.
>
>
> The reason to point out so many details is that I understand there is a
> strong preference in attendees to use java bench.
>
> My other suggestion is that, is it necessary to keep the results in a
> database?. Can't we retain the output in csv, or tsv files, versioned
> with the
> test product verison?
>
> Thanks,
> Damitha
> --
> __________________________________________________________________
>
> Damitha Kumarage
> Technical Lead; WSO2 Inc.
> "Oxygenating the Web Service Platform; " http://www.wso2.com/
>
> blog: " http://damithakumarage.wordpress.com/
> __________________________________________________________________
>
> _______________________________________________
> Architecture mailing list
> Architecture at wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>



--
============================
Srinath Perera, Ph.D.
   WSO2 Inc. http://wso2.com
   Blog: http://srinathsview.blogspot.com/



Reply | Threaded
Open this post in threaded view
|

WSO2 performance testing framework, some changes

Damitha Kumarage-4
Hi,
Sorry for the extremely late reply. See inline comments below
Srinath Perera wrote:
>> 1. httperf is widely accepted as robust and reliable performance testing
>> tools. Comparatively ab and java bench are just basic tools
>>    
>
> What is the current condition of the tool, does hp labs still support
> it? Is it written in C?
>  
Yes hp labs still support it. It is written in C.

>> 2. Computing only one throughput estimate for the entire test may give
>> way to outliers to skew the whole result. httperf avoid this
>>    by sampling the reply throughput once every five seconds. This gives
>> enough samples to compute sufficient confidence intervals.
>>    apache bench or java bench compute throughput estimate for the
>> entire test.
>>    
> +1 this is a very good feature
>
> Does that work well for all our scenarios? e.g. making each message different.
>  
No but I found that the httperf code is manageble  and  I actually did
some hacking into httperf code  to  get such thing done.
Note that we need to make each message different to change timestamp.
But I learnt from Prabath that adding timestampMaxSkew property value to
a larger value in policy we can  avoid such change.

Note also that autobench perl wrapper around httperf that we use, also
support  distributed load generators.

Thanks,
Damitha

> Thanks
> Srinath
>
>  
>> 3. We can configure timeout for httperf. This has several advantages.
>> For example, suppose that  each process can have at most 2000 tcp
>>    connection open at any given time. If the connection request rate
>> from the tool is 100 per second then the test could last at most
>>    20 seconds which is not sufficient for a long running test.(Of
>> course we can increase the file descriptors per process)
>>    We cannot configure timeouts for ab. I faced this problem when
>> performance
>>    testing ESB/Java with ab once, because most of the tests time out
>> for large messages and I could not compare with the ESB/C.
>>    At that time I could successfully perform the tests using httperf.
>> With my experience with it, I understood that it could be fine tuned
>>    for our testing requirments.
>>
>> 4. httperf support options to select smaller limits for socket receive
>> and send buffers. This could be useful in memory constraint environments.
>>
>>   This is but few facts about httperf superior design. There are many
>> more facts are considered in designing httperf like slow system calls,
>>   process scheduling granularity etc. Besides httperf support
>> configuration files, csv, tsv output files, tools for content
>> generation, tools for
>>   GnuPlot , Excel graph generation.
>>
>>
>> The reason to point out so many details is that I understand there is a
>> strong preference in attendees to use java bench.
>>
>> My other suggestion is that, is it necessary to keep the results in a
>> database?. Can't we retain the output in csv, or tsv files, versioned
>> with the
>> test product verison?
>>
>> Thanks,
>> Damitha
>> --
>> __________________________________________________________________
>>
>> Damitha Kumarage
>> Technical Lead; WSO2 Inc.
>> "Oxygenating the Web Service Platform; " http://www.wso2.com/
>>
>> blog: " http://damithakumarage.wordpress.com/
>> __________________________________________________________________
>>
>> _______________________________________________
>> Architecture mailing list
>> Architecture at wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>    
>
>
>
>  


--
__________________________________________________________________

Damitha Kumarage
Technical Lead; WSO2 Inc.
"Oxygenating the Web Service Platform; " http://www.wso2.com/

blog: " http://damithakumarage.wordpress.com/
__________________________________________________________________