Portal stress test
Some people asked me to share my experience about portal stress-testing.
I have done only a few stress test campains (on WPS v5), but here are the some of the main things that you should consider to properly design and conducts your tests:
1/ Estimate the load:
First, try to estimate the load that your portal will have to support. What is very important is the number of concurent users (users which will be working simultaneously), and the corresponding number of hit per seconds (transactions) that will be performed by this audience. You should think about the load at pic hour (i.e during the period of the day when user's activity will be the most important), to size the stress test architecture and especially the number of load injectors you will need.
2/ Define the use_cases:
You have to think about the way users will use your site and how they will navigate between pages (both in anonymous and authenticated mode). My advice is don't take too much time on this phase: define the 4 or 6 main scenarios only (because the more scenario you define, the more the scripting effort will be important). Moreover, it is always difficult to know in advance what will be the real behavior of users on your site.
To identify the use-cases, you can identify the most useful features of your portal: for instance (in an enterprise intranet context) most of the time people are using the directory search, or looking at the stock quote, or using the search engine...so you should focus on these features when specifying the use-cases.
Moreover, if you suspect some Portlets not to be perfectly implemented, and you know that they will have to support a big load, then you can define simple stress test use-case dedicated to these components.
Finally, you will also have to define the ramp up period, that is the way you will have to inject the load (i.e the growing number of users that you want to simulate during a period of time). There are several way to define the ramp up (you should refer to other articles on the web for more informations).
3/ Define the user's profile:
If you have implemented personalization (based on user attributes) or used embedded UserGroups membership for security, you should consider defining a user testing population with distinct profiles (because personalization treatments can affect server performances).
4/ Choose the testing tool:
Depending on your needs, you will have to choose a tool between editor's or open-source solutions. I think open-source solution (like openSTA, or JMeter) can address most of the project needs.
But you will have to make sure that the tool will be able to support the load you want to simulate (hardware could be a limitation, if you need a lot of injectors). Also, you will have to check that the testing tool is able to adapt its script to your portal context (the format of the portal URL might be difficult to analyze, or you might have developed a custom theme with javascript or DHTML which might not be compatible with the way the testing tool works).
5/ Planning
Launching a stress-test phase should be considered as a small project within the project (don't under-estimate the time it will take). You will have to specify test use-cases, develop test scripting, install the testing solution injectors, run the test and finally analyze the results, and most likely you will have to do some tuning.You can plan for 2 month approx for the full load testing phase (1 month for the specification and script dev phase, and 1 month for the test itself, and analysis). The tuning phase duration might of course vary...
In term of ressources: you will need 1 developer which is familiar with the scripting development + 1 developer which is able to modify and optimize the code.
In most cases, you will also need to monitor database and LDAP response under load, and possibly do the tuning actions to improve performances of these components. So a LDAP or DBA will likely be needed, as well as a portal architect (as I said, this should be considered as a small project within the project).
6/ Test a stable version
If you have to modify the portal theme (HTML code) during the optimization phase, be aware that this might impact your testing script (most of the time scripts use the HTML DOM to find a resource in the code, like the URL it has to follows in the scenario ; so one HTML code update can break your scripts). If possible, modify only 1 component at a time (otherwhise you will not be able to know what component modification was responsible for improvement or degradation of performances).
7/ Stress-test analysis
Stress-test result analysis might be difficult, especially if you discover several problems with the same use-case. A good way to find what is wrong is to isolate each component (e.g Portlet) independently and to do a specific test. By example, if you suspect a bug in the theme, then do a test with an empty portal page (without portlet).
8/ Tuning phase
For the tuning phase, my advice is to work with specialist of each component (Database, LDAP, code developer, portal architect). You will save a lot of time if you have the proper expert at the right time. Especially with WPS v6, lots of JCR components (like WCM) rely on the database (so DB administration will likely be required for index creation, etc).
Once again, try to modify only 1 parameter at a time (to clearly identify the result of the optimization).
Finally, if you use caching, consider using a caching strategy at "the highest level" (i.e the web server, or caching HTML output of your Portlet in memory), because this will lead to better performances improvements.
Hope this will help, wish you good luck for your stress test :-)
I have done only a few stress test campains (on WPS v5), but here are the some of the main things that you should consider to properly design and conducts your tests:
1/ Estimate the load:
First, try to estimate the load that your portal will have to support. What is very important is the number of concurent users (users which will be working simultaneously), and the corresponding number of hit per seconds (transactions) that will be performed by this audience. You should think about the load at pic hour (i.e during the period of the day when user's activity will be the most important), to size the stress test architecture and especially the number of load injectors you will need.
2/ Define the use_cases:
You have to think about the way users will use your site and how they will navigate between pages (both in anonymous and authenticated mode). My advice is don't take too much time on this phase: define the 4 or 6 main scenarios only (because the more scenario you define, the more the scripting effort will be important). Moreover, it is always difficult to know in advance what will be the real behavior of users on your site.
To identify the use-cases, you can identify the most useful features of your portal: for instance (in an enterprise intranet context) most of the time people are using the directory search, or looking at the stock quote, or using the search engine...so you should focus on these features when specifying the use-cases.
Moreover, if you suspect some Portlets not to be perfectly implemented, and you know that they will have to support a big load, then you can define simple stress test use-case dedicated to these components.
Finally, you will also have to define the ramp up period, that is the way you will have to inject the load (i.e the growing number of users that you want to simulate during a period of time). There are several way to define the ramp up (you should refer to other articles on the web for more informations).
3/ Define the user's profile:
If you have implemented personalization (based on user attributes) or used embedded UserGroups membership for security, you should consider defining a user testing population with distinct profiles (because personalization treatments can affect server performances).
4/ Choose the testing tool:
Depending on your needs, you will have to choose a tool between editor's or open-source solutions. I think open-source solution (like openSTA, or JMeter) can address most of the project needs.
But you will have to make sure that the tool will be able to support the load you want to simulate (hardware could be a limitation, if you need a lot of injectors). Also, you will have to check that the testing tool is able to adapt its script to your portal context (the format of the portal URL might be difficult to analyze, or you might have developed a custom theme with javascript or DHTML which might not be compatible with the way the testing tool works).
5/ Planning
Launching a stress-test phase should be considered as a small project within the project (don't under-estimate the time it will take). You will have to specify test use-cases, develop test scripting, install the testing solution injectors, run the test and finally analyze the results, and most likely you will have to do some tuning.You can plan for 2 month approx for the full load testing phase (1 month for the specification and script dev phase, and 1 month for the test itself, and analysis). The tuning phase duration might of course vary...
In term of ressources: you will need 1 developer which is familiar with the scripting development + 1 developer which is able to modify and optimize the code.
In most cases, you will also need to monitor database and LDAP response under load, and possibly do the tuning actions to improve performances of these components. So a LDAP or DBA will likely be needed, as well as a portal architect (as I said, this should be considered as a small project within the project).
6/ Test a stable version
If you have to modify the portal theme (HTML code) during the optimization phase, be aware that this might impact your testing script (most of the time scripts use the HTML DOM to find a resource in the code, like the URL it has to follows in the scenario ; so one HTML code update can break your scripts). If possible, modify only 1 component at a time (otherwhise you will not be able to know what component modification was responsible for improvement or degradation of performances).
7/ Stress-test analysis
Stress-test result analysis might be difficult, especially if you discover several problems with the same use-case. A good way to find what is wrong is to isolate each component (e.g Portlet) independently and to do a specific test. By example, if you suspect a bug in the theme, then do a test with an empty portal page (without portlet).
8/ Tuning phase
For the tuning phase, my advice is to work with specialist of each component (Database, LDAP, code developer, portal architect). You will save a lot of time if you have the proper expert at the right time. Especially with WPS v6, lots of JCR components (like WCM) rely on the database (so DB administration will likely be required for index creation, etc).
Once again, try to modify only 1 parameter at a time (to clearly identify the result of the optimization).
Finally, if you use caching, consider using a caching strategy at "the highest level" (i.e the web server, or caching HTML output of your Portlet in memory), because this will lead to better performances improvements.
Hope this will help, wish you good luck for your stress test :-)
posted by Enguerrand SPINDLER at
1:45 AM
|
1 comments
