Executive Summary :
The following explains why it is necessary to have about 6 load generators when doing system-level load tests with LoadRunner.
You do NOT have to read this documentation if:
- If you are not the one who is going to be running system-level tests (or you are only running load tests against your local system)
- You are not involved in setting up the generators.
Otherwise:
This document is 100% necessary for running accurate system-level load tests where a load balancer is involved.
Quick Summary of how to get accurately load balanced tests:
- - Don’t use IP Spoofing for Load Balancing (use multiple machines)
- Set load generator vuser quotas before each test
- Remember that load generators start their vuser quota sequentially, not concurrently
- Remember to group load generators together in LR Scenario manager.
The Problem
Here is a screen shot which illustrates that even when IP spoofed machines are running users, we still have a load imbalance.
Screen shot two:
It is possible to obtain get high user load tests with only one or two load generators, however, test execution will be much longer because you have to throw away the first, imbalanced part of test.
A summary of the reasons many load generators are required to do accurate system-level load testing is:
LoadRunner’s “WAN Spoofing” technique (multiple IP’s assigned to one network card) still leaves the load generator with only one DNS cache. It is this DNS cache, and not the spoofed IP’s, which attaches IP addresses to the request headers. (In other words, LoadRunner’s spoofing is good for assigning a few users to a few scripts for functional testing, however, it is not good for balancing load.) Additionally, if you are testing through a router, routers actually work by MAC addresses. All your virtual machines must have different MAC addresses. LR’s WAN Spoofing does NOT assign different MAC addresses, therefore to the router, the traffic will all appear to be coming from the same machine.
- The extra step of assigning Vuser runtime quotas on each load generator must be taken, otherwise one generator tends to run all the load (and hence no load balancing takes place.)
- Even after the vuser quota has been set, LoadRunner will distribute and start users * sequentially, * per load generator. You think you might be running 200 users against your entire web farm, but in reality for a good part of the test, load is running against only 1 web server.
- For each script, Load generators must be grouped. (The documentation about this is unbelievably bad even by Mercury’s standards.)
The obscurity of these settings and their rationale are additional reasons why:
a) Load testing has historically tended to be done by one person (who else would want to know these obscure settings?)
b) Why load testing on at a system level tends to be very waterfall,, not agile. These settings must be checked and double checked before tests.
c) Although there is no substitute for running multiple-script, high-user level load tests, it is more agile to run some load tests on the developer level, because the number of settings and variables decreases sharply when a load balancer and a web farm are not involved.
d) Once you have a controller and load generators setup, testing tends to be more accurate and less time-consuming if you leave the set up just were it is and don’t try to move it to another environment. When you move it to another environment, with less load generators, all these settings have to change.
Detailed explanations follow:
1. Why Load Balancing May Not Work As Expected When Testing With IP Spoofing Enabled
LoadRunner KB article 11015, “15-Feb-2007 DNS round-robin load balancing problems”
http://support.merc-int.com/kba/User/LoadRunner/11015.html
I believe what this author is saying is that LoadRunner’s IP spoofing works provides some functional testing from different IP’s, but it doesn’t really balance load.
KB “Solution”
“In this case the load balancing has nothing to do w/the source IP of our traffic but rather the name resolution process. What happens on the generator is that when the first vuser starts LR begins processing the script which begins w/a URL. LR passes that URL to the OS which in turn sends a name resolution request (using the OS's tcp/ip stack) to the generator’s DNS server as defined in it's network config.
Once the name is resolved, the IP address is stored in the DNS cache and since this is a function of the OS and not LoadRunner, the generator’s DNS cache supplies the IP address to Load Runner for each additional vuser.
So yes, we're seen as coming from multiple IP addresses, but we're only resolving the name the first time by the first vuser so we never get another IP address.
In this scenario, we'd need to actually come from 2 separate machines to get 2 actual name resolution requests. We'd also need to ensure that in between the first generator’s request that no other users do a request. Because, if so, then our second machine could actually make the third request (from the DNS servers perspective) and get the same IP as the first generator which results in our test traffic not being load balanced as desired.”
So in other words, IP spoofing is not a solution running load balanced load tests. Mercury actually doesn’t advertise IP Spoofing that way as far as I can see.
Here is a screen shot which illustrates that even when IP spoofed machines are running users, we still have a load imbalance.
2. Run-Time Quotas Must Be Set Per Load Generator
Steps 2, 3 and 4 are quite obscure. However, if we move the controller back and forth between, for example, WAN testing and LAN testing, we must execute these steps.
By default, LoadRunner sets the run time quota for new load generators to 500 vusers. This means if you want to run 100 users in the STAGE environment and you have two load generators, unless you change the default, all these users will all run on the same load generator and the load balancer will interpret this traffic as one user and send all traffic to one web server.
If you want the traffic to come from multiple generators, you must enforce vuser quotas on the generators. When the quota is reached, the vusers fail-over to the next load generator.
Here is a screenshot of how you set quotas:
Assigning Quotas to each load generator.
If we really want to run load tests 100% load balanced, this quota must be set each time we change the number of vusers.
For instance if we are moving from our 50 user tests to our 200 and 350 user tests, we must set these quotas on each load generator.
3. Load Generators Kick Off Sequentially, not Concurrently
LoadRunner increases the user load on each load generator * sequentially *. In other words if you are using 100 users, the first 50 users are run by load generator one. If you have two load generators, load will continue to be sent by load generator one until ½ the total number of users is reached. (Vusers/load generators). Then (about ½ hour later) the load will begin to be sent from the second load generator. If you have a load balancer in front of your web servers, this will result in am imbalanced load against your web farm for a good part of the test until the end of the test after all users are ramped up. (see graph below).
In other words, LoadRunner does not start 1 user on loadgen1 and then immediately start a second user on loadgen2. It divides the total number of users and runs all of loadgen1’s users. Only after these are all running and more users are needed does LoadRunner jump over to loadgen2 start it’s users. When using a load balancer, load will be distributed unevenly until all load generators kick in.
The above sort of scenario does not give a true picture of what your production system looks like under load: there will be a lot of errors occurring on web1, meanwhile no load going on at all on web2 until 20 minutes into the test. The final LoadRunner report will show more errors than really happens in production where the web servers would be hit evenly.
You can verify this by looking at LoadRunner’s Load Generator Status Dialog:
In addition to setting vuser quotas, a work-around for this problem is to shorten the time before it takes LoadRunner to failover to the next load generator. In other words, have more load generators.
The more load generators you have, the quicker, during a load test, do you start to see load balancing across the web servers on your web farm.
4. To Load Balance Each Individual Script, the script Must use a Group of Load Generators (not an individual load generator) (Undocumented LoadRunner Feature #562)
This step really just gives you the ability to force each particular script to run from more than one load generator, hence ensuring that this particular script is load balanced. (In other words, we want concurrency (all the scripts) on each server in the web farm. We don’t want script “A” always and only to run on web server “1.” We want scripts A,B, C and D to run on web server one as well as on web server 2. In a load balanced web farm, this will only occur if the load balancer perceives the traffic as coming from different IP’s AND we also direct each script to run on each of the load generators.
In order to force one script to run from more than one load generator, the load generators must be “grouped” in the LoadRunner scenario manager. (This is contrary, as far as I know, to the default behavior of other performance testing applications). However, LoadRunner provides almost no documentation about how to group load generators . I only found one KB article:
http://support.merc-int.com/kba/User/LoadRunner/6211.html
Solution
“In the scenario, once a script has been added, and lets say 100 vusers have been added, 50 to be assigned to LoadGen A and 50 to LoadGen B, simply go to Scenario->Covert Scenario to Percentage Mode..select LoadGens A & B from the Load Generators List and uncheck the option Scenario->Convert Scenario to Percentage Mode. “
Note: You MUST have more than one vuser assigned to each script for this to work. Scripts with only one vuser will have be assigned only one generator.
Set up all your load generators
- Change the scenario to % mode
- Set the scripts to use “All generators”(cannot do this until mode is changed) (Scenario à Convert Load to Percentage Mode)
- Click on the “Generators” button
- Delete any load generators you DO NOT want to participate in the test (this could happen for instance if you move the controller to SIT or a WAN and use only it for load generation.)
- in the “Load Generators” dialog, highlight the generators you DO want participating in the test
- and then, while the desired generators are highlighted, uncheck the “Convert to % mode” (or “Convert Scenario to Vuser Group Mode”)
- Re-check the number of vusers for each script. These loose a few users sometimes in the conversion process.
This leaves each script with multiple load generators to run from.
Converting the scripts to Use All Load Generators
In other words, to get a script with 100 users to run on two load generators, the screen shot below shows what you have to do:
* for more on this see the short appendix at the end of this email
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Appendix: How to Covert LoadRunner to Group Mode (from the LR Help)
Converting a Scenario to the Vuser Group Mode
You can convert a scenario created in the Percentage Mode to the Vuser Group Mode by selecting Scenario > Convert Scenario to the Vuser Group Mode.
Notes:
- You can also convert a scenario in the Vuser Group Mode to the Percentage Mode. For more information, see About Creating a Manual Scenario Using the Percentage Mode .
- LoadRunner displays a warning that you are about to convert your Manual scenario from Vuser Group Mode to Percentage Mode, or vice versa. If you want to convert your scenario, click Yes. If you want to remain in the current mode, click No.
- Always show this dialog box before converting the scenario: Clear this box if you do not want to be prompted with the current warning. To restore the current warning, select Scenario > Show Convert Scenario Warning.
When converting your scenario from Percentage Mode to Vuser Group Mode, note the following:
- Each script will be converted to a Vuser group.
- If you defined multiple load generators for a Vuser script, the Vuser group that is created when converting the scenario will also contain multiple load generators.
- All schedule settings will be kept.
Appendix on MAC addresses
As noted earlier , the problem with DNS may really be a problem with MAC addresses. On a local LAN, once the load balancer gets a hold of the mac address, it goes by that and not the IP. It's "ARPS" the load generator NIC and then knows it by the MAC. Spoofed IP's have only one MAC address.
These instructions assume you have set up all the load generators. You want to stick with the same number of load generators throughout your testing. Otherwise you will get slightly different curves especially at the lower ends.
Troubleshooting
These difficulties are all due to the fact that Mercury only allows one controller and one launcher. It was much easier with multiple copies of VSTS.
Set Vuser limits to a very high number. If you don't do this, you will get errors in the controller: "The maximum number of vusers of this type has already been started on this machine.
 |
 |
 |