GUI-Level Testing Scalability Scenarios

System Test Environment

Silk Performer 2008
Silk Test 2008
4 GB RAM
BroadCom 57xx Gigabit Controller
Dual (2 x 2.4 GHz) processors
167 GB free space hard disk

Note: The results below should be used as a guideline only and your results may differ depending on the application under test and the available resources of specific Silk Performer agent machines.

Test Scenarios

Executing a 1 VUser test over a 10 minute period

40k of Memory for Performer.exe
7.2k of Memory for the PerfRun.exe (Silk Performer replay engine)

During the test, the VUser will launch a terminal service session which in turns then launches the following processes (this scenario is true for every VUser running a test).

Windows terminal services = 6.3k
Partner.exe (Silk Test) = 21k
Agent.exe (Silk Test Agent) = 8k
Perfsm.exe (Silk Performer Session Manager) = 1.09k
Testapp.exe (application under test) = 6.5k (this is the only real variable within the terminal service session in a load test scenario)

Executing a 5 VUser increasing workload test over a 10 minute period, with VUser per process set to 1

CPU% = anywhere between 1% to 51% depending on which state the Silk Test process was in while executing inside the terminal service - mostly around 25% utilization
Memory = consistently around 48% to 50% throughout duration of test
Responsiveness = 100% consistently
PerfRun.exe = 7.2k of Memory

Executing a 5 VUser increasing workload test over a 10 minute period, with VUser per process set to 5

CPU% = anywhere between 1% to 51% depending on which state the Silk Test process was in while executing inside the terminal service - mostly around 25% utilization
Memory = constantly around 46% to 48% throughout duration of test
Responsiveness = 100% constantly
PerfRun.exe (single) = 10.5 k consistently throughout the test

Executing a 10 VUser increasing workload test over a 10 minute period, with VUser per process set to Automatic Calculation (resulting in 10 PerfRun.exe files being launched)

CPU% = anywhere between 1% to 63% depending on which state the Silk Test process was in while executing inside the terminal service, mostly around 23% or 41% utilization
Memory = constantly around 62% throughout duration of test
Responsiveness = 100% constantly
PerfRun.exe (each) = 7.5 k consistently throughout the test

Executing a 20 VUser increasing workload test over a 10 minute period, with VUser per process set to 20

CPU% = anywhere between 17% to 45% depending on which state the Silk Test process was in while executing inside the terminal service, mostly around 25% or 38% utilization
Memory = constantly around 76% throughout duration of test
Responsiveness = 100% constantly
PerfRun.exe (each) = 15 k consistently throughout the test

Executing a 25 VUser increasing workload test over a 15 minute period, with VUser per process set to 25

CPU% = anywhere between 22% to 45% depending on which state the Silk Test process was in while executing inside the terminal service, mostly around 25% or 38% utilization
Memory = constantly around 83% throughout duration of test (when all VUsers were running)
Responsiveness = 100% constantly
PerfRun.exe (one) = 15 k consistently throughout the test

Executing a 36 VUser increasing workload test over a 18 minute period, with VUser per process set to 18 (for example, two PerfRun.exe files executing all 36 VUsers)

CPU% = anywhere between 22% to 45% depending on which state the Silk Test process was in while executing inside the terminal service, mostly around 25% or 38% utilization
Memory = constantly around 98% to 100% throughout duration of test (when all VUsers were running)
Responsiveness = 100% constantly
PerfRun.exe (each) = 19 k consistently throughout the test

Note: The above values were calculated with no logging enabled (no TrueLog, no vuser.log, etc., files being generated) as this would add to the resource consumption of the load test . To summarize, it can be said that both CPU and memory are the limiting scalability factors when executing a GUI-level load test . However, if you specify the setting VUser per process to as high a value as possible (although never higher than 25 VUsers per process) then this will significantly reduce both the CPU and the memory utilization of the load test . It is also worth noting that the greater the memory footprint of the application under test, then the less VUsers you will be able to scale successfully during the test.

Additional Information

Additional information on executing the load test on machines which use Multiple Processors:

As long as you have more replay engines (PerfRun.exe) running than you have CPUs available on the machine, the current implementation of GUI-level support in Silk Performer does not have any limitations regarding the number of processors that can be used on a Silk Performer agent installation. During a load test scenario, each Silk Performer replay engine (PerfRun.exe) will be bound to a particular processor in a round robin method, for example: PerfRun.exe1 to CPU1, PerfRun.exe2 to CPU2, PerfRun.exe3 to CPU3, perfRun.exe4 to CPU4, and so on.