Benchmark

• 1: Agents
• 2: HTTP
• 3: Phases
• 4: Scenario
• 5: Variables
• 6: Templates
• 7: Hooks
• 8: Ergonomics

1 - Agents

This section can be omitted in standalone mode.

Agents section forms either a list or map with arbitrary agent names and either an inline or properties-style definition:

agents:
  someAgent: "inline definition"
  otherAgent:
    foo: bar

The definition is passed to an instance of i.h.api.deployment.Deployer which will interpret the definition. Deployer implementation is registred using the java.util.ServiceLoader and selected through the io.hyperfoil.deployer system property. The default implementation is ssh.

Common properties

SSH deployer

The user account running Hyperfoil Controller must have a public-key authorization set up on agents’ hosts using key $HOME/.ssh/id_rsa. It also has to be able to copy files into the dir directory using SCP - all the required JARs will be copied there and you will find the logs there as well.

ssh deployer accepts either the [user@]host[:port] inline syntax or these properties:

See an example of ssh deployment configuration:

agents:
  agent1: testserver1:22
  agent2: testuser@testserver2
  agent3:
    host: testserver3
    port: 22
    dir: /some/other/path

Kubernetes/Openshift deployer

To activate the kubernetes deployer you should set -Dio.hyperfoil.deployer=k8s; the recommended installation does that automatically.

The agents are configured the same way as with SSH deployment, only the properties differ. Full reference is provided below.

Example:

agents:
  my-agent:
    node: my-worker-node

2 - HTTP

All servers that Hyperfoil should contact must be declared in this section. Before the benchmark starts Hyperfoil agents will open connections to the target servers; if this connection fails the benchmark is terminated immediatelly.

You can either declare single target server (the default one) within this section or more of them:

http:
  host: http://example.com
  ...

http:
- host: http://example.com
  sharedConnections: 100
- host: http://example.com:8080
  sharedConnections: 50

HTTP configuration has these properties:

Shared connections

This number is split between all agents and executor threads evenly; if there are too many agents/executors each will get at least 1 connection.

When a scalar value is used for this property the connection pool has fixed size; Hyperfoil opens all connections when the benchmark starts and should a connection be closed throughout the benchmark, another connection is reopened instead. You can change this behaviour by composing the property of these sub-properties:

Example:

http:
  host: http://example.com
  sharedConnections:
    core: 10
    buffer: 10
    max: 10
    keepAliveTime: 30000

Connection strategies

This property describes the connection pooling model, you can choose from the options below:

KeyManager configuration

All files are loaded when the benchmark is constructed, e.g. on the machine running CLI. You don’t need to upload any files to controller or agent machines.

TrustManager configuration

All files are loaded when the benchmark is constructed, e.g. on the machine running CLI. You don’t need to upload any files to controller or agent machines.

3 - Phases

You might want to simulate several types of workloads at once: e.g. in an eshop users would come browsing or buying products, and operators would restock the virtual warehouse. Also, driving constant load may not be the best way to run the benchmark: often you want to slowly ramp the load up to let the system adjust (scale up, perform JIT, fill pools) and push the full load only after that. When trying to find system limits, you do the same repetitevely - ramp up the load, measure latencies and if the system meets SLAs (latencies below limits) continue ramping up the load until it breaks.

In Hyperfoil, this all is expressed through phases. Phases can run independently of each other; these simulate certain load execute by a group of users. Within one phase all users execute the same scenario (e.g. logging into the system, buying some goods and then logging off).

A phase can be in one of these states:

• not running (scheduled): As the name clearly says, the phase is not yet getting executed.
• running: The agent started running the phase, i.e., performing the configured load.
• finished: Users won’t start new scenarios but we’ll let already-started users complete the scenario.
• terminated: All users are done, all stats are collected and no further requests will be made.
• cancelled: Same as terminated but this phase hasn’t been run at all.

There are different types of phases based on the mode of starting new users:

See the example of phases configuration:

...
phases:
# Over one minute ramp the number of users started each second from 1 to 100
- rampUp:
    increasingRate:
      initialUsersPerSec: 1
      targetUsersPerSec: 100
      # We expect at most 200 users being active at one moment - see below
      maxSessions: 200
      duration: 1m
      scenario: ...
# After rampUp is finished, run for 5 minutes and start 100 new users each second
- steadyState:
    constantRate:
      usersPerSec: 100
      maxSessions: 200
      startAfter: rampUp
      duration: 5m
      # If some users get stuck, forcefully terminate them after 6 minutes from the phase start
      maxDuration: 6m
      scenario: ...
# 2 minutes after the benchmark has started spawn 5 users constantly doing something for 2 minutes
- outOfBand:
    always:
      users: 5
      startTime: 2m
      duration: 2m
      scenario: ...
- final:
    atOnce:
      users: 1
      # Do something at the end: make sure that both rampUp and steadyState are terminated
      startAfterStrict:
      - rampUp
      - steadyState
      scenario: ...

These properties are common for all types of phases:

Below are properties specific for different phase types:

• atOnce:
  • users: Number of users started at the start of the phase.
• always:
  • users: Number of users started at the start of the phase. When a user finishes it is immediatelly restarted (any pause must be part of the scenario).
• constantRate:
  • usersPerSec: Number of users started each second.
  • variance: Randomize delays between starting users following the exponential distribution. That way the starting users behave as the Poisson point process. If this is set to false users will be started with uniform delays. Default is true.
  • maxSessions: Number of preallocated sessions. This number is split between all agents/executors evenly.
• increasingRate / decreasingRate:
  • initialUsersPerSec: Rate of started users at the beginning of the phase.
  • targetUsersPerSec: Rate of started users at the end of the phase.
  • variance: Same as in `constantRate
  • maxSessions: Same as in constantRate.

Hyperfoil initializes all phases before the benchmark starts, pre-allocating memory for sessions. In the open-model phases it’s not possible to know how many users will be active at the same moment (if the server experiences a 3-second hiccup and we have 100 new users per second this should be at least 300 as all the users will be blocked). However we need to provide the estimate for memory pre-allocation. If the estimate gets exceeded the benchmark won’t fail nor block new users from starting, but new sessions will be allocated which might negatively impact results accuracy.

Properties users, usersPerSec, initialUsersPerSec and targetUsersPerSec can be either a scalar number or scale with iterations using the base and increment components. You’ll see an example below.

Forks

As mentioned earlier, users in each phase execute the same scenario. Often it’s convenient to define the ramp-up and steady-state phases just once: the builders allow to declare such ‘sub-phases’ called forks. For all purposes but the benchmark configuration these become regular phases of the same type, duration and dependencies (startAfter, startAfterStrict) as the ‘parent’ phase but slice the users according to their weight:

...
phases:
- steadyState:
    constantRate:
      usersPerSec: 30
      duration: 5m
      forks:
        sellShares:
          # This phase will start 10 users per second
          weight: 1
          scenario: ...
        buyShares:
          # This phase will start 20 users per second
          weight: 2
          scenario: ...

These phases will be later identified as steadyState/sellShares and steadyState/buyShares. Other phases can still reference steadyState (without suffix) as the dependency: there will be a no-op phase steadyState that starts (becomes running) as soon as both the forks finish, finish immediately and terminate once both the forks terminate.

Iterations

In some types of tests it’s useful to repeat given phase with increasing load - we call this concept iterations. In the example below you can see that *usersPerSec are not scalar values; in first iteration the actual value is set to the base value but in each subsequent iteration the value is increased by increment.

...
phases:
- rampUp:
    increasingRate:
      # Create phases rampUp/000, rampUp/001 and rampUp/002
      maxIterations: 3
      # rampUp/000 will go from 1 to 100 users, rampUp will go from 101 to 200 users...
      initialUsersPerSec:
        base: 1
        increment: 100
      targetUsersPerSec:
        base: 100
        increment: 100
      # rampUp/001 will start after steadyState/000 finishes
      startAfter:
        phase: steadyState
        iteration: previous
      duration: 1m
      scenario: ...
- steadyState:
    constantRate:
      maxIterations: 3
      usersPerSec:
        base: 100
        increment: 100
      # steadyState/000 will start after rampUp/000 finishes
      startAfter:
        phase: rampUp
        iteration: same
      duration: 5m

Similar to forks, there will be a no-op phase rampUp that will start after all rampUp/xxx phases finish and terminate after these terminate. Also there’s an implicit dependency between consecutive iterations: subsequent iteration won’t start until previous iteration terminates.

The startAfter property in this example uses a relative reference to iteration in another phase. Each reference has these properties:

| Property | Description | | ——— | | | phase | Name of the referenced phase. | | iteration | Relative number of the iteration; either none (default) which references the top-level phase, same meaning the iteration with same number, or previous with number one lower. | | fork | Reference to particular fork in the phase/iteration. |

Iterations can be combined with forks as well - the result name would be e.g. steadyState/000/sellShares.

Note that the maxSessions parameter is not scaling in iterations: all iterations execute the same scenario, the execution does not overlap and therefore it is possible to share the pool of sessions. Therefore you should provide an estimate for the iteration spawning the highest load.

Staircase

Hyperfoil tries to make opinionated decisions, simplifying common types of benchmark setups. That’s why it offers a simplified syntax for the scenario where you:

• ramp the load to a certain level
• execute steady state for a while
• ramp it up further
• execute another steady state
• repeat previous two steps over and over

This is called a staircase as the load increases in a shape of tilted stairs. Phases such benchmark should consist of are automatically created and linked together, using the same scenario/forks.

staircase as a top-level element in the benchmark is mutually exclusive to scenario and phases elements.

Here is a minimalistic example of such configuration:

name: simple benchmark
http:
  host: http://localhost:8080
staircase:
  initialRampUpDuration: 10s
  steadyStateDuration: 10s
  rampUpDuration: 5s
  initialUsersPerSec: 5
  incrementUsersPerSec: 1
  maxIterations: 3
  scenario:
  - test:
    - httpRequest:
        GET: /foo

This element uses these properties:

4 - Scenario

Scenario

Scenario is a set of sequences. The sequence is a block of sequentially executed steps. Contrary to steps in a sequence the sequences within a scenario do not need to be executed sequentially.

The scenario defines one or more initialSequences that are enabled from the beginning and other sequences that must be enabled by any of the previously executed sequences. To be more precise it is not the sequence that is enabled but a sequence instance as we can run a sequence multiple times in parallel (on different data). The initialSequences enable one instance of each of the referenced sequence.

The session keeps a currently executed step for each of the enabled sequence instances. The step can be blocked (e.g. waiting for a response to come). The session is looping through current steps in each of the enabled sequence instances and if the step is not blocked, it is executed. There’s no guaranteed order in which non-blocked steps from multiple enabled sequence instances will be executed.

Here is an example of scenario:

scenario:
  initialSequences:
  - login:
    - httpRequest:
        POST: /login
    # Enable instance of sequence 'wait5seconds'
    - next: wait5seconds
  sequences:
  - wait5seconds:
    - thinkTime:
        duration: 5s
    - next: logout
  - logout:
    - httpRequest:
        POST: /logout

While this generic approach is useful for complex scenarios with branching logic, simple sequential scenarios can use orderedSequences short-cut enabling sequences in given order:

scenario:
  orderedSequences:
  - login:
    - httpRequest:
        POST: /login
  - wait5seconds:
    - thinkTime:
        duration: 5s
  - logout:
    - httpRequest:
        POST: /logout

This syntax makes the first sequence (login in this case) an initial sequence, adds the subsequent sequences and as the last step of each but the last sequence appends a next step scheduling a new instance of the following sequence.

To make configuration even more concise you can omit the orderedSequences level and start defining the list of sequences under scenario right away:

scenario:
- login:
  - httpRequest:
      POST: /login
- wait5seconds:
  - thinkTime:
      duration: 5s
- logout:
  - httpRequest:
      POST: /logout

An exhaustive list of steps can be found in the steps reference.

5 - Variables

All but the simplest scenarios will use session variables. Hyperfoil sports steps that generate values into these variables (randomInt, randomItem, …), processors that write data from other sources to variables (store, array) and many places that read variables and use the values to perform some operations (httpRequest.path ) or alter control flow.

Hyperfoil uses different types of variables (slots in the session) for integer variables and generic objects (commonly strings). When a numeric value is received as a string (e.g. when parsing response headers) and you want to use it in a step that expects exclusively integral values you have to convert it explicitly, e.g. using the stringToInt action. Steps that read values to form a string can usually consume both types of variables, without any need for conversion.

Besides user-defined variables there are some read-only pseudo-variables that can be used in the scenario as if these were regular variables:

String interpolation

Components that accept string values usually allow you to use a pattern - parts of the string can be replaced in runtime with the value from a session variable. A simple example of pattern would be The quick brown ${wild-animal} jumps over the lazy ${domestic-animal} - variables wild-animal and domestic-animal would get replaced with their respective values.

When you really want to use ${wild-animal} in a value for such component you should escape it with one more dollar sign: This $${variable} won't be replaced will be rendered into This ${variable} won't be replaced.

There are a few transformations that you can perform with a variable value while interpolating the pattern:

• ${urlencode:my-variable} will replace characters in the my-variable using URLEncoder.encode (using UTF-8 encoding).
• ${{ '{%05d' }}:my-number} and other formatter strings ending with d, o, x or X will convert an integer variable using Formatter.
• ${replace/<regexp>/<replacement>/<flags>:my-variable} perform Java regexp replacement on my-variable contents. Note that you can use any character after replace, not just / - this becomes the separator between regexp, replacement and flags. The only flags currently supported is g - replacing all occurences of that string (by default only first occurence is replaced).

Sequence-scoped access

When an array or collection is stored in a session variable you can access the individual elements by appending [.] to the variable name, e.g. my-variable[.]. You can see that we don’t use the actual index into the array: instead we use current sequence instance index. You can read more about running multiple sequences concurrently in the Architecture/Scenario Execution.

6 - Templates

It is often useful to keep a single benchmark in version control but change parts of it depending on the infrastructure where it is executed or intended load. Since version 0.18 Hyperfoil supports parametrization of the benchmark through templates.

Inspired by other (more complex) YAML templating systems we decided to use YAML tags to pre-process the YAML. Templating happens even before applying the YAML nodes onto BenchmarkBuilder, therefore it is not possible to do that programmatically or with the serialized form.

If you are working with CLI or WebCLI there is little difference to regular benchmarks: you upload and edit the benchmark as usual. However it is not possible to auto-detect files before the benchmark is constructed from the template (the reference to a file could be a template, too!), therefore you need to pass all files using option -f to the upload/edit command.

When the benchmark template is uploaded, upon running it (run mybenchmark) you either pass the parameters using option -P or you are interactively asked to provide those params. The parameters are stored in CLI context and on subsequent invocations of run you don’t need to set these. If you want to remove the parameters from the context use option -r/--reset-params. To see both default and current parameters you can use the inspect command.

Param

You should use !param to replace single scalar value:

1name: example
2http:
3  host: http://localhost:8080
4usersPerSec: !param NUM_USERS
5duration: !param DURATION 60s
6scenario: # ...

In this simple constant-rate benchmark you can customize the number of users starting each second as well as the duration. There’s no default for NUM_USERS; you will be asked to provide it when you run the benchmark. On the other hand DURATION has a default value of 60s - anything after the space after parameter name counts as the default value.

Parameters don’t have to be upper-case. The identifier is case-sensitive, though.

run scalar-value-example -PNUM_USERS=5 -PDURATION=60s

Concat

Sometimes you need to replace only part of a string: !concat will let you do that:

1name: example
2http:
3  host: !concat [ "http://", !param SERVER localhost, ":8080" ]
4usersPerSec: 10
5duration: 60s
6scenario: # ...

In this example we will customize the host with the concatenation of http://, parameter SERVER with default localhost and :8080. This example uses inline-form of list, though you can use the regular list (one item per line), too.

Foreach

Chances are you need to generate a list based on a param: you can do this using the !foreach:

 1name: example
 2http: !foreach
 3  items: http://example.com,http://hyperfoil.io
 4  separator: "," # comma is the default separator
 5  param: ITEM   # ITEM is the default parameter name
 6  do:
 7    host: !param ITEM
 8usersPerSec: 10
 9duration: 60s
10scenario: # ...

This splits the items using the separator regexp and produces a list of values or mappings while the param ITEM is set to one of the values from items list. The example above would result in:

name: example
http:
- host: http://example.com
- host: http://hyperfoil.io
usersPerSec: 10
duration: 60s
scenario: # ...

You can also set items to a YAML list; in that case the separator is not used:

myList: !foreach
  items: ["A", !param B, "C"]
  param: FOO
  do: !param FOO

The last example with -PB=bar would result in:

myList:
- A
- bar
- C

Renaming the param used for iteration can be useful in nested loops: without renaming the inner foreach would shadow the outer one.

Anchors and aliases

YAML has a built-in concept for removing repetitive sections: anchors and aliases. With the templating system you can use that universally throughout the file (in versions before 0.18 the support was limited to forks, scenarios and sequences):

foo: &hello-world
  hello: world
anotherFoo:
  sayHi: *hello-world
  myList:
  - *hello-world
  - bar

is interpretted as

foo:
  hello: world
anotherFoo:
  sayHi:
    hello: world
  myList:
  - hello: world
  - bar

7 - Hooks

It might be useful to run certain scripts before and after the run, e.g. starting some infrastructure, preloading database, gathering CPU stats during the test and so on. That’s why Hyperfoil introduces pre- and post-hooks to the run.

Some scripts are not specific to the test being run - these should be deployed on controller as files in *root*/hooks/pre/ and *root*/hooks/post directories where root is controller’s root directory, /tmp/hyperfoil/ by default. Each of these directories should contain executable scripts or binaries that will be run in alphabetic order. We strongly suggest using the format 00-my-script.sh to set the order using first two digits.

Kubernetes/Openshift deployments use the same strategy; the only difference is that the pre and post directories are mapped as volumes from a ConfigMap resource.

Other scripts may be specific to the benchmark executed and therefore you can define them directly in the YAML files. You can either use inline command that will be executed using sh -c your-command --your-options or create a Java class implementing io.hyperfoil.core.hooks.RunHook and register it to be loaded as other Hyperfoil extensions.

name: my-benchmark
pre:
  01-inline: curl http://example.com
  02-custom:
    my-hook:
      foo: bar
post:
  99-some-final-hook: ...
...

The lists of hooks from controller directories and benchmark are merged; if there’s a conflict between two hooks from these two sources the final execution order is not defined (but both get executed).

In case of inline command execution the stderr output will stay on stderr, stdout will be caputered by Hyperfoil and stored in *rundir*/*XXXX*/hooks.json. As the post-hooks are executed after info.json and all.json get written the output cannot be included inside those files. This order of execution was chosen because it’s likely that you will upload these files to a database - yes, using a post-hook.

8 - Ergonomics

This section hosts only single property at this moment: