User Guide

• 1: Installation
• 1.1: Manual startup
• 1.2: K8s/Openshift deployment
• 1.3: Manual k8s/Openshift deployment
• 1.4: Ansible startup
• 2: Benchmark
• 2.1: Agents
• 2.2: HTTP
• 2.3: Phases
• 2.4: Scenario
• 2.5: Variables
• 2.6: Templates
• 2.7: Hooks
• 2.8: Ergonomics
• 3: Examples
• 4: Troubleshooting

1 - Installation

In this section, you’ll find detailed instructions for installing and setting up Hyperfoil using various methods, including manual setup, Ansible, and Kubernetes/Openshift. Follow these guides to choose the best installation procedure for your environment.

1.1 - Manual startup

Hyperfoil controller is started with

bin/controller.sh

Any arguments passed to the scripts will be passed as-is to the java process.

By default io.hyperfoil.deployer is set to ssh which means that the controller will deploy agents over SSH, based on the agents configurion. This requires that the user account running the controller must have public-key SSH authorization set up using key $HOME/.ssh/id_rsa. The user also has to be able to copy files to the directory set in agent definition (by default /tmp/hyperfoil) using SCP - Hyperfoil automatically synchronizes library files in this folder with the currently running instance and then executes the agent.

When you don’t intend to run distributed benchmarks you can start the controller in standalone mode:

bin/standalone.sh

This variant won’t deploy any agents remotely and therefore it does not need any agents: section in the benchmark definition; instead it will use single agent started in the same JVM.

Below is the comprehensive list of all the properties Hyperfoil recognizes. All system properties can be replaced by environment variables, uppercasing the letters and replacing dots and dashes with underscores: e.g. io.hyperfoil.controller.host becomes IO_HYPERFOIL_CONTROLLER_HOST.

If io.hyperfoi.trigger.url is set the controller does not start benchmark run right away after hitting /benchmark/my-benchmark/start ; instead it responds with status 301 and header Location set to concatenation of this string and BENCHMARK=my-benchmark&RUN_ID=xxxx. CLI interprets that response as a request to hit CI instance on this URL, assuming that CI will trigger a new job that will eventually call /benchmark/my-benchmark/start?runId=xxxx with header x-trigger-job. This is useful if the the CI has to synchronize Hyperfoil to other benchmarks that don’t use this controller instance.

Security

Since Hyperfoil accepts and invoked any serialized Java objects you must not run it exposed to public to prevent a very simple remote code execution. Even if using HTTPS and password protection (see below) we recommend to limit access and privileges of the process to absolute minimum.

You can get confidential access to the server using TLS encryption, providing the certificate and keys either using Java Keystore mechanism (properties above starting with io.hyperfoil.controller.keystore) or via PEM files (properties starting with io.hyperfoil.controller.pem). These options are mutually exclusive. In the latter case it is possible to use multiple certificate/key files, separated by comma (,).

Authentication uses Basic authentication scheme accepting any string as username. The password is set using io.hyperfoil.controller.password or respective environment variable. If you’re exposing the server using plaintext HTTP you must set -Dio.hyperfoil.controller.secured.via.proxy=true to confirm that this is a desired configuration (e.g. if the TLS is terminated at proxy and the connection from proxy does not require confidentiality).

1.2 - K8s/Openshift deployment

A convenient alternative to running Hyperfoil on hosts with SSH access is deploying it in Kubernetes or Openshift environment. The recommended way to install it using an operator in your Openshift console - just go to Operators - OperatorHub and search for ‘hyperfoil’, and follow the installation wizard. Alternatively you can deploy the controller manually.

In order to start a Hyperfoil Controller instance in your cluster, create a new namespace hyperfoil: Go to Operators - Installed Operators and open Hyperfoil. In upper left corner select ‘Project: ’ - Create project and fill out the details. Then click on the ‘Hyperfoil’ tab and find the button ‘Create Hyperfoil’.

You should see a YAML definition like this:

apiVersion: hyperfoil.io/v1alpha2
kind: Hyperfoil
metadata:
  name: example-hyperfoil
  namespace: hyperfoil
spec:
  agentDeployTimeout: 60000
  log: myConfigMap/log4j2-superverbose.xml
  route:
    host: hyperfoil.apps.mycloud.example.com
  version: latest

Change the name to just hyperfoil (or whatever you prefer) and delete all the content from the spec section:

apiVersion: hyperfoil.io/v1alpha2
kind: Hyperfoil
metadata:
  name: hyperfoil
  namespace: hyperfoil
spec:

This is a perfectly valid Hyperfoil resource with everything set to default values. You can customize some properties in the spec section further - see the reference.

The operator deploys only the controller; each agent is then started when the run starts as a pod in the same namespace and stopped when the run completes.

When the resource becomes ready (you can check it out through Openshift CLI using oc get hf) the controller pod should be up and running. Now you can open Hyperfoil CLI and connect to the controller. While default Hyperfoil port is 8090, default route setting uses TLS (edge) and therefore Openshift router will expose the service on port 443. If your cluster’s certificate is not recognized (such as when using self-signed certificates) you need to use --insecure (or -k) option.

bin/cli.sh

[hyperfoil]$ connect hyperfoil-hyperfoil.apps.my.cluster.domain:443 --insecure
WARNING: Hyperfoil TLS certificate validity is not checked. Your credentials might get compromised.
Connected!
WARNING: Server time seems to be off by 12124 ms

Now you can upload & run benchmarks as usual - we’re using {% include example_link.md src=‘k8s-hello-world.hf.yaml’ %} in this example. Note that it can take several seconds to spin up containers with agents.

[hyperfoil@hyperfoil-hyperfoil]$ upload examples/k8s-hello-world.hf.yaml
Loaded benchmark k8s-hello-world, uploading...
... done.

[hyperfoil@hyperfoil-hyperfoil]$ run k8s-hello-world
Started run 0000
Run 0000, benchmark k8s-hello-world
Agents: agent-one[STARTING]
Started: 2019/11/18 19:07:36.752    Terminated: 2019/11/18 19:07:41.778
NAME  STATUS      STARTED       REMAINING  COMPLETED     TOTAL DURATION               DESCRIPTION
main  TERMINATED  19:07:36.753             19:07:41.778  5025 ms (exceeded by 25 ms)  5.00 users per second

[hyperfoil@hyperfoil-hyperfoil]$

You can find more details about adjusting the agents in the benchmark format reference.

Running Hyperfoil inside the cluster you are trying to test might skew results due to different network topology compared to driving the load from ‘outside’ (as real users would do). It is your responsibility to validate if your setup and separation between load driver and SUT (system under test) is correct. You have been warned.

Reference

route

auth

1.3 - Manual k8s/Openshift deployment

If you cannot use the operator or if you’re running vanilla Kubernetes you can define all the resource manually. You deploy only the controller; each agent is then started, when the run starts, as a pod in the same namespace and stopped when the run completes.

Following steps install Hyperfoil controller in Openshift, assuming that you have all the required priviledges. With vanilla Kubernetes you might have to replace the route with an appropriate ingress.

1. Create new namespace for hyperfoil

oc new-project hyperfoil

2. Create required resources

curl -s -L k8s.hyperfoil.io | oc apply -f -

role.rbac.authorization.k8s.io/controller created
serviceaccount/controller created
service/hyperfoil created
rolebinding.rbac.authorization.k8s.io/controller created
deploymentconfig.apps.openshift.io/controller created
route.route.openshift.io/hyperfoil created

The route will use hostname following the format hyperfoil-hyperfoil.apps.my.cluster.domain - feel free to customize the hostname as needed.

3. Wait until the image gets downloaded and the container starts

oc get po

NAME                  READY   STATUS              RESTARTS   AGE
controller-1-pqbvs    1/1     Running             0          57s
controller-1-deploy   0/1     Completed           0          72s

4. Open CLI and connect to the controller

While default Hyperfoil port is 8090, Openshift router will expose the service on port 80.

bin/cli.sh

[hyperfoil]$ connect hyperfoil-hyperfoil.apps.my.cluster.domain -p 80
Connected!
WARNING: Server time seems to be off by 12124 ms

5. Upload and run benchmarks as usual

We’re using k8s-hello-world.hf.yaml in this example.

name: k8s-hello-world
agents:
  agent-one:
http:
  host: https://kubernetes.default.svc.cluster.local
duration: 5s
usersPerSec: 5
scenario:
- test:
  - httpRequest:
      GET: /

Note that it can take several seconds to spin up containers with agents.

[hyperfoil@hyperfoil-hyperfoil]$ upload .../k8s-hello-world.hf.yaml
Loaded benchmark k8s-hello-world, uploading...
... done.

[hyperfoil@hyperfoil-hyperfoil]$ run k8s-hello-world
Started run 0000
Run 0000, benchmark k8s-hello-world
Agents: agent-one[STARTING]
Started: 2019/11/18 19:07:36.752    Terminated: 2019/11/18 19:07:41.778
NAME  STATUS      STARTED       REMAINING  COMPLETED     TOTAL DURATION               DESCRIPTION
main  TERMINATED  19:07:36.753             19:07:41.778  5025 ms (exceeded by 25 ms)  5.00 users per second

[hyperfoil@hyperfoil-hyperfoil]$

You can find more details about adjusting the agents in the benchmark format reference.

Running Hyperfoil inside the cluster you are trying to test might skew results due to different network topology compared to driving the load from ‘outside’ (as real users would do). It is your responsibility to validate if your setup and separation between load driver and SUT (system under test) is correct. You have been warned.

1.4 - Ansible startup

You can fetch release, distribute and start the cluster using Ansible Galaxy scripts; setup, test, shutdown

First, get the scripts:

ansible-galaxy install hyperfoil.hyperfoil_setup,{{ site.last_release.galaxy_version }}
ansible-galaxy install hyperfoil.hyperfoil_shutdown,{{ site.last_release.galaxy_version }}
ansible-galaxy install hyperfoil.hyperfoil_test,{{ site.last_release.galaxy_version }}

Now, edit your hosts file, it could look like this:

[hyperfoil-controller]
controller ansible_host=localhost

[hyperfoil-agent]
agent-1 ansible_host=localhost

Prepare your playbook; here is a short example that starts the controller, uploads and starts simple benchmark (the templating engine replaces the agents in benchmark script based on Ansible hosts) and waits for its completion. When it confirms number of requests executed it stops the controller.

- hosts: [ hyperfoil-agent, hyperfoil-controller ]
  tasks: [] # This will only gather facts about all nodes
- hosts: hyperfoil-controller
  roles:
  - hyperfoil.hyperfoil_setup
- hosts: 127.0.0.1
  connection: local
  roles:
  - hyperfoil.hyperfoil_test
  vars:
    test_name: example
# Note that due to the way Ansible lookups work this will work only if hyperfoil-controller == localhost
- hosts: 127.0.0.1
  connection: local
  tasks:
  - name: Find number of requests
    set_fact:
      test_requests: "{{ lookup('csvfile', 'example file=/tmp/hyperfoil/workspace/run/' + test_runid + '/stats/total.csv col=2 delimiter=,')}}"
  - name: Print number of requests
    debug:
      msg: "Executed {{ test_requests }} requests."
- hosts:
  - hyperfoil-controller
  roles:
  - hyperfoil.hyperfoil_shutdown

Finally, run the playbook:

ansible-playbook -i hosts example.yml

2 - Benchmark

In Hyperfoil, defining a benchmark involves structuring scenarios, phases, variables and other components to simulate realistic user behavior and workload patterns. This section provides a detailed breakdown of each component involved in defining a benchmark.

2.1 - Agents

This section can be omitted in standalone mode.

Agents section forms either a list or map with arbitrary agent names and either an inline or properties-style definition:

agents:
  someAgent: "inline definition"
  otherAgent:
    foo: bar

The definition is passed to an instance of i.h.api.deployment.Deployer which will interpret the definition. Deployer implementation is registred using the java.util.ServiceLoader and selected through the io.hyperfoil.deployer system property. The default implementation is ssh.

Common properties

SSH deployer

The user account running Hyperfoil Controller must have a public-key authorization set up on agents’ hosts using key $HOME/.ssh/id_rsa. It also has to be able to copy files into the dir directory using SCP - all the required JARs will be copied there and you will find the logs there as well.

ssh deployer accepts either the [user@]host[:port] inline syntax or these properties:

See an example of ssh deployment configuration:

agents:
  agent1: testserver1:22
  agent2: testuser@testserver2
  agent3:
    host: testserver3
    port: 22
    dir: /some/other/path

Kubernetes/Openshift deployer

To activate the kubernetes deployer you should set -Dio.hyperfoil.deployer=k8s; the recommended installation does that automatically.

The agents are configured the same way as with SSH deployment, only the properties differ. Full reference is provided below.

Example:

agents:
  my-agent:
    node: my-worker-node

2.2 - HTTP

All servers that Hyperfoil should contact must be declared in this section. Before the benchmark starts Hyperfoil agents will open connections to the target servers; if this connection fails the benchmark is terminated immediatelly.

You can either declare single target server (the default one) within this section or more of them:

http:
  host: http://example.com
  ...

http:
- host: http://example.com
  sharedConnections: 100
- host: http://example.com:8080
  sharedConnections: 50

HTTP configuration has these properties:

Shared connections

This number is split between all agents and executor threads evenly; if there are too many agents/executors each will get at least 1 connection.

When a scalar value is used for this property the connection pool has fixed size; Hyperfoil opens all connections when the benchmark starts and should a connection be closed throughout the benchmark, another connection is reopened instead. You can change this behaviour by composing the property of these sub-properties:

Example:

http:
  host: http://example.com
  sharedConnections:
    core: 10
    buffer: 10
    max: 10
    keepAliveTime: 30000

Connection strategies

This property describes the connection pooling model, you can choose from the options below:

KeyManager configuration

All files are loaded when the benchmark is constructed, e.g. on the machine running CLI. You don’t need to upload any files to controller or agent machines.

TrustManager configuration

All files are loaded when the benchmark is constructed, e.g. on the machine running CLI. You don’t need to upload any files to controller or agent machines.

2.3 - Phases

You might want to simulate several types of workloads at once: e.g. in an eshop users would come browsing or buying products, and operators would restock the virtual warehouse. Also, driving constant load may not be the best way to run the benchmark: often you want to slowly ramp the load up to let the system adjust (scale up, perform JIT, fill pools) and push the full load only after that. When trying to find system limits, you do the same repetitevely - ramp up the load, measure latencies and if the system meets SLAs (latencies below limits) continue ramping up the load until it breaks.

In Hyperfoil, this all is expressed through phases. Phases can run independently of each other; these simulate certain load execute by a group of users. Within one phase all users execute the same scenario (e.g. logging into the system, buying some goods and then logging off).

A phase can be in one of these states:

• not running (scheduled): As the name clearly says, the phase is not yet getting executed.
• running: The agent started running the phase, i.e., performing the configured load.
• finished: Users won’t start new scenarios but we’ll let already-started users complete the scenario.
• terminated: All users are done, all stats are collected and no further requests will be made.
• cancelled: Same as terminated but this phase hasn’t been run at all.

There are different types of phases based on the mode of starting new users:

See the example of phases configuration:

...
phases:
# Over one minute ramp the number of users started each second from 1 to 100
- rampUp:
    increasingRate:
      initialUsersPerSec: 1
      targetUsersPerSec: 100
      # We expect at most 200 users being active at one moment - see below
      maxSessions: 200
      duration: 1m
      scenario: ...
# After rampUp is finished, run for 5 minutes and start 100 new users each second
- steadyState:
    constantRate:
      usersPerSec: 100
      maxSessions: 200
      startAfter: rampUp
      duration: 5m
      # If some users get stuck, forcefully terminate them after 6 minutes from the phase start
      maxDuration: 6m
      scenario: ...
# 2 minutes after the benchmark has started spawn 5 users constantly doing something for 2 minutes
- outOfBand:
    always:
      users: 5
      startTime: 2m
      duration: 2m
      scenario: ...
- final:
    atOnce:
      users: 1
      # Do something at the end: make sure that both rampUp and steadyState are terminated
      startAfterStrict:
      - rampUp
      - steadyState
      scenario: ...

These properties are common for all types of phases:

Below are properties specific for different phase types:

• atOnce:
  • users: Number of users started at the start of the phase.
• always:
  • users: Number of users started at the start of the phase. When a user finishes it is immediatelly restarted (any pause must be part of the scenario).
• constantRate:
  • usersPerSec: Number of users started each second.
  • variance: Randomize delays between starting users following the exponential distribution. That way the starting users behave as the Poisson point process. If this is set to false users will be started with uniform delays. Default is true.
  • maxSessions: Number of preallocated sessions. This number is split between all agents/executors evenly.
• increasingRate / decreasingRate:
  • initialUsersPerSec: Rate of started users at the beginning of the phase.
  • targetUsersPerSec: Rate of started users at the end of the phase.
  • variance: Same as in `constantRate
  • maxSessions: Same as in constantRate.

Hyperfoil initializes all phases before the benchmark starts, pre-allocating memory for sessions. In the open-model phases it’s not possible to know how many users will be active at the same moment (if the server experiences a 3-second hiccup and we have 100 new users per second this should be at least 300 as all the users will be blocked). However we need to provide the estimate for memory pre-allocation. If the estimate gets exceeded the benchmark won’t fail nor block new users from starting, but new sessions will be allocated which might negatively impact results accuracy.

Properties users, usersPerSec, initialUsersPerSec and targetUsersPerSec can be either a scalar number or scale with iterations using the base and increment components. You’ll see an example below.

Forks

As mentioned earlier, users in each phase execute the same scenario. Often it’s convenient to define the ramp-up and steady-state phases just once: the builders allow to declare such ‘sub-phases’ called forks. For all purposes but the benchmark configuration these become regular phases of the same type, duration and dependencies (startAfter, startAfterStrict) as the ‘parent’ phase but slice the users according to their weight:

...
phases:
- steadyState:
    constantRate:
      usersPerSec: 30
      duration: 5m
      forks:
        sellShares:
          # This phase will start 10 users per second
          weight: 1
          scenario: ...
        buyShares:
          # This phase will start 20 users per second
          weight: 2
          scenario: ...

These phases will be later identified as steadyState/sellShares and steadyState/buyShares. Other phases can still reference steadyState (without suffix) as the dependency: there will be a no-op phase steadyState that starts (becomes running) as soon as both the forks finish, finish immediately and terminate once both the forks terminate.

Iterations

In some types of tests it’s useful to repeat given phase with increasing load - we call this concept iterations. In the example below you can see that *usersPerSec are not scalar values; in first iteration the actual value is set to the base value but in each subsequent iteration the value is increased by increment.

...
phases:
- rampUp:
    increasingRate:
      # Create phases rampUp/000, rampUp/001 and rampUp/002
      maxIterations: 3
      # rampUp/000 will go from 1 to 100 users, rampUp will go from 101 to 200 users...
      initialUsersPerSec:
        base: 1
        increment: 100
      targetUsersPerSec:
        base: 100
        increment: 100
      # rampUp/001 will start after steadyState/000 finishes
      startAfter:
        phase: steadyState
        iteration: previous
      duration: 1m
      scenario: ...
- steadyState:
    constantRate:
      maxIterations: 3
      usersPerSec:
        base: 100
        increment: 100
      # steadyState/000 will start after rampUp/000 finishes
      startAfter:
        phase: rampUp
        iteration: same
      duration: 5m

Similar to forks, there will be a no-op phase rampUp that will start after all rampUp/xxx phases finish and terminate after these terminate. Also there’s an implicit dependency between consecutive iterations: subsequent iteration won’t start until previous iteration terminates.

The startAfter property in this example uses a relative reference to iteration in another phase. Each reference has these properties:

| Property | Description | | ——— | | | phase | Name of the referenced phase. | | iteration | Relative number of the iteration; either none (default) which references the top-level phase, same meaning the iteration with same number, or previous with number one lower. | | fork | Reference to particular fork in the phase/iteration. |

Iterations can be combined with forks as well - the result name would be e.g. steadyState/000/sellShares.

Note that the maxSessions parameter is not scaling in iterations: all iterations execute the same scenario, the execution does not overlap and therefore it is possible to share the pool of sessions. Therefore you should provide an estimate for the iteration spawning the highest load.

Staircase

Hyperfoil tries to make opinionated decisions, simplifying common types of benchmark setups. That’s why it offers a simplified syntax for the scenario where you:

• ramp the load to a certain level
• execute steady state for a while
• ramp it up further
• execute another steady state
• repeat previous two steps over and over

This is called a staircase as the load increases in a shape of tilted stairs. Phases such benchmark should consist of are automatically created and linked together, using the same scenario/forks.

staircase as a top-level element in the benchmark is mutually exclusive to scenario and phases elements.

Here is a minimalistic example of such configuration:

name: simple benchmark
http:
  host: http://localhost:8080
staircase:
  initialRampUpDuration: 10s
  steadyStateDuration: 10s
  rampUpDuration: 5s
  initialUsersPerSec: 5
  incrementUsersPerSec: 1
  maxIterations: 3
  scenario:
  - test:
    - httpRequest:
        GET: /foo

This element uses these properties:

2.4 - Scenario

Scenario

Scenario is a set of sequences. The sequence is a block of sequentially executed steps. Contrary to steps in a sequence the sequences within a scenario do not need to be executed sequentially.

The scenario defines one or more initialSequences that are enabled from the beginning and other sequences that must be enabled by any of the previously executed sequences. To be more precise it is not the sequence that is enabled but a sequence instance as we can run a sequence multiple times in parallel (on different data). The initialSequences enable one instance of each of the referenced sequence.

The session keeps a currently executed step for each of the enabled sequence instances. The step can be blocked (e.g. waiting for a response to come). The session is looping through current steps in each of the enabled sequence instances and if the step is not blocked, it is executed. There’s no guaranteed order in which non-blocked steps from multiple enabled sequence instances will be executed.

Here is an example of scenario:

scenario:
  initialSequences:
  - login:
    - httpRequest:
        POST: /login
    # Enable instance of sequence 'wait5seconds'
    - next: wait5seconds
  sequences:
  - wait5seconds:
    - thinkTime:
        duration: 5s
    - next: logout
  - logout:
    - httpRequest:
        POST: /logout

While this generic approach is useful for complex scenarios with branching logic, simple sequential scenarios can use orderedSequences short-cut enabling sequences in given order:

scenario:
  orderedSequences:
  - login:
    - httpRequest:
        POST: /login
  - wait5seconds:
    - thinkTime:
        duration: 5s
  - logout:
    - httpRequest:
        POST: /logout

This syntax makes the first sequence (login in this case) an initial sequence, adds the subsequent sequences and as the last step of each but the last sequence appends a next step scheduling a new instance of the following sequence.

To make configuration even more concise you can omit the orderedSequences level and start defining the list of sequences under scenario right away:

scenario:
- login:
  - httpRequest:
      POST: /login
- wait5seconds:
  - thinkTime:
      duration: 5s
- logout:
  - httpRequest:
      POST: /logout

An exhaustive list of steps can be found in the steps reference.

2.5 - Variables

All but the simplest scenarios will use session variables. Hyperfoil sports steps that generate values into these variables (randomInt, randomItem, …), processors that write data from other sources to variables (store, array) and many places that read variables and use the values to perform some operations (httpRequest.path ) or alter control flow.

Hyperfoil uses different types of variables (slots in the session) for integer variables and generic objects (commonly strings). When a numeric value is received as a string (e.g. when parsing response headers) and you want to use it in a step that expects exclusively integral values you have to convert it explicitly, e.g. using the stringToInt action. Steps that read values to form a string can usually consume both types of variables, without any need for conversion.

Besides user-defined variables there are some read-only pseudo-variables that can be used in the scenario as if these were regular variables:

String interpolation

Components that accept string values usually allow you to use a pattern - parts of the string can be replaced in runtime with the value from a session variable. A simple example of pattern would be The quick brown ${wild-animal} jumps over the lazy ${domestic-animal} - variables wild-animal and domestic-animal would get replaced with their respective values.

When you really want to use ${wild-animal} in a value for such component you should escape it with one more dollar sign: This $${variable} won't be replaced will be rendered into This ${variable} won't be replaced.

There are a few transformations that you can perform with a variable value while interpolating the pattern:

• ${urlencode:my-variable} will replace characters in the my-variable using URLEncoder.encode (using UTF-8 encoding).
• ${{ '{%05d' }}:my-number} and other formatter strings ending with d, o, x or X will convert an integer variable using Formatter.
• ${replace/<regexp>/<replacement>/<flags>:my-variable} perform Java regexp replacement on my-variable contents. Note that you can use any character after replace, not just / - this becomes the separator between regexp, replacement and flags. The only flags currently supported is g - replacing all occurences of that string (by default only first occurence is replaced).

Sequence-scoped access

When an array or collection is stored in a session variable you can access the individual elements by appending [.] to the variable name, e.g. my-variable[.]. You can see that we don’t use the actual index into the array: instead we use current sequence instance index. You can read more about running multiple sequences concurrently in the Architecture/Scenario Execution.

2.6 - Templates

It is often useful to keep a single benchmark in version control but change parts of it depending on the infrastructure where it is executed or intended load. Since version 0.18 Hyperfoil supports parametrization of the benchmark through templates.

Inspired by other (more complex) YAML templating systems we decided to use YAML tags to pre-process the YAML. Templating happens even before applying the YAML nodes onto BenchmarkBuilder, therefore it is not possible to do that programmatically or with the serialized form.

If you are working with CLI or WebCLI there is little difference to regular benchmarks: you upload and edit the benchmark as usual. However it is not possible to auto-detect files before the benchmark is constructed from the template (the reference to a file could be a template, too!), therefore you need to pass all files using option -f to the upload/edit command.

When the benchmark template is uploaded, upon running it (run mybenchmark) you either pass the parameters using option -P or you are interactively asked to provide those params. The parameters are stored in CLI context and on subsequent invocations of run you don’t need to set these. If you want to remove the parameters from the context use option -r/--reset-params. To see both default and current parameters you can use the inspect command.

Param

You should use !param to replace single scalar value:

1name: example
2http:
3  host: http://localhost:8080
4usersPerSec: !param NUM_USERS
5duration: !param DURATION 60s
6scenario: # ...

In this simple constant-rate benchmark you can customize the number of users starting each second as well as the duration. There’s no default for NUM_USERS; you will be asked to provide it when you run the benchmark. On the other hand DURATION has a default value of 60s - anything after the space after parameter name counts as the default value.

Parameters don’t have to be upper-case. The identifier is case-sensitive, though.

run scalar-value-example -PNUM_USERS=5 -PDURATION=60s

Concat

Sometimes you need to replace only part of a string: !concat will let you do that:

1name: example
2http:
3  host: !concat [ "http://", !param SERVER localhost, ":8080" ]
4usersPerSec: 10
5duration: 60s
6scenario: # ...

In this example we will customize the host with the concatenation of http://, parameter SERVER with default localhost and :8080. This example uses inline-form of list, though you can use the regular list (one item per line), too.

Foreach

Chances are you need to generate a list based on a param: you can do this using the !foreach:

 1name: example
 2http: !foreach
 3  items: http://example.com,http://hyperfoil.io
 4  separator: "," # comma is the default separator
 5  param: ITEM   # ITEM is the default parameter name
 6  do:
 7    host: !param ITEM
 8usersPerSec: 10
 9duration: 60s
10scenario: # ...

This splits the items using the separator regexp and produces a list of values or mappings while the param ITEM is set to one of the values from items list. The example above would result in:

name: example
http:
- host: http://example.com
- host: http://hyperfoil.io
usersPerSec: 10
duration: 60s
scenario: # ...

You can also set items to a YAML list; in that case the separator is not used:

myList: !foreach
  items: ["A", !param B, "C"]
  param: FOO
  do: !param FOO

The last example with -PB=bar would result in:

myList:
- A
- bar
- C

Renaming the param used for iteration can be useful in nested loops: without renaming the inner foreach would shadow the outer one.

Anchors and aliases

YAML has a built-in concept for removing repetitive sections: anchors and aliases. With the templating system you can use that universally throughout the file (in versions before 0.18 the support was limited to forks, scenarios and sequences):

foo: &hello-world
  hello: world
anotherFoo:
  sayHi: *hello-world
  myList:
  - *hello-world
  - bar

is interpretted as

foo:
  hello: world
anotherFoo:
  sayHi:
    hello: world
  myList:
  - hello: world
  - bar

2.7 - Hooks

It might be useful to run certain scripts before and after the run, e.g. starting some infrastructure, preloading database, gathering CPU stats during the test and so on. That’s why Hyperfoil introduces pre- and post-hooks to the run.

Some scripts are not specific to the test being run - these should be deployed on controller as files in *root*/hooks/pre/ and *root*/hooks/post directories where root is controller’s root directory, /tmp/hyperfoil/ by default. Each of these directories should contain executable scripts or binaries that will be run in alphabetic order. We strongly suggest using the format 00-my-script.sh to set the order using first two digits.

Kubernetes/Openshift deployments use the same strategy; the only difference is that the pre and post directories are mapped as volumes from a ConfigMap resource.

Other scripts may be specific to the benchmark executed and therefore you can define them directly in the YAML files. You can either use inline command that will be executed using sh -c your-command --your-options or create a Java class implementing io.hyperfoil.core.hooks.RunHook and register it to be loaded as other Hyperfoil extensions.

name: my-benchmark
pre:
  01-inline: curl http://example.com
  02-custom:
    my-hook:
      foo: bar
post:
  99-some-final-hook: ...
...

The lists of hooks from controller directories and benchmark are merged; if there’s a conflict between two hooks from these two sources the final execution order is not defined (but both get executed).

In case of inline command execution the stderr output will stay on stderr, stdout will be caputered by Hyperfoil and stored in *rundir*/*XXXX*/hooks.json. As the post-hooks are executed after info.json and all.json get written the output cannot be included inside those files. This order of execution was chosen because it’s likely that you will upload these files to a database - yes, using a post-hook.

2.8 - Ergonomics

This section hosts only single property at this moment:

3 - Examples

If you haven’t checked the Getting started guide we strongly recommend going there first.

Below you’ll see commented examples of configuration; contrary to the Getting started guide these don’t present scenarios but rather list the various configuration options by example.

httpRequest

You will most likely use step httpRequest in each of your scenarios, and there’s many ways to send a request.

# This example should demonstrate various ways to configure one of the most important
# steps - the httpRequest.
name: http-requests
http:
  host: http://example.com
ergonomics: # Disable stopping the scenario on 4xx or 5xx response
  autoRangeCheck: false
usersPerSec: 1
duration: 1
scenario:
- jsonBody:
  - httpRequest:
      POST: /foo/bar
      # Hyperfoil doesn't know what's the content of the body string, if the server
      # requires correct content-type header you have to provide it yourselves
      headers:
        content-type: application/json
      # Here we specify a multi-line string. For more info about multiline strings,
      # compacting/chopping of newlines etc. please check out https://yaml-multiline.info/
      body: |
        {
           "foo" : "bar"
        }        
- formBody:
  - set: myVar <- foobar
  - httpRequest:
      POST: /myform
      # Here we don't need to add any headers as the form: knows that you're sending
      # a HTML form and it can add 'content-type: application/x-www-form-urlencoded'
      # automatically.
      body:
        # This will generate body 'foo=bar&bar=foobar&goo=foofoobarbar'. Any non-ascii
        # or otherwise illegal characters are correctly URL-encoded.
        form:
        - name: foo
          value: bar
        - name: bar
          fromVar: myVar
        - name: goo
          pattern: foo${myVar}bar
- bodyFromFile:
  - httpRequest:
      POST: /foo/bar
      body:
        # This simply loads the file and sends it as the body without any conversion.
        # It does not add any headers nor make it a multipart upload as the browser would do.
        fromFile: usernames.txt
- customHeaders:
  - set: token <- dGhpcyBpcyBhIG5pY2UgYW5kIHNlY3VyZSB0b2tlbgo=
  - set: etag <- "ETag received in some previous request"
  - httpRequest:
      GET: /secured/page
      # Note that HTTP headers are case-insensitive (use your preferred capitalization)
      headers:
        # Headers can be set inline
        accept: text/html
        # Session variables are replaced using the pattern syntax
        authorization: Bearer ${token}
        # Values from session variables can be also loaded using fromVar
        if-match:
          fromVar: etag
- nonDefaultMetric:
  - httpRequest:
      GET: /cats
      # By default the metric name equals to name of the sequence ('nonDefaultMetric' here).
      # We can override that either with a constant value...
      metric: mammals
  - randomItem:
      toVar: animal
      list:
      - cats
      - dogs
      - locusts
  - httpRequest:
      GET: /foo/${animal}
      # ... or a regexp switch on the actual authority+path (e.g. example.com:8080/foo/cats).
      # If the benchmark uses single (default) HTTP target the authority is omitted.
      metric:
      - .*cats -> mammals
      - .*dogs -> mammals
      - -> insects
- toughSLAs:
  - httpRequest:
      GET: /index.html
      handler:
        status:
          # Any request that is not responded with status code will be marked as invalid.
          range: 200
      # When you need only one SLA you can use mapping without the list (just forget the dash).
      sla:
      # This first SLA is evaluated when the phase completes from all requests that happened
      # during the phase.
      # Errors are connection failures, timeouts, 4xx and 5xx responses
      - errorRatio: 0.1
        # You can set custom criteria for what is considered valid/invalid as with the status
        # handler above. By default any response with status that is not within 200-399
        # is deemed invalid (as well as error).
        invalidRatio: 0.2
        blockedRatio: 0.0
        meanResponseTime: 10 ms
        # 90% requests should be under 100 ms, only 1% can be over 1 second
        limits:
          0.9: 100 ms
          0.99: 1 s
      # Following SLA is evaluated when all the statistics for the past second arrive,
      # accumulating results from the window (last 10 seconds). Therefore it can detect shorter
      # peaks of degraded performance.
      - window: 10s
        meanResponseTime: 50 ms

Some scenarios need to access multiple HTTP endpoints; following example shows an example configuration for that:

# This example manifests running a benchmark against multiple domains
name: more-servers
http:
- host: http://example.com
  sharedConnections: 100
- host: https://hyperfoil.io
  # With HTTPS, most modern servers will negotiate HTTP 2.0 as the application protocol.
  # Since HTTP 2.0 uses multiple streams over single TCP (TLS in this case) connection
  # you can usually set lower number of connections.
  sharedConnections: 10
# You may want to route requests through a proxy/load balancer or simply use a domain
# that is not resolvable. Configuration below will actually send the requests to addresses
# set below, but the requests will use in the 'host: foobar.com' in the headers
# and in SNI if this goes over TLS (HTTPS).
- host: http://foobar.com
  # Hyperfoil will split the connections evenly to the defined addresses
  # (an entry is considered single address for this purpose even if DNS registers
  # multiple IP addresses for the hostname).
  sharedConnections: 30
  addresses:
  # Both hostnames and IP addresses are allowed
  - proxy.my-locally-defined-domain.test
  - 192.168.1.10
  # You can set a custom port as well
  - 192.168.1.11:8080
usersPerSec: 1
duration: 1
scenario:
- test:
  - httpRequest:
      # Authority is the combination of hostname and port.
      authority: hyperfoil.io:443
      GET: /docs
  - randomItem:
      toVar: hostname
      list:
      - example.com
      - foobar.com
  - httpRequest:
      # The target must be configured in the 'http' section above; the correctness
      # is usually validated when parsing/building the benchmark but sometimes it is
      # only possible at runtime, potentially resulting in errors during execution.
      authority: ${hostname}:80
      GET: /foo

4 - Troubleshooting

It doesn’t work. Can you help me?

The first step to identifying any issue is getting a verbose log - setting logging level to TRACE. How exactly you do that depends on the way you deploy Hyperfoil:

1. If you use CLI and the start-local command, just run it as start-local -l TRACE which sets the default logging level. You’ll find the log in /tmp/hyperfoil.local.log by default.

2. If you run Hyperfoil manually in standalone mode (non-clustered) the agent will run in the same JVM as the controller. You need to add -Dlog4j.configurationFile=file:///path/to/log4j2-trace.xml option when starting standalone.sh. If you start Hyperfoil through Ansible the same is set using hyperfoil_log_config variable.

3. If you run Hyperfoil in clustered mode, the failing code is probably in the agents. You need to pass the logging settings to agents using the deployer; with SSH deployer you need to add -Dlog4j.configurationFile=file:///path/to/log4j2-trace.xml to the extras property, in Kubernetes/Openshift there is the log option that lets you set the logging configuration through a config-map.

An example of Log4j2 configuration file with TRACE logging on is here:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
   <Appenders>
      <Console name="CONSOLE" target="SYSTEM_OUT">
         <PatternLayout pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %m%n"/>
      </Console>
   </Appenders>
   <Loggers>
      <Root level="TRACE">
         <AppenderRef ref="CONSOLE"/>
      </Root>
   </Loggers>
</Configuration>

TRACE-level logging can be very verbose to a point where it will pose a bottleneck. It’s recommended to isolate your problem at lower request rate if that’s possible.

If you need to print variable values for debugging, check out log step.

My phase fails with SLA failure ‘Progress was blocked waiting for a free connection. Hint: increase http.sharedConnections.’

By default Hyperfoil uses single connection to each HTTP(s) host; the default is set so low to force you thinking about connection limits early during test development. If you don’t override this value as in:

http:
  host: http://hyperfoil.io
  sharedConnections: 1000

you get the error above, as the default SLA does not allow a session (virtual user) to be blocked due to not being able to acquire a connection from the connection pool immediatelly. If you can’t increase number of connections (or use HTTP2 that allows multiple requests to multiplex within single connection), you can set

- httpRequest:
    sla:
    - blockedRatio: 1000 # any value big enough

on each request to drop the default SLA. The blockedRatio value is a threshold ratio between time spent waiting for a free connection and waiting for the response.

You could also wonder why the sessions are missing a connection when the scenario should guarantee there’s always a free connection e.g. when using always phase type with same number of users and connections. However this may not hold when the connection is closed (either explicitly or after receiving a 5xx response) - while Hyperfoil starts replacing that connection immediatelly it takes a moment. If you expects connections to be closed add a few (10%) extra connections. Another reason could be poor balancing of connections and sessions to threads (should be gone in version 0.8).

When I set ‘Host’ header for HTTP request I get warnings

Hyperfoil automatically inserts the ‘Host’ header to each request and when you try to override that for certain request it emits a warning:

Setting `host` header explicitly is not recommended. Use the HTTP host and adjust actual target using `addresses` property.

With this warning on we don’t inject the header as it might be intended, e.g. when the target server does not parse headers in a case-sensitive way (as it should!) and you have to use certain case. However, if you want to run your requests to a different IP than the host resolves to (e.g. hit 127.0.0.1:8080 with Host: example.com) you should rather use

http:
  host: http://hyperfoil.io
  addresses:
  - 127.0.0.1:8080

When I use a session variable I am seeing the error “Variable foo is not set!”

Errors:
in-vm: Variable foo is not set!

On occasion a scenario step has been seen to execute out of sequence. To ensure the variable is set beforehand use initialSequences with the step that populates the variable.