Here you can find an exhaustive documentation for the Hyperfoil tool, let’s start!
1 - Overview
Generic overview on the Hyperfoil tool
There’s plenty of web benchmark tools around and it might be hard to pick one, and investing time into not-established tools is risky. Hyperfoil was not created for the pure joy of coding but as a solution to set of problems which could not be solved all by single other existing tool.
Free software
Free software allows you to take your benchmark and publish it for everyone to verify. With proprietary licenses that wouldn’t be so easy.
Generating load with single node stops scaling at certain point and you need to orchestrate the benchmark across a cluster of nodes. Simple command-line tools usually ignore this completely (you’re supposed to start them, gather and merge the data in a bash scripts). Other tools use open-core model with the clustering part being a paid/proprietary option. There are frameworks that have clustering built in as well.
Hyperfoil uses leader-follower model with Vert.x Event bus as the clustering middleware. While running from single VM is possible (and quite easy) as well, the design is distributed by default.
Accuracy
The point of web benchmark is usually finding out what happens when your system is accessed by thousands of concurrent users - each doing a page load every few seconds. However many traditional load drivers simplify the scenario to few dozen of virtual users (VUs) that execute one request after another or with very short delays in between - this is referred to as the Closed System Model (as the set of VUs is finite). This discrepancy leads to problem known as coordinated omission and results in significantly skewed latency results and pathological conditions (queues overflow…) not being triggered.
Hyperfoil embraces Open System Model by default - virtual users are completely independent until it runs out of resources, recording that situation in consequence. Hyperfoil runs a state-machine for each VU and all requests are executed asynchronously.
Versatility
While you can design your benchmark to just hit single endpoint (URL) with desired rate this is likely not what the users would be doing. Randomizing some parts of the query or looping through a list of endpoint might be better but the resulting scenario might be still too simplified.
Hyperfoil introduces a DSL expressed in YAML with which you can sketch the scenario in a fine detail, including timing, concurrent requests, processing responses and so forth. We’re not trying to invent a new programming language, though, so if the DSL gets too complex you can write the logic in Java or any other JVM language.
This document explains some core terms used throughout the documentation.
Controller and agents
While it is possible to run benchmarks directly from CLI, in its nature Hyperfoil is a distributed tool with leader-follower architecture. Controller has the leader role; this is a Vert.x-based server with REST API. When a benchmark is started controller deploys agents (according to the benchmark definition), pushes the benchmark definition to these agents and orchestrates benchmark phases. Agents execute the benchmark, periodically sending statistics to the controller. This way the controller can combine and evaluate statistics from all agents on the fly. When the benchmark is completed all agents terminate.
All communication between the controller and agents happens over Vert.x eventbus - therefore it is independent on the deployment type.
Phases
Conceptually the benchmark consists of several phases. Phases can run independently of each other;
these simulate certain load executed by a group of users (e.g. visitors vs. admins). Within one phase all users execute the same scenario (e.g. logging into the system, selling all their stock and then logging off).
Phases are also using for scaling the load during the benchmark; when looking for maximum throughput you schedule several iterations of given phase, gradually increasing the number of users that run the scenario.
A phase can be in one of these states:
not running (scheduled)
running
finished: users won’t start new scenarios but we’ll let already-started users complete the scenario
terminated: all users are done, all stats are collected and no further requests will be made
The state of phase on every agent is managed by Controller; this is also the finest grained unit of work it understands (controller has no information about state of each user).
Sessions
The state of each user’s scenario is saved in the session; sometimes we speak about (re)starting sessions instead of starting new users. Hyperfoil tries to keep allocations during benchmark as low as possible and therefore it pre-allocates all memory for the scenario execution ahead. This is why all resources the benchmark uses are capped - it needs to know the sizes of pools.
It is also necessary to know how many sessions we should preallocate - maximum concurrency of the system. If this threshold is exceeded Hyperfoil allocates further session as needed, but this is not the optimal mode of operation. It means that either you’ve underestimated the resources need or you’ve put a load on the system that it can’t handle anymore, requests are not being completed and scenarios are not finished - which means that session objects cannot be recycled for reuse by next user.
Scenario
Scenario consists of one or more sequences that are composed of steps. Steps are similar to statements in programming language and sequences are an equivalent of blocks of code.
While most of the time the scenario will consist of sequential operations as the user is not multi-tasking, the browser (or other system you’re simulating) actually executes some operations in parallel - e.g. during page load it loads images concurrently. Therefore at any time the session contains one or more active sequence instances; when all sequence instances are done, the session has finished and can be recycled for a new user. Most of the time the scenario will start with only one active instance and as it progresses, it might create instances of other sequences (e.g after evaluating a condition it creates a sequence instance according to the branching logic).
1.2 - FAQ
Frequently Asked Questions
Why is Hyperfoil written in Java?
People are often concerned about JVM performance or predictability. While nowadays JVM is very good in the sense of throughput, dealing with jitter can be challenging. We are Java engineers, though, and we believe that these issues can be mitigated with a right design. That’s why we try to be very careful on the execution hot-path.
We could achieve even better properties with C/C++, but the development effectivity would suffer. We could be succesful in Go, but we’re not as intimate with its internals. Other languages and frameworks would pose its own challenges. So far, the choice of Java was not found to be a limiting factor.
Why are you inventing your own YAML-based DSL instead of using Javascipt/Lua/…?
While some people might be more comfortable with describing their complex scenarios in a familiar language, running a script execution engine would have impact on performance and could put us out of control. We are not trying to invent a new language, written in YAML structure. Instead we propose a set of components common to many scenarios that could be recombined as it suits you. If you ever feel that the YAML is becoming cumbersome, try to move your complex benchmark logic to Java code and use it that way, instead.
I just want to know what is the maximum throughput!
Maximum throughput is a single number which makes comparison very easy. Was my code change for better or worse? However finding the sweet spot is not as simple as throwing in few hundred concurrent threads running one request after another and taking the readings. With too high concurrency you can get worse results due to contention and longer queues, so you need to try different concurrency levels anyway.
There’s nothing wrong with this type of test as long as you know what you’re doing, and that the response latencies might be far off reality. It’s actually very good test when you look only for regressions - and Hyperfoil supports that, too.
Hyperfoil is so hard to set up, I’ll just use …
Some tools can be run from the shell, with everything set just through options and arguments. That is quite handy for a quick test - and if the tool is sufficient for the job, use it. Or you can try runnning the same through Hyperfoil - e.g. for wrk/wrk2 we offer a facade (CLI command wrk or bin/wrk.sh) that creates a benchmark with the same behaviour but you also get all the detailed results as from any other run - see the migration guide. Once that your requirements outgrow what’s possible in these simple tools, you can embrace the full power of benchmark composition.
What does that ‘Exceeded session limit’ error mean?
With open-model phase types (constantRate, increasingRate, decreasingRate) the concurrency should not be limited. However as Hyperfoil tries not to allocate any memory during the benchmark we need to reserve space ahead for all sessions that could run concurrently - we call this the session limit. By default this limit is equal to number of users per second (assuming that the scenario won’t take more than 1 second).
When you get the ‘Exceeded session limit’ error this means that some of the requests took a long time (or you have delays as part of the scenario) and Hyperfoil ran out of session pool. In that case you can change the limit using maxSessions property on the phase to the expected maximum concurrency. E.g. if you expect that the scenario will take 3 seconds and you’re running at usersPerSec: 100 you should set maxSessions: 300 (or rather more to give it a buffer for unexpected jitter).
If increasing the limit doesn’t help it usually means that the load at the tested system is too high and the responses are not arriving as fast as you fire the requests. In that case you should lower the load.
2 - Getting Started
Collection of quickstarts aiming to make the Hyperfoil adoption a lot easier.
Embark on this journey with our collection of 8 quickstarts, guiding you through a wide range of potential use cases that you might encounter in daily operations.
Info
Do not hesitate to suggest additional use cases that you believe would be valuable for this section.
2.1 - First benchmark
Download, set up, and run your first Hyperfoil benchmark
For our first benchmark we’ll start an embedded server (controller) within the CLI:
[hyperfoil]$ start-local
Starting controller in default directory (/tmp/hyperfoil)Controller started, listening on 127.0.0.1:41621
Connecting to the controller...
Connected!
3. Upload the minimalistic benchmark and run it
As you can see below, the benchmark is really minimalistic as it is doing only single request to http://hyperfoil.io.
# This is the name of the benchmark. It's recommended to keep this in sync with# name of this file, adding extension `.hf.yaml`.name:single-request# We must define at least one HTTP target, in this case it becomes a default# for all HTTP requests.http:host:http://hyperfoil.io# Simulation consists of phases - potentially independent workloads.# We'll discuss phases more in detail in next quickstarts.phases:# `example` is the name of the single phase in this benchmark.- example:# `atOnce` with `users: 1` results in running the scenario below just onceatOnce:users:1scenario:# The only sequence in this scenario is called `test`.- test:# In the only step in this sequence we'll do a HTTP GET request# to `http://hyperfoil.io/`- httpRequest:GET:/# Inject helpers to make this request synchronous, i.e. keep# the sequence blocked until Hyperfoil processes the response.sync:true
Create the same benchmark in your local environment or download it.
After that, upload it using the upload command as follows:
[hyperfoil@in-vm]$ upload .../single-request.hf.yaml
Loaded benchmark single-request, uploading...
... done.
[hyperfoil@in-vm]$ run single-request
Started run 0001Run 0001, benchmark single-request
Agents: in-vm[STARTING]Started: 2019/11/15 16:11:43.725 Terminated: 2019/11/15 16:11:43.899
<span class="hfcaption">NAME STATUS STARTED REMAINING COMPLETED TOTAL DURATION DESCRIPTION
example TERMINATED 16:11:43.725 16:11:43.899 174 ms (exceeded by 174 ms)1 users at once
4. Check out performance results:
[hyperfoil@in-vm]$ stats
Total stats from run 000A
<span class="hfcaption">PHASE METRIC REQUESTS MEAN p50 p90 p99 p99.9 p99.99 2xx 3xx 4xx 5xx CACHE TIMEOUTS ERRORS BLOCKED
example test1 172.49 ms 173.02 ms 173.02 ms 173.02 ms 173.02 ms 173.02 ms 01000000 ns
Doing one request is not much of a benchmark and the statistics above are moot, but hey, this is a quickstart.
In the future you might find editing with schema useful but at this point any editor with YAML syntax highlighting will do the job.
Learn how to create more steps and how to gather run statistics
In previous quickstart you created a benchmark
that fires only one HTTP request. Our next example is going to hit random URLs at this server with 10 requests per second. We’ll see how to generate random data and collect statistics for different URLs.
Let’s start a container that will serve the requests:
podman run --rm -p 8080:8083 quay.io/hyperfoil/hyperfoil-examples
If you prefer running this in Docker just replace podman with docker. You can explore the handling of requests from this example on GitHub.
Here is the benchmark we’re going to run:
name:random-urlshttp:host:http://localhost:8080sharedConnections:10# 10 users will be starting the scenario every secondusersPerSec:10duration:5sscenario:- test:# Step `randomItem` randomly picks one item from the list below...- randomItem:list:- index.html- foo.png- bar.png- this-returns-404.png# ... and stores it in users's session under key `my-random-path`toVar:my-random-path- httpRequest:# HTTP request will read the variable from the session and format# the path for the GET requestGET:/quickstarts/random-urls/${my-random-path}# We'll use different statistics metric for webpages and imagesmetric:- .*\.html -> pages- .*\.png -> images- -> other# Handler processes the responsehandler:# We'll check that the response was successful (status 200-299)status:range:2xx# When the response is fully processed we'll set variable `completed`# in the session.onCompletion:set:completed <- yes# For demonstration purposes we will set `sync: false`.# Next step is executed immediately after we fire the request, not# waiting for the response.sync:false# We'll wait for the `completed` var to be set in this step, though.- awaitVar:completed
So let’s run this through CLI:
[hyperfoil]$ start-local
...
[hyperfoil@in-vm]$ upload .../random-urls.hf.yaml
...
[hyperfoil@in-vm]$ run
Started run 0002Run 0002, benchmark random-urls
Agents: in-vm[STARTING]Started: 2019/11/15 17:49:45.859 Terminated: 2019/11/15 17:49:50.904
NAME STATUS STARTED REMAINING COMPLETED TOTAL DURATION DESCRIPTION
main TERMINATED 17:49:45.859 17:49:50.903 5044 ms (exceeded by 44 ms) 10.00 users per second
[hyperfoil@in-vm]$ stats
Total stats from run 0002PHASE METRIC REQUESTS MEAN p50 p90 p99 p99.9 p99.99 2xx 3xx 4xx 5xx CACHE TIMEOUTS ERRORS BLOCKED
main images 34 3.25 ms 3.39 ms 4.39 ms 12.58 ms 12.58 ms 12.58 ms 1213120000 1.11 ms
main pages 13 2.89 ms 3.19 ms 4.15 ms 4.33 ms 4.33 ms 4.33 ms 130000000 ns
main/images: Progress was blocked waiting for a free connection. Hint: increase http.sharedConnections.
There are several things worth mentioning in this example:
The command run does not have any argument. In this case, the benchmark name random-urls is optional as you’ve just uploaded it and CLI knows that you are going to work with it. The same holds for stats - you don’t have to write down run ID 0002 when displaying statistics as the implicit run ID is set automatically in the run/status command.
The test did only 47 requests in 5 seconds, instead of 50. It does not execute one request every 100 ms sharp, it randomizes the times of requests as well; this simulates the Poisson point process. Longer runs would have lower variance in the total numbers.
In metric images the test reports 1.11 ms being blocked and there’s SLA failure below the stats. This is happening because in the default configuration Hyperfoil opens only one connection to the target server. All (possibly concurrent) requests have to share the common pool of 1 connection and if some request cannot be executed immediatelly we report this as blocked time. All practical benchmarks should increase the pool size to a value that reflects simulated load and prevent this situation.
The test took 44 ms longer than the configured 5 seconds. We terminate the test only after all responses for sent requests arrive (or time out).
The previous example was the first ‘real’ benchmark, but it didn’t do anything different from what you could run through wrk, ab, siege or similar tools.
Of course, the results were not suffering from the coordinated omission problem, but Hyperfoil can do more. Let’s try a more complex scenario:
name:choose-moviehttp:host:http://localhost:8080# Use 80 concurrent HTTP connections to the server. Default is 1,# therefore we couldn't issue two concurrent requests (as HTTP pipelining# is disabled by default and we use HTTP 1.1 connections).sharedConnections:80usersPerSec:10duration:5s# Each session will take at least 3 seconds (see the sleep time below),# and we'll be running ~10 per second. That makes 30, let's give it# some margin and set this to 40.maxSessions:40scenario:# In previous scenarios we have used only single sequence and we could# define the list of sequences right away. In this scenario, we're going# to be using 3 different sequences.# Initial sequences are scheduled at session start and are not linked# to the other sessions.initialSequences:- home:# Pick a random username from a file- randomItem:file:usernames.txttoVar:username# The page would load a profile, e.g. to display full name.- httpRequest:GET:/quickstarts/choose-movie/profile?user=${username}sync:falsemetric:profile# Fetch movies user could watch- httpRequest:GET:/quickstarts/choose-movie/moviessync:falsemetric:movieshandler:body:# Parse the returned JSON that is an array and for each# element fire the processor.json:query:.[]processor:# Store each element in a collection `movies`array:toVar:movies# Store as byte[] to avoid encoding UTF-8 into Stringformat:BYTES# Every data structure in session has maximum size.# This space is pre-allocated.maxSize:10# This step waits until responses for all sent requests are received and processed.- awaitAllResponses# Wait 3 seconds to simulate user-interaction- thinkTime:duration:3s# Set variable `quality` and `movieNames` to an uninitialized array# of 10 elements. We will use them later on.- set:var:qualityobjectArray:size:10- set:var:movieNamesobjectArray:size:10# For each element in variable `movies` schedule one (new) instance# of sequence `movies`, defined below. These instances differ in# one intrinsic "variable" - their index.- foreach:fromVar:moviessequence:addComment# Schedule one more sequence- newSequence:watchMovie# These sequences are defined but don't get scheduled at session start. We activate# them explicitly (and multiple times in parallel) in foreach step above.sequences:# Sequences that can run multiple instances concurrently must declare the maximum# concurrency level explicitly using the brackets.- addComment[10]:# Variables `movies` hosts an array, and in the foreach step# we've created one sequence for each element. We'll access# the element through the '[.]' notation below.- json:fromVar:movies[.]query:.quality# We'll extract quality to another collection under# this sequence's index. We shouldn't use global variable# as the execution of sequences may interleave.toVar:quality[.]# For high-quality movies we won't post insults (we haven't seen# the movie yet anyway). Therefore, we'll stop executing# the sequence prematurely.- breakSequence:intCondition:fromVar:quality[.]# Note: ideally we could filter the JSON directly using query# .[] | select(.quality >= 80)# but this feature is not implemented yet.greaterOrEqualTo:80- json:fromVar:movies[.]query:.nametoVar:movieNames[.]- httpRequest:# URLs with spaces and other characters don't work well;# let's encode it (e.g. space -> %20)POST:/quickstarts/choose-movie/movie/${urlencode:movieNames[.]}/commentsbody:text:This movie sucks.# The sync shortcut actually sets up a bit in the session state# cleared before the request and set when the request is complete,# automatically waiting it after this step.# You can write your own handlers (using sequence-scoped vars)# to change this behaviour.sync:true# Set value to variable `commented`. The actual value does not matter.- set:commented <- true- watchMovie:# This sequence is blocked in its first step until the variable gets# set. Therefore we could define it in `initialSequences` and omit# the `newSequence` step at the end of `home` sequence.- awaitVar:commented# Choose one of the movies (including the bad ones, for simplicity)- randomItem:selectedMovie <- movies- json:fromVar:selectedMoviequery:.name# This sequence is executed only once so we can use global var.toVar:movieName# Finally, go watch the movie!- httpRequest:GET:/quickstarts/choose-movie/movie/${urlencode:movieName}/watchsync:true
Start the server and fire the scenario the usual way:
# start the server to interact withpodman run --rm -d -p 8080:8083 quay.io/hyperfoil/hyperfoil-examples
# start Hyperfoil CLIbin/cli.sh
[hyperfoil]$ start-local
...
[hyperfoil@in-vm]$ upload .../choose-movie.hf.yaml
...
[hyperfoil@in-vm]$ run
...
Is this scenario too simplistic? Let’s define phases…
2.4 - Phases - basics
Deep dive into the basics of phases
So far the benchmark contained only one type of load; certain number of users hitting the system, doing always the same (though data could be randomized). In practice you might want to simulate several types of workloads at once: in an eshop users would come browsing or buying products, and operators would restock the virtual warehouse.
Also, driving constant load may not be the best way to run the benchmark: often you want to slowly ramp the load up to let the system adjust (scale up, perform JIT, fill pools) and push the full load only after that. When trying to find system limits, you do the same repetitevely - ramp up the load, measure latencies and if the system meets SLAs (latencies below limits) continue ramping up the load until it breaks.
In Hyperfoil, this all is expressed through phases. We’ve already seen phases in the first quickstart as we wanted to execute a non-default type of load - running the workload only once. Let’s take a look on the “eshop” case first:
# This benchmark simulates operations in an eshop, with browsing/shopping users# and operators restocking the warehouse.name:eshophttp:host:http://localhost:8080sharedConnections:80phases:# This defines a workload where users just look through the pages.- browsingUser:# This is the default type of workload, starting constant number of users# each second. Note that we don't speak about 'requests per second' as# the scenario may issue any number of requests.constantRate:duration:10susersPerSec:10scenario:# Browse is the name of our only sequence. We will avoid steps generating# random data for browsing for the sake of brevity.- browse:- httpRequest:GET:/quickstarts/eshop/items# Workload simulating users that are going to buy something- buyingUser:constantRate:# The length of this phase is not synchronized with other phases.# You might think that this is too flexible at first.duration:10susersPerSec:5scenario:- browse:- httpRequest:GET:/quickstarts/eshop/itemshandler:body:json:query:.[].id# This is a shortcut to store in array-typed variable# `itemIds` holding at most 10 elements.toArray:itemIds[10]- buy:# Pick id for a random item- randomItem:itemId <- itemIds- httpRequest:POST:/quickstarts/eshop/items/${itemId}/buy- operator:# This is a different type of phase, running fixed number of users.# It is what most benchmarks do when you set number of threads; here we use# that as we know that we have fixed number of employees (operators) who# are restocking the warehouse.always:users:5duration:10sscenario:- restock:# Select an id for random item to restock# Variables in different scenarios are completely unrelated.- randomInt:itemId <- 1 .. 999- randomInt:units <- 1 .. 10- httpRequest:POST:/quickstarts/eshop/items/${itemId}/restockbody:# We are using url-encoded form dataform:- name:addUnitsfromVar:units# Operators need some pauses - otherwise we would start another# scenario execution (and fire another request) right away.- thinkTime:duration:2s
Start the same server as you did in the previous quickstarts:
podman run --rm -p 8080:8083 quay.io/hyperfoil/hyperfoil-examples
In next quickstart you’ll learn how to repeat and link the phases.
2.5 - Phases - advanced
Delve into more advanced phase configuration
Previous quickstart presented a benchmark with three phases that all started at the same moment (when the benchmark was started) and had the same duration - different phases represented different workflows (types of user). In this example we will adjust the benchmark to scale the load gradually up.
At this point it would be useful to mention the lifecycle of phases; phase is in one of these states:
not started: As the name clearly says, the phase is not yet started.
running: The agent started running the phase, i.e., performing the configured load.
finished: When the duration elapses, no more new users are started. However, some might be still executing their scenarios.
terminated: When all users complete their scenarios the phase becomes terminated. Users may be forcibly interrupted by setting maxDuration on the phase.
cancelled If the benchmark cannot continue further, all remaining stages are cancelled.
Let’s take a look into the example, where we’ll slowly (over 5 seconds) increase load to 10+5 users/sec, run with this load for 10 seconds, again increase it by another 10+5 users/sec and so forth until we reach 100+50 users per second. As we define maxIterations for these phases the benchmark will actually contain phases browsingUserRampUp/0, browsingUserRampUp/1, browsingUserRampUp/2 and so forth.
name:eshop-scalehttp:host:http://localhost:8080sharedConnections:80phases:- browsingUserRampUp:# This type of phase is similar to constantRate in the way how new users# are started but gradually increases the rate from `initialUsersPerSec`# to `targetUsersPerSec`.increasingRate:duration:5s# In Hyperfoil, everything is pre-allocated = limited in size. Here we'll# set that we won't run more than 10 iterations of this phase.maxIterations:10# Number of started users per sec increases with the iteration; in first# iteration we'll go from 0 to 10 users/second, in second from 10 to 20# and in last (10th) we'll reach 100 users/second.initialUsersPerSec:base:0increment:10targetUsersPerSec:base:10increment:10# Nth iteration of this phase will start when (N-1)th iteration of other# steady-state phases are finished. First iteration can start# immediatelly, of course.startAfter:- phase:browsingUserSteadyiteration:previous- phase:buyingUserSteadyiteration:previous# The &browsingUser syntax below creates YAML alias: we can later# reference this scenario and it will be used verbatim in another phase.# It is possible to use aliases for both scenarios and sequences.scenario:&browsingUser# We'll use the same scenario as in eshop.hf.yaml- browse:- httpRequest:GET:/quickstarts/eshop/items- browsingUserSteady:constantRate:duration:10smaxIterations:10usersPerSec:base:10increment:10# Nth iteration of this phase will start when Nth iteration of ramp-up# phases is finished.# Note that there's implicit rule that Nth iteration of given phase will# start only after (N-1)th iteration terminates.startAfter:- phase:browsingUserRampUpiteration:same- phase:buyingUserRampUpiteration:same# This refers to the alias created above; in steady state we'll use the# same scenario.scenario:*browsingUser# These two phases will be very similar to browsingUserSteady and RampUp- buyingUserRampUp:increasingRate:duration:5smaxIterations:10initialUsersPerSec:base:0increment:5targetUsersPerSec:base:5increment:5startAfter:- phase:browsingUserSteadyiteration:previous- phase:buyingUserSteadyiteration:previous# Again we'll use the same scenario as in eshop.hf.yamlscenario:&buyingUser- browse:- httpRequest:GET:/quickstarts/eshop/itemshandler:body:json:query:.[].idtoArray:itemIds[10]- buy:- randomItem:itemId <- itemIds- httpRequest:POST:/quickstarts/eshop/items/${itemId}/buy- buyingUserSteady:constantRate:duration:10smaxIterations:10usersPerSec:base:5increment:5startAfter:- phase:browsingUserRampUpiteration:same- phase:buyingUserRampUpiteration:samescenario:*buyingUser# Operator phase is omitted for brevity as we wouldn't scale that up
Don’t forget to start the mock server as we’ve used in the previous quickstart.
podman run --rm -p 8080:8083 quay.io/hyperfoil/hyperfoil-examples
Synchronizing multiple workloads across iteration can become a bit cumbersome. That’s why we can keep similar types of workflow together, and split the phase into forks. In fact forks will become different phases, but these will be linked together so that you can refer to all of them as to a single phase. Take a look at the benchmark rewritten to use forks:
name:eshop-forkshttp:host:http://localhost:8080sharedConnections:80phases:- rampUp:increasingRate:duration:5smaxIterations:10# Note that we have increased both the base and increment from 10 and 5# to 15. This value is split between the forks based on their weight.initialUsersPerSec:base:0increment:15targetUsersPerSec:base:15increment:15startAfter:phase:steadyStateiteration:previousforks:browsingUser:weight:2scenario:&browsingUser- browse:- httpRequest:GET:/quickstarts/eshop/itemsbuyingUser:weight:1scenario:&buyingUser- browse:- httpRequest:GET:/quickstarts/eshop/itemshandler:body:json:query:.[].idtoArray:itemIds[10]- buy:- randomItem:itemId <- itemIds- httpRequest:POST:/quickstarts/eshop/items/${itemId}/buy- steadyState:constantRate:duration:10smaxIterations:10usersPerSec:base:15increment:15startAfter:phase:rampUpiteration:sameforks:browsingUser:weight:2scenario:*browsingUserbuyingUser:weight:1scenario:*buyingUser# Operator phase is omitted for brevity as we wouldn't scale that up
This definition will create phases rampUp/0/browsingUser, rampUp/0/buyingUser, rampUp/1/browsingUser etc. - you’ll see them in statistics.
You could orchestrate the phases as it suits you, using startAfter, startAfterStrict (this requires the referenced phase to me terminated instead of finished as with startAfter) or startTime with relative time since benchmark start.
This sums up basic principles, in next quickstart you’ll see how to start and use Hyperfoil in distributed mode.
2.6 - Running the server
Learn how to start the Hyperfoil server in standalone mode
Until now we have always started our benchmarks using an embedded controller in the CLI, using the start-local command. This spawns a server in the CLI JVM. CLI communicates with it using standard REST API, though the server port is randomized and listens on localhost only. All the benchmarks and run results are also stored in /tmp/hyperfoil/ - you can change the directory as an argument to the start-local command.
While the embedded controller might be convenient for a quick test or when developing the scenario it’s not something that you’d use for a full-fledged benchmark.
When testing a reasonably performing system you need multiple nodes driving the load - we call them agents. These agents sync up, receive commands and report statistics to a master node, the controller. This node exposes a RESTful API to upload & start the benchmark, watch its progress and download results.
There are two other scripts in the bin/ directory:
standalone.sh starts both the controller and (one) agent in a single JVM. This is not too different from the controller embedded in CLI.
controller.sh starts clustered Vert.x and deploys the controller. Agents are started as needed in different nodes. You’ll see this in the next quickstart.
Open two terminals; in one terminal start the standalone server and in second terminal start the CLI.
bin/standalone.sh
and
bin/cli.sh
Then, let’s try to connect to the server (by default running on http://localhost:8090) and upload the single-request benchmark:
# This is the name of the benchmark. It's recommended to keep this in sync with# name of this file, adding extension `.hf.yaml`.name:single-request# We must define at least one HTTP target, in this case it becomes a default# for all HTTP requests.http:host:http://hyperfoil.io# Simulation consists of phases - potentially independent workloads.# We'll discuss phases more in detail in next quickstarts.phases:# `example` is the name of the single phase in this benchmark.- example:# `atOnce` with `users: 1` results in running the scenario below just onceatOnce:users:1scenario:# The only sequence in this scenario is called `test`.- test:# In the only step in this sequence we'll do a HTTP GET request# to `http://hyperfoil.io/`- httpRequest:GET:/# Inject helpers to make this request synchronous, i.e. keep# the sequence blocked until Hyperfoil processes the response.sync:true
From the second terminal, the one running the Hyperfoil CLI, issue the following commands:
[hyperfoil@localhost]$ connect
Connected! Server has these agents connected:
* localhost[REGISTERED][hyperfoil@localhost]$ upload .../single-request.hf.yaml
Loaded benchmark single-request, uploading...
... done.
[hyperfoil@localhost]$ run single-request
Started run 0001
When you switch to the first terminal (the one running the controller), you can see in the logs that the benchmark definition was stored on the server, the benchmark has been executed and its results have been stored to disk. Hyperfoil by default stores benchmarks in directory /tmp/hyperfoil/benchmark and data about runs in /tmp/hyperfoil/run; check it out:
Phase Name Requests Responses Mean Min p50.0 p90.0 p99.0 p99.9 p99.99 Max MeanSendTime ConnFailure Reset Timeouts 2xx 3xx 4xx 5xx Other Invalid BlockedCount BlockedTime MinSessions MaxSessions
example test11267911168267386880268435455268435455268435455268435455268435455268435455265587900001000000
Reading CSV/JSON files directly is not too comfortable; you can check the details through CLI as well:
[hyperfoil@localhost]$ stats
Total stats from run 002D
Phase Sequence Requests Mean p50 p90 p99 p99.9 p99.99 2xx 3xx 4xx 5xx Timeouts Errors
example:
test: 1 267.91 ms 268.44 ms 268.44 ms 268.44 ms 268.44 ms 268.44 ms 010000
By the time you type the stats command into CLI the benchmark is already completed and the CLI shows stats for the whole run. Let’s try running the {% include example_link.md src=‘eshop-scale.hf.yaml’ %} we’ve seen in previous quickstart; this will give us some time to observe on-line statistics as the benchmark is progressing:
podman run --rm -p 8080:8083 quay.io/hyperfoil/hyperfoil-examples
[hyperfoil@localhost]$ upload .../eshop-scale.hf.yaml
Loaded benchmark eshop-scale, uploading...
... done.
[hyperfoil@localhost]$ run eshop-scale
Started run 0002Run 0002, benchmark eshop-scale
...
Here the console would automatically jump into the status command, displaying the progress of the benchmark online. Press Ctrl+C to cancel that (it won’t stop the benchmark run) and run the stats command:
[hyperfoil@localhost]$ stats
Recent stats from run 0002Phase Sequence Requests Mean p50 p90 p99 p99.9 p99.99 2xx 3xx 4xx 5xx Timeouts Errors
buyingUserSteady/000:
buy: 8 1.64 ms 1.91 ms 3.05 ms 3.05 ms 3.05 ms 3.05 ms 800000 browse: 8 2.13 ms 2.65 ms 3.00 ms 3.00 ms 3.00 ms 3.00 ms 800000browsingUserSteady/000:
browse: 8 2.74 ms 2.69 ms 2.97 ms 2.97 ms 2.97 ms 2.97 ms 800000Press Ctr+C to stop watching...
You can go back to the run progress using the status command (hint: use status --all to display all phases, including those not started or already terminated):
[hyperfoil@localhost]$ status
Run 0002, benchmark eshop-scale
Agents: localhost[INITIALIZED]Started: 2019/04/15 16:27:24.526
NAME STATUS STARTED REMAINING FINISHED TOTAL DURATION
browsingUserRampUp/006 RUNNING 16:28:54.565 2477 ms
buyingUserRampUp/006 RUNNING 16:28:54.565 2477 ms
Press Ctrl+C to stop watching...
Since we are showing this quickstart running the controller and CLI on the same machine it’s easy to fetch results locally from /tmp/hyperfoil/run/XXXX/.... To save you SSHing into the controller host and finding the directories in a ’true remote’ case there’s the export command; This fetches statistics to your computer where you’re running CLI. You can chose between default JSON format (e.g. export 0002 -f json -d /path/to/dir) and CSV format (export 0002 -f csv -d /path/to/dir) - the latter packs all CSV files into single ZIP file for your convenience.
When you find out that the benchmark is not going well, you can terminate it prematurely:
[hyperfoil@localhost]$ killKill run 0002, benchmark eshop-scale(phases: 2 running, 0 finished, 40 terminated)[y/N]: y
Killed.
In the next quickstart we will deal with starting clustered Hyperfoil.
2.7 - Clustered mode
Learn how to start the Hyperfoil server in clustered mode
Previously we’ve learned to start Hyperfoil in standalone server mode, and to do some runs through CLI. In this quickstart we’ll see how to run your benchmark distributed to several agent nodes.
Hyperfoil operates as a cluster of Vert.x. When the benchmark is started, it deploys agents on other nodes according to the benchmark configuration - these are Vert.x nodes, too. Together controller and agents form a cluster and communicate over the event bus.
In this quickstart we’ll use the SSH deployer; make sure your machine has SSH server running on port 22 and you can login using your pubkey ~/.ssh/id_rsa. The SSH deployer copies the necessary JARs to /tmp/hyperfoil/agentlib/ and starts the agent there. For instructions to run Hyperfoil in Kubernetes or Openshift please consult the Installation docs.
When we were running in the standalone or local mode we did not have to set any agents in the benchmark definition. That changes now as we need to inform the controller where the agents should be deployed. Let’s see a benchmark - two-agents.hf.yaml that has those agents defined.
name:two-agents# List of agents the Controller should deployagents:# This defines the agent using SSH connection to localhost, port 22agent-one:localhost:22# Another agent on localhost, this time defined using propertiesagent-two:host:localhostport:22http:host:http://localhost:8080usersPerSec:10duration:10sscenario:- test:- httpRequest:GET:/
The load the benchmark generates is evenly split among the agents, so if you want to use another agent, you don’t need to do any calculations - just add the agent and you’re good to go.
Open three terminals; in the first start the controller using bin/controller.sh, in second one open the CLI with bin/cli.sh and in the third one start the example workload server:
podman run --rm -p 8080:8083 quay.io/hyperfoil/hyperfoil-examples
Connect, upload, start and check out the benchmark using CLI exactly the same way as we did in the previous quickstart:
[hyperfoil@localhost]$ connect
Connected!
[hyperfoil@localhost]$ upload .../two-agents.hf.yaml
Loaded benchmark two-agents, uploading...
... done.
[hyperfoil@localhost]$ run two-agents
Started run 004A
[hyperfoil@localhost]$ status
Run 004A, benchmark two-agents
Agents: agent-one[STARTING], agent-two[STARTING]Started: 2019/04/17 17:08:19.703 Terminated: 2019/04/17 17:08:29.729
NAME STATUS STARTED REMAINING FINISHED TOTAL DURATION
main TERMINATED 17:08:19.708 17:08:29.729 10021 ms (exceeded by 21 ms)[hyperfoil@localhost]$ stats
Total stats from run 004A
Phase Sequence Requests Mean p50 p90 p99 p99.9 p99.99 2xx 3xx 4xx 5xx Timeouts Errors
main:
test: 106 3.12 ms 2.83 ms 3.23 ms 19.53 ms 25.30 ms 25.30 ms 10600000
You see that we did 106 requests which fits the assumption about running 10 user sessions per second over 10 seconds, while we have used 2 agents.
Vert.x clustering is using Infinispan and JGroups; depending on your networking setup it might not work out-of-the-box. If you experience any trouble, check out the FAQ.
Next quickstart will get back to the scenario definition; we’ll show you how to extend Hyperfoil with custom steps and handlers.
2.8 - Custom components
Hyperfoil offers some basic steps to do HTTP requests, generate data, alter control flow in the scenario etc., but your needs may surpass the features implemented so far. Also, it might be just easier to express your logic in Java code than combining steps in the YAML. The downside is reduced ability to reuse and more tight dependency on Hyperfoil APIs.
This quickstart will show you how to extend Hyperfoil with custom steps and handlers. As we use the standard Java ServiceLoader approach, after you build the module you should drop it into extensions directory. (Note: if you upload the benchmarks through CLI you need to put it to both the machine where you run the CLI and to the controller.)
Each extension will consist of two classes:
Builder, is loaded as service and creates the immutable extension instance
extension (Step, Action or handler)
Let’s start with a io.hyperfoil.api.config.Step implementation. The interface has single method invoke(Session) that should return true if the step was executed and false if its execution has been blocked and should be retried later. In case that the execution is blocked the invocation must not have any side effects - e.g. if the step is fetching objects from some pools and one of the pools is depleted, it should release the already acquired objects back to the pool.
We’ll create a step that will divide variable from a session by a (configurable) constant and store the result in another variable.
Java
publicclassDivideStepimplementsStep{// All fields in a step are immutable, any state must be stored in the Session
privatefinalReadAccessfromVar;privatefinalIntAccesstoVar;privatefinalintdivisor;publicDivideStep(ReadAccessfromVar,IntAccesstoVar,intdivisor){// Variables in session are not accessed directly using map lookup but
// through the Access objects. This is necessary as the scenario can use
// some simple expressions that are parsed when the scenario is built
// (in this constructor), not at runtime.
this.fromVar=fromVar;this.toVar=toVar;this.divisor=divisor;}@Overridepublicbooleaninvoke(Sessionsession){// This step will block until the variable is set, rather than
// throwing an error or defaulting the value.
if(!fromVar.isSet(session)){returnfalse;}// Session can store either objects or integers. Using int variables is
// more efficient as it prevents repeated boxing and unboxing.
intvalue=fromVar.getInt(session);toVar.setInt(session,value/divisor);returntrue;}...
Then we need a builder class that will allow us to configure the step. To keep related classes together we will define it as inner static class:
Java
publicclassDivideStepimplementsStep{...// Make this builder loadable as service
@MetaInfServices(StepBuilder.class)// This is the step name that will be used in the YAML
@Name("divide")publicstaticclassBuilderextendsBaseStepBuilder<Builder>implementsInitFromParam<Builder>{// Contrary to the step fields in builder are mutable
privateStringfromVar;privateStringtoVar;privateintdivisor;// Let's permit a short-form definition that will store the result
// in the same variable. Note that the javadoc @param is used to generate external documentation.
/**
* @param param Use myVar /= constant
*/@OverridepublicBuilderinit(Stringparam){intdivIndex=param.indexOf("/=");if(divIndex<0){thrownewBenchmarkDefinitionException("Invalid inline definition: "+param);}try{divisor(Integer.parseInt(param.substring(divIndex+2).trim()));}catch(NumberFormatExceptione){thrownewBenchmarkDefinitionException("Invalid inline definition: "+param,e);}Stringvar=param.substring(0,divIndex).trim();returnfromVar(var).toVar(var);}// All fields are set in fluent setters - this helps when the scenario
// is defined through programmatic configuration.
// When parsing YAML the methods are invoked through reflection;
// the attribute name is used for the method lookup.
publicBuilderfromVar(StringfromVar){this.fromVar=fromVar;returnthis;}publicBuildertoVar(StringtoVar){this.toVar=toVar;returnthis;}// The parser can automatically convert primitive types and enums.
publicBuilderdivisor(intdivisor){this.divisor=divisor;returnthis;}@OverridepublicList<Step>build(){// You can ignore the sequence parameter; this is used only in steps
// that require access to the parent sequence at runtime.
if(fromVar==null||toVar==null||divisor==0){// Here is a good place to check that the attributes are sane.
thrownewBenchmarkDefinitionException("Missing one of the required attributes!");}// The builder has a bit more flexibility and it can create more than
// one step at once.
returnCollections.singletonList(newDivideStep(SessionFactory.readAccess(fromVar),SessionFactory.intAccess(toVar),divisor));}}...
As the comments say, the builder is using fluent setter syntax to set the attributes. When you want to nest attributes under another builder, you can just add parameter-less method FooBuilder foo() the returns an instance of FooBuilder; the parser will fill this instance as well. There are some interfaces your builder can implement to accept lists or different structures, but the description is out of scope of this quickstart.
The builder class has two annotations: @Name which specifies the name we’ll use in YAML as step name, and @MetaInfServices with StepBuilder.class as the parameter. If you were to implement other type of extension, this would be Action.Builder.class, Request.ProcessorBuilder.class etc. In order to record the service in META-INF directory in the jar you must also add this dependency to your module:
The whole class can be inspected here and it is already included in the extensions directory. You can try running bin/standalone.sh, upload and run divide.hf.yaml. You should see about 5 log messages in the server log.
# This benchmark demonstrates custom stepsname:dividehttp:host:http://localhost:8080usersPerSec:1duration:5sscenario:- test:- setInt:foo <- 33- divide:foo /= 3- log:message:Foo is {}vars:- foo
There are several other integration points but Step:
io.hyperfoil.api.session.Action is very similar to step, but it does not allow blocking. Implement Action.BuilderFactory to define new actions.
StatusHandler, HeaderHandler and BodyHandler in io.hyperfoil.api.http package process different stages of HTTP response parsing. All these have BuilderFactory inner interface for you to implement.
io.hyperfoil.api.connection.Processor performs later generic stages of response processing. As this interface is generic, there are two factories that you could use: i.h.a.c.Request.ProcessorBuilderFactory and i.h.a.c.HttpRequest.ProcessorBuilderFactory.
There is quite some boilerplate code in the process of creating a new component; that’s why you can use Hyperfoil Codegen Maven plugin to scaffold the basic outline for you. Go to the module where you want the component generated and run:
The plugin will ask you for the package name, component name and type and write down the source code skeleton. You can provide the parameters right on commandline like
This is the last quickstart in this series; if you seek more info check out the documentation or talk to us on GitHub Discussions.
3 - User Guide
Comprehensive set of resourcs for everything you need to get started with Hyperfoil
Welcome to the Hyperfoil User Guide, your comprehensive resource for everything you need to get started. This section covers installation, detailed instructions on defining benchmarks, and troubleshooting tips to help you resolve common issues. Whether you’re a beginner or an advanced user, you’ll find valuable information to enhance your performance testing with Hyperfoil.
3.1 - Installation
Detailed instructions for installing Hyperfoil manually, on Kubernetes/Openshift, or with Ansible.
In this section, you’ll find detailed instructions for installing and setting up Hyperfoil using various methods, including manual setup, Ansible, and Kubernetes/Openshift. Follow these guides to choose the best installation procedure for your environment.
3.1.1 - Manual startup
Explore manual startup options for the Hyperfoil controller.
Hyperfoil controller is started with
bin/controller.sh
Any arguments passed to the scripts will be passed as-is to the java process.
By default io.hyperfoil.deployer is set to ssh which means that the controller will deploy agents over SSH, based on the agents configurion. This requires that the user account running the controller must have public-key SSH authorization set up using key $HOME/.ssh/id_rsa. The user also has to be able to copy files to the directory set in agent definition (by default /tmp/hyperfoil) using SCP - Hyperfoil automatically synchronizes library files in this folder with the currently running instance and then executes the agent.
When you don’t intend to run distributed benchmarks you can start the controller in standalone mode:
bin/standalone.sh
This variant won’t deploy any agents remotely and therefore it does not need any agents: section in the benchmark definition; instead it will use single agent started in the same JVM.
Below is the comprehensive list of all the properties Hyperfoil recognizes. All system properties can be replaced by environment variables, uppercasing the letters and replacing dots and dashes with underscores: e.g. io.hyperfoil.controller.host becomes IO_HYPERFOIL_CONTROLLER_HOST.
Property
Default
Description
io.hyperfoil.controller.host
0.0.0.0
Host for Controller REST server
io.hyperfoil.controller.port
8090
Port for Controller REST server
io.hyperfoil.rootdir
/tmp/hyperfoil
Root directory for stored files
io.hyperfoil.benchmarkdir
root/benchmark
Benchmark files (YAML and serialized)
io.hyperfoil.rundir
root/run
Run result files (configs, stats…)
io.hyperfoil.deployer
ssh
Implementation for agents deployment
io.hyperfoil.deployer.timeout
15000 ms
Timeout for agents to start
io.hyperfoil.agent.debug.port
If set, agent will be started with JVM debug port open
io.hyperfoil.agent.debug.suspend
n
Suspend parameter for the debug port
io.hyperfoil.controller.cluster.ip
first non-loopback
Hostname/IP used for clustering with agents
io.hyperfoil.controller.cluster.port
7800
Default JGroups clustering port
io.hyperfoil.controller.external.uri
Externally advertised URI of REST server
io.hyperfoil.controller.keystore.path
File path to Java Keystore
io.hyperfoil.controller.keystore.password
Java Keystore password
io.hyperfoil.controller.pem.keys
File path(s) to private TLS key(s) in PEM format
io.hyperfoil.controller.pem.certs
File path(s) to server TLS certificate(s) in PEM format
io.hyperfoil.controller.password
Password used for Basic authentication
io.hyperfoil.controller.secured.via.proxy
This must be set to true for Basic auth without TLS encryption
io.hyperfoil.trigger.url
See below
If io.hyperfoi.trigger.url is set the controller does not start benchmark run right away after hitting /benchmark/my-benchmark/start ; instead it responds with status 301 and header Location set to concatenation of this string and BENCHMARK=my-benchmark&RUN_ID=xxxx. CLI interprets that response as a request to hit CI instance on this URL, assuming that CI will trigger a new job that will eventually call /benchmark/my-benchmark/start?runId=xxxx with header x-trigger-job. This is useful if the the CI has to synchronize Hyperfoil to other benchmarks that don’t use this controller instance.
Security
Since Hyperfoil accepts and invoked any serialized Java objects you must not run it exposed to public to prevent a very simple remote code execution. Even if using HTTPS and password protection (see below) we recommend to limit access and privileges of the process to absolute minimum.
You can get confidential access to the server using TLS encryption, providing the certificate and keys either using Java Keystore mechanism (properties above starting with io.hyperfoil.controller.keystore) or via PEM files (properties starting with io.hyperfoil.controller.pem). These options are mutually exclusive. In the latter case it is possible to use multiple certificate/key files, separated by comma (,).
Authentication uses Basic authentication scheme accepting any string as username. The password is set using io.hyperfoil.controller.password or respective environment variable. If you’re exposing the server using plaintext HTTP you must set -Dio.hyperfoil.controller.secured.via.proxy=true to confirm that this is a desired configuration (e.g. if the TLS is terminated at proxy and the connection from proxy does not require confidentiality).
3.1.2 - K8s/Openshift deployment
Deploy Hyperfoil in Kubernetes or Openshift environment using the out-of-the-box Hyperfoil operator
A convenient alternative to running Hyperfoil on hosts with SSH access is deploying it in Kubernetes or Openshift environment. The recommended way to install it using an operator in your Openshift console - just go to Operators - OperatorHub and search for ‘hyperfoil’, and follow the installation wizard. Alternatively you can deploy the controller manually.
In order to start a Hyperfoil Controller instance in your cluster, create a new namespace hyperfoil: Go to Operators - Installed Operators and open Hyperfoil. In upper left corner select ‘Project: ’ - Create project and fill out the details. Then click on the ‘Hyperfoil’ tab and find the button ‘Create Hyperfoil’.
This is a perfectly valid Hyperfoil resource with everything set to default values. You can customize some properties in the spec section further - see the reference.
The operator deploys only the controller; each agent is then started when the run starts as a pod in the same namespace and stopped when the run completes.
When the resource becomes ready (you can check it out through Openshift CLI using oc get hf) the controller pod should be up and running. Now you can open Hyperfoil CLI and connect to the controller. While default Hyperfoil port is 8090, default route setting uses TLS (edge) and therefore Openshift router will expose the service on port 443. If your cluster’s certificate is not recognized (such as when using self-signed certificates) you need to use --insecure (or -k) option.
bin/cli.sh
[hyperfoil]$ connect hyperfoil-hyperfoil.apps.my.cluster.domain:443 --insecure
WARNING: Hyperfoil TLS certificate validity is not checked. Your credentials might get compromised.
Connected!
WARNING: Server time seems to be off by 12124 ms
Now you can upload & run benchmarks as usual - we’re using {% include example_link.md src=‘k8s-hello-world.hf.yaml’ %} in this example. Note that it can take several seconds to spin up containers with agents.
[hyperfoil@hyperfoil-hyperfoil]$ upload examples/k8s-hello-world.hf.yaml
Loaded benchmark k8s-hello-world, uploading...
... done.
[hyperfoil@hyperfoil-hyperfoil]$ run k8s-hello-world
Started run 0000Run 0000, benchmark k8s-hello-world
Agents: agent-one[STARTING]Started: 2019/11/18 19:07:36.752 Terminated: 2019/11/18 19:07:41.778
NAME STATUS STARTED REMAINING COMPLETED TOTAL DURATION DESCRIPTION
main TERMINATED 19:07:36.753 19:07:41.778 5025 ms (exceeded by 25 ms) 5.00 users per second
[hyperfoil@hyperfoil-hyperfoil]$
Running Hyperfoil inside the cluster you are trying to test might skew results due to different network topology compared to driving the load from ‘outside’ (as real users would do). It is your responsibility to validate if your setup and separation between load driver and SUT (system under test) is correct. You have been warned.
Reference
Property
Description
version
Tag for controller image (e.g. 0.14 for a released version or 0.15-SNAPSHOT for last build from main (master) branch). Defaults to latest.
image
Controller image. If ‘version’ is defined, too, the tag is replaced (or appended). Defaults to ‘quay.io/hyperfoil/hyperfoil’
Name of the config map and optionally its entry (separated by ‘/’: e.g myconfigmap/log4j2-superverbose.xml) storing Log4j2 configuration file. By default the Controller uses its embedded configuration.
List of config map names holding hooks that run before the run starts.
postHooks
List of config map names holding hooks that run when the run finishes.
persistentVolumeClaim
Name of the PVC Hyperfoil should mount for its workdir.
route
Property
Description
host
Host for the route leading to Controller REST endpoint. Example: hyperfoil.apps.cloud.example.com
type
Either ‘http’ (for plain-text routes - not recommended), ’edge’, ‘reencrypt’ or ‘passthrough’
tls
Name of the secret hosting tls.crt, tls.key and optionally ca.crt. This is mandatory for passthrough routes and optional for edge and reencrypt routes
auth
Property
Description
secret
Name of secret used for basic authentication. Must contain key password; Hyperfoil accepts any username for login.
3.1.3 - Manual k8s/Openshift deployment
Manually deploy Hyperfoil in Kubernetes or Openshift environment
If you cannot use the operator or if you’re running vanilla Kubernetes you can define all the resource manually. You deploy only the controller; each agent is then started, when the run starts, as a pod in the same namespace and stopped when the run completes.
Following steps install Hyperfoil controller in Openshift, assuming that you have all the required priviledges. With vanilla Kubernetes you might have to replace the route with an appropriate ingress.
1. Create new namespace for hyperfoil
oc new-project hyperfoil
2. Create required resources
curl -s -L k8s.hyperfoil.io | oc apply -f -
role.rbac.authorization.k8s.io/controller created
serviceaccount/controller created
service/hyperfoil created
rolebinding.rbac.authorization.k8s.io/controller created
deploymentconfig.apps.openshift.io/controller created
route.route.openshift.io/hyperfoil created
The route will use hostname following the format hyperfoil-hyperfoil.apps.my.cluster.domain - feel free to customize the hostname as needed.
3. Wait until the image gets downloaded and the container starts
oc get po
NAME READY STATUS RESTARTS AGE
controller-1-pqbvs 1/1 Running 0 57s
controller-1-deploy 0/1 Completed 0 72s
4. Open CLI and connect to the controller
While default Hyperfoil port is 8090, Openshift router will expose the service on port 80.
bin/cli.sh
[hyperfoil]$ connect hyperfoil-hyperfoil.apps.my.cluster.domain -p 80Connected!
WARNING: Server time seems to be off by 12124 ms
5. Upload and run benchmarks as usual
We’re using k8s-hello-world.hf.yaml in this example.
Note that it can take several seconds to spin up containers with agents.
[hyperfoil@hyperfoil-hyperfoil]$ upload .../k8s-hello-world.hf.yaml
Loaded benchmark k8s-hello-world, uploading...
... done.
[hyperfoil@hyperfoil-hyperfoil]$ run k8s-hello-world
Started run 0000Run 0000, benchmark k8s-hello-world
Agents: agent-one[STARTING]Started: 2019/11/18 19:07:36.752 Terminated: 2019/11/18 19:07:41.778
NAME STATUS STARTED REMAINING COMPLETED TOTAL DURATION DESCRIPTION
main TERMINATED 19:07:36.753 19:07:41.778 5025 ms (exceeded by 25 ms) 5.00 users per second
[hyperfoil@hyperfoil-hyperfoil]$
Running Hyperfoil inside the cluster you are trying to test might skew results due to different network topology compared to driving the load from ‘outside’ (as real users would do). It is your responsibility to validate if your setup and separation between load driver and SUT (system under test) is correct. You have been warned.
3.1.4 - Ansible startup
Deploy Hyperfoil using Ansible Galaxy scripts
You can fetch release, distribute and start the cluster using Ansible Galaxy scripts; setup, test, shutdown
You can add more agents by duplicating the last line with agent-2 etc.
Prepare your playbook; here is a short example that starts the controller, uploads and starts simple benchmark (the templating engine replaces the agents in benchmark script based on Ansible hosts) and waits for its completion. When it confirms number of requests executed it stops the controller.
- hosts:[hyperfoil-agent, hyperfoil-controller ]tasks:[]# This will only gather facts about all nodes- hosts:hyperfoil-controllerroles:- hyperfoil.hyperfoil_setup- hosts:127.0.0.1connection:localroles:- hyperfoil.hyperfoil_testvars:test_name:example# Note that due to the way Ansible lookups work this will work only if hyperfoil-controller == localhost- hosts:127.0.0.1connection:localtasks:- name:Find number of requestsset_fact:test_requests:"{{ lookup('csvfile', 'example file=/tmp/hyperfoil/workspace/run/' + test_runid + '/stats/total.csv col=2 delimiter=,')}}"- name:Print number of requestsdebug:msg:"Executed {{ test_requests }} requests."- hosts:- hyperfoil-controllerroles:- hyperfoil.hyperfoil_shutdown
Finally, run the playbook:
ansible-playbook -i hosts example.yml
3.2 - Benchmark
Detailed breakdown of each component involved in defining a benchmark
In Hyperfoil, defining a benchmark involves structuring scenarios, phases, variables and other components to simulate realistic user behavior and workload patterns. This section provides a detailed breakdown of each component involved in defining a benchmark.
3.2.1 - Agents
Entities responsible for executing benchmark and collecting statistics
The definition is passed to an instance of i.h.api.deployment.Deployer which will interpret the definition. Deployer implementation is registred using the java.util.ServiceLoader and selected through the io.hyperfoil.deployer system property. The default implementation is ssh.
Common properties
Property
Default
Description
threads
from benchmark
Number of threads used by the agent (overrides threads in benchmark root).
extras
Custom options passed to the JVM (system properties, JVM options…)
SSH deployer
The user account running Hyperfoil Controller must have a public-key authorization set up on agents’ hosts using key $HOME/.ssh/id_rsa. It also has to be able to copy files into the dir directory using SCP - all the required JARs will be copied there and you will find the logs there as well.
ssh deployer accepts either the [user@]host[:port] inline syntax or these properties:
Property
Default
Description
user
Current username
host
This property is mandatory.
port
22
sshKey
id_rsa
Optionally define a different named key in the $HOME/.ssh directory
dir
Directory set by system property io.hyperfoil.rootdir or /tmp/hyperfoil
Working directory for the agent. This directory can be shared by multiple agents running on the same physical machine.
cpu
(all cpus)
If set the CPUs where the agent can run is limited using taskset -c <cpu>. Example: 0-2,6
To activate the kubernetes deployer you should set -Dio.hyperfoil.deployer=k8s; the recommended installation does that automatically.
The agents are configured the same way as with SSH deployment, only the properties differ. Full reference is provided below.
Example:
agents:my-agent:node:my-worker-node
Property
Default
Description
node
Configures the labels for the nodeSelector. If the value does not contain equals sign (=) or comma (,) this sets the desired value of label kubernetes.io/hostname. You can also set multiple custom labels separated by commas, e.g. foo=bar,kubernetes.io/os=linux.
stop
true
By default the controller stops all agents immediatelly after the run terminates. In case of errors this is not too convenient as you might want to perform further analysis. To prevent automatic agent shutdown set this to false.
log
Name of config map (e.g. my-config-map) or config map and its entry (e.g. my-config-map/log4j2.xml) that contains the Log4j2 configuration file. Default entry from the config map is log4j2.xml. Hyperfoil will mount this configmap as a volume to this agent.
image
quay.io/hyperfoil/hyperfoil:controller-version
Different version of Hyperfoil in the agents
imagePullPolicy
Always
Image pull policy for agents
fetchLogs
true
Automatically watch agents’ logs and store them in the run directory.
3.2.2 - HTTP
This section defines servers that agents contact during benchmarks, allowing configurations for multiple targets with specific connection settings
All servers that Hyperfoil should contact must be declared in this section. Before the benchmark starts Hyperfoil agents will open connections to the target servers; if this connection fails the benchmark is terminated immediatelly.
You can either declare single target server (the default one) within this section or more of them:
Supply list of IPs or IP:port targets that will be used for the connections instead of resolving the host in DNS and using port as set - host and port will be used only for Host headers and SNI. If this list contains more addresses the connections will be split evenly.
requestTimeout
30 seconds
Default request timeout, this can be overridden in each httpRequest.
allowHttp1x
true
Allow HTTP 1.1 for connections (e.g. during ALPN).
allowHttp2x
true
Allow HTTP 2.0 for connections (e.g. during ALPN). If both 1.1 and 2.0 are allowed and https is not used (which would trigger ALPN) Hyperfoil will use HTTP 1.1. If only 2.0 is allowed Hyperfoil will start with HTTP 1.1 and perform protocol upgrade to 2.0.
directHttp2
false
Start with H2C HTTP 2.0 without protocol upgrade. Makes sense only for plain text (http) connections. Currently not implemented.
maxHttp2Streams
100
Maximum number of requests concurrently enqueued on single HTTP 2.0 connection.
pipeliningLimit
1
Maximum number of requests pipelined on single HTTP 1.1 connection.
rawBytesHandlers
true
Enable or disable using handlers that process HTTP response raw bytes.
TLS trust manager for setting up server certificates.
useHttpCache
true
Make use of HTTP cache on client-side. If multiple authorities are involved, disable the HTTP cache for all of them to achieve the desired outcomes. The default is true except for wrk/wrk2 wrappers where it is set to false.
Shared connections
This number is split between all agents and executor threads evenly; if there are too many agents/executors each will get at least 1 connection.
When a scalar value is used for this property the connection pool has fixed size; Hyperfoil opens all connections when the benchmark starts and should a connection be closed throughout the benchmark, another connection is reopened instead. You can change this behaviour by composing the property of these sub-properties:
Property
Description
core
Number of connections that will be opened when the benchmark starts. Number of connections in the pool should never drop below this value (another connection will be opened instead).
max
Maximum number of connections in the pool.
buffer
Hyperfoil will try to keep at least active + buffer connections in the pool where active is the number of currently used connection (those with at least 1 in-flight request)
keepAliveTime
When a connection is not used for more than this value (in milliseconds) it will be closed. Non-positive value means that the connection is never closed because of being idle.
This property describes the connection pooling model, you can choose from the options below:
Strategy
Description
SHARED_POOL
Connections are created in a pool and then borrowed by the session. When the request is complete the connection is returned to the shared pool.
SESSION_POOLS
Connections are created in a shared pool. When the request is completed it is not returned to the shared pool but to a session-local pool. Subsequent requests by this session first try to acquire the connection from this local pool. When the session completes all connections from the session-local pool are returned to the shared pool.
OPEN_ON_REQUEST
Connections are created before request or borrowed from a session-local pool. When the request is completed the connection is returned to this pool. When the session completes all connections from the session-local pool are closed.
ALWAYS_NEW
Always create the connection before the request and close it when it is complete. No pooling of connections.
KeyManager configuration
All files are loaded when the benchmark is constructed, e.g. on the machine running CLI. You don’t need to upload any files to controller or agent machines.
Property
Default
Description
storeType
JKS
Implementation of the store.
storeFile
Path to a file with the store.
password
Password for accessing the store file.
alias
Keystore alias.
certFile
Path to a file with the client certificate.
keyFile
Path to a file with client’s private key.
TrustManager configuration
All files are loaded when the benchmark is constructed, e.g. on the machine running CLI. You don’t need to upload any files to controller or agent machines.
Property
Default
Description
storeType
JKS
Implementation of the store.
storeFile
Path to a file with the store.
password
Password for accessing the store file.
certFile
Path to a file with the server certificate.
3.2.3 - Phases
Defines a unit of workload simulation within a benchmark, representing a specific load pattern or behavior
You might want to simulate several types of workloads at once: e.g. in an eshop users would come browsing or buying products, and operators would restock the virtual warehouse. Also, driving constant load may not be the best way to run the benchmark: often you want to slowly ramp the load up to let the system adjust (scale up, perform JIT, fill pools) and push the full load only after that. When trying to find system limits, you do the same repetitevely - ramp up the load, measure latencies and if the system meets SLAs (latencies below limits) continue ramping up the load until it breaks.
In Hyperfoil, this all is expressed through phases. Phases can run independently of each other;
these simulate certain load execute by a group of users. Within one phase all users execute the same scenario
(e.g. logging into the system, buying some goods and then logging off).
A phase can be in one of these states:
not running (scheduled): As the name clearly says, the phase is not yet getting executed.
running: The agent started running the phase, i.e., performing the configured load.
finished: Users won’t start new scenarios but we’ll let already-started users complete the scenario.
terminated: All users are done, all stats are collected and no further requests will be made.
cancelled: Same as terminated but this phase hasn’t been run at all.
There are different types of phases based on the mode of starting new users:
Type
Description
constantRate
The benchmark will start certain number of users according to a schedule regardless of previously started users completing the scenario. This is the open-model.
increasingRate
Similar to constantRate but ramps up the number of started users throughout the execution of the phase.
decreasingRate
The same as increasingRate but requires initialUsersPerSec > targetUsersPerSec.
atOnce
All users are be started when the phase starts running and once the scenario is completed the users won’t retry the scenario.
always
There is fixed number of users and once the scenario is completed the users will start executing the scenario from beginning. This is called a closed-model and is similar to the way many benchmarks with fixed number of threads work.
noop
This phase cannot have any scenario (or forks). It might be useful to add periods of inactivity into the benchmark.
See the example of phases configuration:
...phases:# Over one minute ramp the number of users started each second from 1 to 100- rampUp:increasingRate:initialUsersPerSec:1targetUsersPerSec:100# We expect at most 200 users being active at one moment - see belowmaxSessions:200duration:1mscenario:...# After rampUp is finished, run for 5 minutes and start 100 new users each second- steadyState:constantRate:usersPerSec:100maxSessions:200startAfter:rampUpduration:5m# If some users get stuck, forcefully terminate them after 6 minutes from the phase startmaxDuration:6mscenario:...# 2 minutes after the benchmark has started spawn 5 users constantly doing something for 2 minutes- outOfBand:always:users:5startTime:2mduration:2mscenario:...- final:atOnce:users:1# Do something at the end: make sure that both rampUp and steadyState are terminatedstartAfterStrict:- rampUp- steadyStatescenario:...
These properties are common for all types of phases:
Property
Description
startTime
Time relative to benchmark start when this phase should be scheduled. In other words, it’s the earliest moment when it could be scheduled, other conditions (below) may delay that even further.
startAfter
Phases that must be finished before this phase can start. You can use either single phase name, list of phases or a reference to certain iteration.
startAfterStrict
Phases that must be terminated before this phase can start. Use the same syntax as for startAfter.
duration
Intended duration for the phase (must be defined but for the atOnce type). After this time elapses no new sessions will be started; there might be some running sessions still executing operations, though.
maxDuration
After this time elapses all sessions are forcefully terminated.
isWarmup
This marker property is propagated to results JSON and allows the reporter to hide some phases by default.
maxUnfinishedSessions
Maximum number of session that are allowed to be open when the phase finishes. When there are more open sessions all the other sessions are cancelled and the benchmark is terminated. Unlimited by default.
maxIterations
Maximum number of iterations this phase will be scaled to. More about that below.
Below are properties specific for different phase types:
atOnce:
users: Number of users started at the start of the phase.
always:
users: Number of users started at the start of the phase. When a user finishes it is immediatelly restarted (any pause must be part of the scenario).
constantRate:
usersPerSec: Number of users started each second.
variance: Randomize delays between starting users following the exponential distribution. That way the starting users behave as the Poisson point process. If this is set to false users will be started with uniform delays. Default is true.
maxSessions: Number of preallocated sessions. This number is split between all agents/executors evenly.
increasingRate / decreasingRate:
initialUsersPerSec: Rate of started users at the beginning of the phase.
targetUsersPerSec: Rate of started users at the end of the phase.
variance: Same as in `constantRate
maxSessions: Same as in constantRate.
Hyperfoil initializes all phases before the benchmark starts, pre-allocating memory for sessions.
In the open-model phases it’s not possible to know how many users will be active at the same moment
(if the server experiences a 3-second hiccup and we have 100 new users per second this should be at least 300
as all the users will be blocked). However we need to provide the estimate for memory pre-allocation.
If the estimate gets exceeded the benchmark won’t fail nor block new users from starting, but new sessions
will be allocated which might negatively impact results accuracy.
Properties users, usersPerSec, initialUsersPerSec and targetUsersPerSec can be either a scalar number or scale with iterations using the base and increment components. You’ll see an example below.
Forks
As mentioned earlier, users in each phase execute the same scenario. Often it’s convenient
to define the ramp-up and steady-state phases just once: the builders allow to declare such ‘sub-phases’ called forks.
For all purposes but the benchmark configuration these become regular phases of the same type, duration and dependencies (startAfter, startAfterStrict) as the ‘parent’ phase but slice the users according to their weight:
...phases:- steadyState:constantRate:usersPerSec:30duration:5mforks:sellShares:# This phase will start 10 users per secondweight:1scenario:...buyShares:# This phase will start 20 users per secondweight:2scenario:...
These phases will be later identified as steadyState/sellShares and steadyState/buyShares. Other phases can still
reference steadyState (without suffix) as the dependency: there will be a no-op phase steadyState that starts (becomes running) as soon as both the forks finish, finish immediately and terminate once both the forks terminate.
Iterations
In some types of tests it’s useful to repeat given phase with increasing load - we call this concept iterations. In the example below you can see that *usersPerSec are not scalar values; in first iteration the actual value is set to the base value but in each subsequent iteration the value is increased by increment.
...phases:- rampUp:increasingRate:# Create phases rampUp/000, rampUp/001 and rampUp/002maxIterations:3# rampUp/000 will go from 1 to 100 users, rampUp will go from 101 to 200 users...initialUsersPerSec:base:1increment:100targetUsersPerSec:base:100increment:100# rampUp/001 will start after steadyState/000 finishesstartAfter:phase:steadyStateiteration:previousduration:1mscenario:...- steadyState:constantRate:maxIterations:3usersPerSec:base:100increment:100# steadyState/000 will start after rampUp/000 finishesstartAfter:phase:rampUpiteration:sameduration:5m
Similar to forks, there will be a no-op phase rampUp that will start after all
rampUp/xxx phases finish and terminate after these terminate. Also there’s an implicit dependency
between consecutive iterations: subsequent iteration won’t start until previous iteration terminates.
The startAfter property in this example uses a relative reference to iteration in another phase. Each reference has these properties:
| Property | Description |
| ——— | |
| phase | Name of the referenced phase. |
| iteration | Relative number of the iteration; either none (default) which references the top-level phase, same meaning the iteration with same number, or previous with number one lower. |
| fork | Reference to particular fork in the phase/iteration. |
Iterations can be combined with forks as well - the result name would be e.g. steadyState/000/sellShares.
Note that the maxSessions parameter is not scaling in iterations: all iterations execute the same
scenario, the execution does not overlap and therefore it is possible to share the pool of sessions.
Therefore you should provide an estimate for the iteration spawning the highest load.
Staircase
Hyperfoil tries to make opinionated decisions, simplifying common types of benchmark setups. That’s why it offers a simplified syntax for the scenario where you:
ramp the load to a certain level
execute steady state for a while
ramp it up further
execute another steady state
repeat previous two steps over and over
This is called a staircase as the load increases in a shape of tilted stairs. Phases such benchmark should consist of are automatically created and linked together, using the same scenario/forks.
staircase as a top-level element in the benchmark is mutually exclusive to scenario and phases elements.
Here is a minimalistic example of such configuration:
Defines the behavior and sequence of actions that virtual users (VU) perform during a benchmark execution
Scenario
Scenario is a set of sequences. The sequence is a block of sequentially executed steps. Contrary to steps in a sequence the sequences within a scenario do not need to be executed sequentially.
The scenario defines one or more initialSequences that are enabled from the beginning and other sequences that
must be enabled by any of the previously executed sequences. To be more precise it is not the sequence
that is enabled but a sequence instance as we can run a sequence multiple times in parallel (on different data).
The initialSequences enable one instance of each of the referenced sequence.
The session keeps a currently executed step for each of the enabled sequence instances. The step can be blocked
(e.g. waiting for a response to come). The session is looping through current steps in each of the enabled
sequence instances and if the step is not blocked, it is executed. There’s no guaranteed order in which non-blocked
steps from multiple enabled sequence instances will be executed.
While this generic approach is useful for complex scenarios with branching logic, simple sequential scenarios can use orderedSequences short-cut enabling sequences in given order:
This syntax makes the first sequence (login in this case) an initial sequence, adds the subsequent sequences and as the last step of each but the last sequence appends a next step scheduling a new instance of the following sequence.
To make configuration even more concise you can omit the orderedSequences level and start defining the list of sequences under scenario right away:
An exhaustive list of steps can be found in the steps reference.
3.2.5 - Variables
Data placeholders within sessions that hold values throughout the execution of a benchmark scenario
All but the simplest scenarios will use session variables. Hyperfoil sports steps that generate values into these variables (randomInt, randomItem, …), processors that write data from other sources to variables (store, array) and many places that read variables and use the values to perform some operations (httpRequest.path ) or alter control flow.
Hyperfoil uses different types of variables (slots in the session) for integer variables and generic objects (commonly strings). When a numeric value is received as a string (e.g. when parsing response headers) and you want to use it in a step that expects exclusively integral values you have to convert it explicitly, e.g. using the stringToInt action. Steps that read values to form a string can usually consume both types of variables, without any need for conversion.
Besides user-defined variables there are some read-only pseudo-variables that can be used in the scenario as if these were regular variables:
Number of agent nodes or 1 when running in in-VM mode (standalone or CLI)
hyperfoil.agent.thread.id
integer
Zero-based index of current executor thread within this agent.
hyperfoil.agent.thread s
integer
Number of executor threads running in this agent.
hyperfoil.global.thread.id
integer
Zero-based index of current executor thread across all agents (unique).
hyperfoil.global.threads
integer
Total number of executor threads on all agents.
hyperfoil.phase.name
object
Full name of the currently executed phase (possibly including fork and iteration number).
hyperfoil.phase.id
integer
Index of the currently executed phase.
hyperfoil.phase.iteration
integer
Iteration number of the currently executed phase.
hyperfoil.run.id
object
Identifier of the current run, e.g. 0123.
hyperfoil.session.id
integer
Unique index of this virtual user (session). Note that in benchmarks with multiple phases the indices might not be zero-based.
String interpolation
Components that accept string values usually allow you to use a pattern - parts of the string can be replaced in runtime with the value from a session variable. A simple example of pattern would be The quick brown ${wild-animal} jumps over the lazy ${domestic-animal} - variables wild-animal and domestic-animal would get replaced with their respective values.
When you really want to use ${wild-animal} in a value for such component you should escape it with one more dollar sign: This $${variable} won't be replaced will be rendered into This ${variable} won't be replaced.
There are a few transformations that you can perform with a variable value while interpolating the pattern:
${urlencode:my-variable} will replace characters in the my-variable using URLEncoder.encode (using UTF-8 encoding).
${{ '{%05d' }}:my-number} and other formatter strings ending with d, o, x or X will convert an integer variable using Formatter.
${replace/<regexp>/<replacement>/<flags>:my-variable} perform Java regexp replacement on my-variable contents. Note that you can use any character after replace, not just / - this becomes the separator between regexp, replacement and flags. The only flags currently supported is g - replacing all occurences of that string (by default only first occurence is replaced).
Sequence-scoped access
When an array or collection is stored in a session variable you can access the individual elements by appending [.] to the variable name, e.g. my-variable[.]. You can see that we don’t use the actual index into the array: instead we use current sequence instance index. You can read more about running multiple sequences concurrently in the Architecture/Scenario Execution.
3.2.6 - Templates
Templates in Hyperfoil allow for efficient benchmark parametrization, enabling users to customize benchmarks based on specific execution environments or intended loads
It is often useful to keep a single benchmark in version control but change parts of it depending on the infrastructure where it is executed or intended load. Since version 0.18 Hyperfoil supports parametrization of the benchmark through templates.
Inspired by other (more complex) YAML templating systems we decided to use YAML tags to pre-process the YAML. Templating happens even before applying the YAML nodes onto BenchmarkBuilder, therefore it is not possible to do that programmatically or with the serialized form.
If you are working with CLI or WebCLI there is little difference to regular benchmarks: you upload and edit the benchmark as usual. However it is not possible to auto-detect files before the benchmark is constructed from the template (the reference to a file could be a template, too!), therefore you need to pass all files using option -f to the upload/edit command.
When the benchmark template is uploaded, upon running it (run mybenchmark) you either pass the parameters using option -P or you are interactively asked to provide those params. The parameters are stored in CLI context and on subsequent invocations of run you don’t need to set these. If you want to remove the parameters from the context use option -r/--reset-params. To see both default and current parameters you can use the inspect command.
Param
You should use !param to replace single scalar value:
In this simple constant-rate benchmark you can customize the number of users starting each second as well as the duration. There’s no default for NUM_USERS; you will be asked to provide it when you run the benchmark. On the other hand DURATION has a default value of 60s - anything after the space after parameter name counts as the default value.
Parameters don’t have to be upper-case. The identifier is case-sensitive, though.
run scalar-value-example -PNUM_USERS=5 -PDURATION=60s
Concat
Sometimes you need to replace only part of a string: !concat will let you do that:
1name:example2http:3host:!concat [ "http://", !param SERVER localhost, ":8080" ]4usersPerSec:105duration:60s6scenario:# ...
In this example we will customize the host with the concatenation of http://, parameter SERVER with default localhost and :8080. This example uses inline-form of list, though you can use the regular list (one item per line), too.
Foreach
Chances are you need to generate a list based on a param: you can do this using the !foreach:
1name:example 2http:!foreach 3items:http://example.com,http://hyperfoil.io 4separator:","# comma is the default separator 5param:ITEM # ITEM is the default parameter name 6do: 7host:!param ITEM 8usersPerSec:10 9duration:60s10scenario:# ...
This splits the items using the separator regexp and produces a list of values or mappings while the param ITEM is set to one of the values from items list. The example above would result in:
Renaming the param used for iteration can be useful in nested loops: without renaming the inner foreach would shadow the outer one.
Anchors and aliases
YAML has a built-in concept for removing repetitive sections: anchors and aliases. With the templating system you can use that universally throughout the file (in versions before 0.18 the support was limited to forks, scenarios and sequences):
foo:&hello-worldhello:worldanotherFoo:sayHi:*hello-worldmyList:- *hello-world- bar
is interpretted as
foo:
hello: world
anotherFoo:
sayHi:
hello: world
myList:
- hello: world
- bar
3.2.7 - Hooks
Mechanisms that allow users to run specific scripts or commands automatically before and after executing a benchmark run
It might be useful to run certain scripts before and after the run, e.g. starting some infrastructure, preloading database, gathering CPU stats during the test and so on. That’s why Hyperfoil introduces pre- and post-hooks to the run.
Some scripts are not specific to the test being run - these should be deployed on controller as files in *root*/hooks/pre/ and *root*/hooks/post directories where root is controller’s root directory, /tmp/hyperfoil/ by default. Each of these directories should contain executable scripts or binaries that will be run in alphabetic order. We strongly suggest using the format 00-my-script.sh to set the order using first two digits.
Kubernetes/Openshift deployments use the same strategy; the only difference is that the pre and post directories are mapped as volumes from a ConfigMap resource.
Other scripts may be specific to the benchmark executed and therefore you can define them directly in the YAML files. You can either use inline command that will be executed using sh -c your-command --your-options or create a Java class implementing io.hyperfoil.core.hooks.RunHook and register it to be loaded as other Hyperfoil extensions.
The lists of hooks from controller directories and benchmark are merged; if there’s a conflict between two hooks from these two sources the final execution order is not defined (but both get executed).
In case of inline command execution the stderr output will stay on stderr, stdout will be caputered by Hyperfoil and stored in *rundir*/*XXXX*/hooks.json. As the post-hooks are executed after info.json and all.json get written the output cannot be included inside those files. This order of execution was chosen because it’s likely that you will upload these files to a database - yes, using a post-hook.
3.2.8 - Ergonomics
Configuration options that enhance usability and automation of benchmarking sessions
This section hosts only single property at this moment:
Property
Default
Description
repeatCookies
true
Automatically parse cookies from HTTP responses, store them in session and resend them with subsequent requests.
userAgentFromSession
true
Add user-agent header to each request, holding the agent name and session id.
autoRangeCheck
true
Mark 4xx and 5xx responses as invalid. You can also turn this off in each step.
stopOnInvalid
true
When the session receives an invalid response it does not execute any further steps, cancelling all requests and stopping immediately.
If you haven’t checked the Getting started guide we strongly recommend going there first.
Below you’ll see commented examples of configuration; contrary to the Getting started guide these don’t present scenarios but rather list the various configuration options by example.
httpRequest
You will most likely use step httpRequest in each of your scenarios, and there’s many ways to send a request.
# This example should demonstrate various ways to configure one of the most important# steps - the httpRequest.name:http-requestshttp:host:http://example.comergonomics:# Disable stopping the scenario on 4xx or 5xx responseautoRangeCheck:falseusersPerSec:1duration:1scenario:- jsonBody:- httpRequest:POST:/foo/bar# Hyperfoil doesn't know what's the content of the body string, if the server# requires correct content-type header you have to provide it yourselvesheaders:content-type:application/json# Here we specify a multi-line string. For more info about multiline strings,# compacting/chopping of newlines etc. please check out https://yaml-multiline.info/body:| {
"foo" : "bar"
}- formBody:- set:myVar <- foobar- httpRequest:POST:/myform# Here we don't need to add any headers as the form: knows that you're sending# a HTML form and it can add 'content-type: application/x-www-form-urlencoded'# automatically.body:# This will generate body 'foo=bar&bar=foobar&goo=foofoobarbar'. Any non-ascii# or otherwise illegal characters are correctly URL-encoded.form:- name:foovalue:bar- name:barfromVar:myVar- name:goopattern:foo${myVar}bar- bodyFromFile:- httpRequest:POST:/foo/barbody:# This simply loads the file and sends it as the body without any conversion.# It does not add any headers nor make it a multipart upload as the browser would do.fromFile:usernames.txt- customHeaders:- set:token <- dGhpcyBpcyBhIG5pY2UgYW5kIHNlY3VyZSB0b2tlbgo=- set:etag <- "ETag received in some previous request"- httpRequest:GET:/secured/page# Note that HTTP headers are case-insensitive (use your preferred capitalization)headers:# Headers can be set inlineaccept:text/html# Session variables are replaced using the pattern syntaxauthorization:Bearer ${token}# Values from session variables can be also loaded using fromVarif-match:fromVar:etag- nonDefaultMetric:- httpRequest:GET:/cats# By default the metric name equals to name of the sequence ('nonDefaultMetric' here).# We can override that either with a constant value...metric:mammals- randomItem:toVar:animallist:- cats- dogs- locusts- httpRequest:GET:/foo/${animal}# ... or a regexp switch on the actual authority+path (e.g. example.com:8080/foo/cats).# If the benchmark uses single (default) HTTP target the authority is omitted.metric:- .*cats -> mammals- .*dogs -> mammals- -> insects- toughSLAs:- httpRequest:GET:/index.htmlhandler:status:# Any request that is not responded with status code will be marked as invalid.range:200# When you need only one SLA you can use mapping without the list (just forget the dash).sla:# This first SLA is evaluated when the phase completes from all requests that happened# during the phase.# Errors are connection failures, timeouts, 4xx and 5xx responses- errorRatio:0.1# You can set custom criteria for what is considered valid/invalid as with the status# handler above. By default any response with status that is not within 200-399# is deemed invalid (as well as error).invalidRatio:0.2blockedRatio:0.0meanResponseTime:10ms# 90% requests should be under 100 ms, only 1% can be over 1 secondlimits:0.9:100ms0.99:1s# Following SLA is evaluated when all the statistics for the past second arrive,# accumulating results from the window (last 10 seconds). Therefore it can detect shorter# peaks of degraded performance.- window:10smeanResponseTime:50ms
Some scenarios need to access multiple HTTP endpoints; following example shows an example configuration for that:
# This example manifests running a benchmark against multiple domainsname:more-servershttp:- host:http://example.comsharedConnections:100- host:https://hyperfoil.io# With HTTPS, most modern servers will negotiate HTTP 2.0 as the application protocol.# Since HTTP 2.0 uses multiple streams over single TCP (TLS in this case) connection# you can usually set lower number of connections.sharedConnections:10# You may want to route requests through a proxy/load balancer or simply use a domain# that is not resolvable. Configuration below will actually send the requests to addresses# set below, but the requests will use in the 'host: foobar.com' in the headers# and in SNI if this goes over TLS (HTTPS).- host:http://foobar.com# Hyperfoil will split the connections evenly to the defined addresses# (an entry is considered single address for this purpose even if DNS registers# multiple IP addresses for the hostname).sharedConnections:30addresses:# Both hostnames and IP addresses are allowed- proxy.my-locally-defined-domain.test- 192.168.1.10# You can set a custom port as well- 192.168.1.11:8080usersPerSec:1duration:1scenario:- test:- httpRequest:# Authority is the combination of hostname and port.authority:hyperfoil.io:443GET:/docs- randomItem:toVar:hostnamelist:- example.com- foobar.com- httpRequest:# The target must be configured in the 'http' section above; the correctness# is usually validated when parsing/building the benchmark but sometimes it is# only possible at runtime, potentially resulting in errors during execution.authority:${hostname}:80GET:/foo
3.4 - Troubleshooting
Common technical issues that you could hit during benchmark development
It doesn’t work. Can you help me?
The first step to identifying any issue is getting a verbose log - setting logging level to TRACE. How exactly you do that depends on the way you deploy Hyperfoil:
If you use CLI and the start-local command, just run it as start-local -l TRACE which sets the default logging level. You’ll find the log in /tmp/hyperfoil/hyperfoil.local.log by default.
If you run Hyperfoil manually in standalone mode (non-clustered) the agent will run in the same JVM as the controller. You need to add -Dlog4j.configurationFile=file:///path/to/log4j2-trace.xml option when starting standalone.sh. If you start Hyperfoil through Ansible the same is set using hyperfoil_log_config variable.
If you run Hyperfoil in clustered mode, the failing code is probably in the agents. You need to pass the logging settings to agents using the deployer; with SSH deployer you need to add -Dlog4j.configurationFile=file:///path/to/log4j2-trace.xml to the extras property, in Kubernetes/Openshift there is the log option that lets you set the logging configuration through a config-map.
An example of Log4j2 configuration file with TRACE logging on is here:
TRACE-level logging can be very verbose to a point where it will pose a bottleneck. It’s recommended to isolate your problem at lower request rate if that’s possible.
If you need to print variable values for debugging, check out log step.
My phase fails with SLA failure ‘Progress was blocked waiting for a free connection. Hint: increase http.sharedConnections.’
By default Hyperfoil uses single connection to each HTTP(s) host; the default is set so low to force you thinking about connection limits early during test development. If you don’t override this value as in:
you get the error above, as the default SLA does not allow a session (virtual user) to be blocked due to not being able to acquire a connection from the connection pool immediatelly. If you can’t increase number of connections (or use HTTP2 that allows multiple requests to multiplex within single connection), you can set
- httpRequest:
sla:
- blockedRatio: 1000 # any value big enough
on each request to drop the default SLA. The blockedRatio value is a threshold ratio between time spent waiting for a free connection and waiting for the response.
You could also wonder why the sessions are missing a connection when the scenario should guarantee there’s always a free connection e.g. when using always phase type with same number of users and connections. However this may not hold when the connection is closed (either explicitly or after receiving a 5xx response) - while Hyperfoil starts replacing that connection immediatelly it takes a moment. If you expects connections to be closed add a few (10%) extra connections. Another reason could be poor balancing of connections and sessions to threads (should be gone in version 0.8).
When I set ‘Host’ header for HTTP request I get warnings
Hyperfoil automatically inserts the ‘Host’ header to each request and when you try to override that for certain request it emits a warning:
Setting `host` header explicitly is not recommended. Use the HTTP host and adjust actual target using `addresses` property.
With this warning on we don’t inject the header as it might be intended, e.g. when the target server does not parse headers in a case-sensitive way (as it should!) and you have to use certain case. However, if you want to run your requests to a different IP than the host resolves to (e.g. hit 127.0.0.1:8080 with Host: example.com) you should rather use
When I use a session variable I am seeing the error “Variable foo is not set!”
Errors:
in-vm: Variable foo is not set!
On occasion a scenario step has been seen to execute out of sequence. To ensure the variable is set beforehand use initialSequences with the step that populates the variable.
My benchmark hangs indefinitely
If you experience your benchmark hanging indefinitely,
This is usually reflected by having some phases marked as FINISHED and never terminated, i.ei, the REMAINING column
keeps decreasing over time.
This is most likely caused by a wrong assumption made as part of the benchmark configuration.
The indefinite hanging is usually caused by the awaitVar step that indefinitely waits for some variable to be set
even if that will never happen because of how the benchmark is configured.
How do I know whether this is the problem I am facing?
First of all just check whether the benchmark uses awaitVar and if so, double check whether there might be cases
where that variable the step is waiting for could be NOT set.
Can you give me an example?
Consider a simple use case where you are calling an endpoint and save the result into a variable:
This is a very simple example where we made the wrong assumption/configuration: we are waiting for a variable
token that is going to be set by the http request handler, but what happen if the request fails? Given that we disabled
the autoRangeCheck, that request won’t be marked as invalid and the benchmark will proceed executing the next step
anyway (the awaitVar) that will wait for a state that will never happen.
Moreover, in this specific case we don’t even need the awaitVar as by default the httpRequest steps are synchronouds by
default - you change this behavior by setting sync: false at httpRequest level. See httpRequest configuration
for more details.
In conclusion, a couple of considerations:
Use awaitVar only if dealing with async processes, by default this is not necessary as requests are sync
Change the default ergonomics, e.g., autoRangeCheck or stopOnInvalid, with caution as they can heavily impact the overall behavior and
your expectations
4 - How To
A collection of practical advices for common things in Hyperfoil
This section contains practical advices for common things you could want to use in a benchmark.
4.1 - JSON schema support
How to make your life easier while building your own benchmarks
For your convenience we recommend using editor with YAML validation against JSON schema; you can point your editor to schema.json. We can recommend Visual Studio Code
with redhat.vscode-yaml plugin.
You need to edit settings file to map benchmark configuration files (with .hf.yaml extension) to the schema, adding:
Hyperfoil benchmarks can refer to external files. When you use the upload command in CLI the files are automatically attached the the benchmark YAML (relative paths are resolved relative to this file). Later on when editing the file you can choose to re-upload some of these. Once the benchmark is built the files are loaded to the in-memory representation - Hyperfoil won’t access these files during runtime. With clustered benchmarks these files don’t need to be on the agents either - the controller sends serialized in-memory representation to the agents and that contains everything needed for the actual execution.
When testing a workload you will likely skip user registration and come with a list of username+password keys. A convenient way is to keep these in a CSV file that looks like
We will create a step that selects a random line from such file and stores it in session variables username and password:
- randomCsvRow:file:credentials.csv# Path relative to the benchmarkcolumns:0:username1:password
The columns property has a mapping of zero-based indices of columns in the CSV file. This way you can use a file with some extra information.
In case you don’t need to split lines into separate variables you can use the randomItem:
- randomItem:file:usernames.txttoVar:username
We have selected the credentials, now it’s time to execute the login. When users submit a form on HTML page this usually results in a POST request with content-type: application/x-www-form-urlencoded header and the form contents encoded in the request body. Hyperfoil lets you specify this conveniently using the form body formatter:
Successful response from the server carries a token in most cases but the actual method can vary. If the server sets a cookie Hyperfoil automatically records this and sends it on subsequent requests (this can be switched off using ergonomics.repeatCookies). If the server sends the token in a response header, e.g. x-token you can store it using the header handler into the token session variable:
If the server returns this token as a part of JSON object - e.g { "token": "abc123" } you would process response body using the json processor in body handler:
Below you can find a runnable example that we have prepared for you and that you can download here.
name:credentialshttp:- host:https://localhost:8084phases:- use-cookie:atOnce:users:1scenario:# Make sure that without the cookie (before login) the request fails with 401- before-login:- httpRequest:GET:/howto/credentials/securehandler:autoRangeCheck:false# Don't fail with 4xx-5xxstatus:range:401- login-with-form:- randomCsvRow:file:credentials.csvcolumns:0:username1:password- httpRequest:POST:/howto/credentials/loginbody:form:- name:usernamefromVar:username- name:passwordfromVar:password# Here we already have the cookie set to a httpRequest will succeed- after-login:- httpRequest:GET:/howto/credentials/secure- use-bearer-token:atOnce:users:1scenario:- request-login-page:# This first request will response with WWW-Authenticate header- httpRequest:GET:/howto/credentials/loginhandler:autoRangeCheck:falsestatus:range:401- login-with-basic-auth:- randomCsvRow:file:credentials.csvcolumns:0:username1:password- template:pattern:${username}:${password}toVar:concatenated- httpRequest:GET:/howto/credentials/loginheaders:# Our example runs over HTTP2 and that mandates lower-case header namesauthorization:Basic ${base64encode:concatenated}# With Authorization header the server will reply with token# and redirect us to the secured page. `httpRequest` implements# automatic redirection through `handler.followRedirect` but that# wouldn't send the token, so we'll do that manuallyhandler:header:- filter:header:x-tokenprocessor:- store:tokenstatus:range:30x- after-login:- httpRequest:GET:/howto/credentials/secureheaders:authorization:Bearer ${token}
There are two scenarios: use-cookie performs the login using the POST from form as shown above and then keeps the secret token in a cookie; use-bearer-token performs HTTP Basic authentication, receives a token in headers and then uses that for HTTP Bearer authentication.
podman run --rm -p 8084:8084 quay.io/hyperfoil/hyperfoil-examples
In second console start the CLI with in-vm controller (or standalone and open the WebCLI in your browser). We are running in host-network mode to be able to reach localhost from within the container.
podman run --rm -it --net host quay.io/hyperfoil/hyperfoil cli
[hyperfoil]$ start-local
Starting controller in default directory (/tmp/hyperfoil)Controller started, listening on 127.0.0.1:35243
Connecting to the controller...
Connected to 127.0.0.1:35243!
[hyperfoil@in-vm]$ upload https://hyperfoil.io/benchmarks/credentials.hf.yaml
Loaded benchmark credentials, uploading...
... done.
[hyperfoil@in-vm]$ run
Started run 0000Run 0000, benchmark credentials
Agents: in-vm[STOPPED]Started: 2022/10/27 15:48:39.271 Terminated: 2022/10/27 15:48:39.301
NAME STATUS STARTED REMAINING COMPLETED TOTAL DURATION DESCRIPTION</span>
use-bearer-token TERMINATED 15:48:39.271 15:48:39.301 30 ms (exceeded by 31 ms)1 users at once
use-cookie TERMINATED 15:48:39.271 15:48:39.300 29 ms (exceeded by 30 ms)1 users at once
[hyperfoil@in-vm]$ stats
Total stats from run 0000PHASE METRIC THROUGHPUT REQUESTS MEAN p50 p90 p99 p99.9 p99.99 TIMEOUTS ERRORS BLOCKED 2xx 3xx 4xx 5xx CACHE</span>
use-bearer-token after-login 33.33 req/s 1 3.79 ms 3.80 ms 3.80 ms 3.80 ms 3.80 ms 3.80 ms 000 ns 10000use-bearer-token login-with-basic-auth 33.33 req/s 1 4.57 ms 4.59 ms 4.59 ms 4.59 ms 4.59 ms 4.59 ms 000 ns 01000use-bearer-token request-login-page 33.33 req/s 1 7.49 ms 7.50 ms 7.50 ms 7.50 ms 7.50 ms 7.50 ms 000 ns 00100use-cookie after-login 34.48 req/s 1 3.69 ms 3.70 ms 3.70 ms 3.70 ms 3.70 ms 3.70 ms 000 ns 10000use-cookie before-login 34.48 req/s 1 9.60 ms 9.63 ms 9.63 ms 9.63 ms 9.63 ms 9.63 ms 000 ns 00100use-cookie login-with-form 34.48 req/s 1 4.96 ms 4.98 ms 4.98 ms 4.98 ms 4.98 ms 4.98 ms 000 ns 10000
The list of possibilities is endless; if your use case does not fit any of the above please check out the reference. You can also have a look at a full example in the second part of the Beginner’s Guide.
4.3 - Hyperfoil run script
How to quickly run Hyperfoil benchmarks
Starting from release 0.27, Hyperfoil includes an easy-to-use script that simplifies running benchmarks, allowing users to try tests faster when in-vm controller server is acceptable.
This script is particularly beneficial when you need to quickly test, validate or refine your benchmark
definitions, ensuring they run as expected without needing to manually orchestrate the controller and
agent processes. It also enables seamless integration into automation scripts or CI/CD pipelines, where
you can configure benchmarks to run as part of routine testing, with results saved for further analysis.
By simplifying the benchmark execution process, this script accelerates your workflow and allows for
more streamlined performance testing with Hyperfoil.
Key features
In-vm controller: The script launches an in-VM Hyperfoil controller, so there’s no need for users to
set up or manage an external controller.
Benchmark upload & execution: Once the controller is running, the script automatically uploads the
benchmark you provide and triggers its execution. This minimizes manual setup, allowing users to
focus on their test scenarios.
No CLI interactions: Running the script does not require any CLI interaction, making this scipt
suitable for further automation.
Automatic report generation: By adding the --output <path-to-dir> option, the script will generate
and save an HTML report of the test results in the specified directory, making it easy to review
performance data immediately after the benchmark completes.
Usage
The syntax of this script is basically a superset of the run command, where the main argument is not the name of the benchmark but the benchmark file itself.
Usage: run [<options>] <benchmark>
Load and start a benchmark on Hyperfoil controller server, the argument can be the benchmark definition directly.
Options:
-o, --output Output destination path for the HTML report
--print-stack-trace
-d, --description Run description
-P, --param Parameters in case the benchmark is a template. Can be set multiple times. Use `-PFOO=` to set the parameter to empty value and `-PFOO` to remove it and use default if available.
-E, --empty-params Template parameters that should be set to empty string.
-r, --reset-params Reset all parameters in context.
Argument:
Benchmark filename.
From the unzipped Hyperfoil distribution, you can simply run the script using the following format:
$ ./distribution/target/distribution/bin/run.sh -o /tmp/reports /tmp/first-benchmark.yml
Loaded benchmark first-benchmark, uploading...
... done.
Started run 0021Monitoring run 0021, benchmark first-benchmark
Started: 2024/09/30 19:19:38.689
Terminated: 2024/09/30 19:19:49.532
Report written to /tmp/reports/0021.html
Alternatively you could also run the same directly using the Hyperfoil docker image:
docker run -it -v /tmp/benchmark/:/benchmarks:Z -v /tmp/reports:/tmp/reports:rw,Z -it --network=host quay.io/hyperfoil/hyperfoil run -o /tmp/reports /benchmarks/first-benchmark.yml
and the output will be the same:
$ docker run -it -v /tmp/benchmarks/:/benchmarks:Z -v /tmp/reports:/tmp/reports:rw,Z -it --network=host quay.io/hyperfoil/hyperfoil run -o /tmp/reports /benchmarks/first-benchmark.yml
Loaded benchmark first-benchmark, uploading...
... done.
Started run 0000Monitoring run 0000, benchmark first-benchmark
Started: 2024/09/30 17:21:22.484
Terminated: 2024/09/30 17:21:32.490
Report written to /tmp/reports/0000.html
5 - Migration
Set of guides to migrate from other tools to Hyperfoil
This section includes some guides to make the migration from other benchmarking tools easier.
5.1 - Migrating from wrk/wrk2
How and why should I migrate from wrk/wrk2 to Hyperfoil?
Both Will Glozer’s wrk and Gil Tene’s wrk2 are great tools but maybe you’ve realized that you need more functionalities, need to hit more endpoints in parallel or simply have to scale horizontally to more nodes. Hyperfoil offers an adapter that tries to mimic the behaviour of these load drivers. This guide will show how to use these and translate the test into a full-fledged Hyperfoil benchmark.
Warning
Please make sure that you are using Hyperfoil version >= 0.22
Running 10s test @ http://example.com/
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 98.61ms 9.58ms 402.65ms 98.92%
Req/Sec 101.70 3.35 110.00 80.00
1017 requests in 10.023s, 1.59MB readRequests/sec: 101.47
Transfer/sec: 162.84kB
Not much of a difference as you can see. Note that if you want to test something in localhost you’d need to use host networking (--net host). You could also run it from CLI using the wrk/wrk2 command. In that case you’d either connect to a controller using the connect command or start a controller inside the CLI with start-local command. When the (remote) controller is clustered you can use -A/--agent option (e.g. -Amy-agent=host=my-server.my-company.com,port=22) to drive the load from a different node.
Unlike original tools we support HTTP2 - this is disabled by default and you need to pass --enable-http2 to allow it explicitly.
When you run the command from the CLI the benchmark stays in controller, so you can have a look on the statistics the same way as with any other Hyperfoil run (stats, report, compare, …). You can also execute another run of the benchmark and observe the status as it progresses:
podman run -it --rm quay.io/hyperfoil/hyperfoil cli
[hyperfoil]$ start-local --quiet
[hyperfoil@in-vm]$ wrk -t 2 -c 10 -H 'accept: application/json' -d 10s http://example.com
Running 10s test @ http://example.com
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 97.99ms 618.36?s 101.19ms 62.63%
Req/Sec 102.50 2.94 110.00 90.00
1025 requests in 10.006s, 1.61MB readRequests/sec: 102.44
Transfer/sec: 164.41kB
[hyperfoil@in-vm]$ stats
Total stats from run 0000Extensions (use -e to show): transfer
PHASE METRIC THROUGHPUT REQUESTS MEAN p50 p90 p99 p99.9 p99.99 TIMEOUTS ERRORS BLOCKED 2xx 3xx 4xx 5xx CACHE
calibration request 96.58 req/s 608 99.39 ms 98.57 ms 100.14 ms 117.96 ms 406.85 ms 406.85 ms 000 ns 6080000test request 101.44 req/s 1025 97.99 ms 98.04 ms 99.09 ms 100.14 ms 101.19 ms 101.19 ms 000 ns 10150000[hyperfoil@in-vm]$ run wrk
Started run 0001Run 0001, benchmark wrk
Agents: in-vm[READY]Started: 2022/09/08 17:59:05.283
NAME STATUS STARTED REMAINING COMPLETED TOTAL DURATION DESCRIPTION
test RUNNING 17:59:11.372 88 ms (88 ms)10 users always
Under the hood the command creates a benchmark that should reflect what the original wrk/wrk2 implementation does. These do not have a YAML representation so you cannot view it with info/edit but you can use inspect to dig deep into the programmatic representation. The source YAML for the wrk invocation above would look like this:
name:wrkthreads:2# option -thttp:host:http://example.comallowHttp2:falsesharedConnections:10# option -cergonomics:repeatCookies:falseuserAgentFromSession:falsephases:- calibration:always:users:10# option -cduration:6smaxDuration:70s# This is duration + default timeout 60sscenario:&scenario- request:- httpRequest:GET:/timeout:60s# option --timeoutheaders:- accept:application/json# option -Hhandler:rawBytes:...# Handler to record the data for 'Transfer/sec'- test:always:users:10# option -cduration:10s# option -dmaxDuration:10s# option -dstartAfterStrict:calibrationscenario:*scenario
As you can see there’s a ‘calibration’ phase - wrk implementations might use it for different purposes but in our case it’s convenient at least for a basic JVM warmup. The ’test’ phase starts only when all requests from calibration phase complete. The scenario and number of users is identical, though.
With wrk2 we use open model, you have to set request rate using the -R option - with -R 20; the phases would look like:
phases:- calibration:constantRate:usersPerSec:20variance:falsemaxSessions:300# request rate x 15duration:10s# option -d# ...
Hopefully this gives you some headstart and you can get familiar with Hyperfoil before diving into the details of benchmark syntax.
6 - Reference
List of all possible steps and handlers that can be used for the benchmark development
Before exploring this reference you should be familiar with the basic structure of a benchmark. If you’re not sure what is the difference between the phase, scenario and sequence check out the concepts in user guide.
This reference lists all the steps and handlers used in a scenario. The lists below are not finite; you can also easily develop and use your own components, but Hyperfoil provides generic components for the most common tasks out of the box.
This documentation is auto-generated from Javadoc in source code, explaining format for each key-value pair in benchmark YAML. If there is an issue with these docs (e.g. property showing no description) please file an issue on GitHub.
This is the basic structure of the docs:
EXAMPLE
Example description.
Property
Type
Description
Key
Class
Explanation for the value
YAML syntax
EXAMPLE:
Key: Value
For example, the POST definition in httpRequest step looks like this:
POST
Generic builder for generating a string.
Property
Type
Description
fromVar
String
Load the string from session variable.
pattern
String
Use pattern replacing session variables.
value
String
String value used verbatim.
YAML syntax
POST:
pattern: /user/${userId}/info
You might be wondering why the documentation above does not mention anything about issuing a HTTP request. In fact the top-level POST property httpRequest says “Issue HTTP POST request to given path.” but the POST() method returns a generic string builder; this generic builder is used as the path for the HTTP request with POST method.
If the ’type’ is not a scalar value, the key in ‘property’ works as a link to further property mapping. It’s also possible that the property has multiple options, e.g. accepting both property mapping and list of values.
For brevity some components have inline definition like this:
Steps are the basic building blocks that form each sequence of a scenario, similar to statements in a programming language. Steps are potentially blocking (the sequence cannot continue with next step until previous one finishes).
Note that every action can be also used as a step that simply never blocks, as actions do not require any extra input.
6.1.1 - addItem
Appends value to an array stored in another variable.
Appends value to an array stored in another variable.
Property
Type
Description
fromVar
String
Fetch value from session variable.
toVar
String
Destination variable with the array.
value
String
Verbatim value.
6.1.2 - addToInt
Add value to integer variable in the session.
Add value to integer variable in the session.
Inline definition
Accepting one of: var++, var–, var += value,
var -= value.
Property
Type
Description
orElseSetTo
int
If the variable is currently not set, set it to this value instead of addition.
value
int
Value added (can be negative).
var
String
Variable name.
6.1.3 - addToSharedCounter
Adds value to a counter shared by all sessions in the same executor.
Adds value to a counter shared by all sessions in the same executor.
Inline definition
Use on of: counter++, counter–, counter += <value>,
counter -= <value>
Property
Type
Description
fromVar
String
Input variable name.
key
String
Identifier for the counter.
operator
enum
Operation to perform on the counter. Default is ADD. Options:
ADD
SUBTRACT
value
int
Value (integer).
6.1.4 - awaitAllResponses
Block current sequence until all requests receive the response.
Block current sequence until all requests receive the response.
6.1.5 - awaitDelay
Block this sequence until referenced delay point.
Block this sequence until referenced delay point.
Inline definition
Delay point created in scheduleDelay.key.
Property
Type
Description
key
String
Delay point created in scheduleDelay.key.
6.1.6 - awaitInt
Block current sequence until condition becomes true.
Block current sequence until condition becomes true.
Retrieves value from a counter shared by all sessions in the same executor and stores that in a session variable.
Retrieves value from a counter shared by all sessions in the same executor and stores that in a session variable. If the value exceeds allowed integer range (-2^31 .. 2^31 - 1) it is capped.
Inline definition
Both the key and variable name.
Property
Type
Description
key
String
Identifier for the counter.
toVar
String
Session variable for storing the value.
6.1.17 - getSize
Calculates size of an array/collection held in variable into another variable
Calculates size of an array/collection held in variable into another variable
Allows categorizing request statistics into metrics based on the request path.
Property
Type
Description
<list of strings>
<list of strings>
Allows categorizing request statistics into metrics based on the request path. The expressions are evaluated in the order as provided in the list. Use one of:
regexp -> replacement, e.g. ([^?])(?.)? -> $1 to drop the query part.
regexp (don’t do any replaces and use the full path), e.g. .*.jpg
-> name (metric applied if none of the previous expressions match).
This request is synchronous; execution of the sequence does not continue until the full response is received. If this step is executed from multiple parallel instances of this sequence the progress of all sequences is blocked until there is a request in flight without response.
Default is true.
timeout
String
Request timeout - after this time the request will be marked as failed and connection will be closed.
Defaults to value set globally in http section.
TRACE
String
Issue HTTP TRACE request to given path. This can be a pattern.
Build form as if we were sending the request using HTML form. This option automatically adds Content-Type: application/x-www-form-urlencoded to the request headers.
fromFile
String
Send contents of the file. Note that this method does NOT set content-type automatically.
fromVar
String
Use variable content as request body.
pattern
String
Pattern replacing ${sessionvar} with variable contents in a string.
Configure a custom metric for the compensated results.
targetRate
double
Desired rate of new virtual users per second. This is similar to constantRate.usersPerSec phase settings but works closer to legacy benchmark drivers by fixing the concurrency.
Desired rate of new virtual users per second. This is similar to constantRate.usersPerSec phase settings but works closer to legacy benchmark drivers by fixing the concurrency.
compensation.metric
Configure a custom metric for the compensated results.
Property
Type
Description
<list of strings>
<list of strings>
Allows categorizing request statistics into metrics based on the request path. The expressions are evaluated in the order as provided in the list. Use one of:
regexp -> replacement, e.g. ([^?])(?.)? -> $1 to drop the query part.
regexp (don’t do any replaces and use the full path), e.g. .*.jpg
-> name (metric applied if none of the previous expressions match).
compensation.targetRate
Property
Type
Description
base
double
Base value used for first iteration.
increment
double
Value by which the base value is incremented for each (but the very first) iteration.
compression
Property
Type
Description
encoding
String
Encoding used for Accept-Encoding/TE header. The only currently supported is gzip.
type
enum
Type of compression (resource vs. transfer based). Options:
CONTENT_ENCODINGUse Accept-Encoding in request and expect Content-Encoding in response.
TRANSFER_ENCODINGUse TE in request and expect Transfer-Encoding in response.
Automatically fire requests when the server responds with redirection. Default value depends on ergonomics.followRedirect (see User Guide). Options:
NEVERDo not insert any automatic redirection handling.
LOCATION_ONLYRedirect only upon status 3xx accompanied with a ’location’ header. Status, headers, body and completions handlers are suppressed in this case (only raw-bytes handlers are still running). This is the default option.
HTML_ONLYHandle only HTML response with META refresh header. Status, headers and body handlers are invoked both on the original response and on the response from subsequent requests. Completion handlers are suppressed on this request and invoked after the last response arrives (in case of multiple redirections).
ALWAYSImplement both status 3xx + location and HTML redirects.
Inject completion handler that will stop the session if the request has been marked as invalid. Default value depends on ergonomics.stopOnInvalid (see User Guide).
Passes the headers to nested handler if the condition holds. Note that the condition may be evaluated multiple times and therefore any nested handlers should not change the results of the condition.
countHeaders
CountHeadersHandler.Builder
Stores number of occurences of each header in custom statistics (these can be displayed in CLI using the stats -c command).
Records alternative metric based on values from a header (e.g. when a proxy reports processing time).
handler.header.conditional
Passes the headers to nested handler if the condition holds. Note that the condition may be evaluated multiple times and therefore any nested handlers should not change the results of the condition.
Passes the headers to nested handler if the condition holds. Note that the condition may be evaluated multiple times and therefore any nested handlers should not change the results of the condition.
countHeaders
CountHeadersHandler.Builder
Stores number of occurences of each header in custom statistics (these can be displayed in CLI using the stats -c command).
Perform a sequence of actions if the range matches. Use range as the key and action in the mapping. Possible values of the status should be separated by commas (,). Ranges can be set using low-high (inclusive) (e.g. 200-299), or replacing lower digits with ‘x’ (e.g. 2xx).
handler.status.counter
Counts how many times given status is received.
Property
Type
Description
add
int
Number to be added to the session variable.
expectStatus
int
Expected status (others are ignored). All status codes match by default.
init
int
Initial value for the session variable.
set
int
Do not accumulate (add), just set the variable to this value.
var
String
Variable name.
handler.status.multiplex
Multiplexes the status based on range into different status handlers.
Run another handler if the range matches. Use range as the key and another status handler in the mapping. Possible values of the status should be separated by commas (,). Ranges can be set using low-high (inclusive) (e.g. 200-299), or replacing lower digits with ‘x’ (e.g. 2xx).
handler.status.multiplex.<any>
Run another handler if the range matches. Use range as the key and another status handler in the mapping. Possible values of the status should be separated by commas (,). Ranges can be set using low-high (inclusive) (e.g. 200-299), or replacing lower digits with ‘x’ (e.g. 2xx).
Allows categorizing request statistics into metrics based on the request path.
Property
Type
Description
<list of strings>
<list of strings>
Allows categorizing request statistics into metrics based on the request path. The expressions are evaluated in the order as provided in the list. Use one of:
regexp -> replacement, e.g. ([^?])(?.)? -> $1 to drop the query part.
regexp (don’t do any replaces and use the full path), e.g. .*.jpg
-> name (metric applied if none of the previous expressions match).
Defines a Service Level Agreement (SLA) - conditions that must hold for benchmark to be deemed successful.
Property
Type
Description
blockedRatio
double
Maximum allowed ratio of time spent waiting for usable connection to sum of response latencies and blocked time. Default is 0 - client must not be blocked. Set to 1 if the client can block without limits.
errorRatio
double
Maximum allowed ratio of errors: connection failures or resets, timeouts and internal errors. Valid values are 0.0 - 1.0 (inclusive). Note: 4xx and 5xx statuses are NOT considered errors for this SLA parameter. Use invalidRatio for that.
invalidRatio
double
Maximum allowed ratio of requests with responses marked as invalid. Valid values are 0.0 - 1.0 (inclusive). Note: With default settings 4xx and 5xx statuses are considered invalid. Check out ergonomics.autoRangeCheck or httpRequest.handler.autoRangeCheck to change this.
Custom transformation executed on the value of the selected item. Note that the output value must contain quotes (if applicable) and be correctly escaped.
replace (alternative)
String
Replace value of selected item with value generated through a pattern. Note that the result must contain quotes and be correctly escaped.
toArray
String
Shortcut to store selected parts in an array in the session. Must follow the pattern variable[maxSize]
toVar
String
Shortcut to store first match in given variable. Further matches are ignored.
unquote
boolean
Automatically unquote and unescape the input values. By default true.
replace
Custom transformation executed on the value of the selected item. Note that the output value must contain quotes (if applicable) and be correctly escaped.
This transformer stores the (defragmented) input into a variable, using requested format. After that it executes all the actions and fetches transformed value using the pattern.
This transformer stores the (defragmented) input into a variable, using requested format. After that it executes all the actions and fetches transformed value using the pattern.
This step is supposed to be inserted as the first step of a sequence and the counterVar should not be set during the first invocation.
Before the loop the counterVar is initialized to zero, and in each after the last step in the steps sequence the counter is incremented. If the result is lesser than repeats this sequence is restarted from the beginning (this is also why the step must be the first step in the sequence).
It is legal to place steps after the looped steps.
Unconditionally mark currently processed request as invalid.
Unconditionally mark currently processed request as invalid.
6.1.24 - newSequence
Instantiates a sequence for each invocation.
Instantiates a sequence for each invocation.
Inline definition
Sequence name.
Property
Type
Description
concurrencyPolicy
enum
Options:
FAIL
WARN
forceSameIndex
boolean
Forces that the sequence will have the same index as the currently executing sequence. This can be useful if the sequence is passing some data to the new sequence using sequence-scoped variables. Note that the new sequence must have same concurrency factor as the currently executing sequence.
sequence
String
Name of the instantiated sequence.
6.1.25 - noop
Does nothing. Only for demonstration purposes.
Does nothing. Only for demonstration purposes.
6.1.26 - publishAgentData
Makes the data available to all sessions in the same agent, including those using different executors.
Makes the data available to all sessions in the same agent, including those using different executors.
Inline definition
Both name of source variable and the key used to read the data.
Property
Type
Description
fromVar
String
Source session variable name.
name
String
Arbitrary unique identifier for the data.
6.1.27 - publishGlobalCounters
Gathers values from session variables and publishes them globally (to all agents).
Gathers values from session variables and publishes them globally (to all agents). You can name the counters individually (example 1) or use the variable names (example 2):
<code>
# Example 1:
- publishGlobalCounters:
key: myKey
vars: [ foo, bar ]
# Example 2:
- publishGlobalCounters:
key: someOtherKey
vars:
- foo: myFoo
- bar: bbb
</code>
Move values from a map shared across all sessions using the same executor into session variables.
Move values from a map shared across all sessions using the same executor into session variables.
The executor can host multiple shared maps, each holding an entry with several variables. This step moves variables from either a random entry (if no match is set) or with an entry that has the same value for given variable as the current session. When data is moved to the current session the entry is dropped from the shared map. If the map contains records for which the {@link #vars()} don’t contain a destination variable the contents is lost.
Property
Type
Description
key
String
Key identifying the shared map.
match
String
Name of the session variable that stores value identifying the entry in the shared map.
Store values from session variables into a map shared across all sessions using the same executor into session variables.
Store values from session variables into a map shared across all sessions using the same executor into session variables.
The executor can host multiple shared maps, each holding an entry with several variables. This step creates one entry in the map, copying values from session variables into the entry.
Potentially weighted list of items to choose from.
toVar
String
Variable where the chosen item should be stored.
list
Property
Type
Description
<any>
<list of strings>
Item as the key and weight (arbitrary floating-point number, defaults to 1.0) as the value.
<list of strings>
<list of strings>
Item as the key and weight (arbitrary floating-point number, defaults to 1.0) as the value.
6.1.34 - randomUUID
Stores random string into session variable based on the UUID generator.
Stores random string into session variable based on the UUID generator.
Property
Type
Description
toVar
String
Variable name to store the result.
6.1.35 - readAgentData
Reads data from agent-wide scope into session variable.
Reads data from agent-wide scope into session variable. The data must be published in a phase that has terminated before this phase starts: usually this is achieved using the startAfterStrict property on the phase.
Inline definition
Both the identifier and destination session variable.
Property
Type
Description
name
String
Unique identifier for the data.
toVar
String
Destination session variable name.
6.1.36 - removeItem
Removes element from an array of variables.
Removes element from an array of variables.
Property
Type
Description
fromVar
String
Variable containing an existing array of object variables.
Set index at which the item should be removed. Elements to the right of this are moved to the left.
index
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
6.1.37 - restartSequence
Restarts current sequence from beginning.
Restarts current sequence from beginning.
6.1.38 - scheduleDelay
Define a point in future until which we should wait. Does not cause waiting.
Define a point in future until which we should wait. Does not cause waiting.
Property
Type
Description
duration
String
Duration of the delay with appropriate suffix (e.g. ms or s).
fromLast
<none>
Set previous delay point reference as the reference for next delay point; it will be computed as (previous delay point or now) + duration. Note: property does not have any value
fromNow
<none>
Set this step invocation as the delay point reference; it will be computed as now + duration. Note: property does not have any value
key
String
Key that is referenced later in awaitDelay step. If you’re introducing the delay through thinkTime do not use this property.
max
String
Upper cap on the duration (if randomized).
min
String
Lower cap on the duration (if randomized).
random
enum
Randomize the duration. Options:
CONSTANTDo not randomize; use constant duration.
LINEARUse linearly random duration between min and max (inclusively).
NEGATIVE_EXPONENTIALUse negative-exponential duration with expected value of duration, capped at min and max (inclusively).
type
enum
Alternative way to set delay reference point. See fromNow and fromLast property setters. Options:
Creates integer arrays to be stored in the session.
Property
Type
Description
fromVar
String
Contents of the new array. If the variable contains an array or a list, items will be copied to the elements with the same index up to the size of this array. If the variable contains a different value all elements will be initialized to this value.
size
int
Size of the array.
objectArray
Creates object arrays to be stored in the session.
Property
Type
Description
fromVar
String
Contents of the new array. If the variable contains an array or a list, items will be copied to the elements with the same index up to the size of this array. If the variable contains a different value all elements will be initialized to this value.
Pattern to be encoded, e.g. foo${variable}bar${another-variable}
toVar
String
Variable name to store the result.
6.1.47 - thinkTime
Block current sequence for specified duration.
Block current sequence for specified duration.
Inline definition
Duration of the delay with appropriate suffix (e.g. ms or s).
Property
Type
Description
duration
String
Duration of the delay with appropriate suffix (e.g. ms or s).
fromLast
<none>
Set previous delay point reference as the reference for next delay point; it will be computed as (previous delay point or now) + duration. Note: property does not have any value
fromNow
<none>
Set this step invocation as the delay point reference; it will be computed as now + duration. Note: property does not have any value
key
String
Key that is referenced later in awaitDelay step. If you’re introducing the delay through thinkTime do not use this property.
max
String
Upper cap on the duration (if randomized).
min
String
Lower cap on the duration (if randomized).
random
enum
Randomize the duration. Options:
CONSTANTDo not randomize; use constant duration.
LINEARUse linearly random duration between min and max (inclusively).
NEGATIVE_EXPONENTIALUse negative-exponential duration with expected value of duration, capped at min and max (inclusively).
type
enum
Alternative way to set delay reference point. See fromNow and fromLast property setters. Options:
FROM_LAST
FROM_NOW
6.1.48 - timestamp
Stores the current time in milliseconds as string to a session variable.
Stores the current time in milliseconds as string to a session variable.
Inline definition
Variable name.
Property
Type
Description
localeCountry
String
2-letter ISO country code used in the formatter locale. Defaults to ‘US’.
pattern
String
Format the timestamp using SimpleDateFormat pattern.
toVar
String
Target variable name.
6.1.49 - unset
Undefine variable name.
Undefine variable name.
Inline definition
Variable name.
Property
Type
Description
var
Object
Variable name.
6.2 - Processors
Processors can work either as consumers of input bytes (e.g. storing part of the input into session variables), as filters (passing modified version of the input to delegated processors) or combination of the above.
Some processors can expect extra context in the session, such as an ongoing (HTTP) request. It is possible to use processors that don’t expect specific type of request in places where a more specific type is provided; opposite is not allowed.
Also it is possible to use an action instead of a processor; Hyperfoil automatically inserts an adapter. That’s why the list below includes actions as well.
6.2.1 - addItem
Appends value to an array stored in another variable.
Appends value to an array stored in another variable.
Property
Type
Description
fromVar
String
Fetch value from session variable.
toVar
String
Destination variable with the array.
value
String
Verbatim value.
6.2.2 - addToInt
Add value to integer variable in the session.
Add value to integer variable in the session.
Inline definition
Accepting one of: var++, var–, var += value,
var -= value.
Property
Type
Description
orElseSetTo
int
If the variable is currently not set, set it to this value instead of addition.
value
int
Value added (can be negative).
var
String
Variable name.
6.2.3 - addToSharedCounter
Adds value to a counter shared by all sessions in the same executor.
Adds value to a counter shared by all sessions in the same executor.
Inline definition
Use on of: counter++, counter–, counter += <value>,
counter -= <value>
Property
Type
Description
fromVar
String
Input variable name.
key
String
Identifier for the counter.
operator
enum
Operation to perform on the counter. Default is ADD. Options:
ADD
SUBTRACT
value
int
Value (integer).
6.2.4 - array
Stores data in an array stored as session variable.
Stores data in an array stored as session variable.
Inline definition
Use format toVar[maxSize].
Property
Type
Description
format
enum
Format into which should this processor convert the buffers before storing. Default is STRING. Options:
BYTEBUFStore the buffer directly. Beware that this may cause memory leaks!
BYTESStore data as byte array.
STRINGInterprets the bytes as UTF-8 string.
maxSize
int
Maximum size of the array.
silent
boolean
Do not log warnings when the maximum size is exceeded.
toVar
String
Variable name.
6.2.5 - clearHttpCache
Drops all entries from HTTP cache in the session.
Drops all entries from HTTP cache in the session.
6.2.6 - closeConnection
Prevents reuse connection after the response has been handled.
Prevents reuse connection after the response has been handled.
6.2.7 - collection
Collects results of processor invocation into a unbounded list.
Collects results of processor invocation into a unbounded list. WARNING: This processor should be used rarely as it allocates memory during the benchmark.
Inline definition
Variable name to store the list.
Property
Type
Description
format
enum
Format into which should this processor convert the buffers before storing. Default is STRING. Options:
BYTEBUFStore the buffer directly. Beware that this may cause memory leaks!
BYTESStore data as byte array.
STRINGInterprets the bytes as UTF-8 string.
toVar
String
Variable name.
6.2.8 - conditional
Passes the data to nested processor if the condition holds.
Passes the data to nested processor if the condition holds. Note that the condition may be evaluated multiple times and therefore any nested processors should not change the results of the condition.
Retrieves value from a counter shared by all sessions in the same executor and stores that in a session variable.
Retrieves value from a counter shared by all sessions in the same executor and stores that in a session variable. If the value exceeds allowed integer range (-2^31 .. 2^31 - 1) it is capped.
Inline definition
Both the key and variable name.
Property
Type
Description
key
String
Identifier for the counter.
toVar
String
Session variable for storing the value.
6.2.14 - getSize
Calculates size of an array/collection held in variable into another variable
Calculates size of an array/collection held in variable into another variable
Compared variable must not be equal to this value.
stringFilter.length.equalTo
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
stringFilter.length.greaterOrEqualTo
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
stringFilter.length.greaterThan
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
stringFilter.length.lessOrEqualTo
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
stringFilter.length.lessThan
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
stringFilter.length.notEqualTo
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
6.2.15 - gzipInflator
Decompresses a GZIP data and pipes the output to delegated processors.
Decompresses a GZIP data and pipes the output to delegated processors. If the data contains multiple concatenated GZIP streams it will pipe multiple decompressed objects with isLastPart set to true at the end of each stream.
Property
Type
Description
encodingVar
Object
Variable used to pass header value from header handlers.
If neither delete or replace was set this processor will be called with the selected parts. In the other case the processor will be called with chunks of full (modified) JSON.
Note that the processor.before() and processor.after() methods are called only once for each request, not for the individual filtered items.
Custom transformation executed on the value of the selected item. Note that the output value must contain quotes (if applicable) and be correctly escaped.
replace (alternative)
String
Replace value of selected item with value generated through a pattern. Note that the result must contain quotes and be correctly escaped.
toArray
String
Shortcut to store selected parts in an array in the session. Must follow the pattern variable[maxSize]
toVar
String
Shortcut to store first match in given variable. Further matches are ignored.
unquote
boolean
Automatically unquote and unescape the input values. By default true.
replace
Custom transformation executed on the value of the selected item. Note that the output value must contain quotes (if applicable) and be correctly escaped.
This transformer stores the (defragmented) input into a variable, using requested format. After that it executes all the actions and fetches transformed value using the pattern.
This transformer stores the (defragmented) input into a variable, using requested format. After that it executes all the actions and fetches transformed value using the pattern.
Unconditionally mark currently processed request as invalid.
Unconditionally mark currently processed request as invalid.
6.2.20 - newSequence
Instantiates a sequence for each invocation.
Instantiates a sequence for each invocation.
Inline definition
Sequence name.
Property
Type
Description
concurrencyPolicy
enum
Options:
FAIL
WARN
forceSameIndex
boolean
Forces that the sequence will have the same index as the currently executing sequence. This can be useful if the sequence is passing some data to the new sequence using sequence-scoped variables. Note that the new sequence must have same concurrency factor as the currently executing sequence.
sequence
String
Name of the instantiated sequence.
6.2.21 - parseHtml
Parses HTML tags and invokes handlers based on criteria.
Parses HTML tags and invokes handlers based on criteria.
Action performed when the download of all resources completes.
onEmbeddedResource.fetchResource.metric
Metrics selector for downloaded resources.
Property
Type
Description
<list of strings>
<list of strings>
Allows categorizing request statistics into metrics based on the request path. The expressions are evaluated in the order as provided in the list. Use one of:
regexp -> replacement, e.g. ([^?])(?.)? -> $1 to drop the query part.
regexp (don’t do any replaces and use the full path), e.g. .*.jpg
-> name (metric applied if none of the previous expressions match).
onTagAttribute
Property
Type
Description
attribute
String
Name of the attribute in this element you want to process, e.g. action
format
enum
Conversion to apply on the matching parts with ’toVar’ or ’toArray’ shortcuts. Options:
BYTEBUFStore the buffer directly. Beware that this may cause memory leaks!
Name of the tag this handler should look for, e.g. form
toArray
String
Shortcut to store selected parts in an array in the session. Must follow the pattern variable[maxSize]
toVar
String
Shortcut to store first match in given variable. Further matches are ignored.
6.2.22 - publishAgentData
Makes the data available to all sessions in the same agent, including those using different executors.
Makes the data available to all sessions in the same agent, including those using different executors.
Inline definition
Both name of source variable and the key used to read the data.
Property
Type
Description
fromVar
String
Source session variable name.
name
String
Arbitrary unique identifier for the data.
6.2.23 - publishGlobalCounters
Gathers values from session variables and publishes them globally (to all agents).
Gathers values from session variables and publishes them globally (to all agents). You can name the counters individually (example 1) or use the variable names (example 2):
<code>
# Example 1:
- publishGlobalCounters:
key: myKey
vars: [ foo, bar ]
# Example 2:
- publishGlobalCounters:
key: someOtherKey
vars:
- foo: myFoo
- bar: bbb
</code>
Stores defragmented data in a queue. For each item in the queue a new sequence instance will be started (subject the concurrency constraints) with sequence index that allows it to read an object from an array using sequence-scoped access.
Property
Type
Description
concurrency
int
Maximum number of started sequences that can be running at one moment.
format
enum
Conversion format from byte buffers. Default format is STRING. Options:
BYTEBUFStore the buffer directly. Beware that this may cause memory leaks!
BYTESStore data as byte array.
STRINGInterprets the bytes as UTF-8 string.
maxSize
int
Maximum number of elements that can be stored in the queue.
Custom action that should be performed when the last consuming sequence reports that it has processed the last element from the queue. Note that the sequence is NOT automatically augmented to report completion.
sequence
String
Name of the started sequence.
var
String
Variable storing the array that it used as a output object from the queue.
6.2.25 - readAgentData
Reads data from agent-wide scope into session variable.
Reads data from agent-wide scope into session variable. The data must be published in a phase that has terminated before this phase starts: usually this is achieved using the startAfterStrict property on the phase.
Inline definition
Both the identifier and destination session variable.
Property
Type
Description
name
String
Unique identifier for the data.
toVar
String
Destination session variable name.
6.2.26 - removeItem
Removes element from an array of variables.
Removes element from an array of variables.
Property
Type
Description
fromVar
String
Variable containing an existing array of object variables.
Creates integer arrays to be stored in the session.
Property
Type
Description
fromVar
String
Contents of the new array. If the variable contains an array or a list, items will be copied to the elements with the same index up to the size of this array. If the variable contains a different value all elements will be initialized to this value.
size
int
Size of the array.
objectArray
Creates object arrays to be stored in the session.
Property
Type
Description
fromVar
String
Contents of the new array. If the variable contains an array or a list, items will be copied to the elements with the same index up to the size of this array. If the variable contains a different value all elements will be initialized to this value.
Retrieves value from a counter shared by all sessions in the same executor and stores that in a session variable.
Retrieves value from a counter shared by all sessions in the same executor and stores that in a session variable. If the value exceeds allowed integer range (-2^31 .. 2^31 - 1) it is capped.
Inline definition
Both the key and variable name.
Property
Type
Description
key
String
Identifier for the counter.
toVar
String
Session variable for storing the value.
6.3.10 - getSize
Calculates size of an array/collection held in variable into another variable
Calculates size of an array/collection held in variable into another variable
Unconditionally mark currently processed request as invalid.
Unconditionally mark currently processed request as invalid.
6.3.13 - newSequence
Instantiates a sequence for each invocation.
Instantiates a sequence for each invocation.
Inline definition
Sequence name.
Property
Type
Description
concurrencyPolicy
enum
Options:
FAIL
WARN
forceSameIndex
boolean
Forces that the sequence will have the same index as the currently executing sequence. This can be useful if the sequence is passing some data to the new sequence using sequence-scoped variables. Note that the new sequence must have same concurrency factor as the currently executing sequence.
sequence
String
Name of the instantiated sequence.
6.3.14 - publishAgentData
Makes the data available to all sessions in the same agent, including those using different executors.
Makes the data available to all sessions in the same agent, including those using different executors.
Inline definition
Both name of source variable and the key used to read the data.
Property
Type
Description
fromVar
String
Source session variable name.
name
String
Arbitrary unique identifier for the data.
6.3.15 - publishGlobalCounters
Gathers values from session variables and publishes them globally (to all agents).
Gathers values from session variables and publishes them globally (to all agents). You can name the counters individually (example 1) or use the variable names (example 2):
<code>
# Example 1:
- publishGlobalCounters:
key: myKey
vars: [ foo, bar ]
# Example 2:
- publishGlobalCounters:
key: someOtherKey
vars:
- foo: myFoo
- bar: bbb
</code>
Reads data from agent-wide scope into session variable.
Reads data from agent-wide scope into session variable. The data must be published in a phase that has terminated before this phase starts: usually this is achieved using the startAfterStrict property on the phase.
Inline definition
Both the identifier and destination session variable.
Property
Type
Description
name
String
Unique identifier for the data.
toVar
String
Destination session variable name.
6.3.17 - removeItem
Removes element from an array of variables.
Removes element from an array of variables.
Property
Type
Description
fromVar
String
Variable containing an existing array of object variables.
Creates integer arrays to be stored in the session.
Property
Type
Description
fromVar
String
Contents of the new array. If the variable contains an array or a list, items will be copied to the elements with the same index up to the size of this array. If the variable contains a different value all elements will be initialized to this value.
size
int
Size of the array.
objectArray
Creates object arrays to be stored in the session.
Property
Type
Description
fromVar
String
Contents of the new array. If the variable contains an array or a list, items will be copied to the elements with the same index up to the size of this array. If the variable contains a different value all elements will be initialized to this value.
Integer session variable with an index into the collection.
toVar
String
Session variable with the collection.
value
String
Verbatim value.
index
Inline definition
Uses the argument as a constant value.
Property
Type
Description
fromVar
String
Input variable name.
value
int
Value (integer).
6.3.22 - setSharedCounter
Sets value in a counter shared by all sessions in the same executor.
Sets value in a counter shared by all sessions in the same executor.
Inline definition
Both the name of the counter and variable name.
Property
Type
Description
fromVar
String
Input variable name.
key
String
Identifier for the counter.
value
int
Value (integer).
6.3.23 - stringToInt
Parse string into integer and store it in a variable.
Parse string into integer and store it in a variable.
If the parsing fails the target variable is not modified.
Inline definition
Use fromVar -> toVar
Property
Type
Description
fromVar
String
Source variable name.
toVar
String
Target variable name.
6.3.24 - unset
Undefine variable name.
Undefine variable name.
Inline definition
Variable name.
Property
Type
Description
var
Object
Variable name.
7 - Extensions
How to develop your own extensions
You have probably already read the Custom steps and handlers quickstart which shows how to create a simple component. It can get more tricky when the component embeds other components, though.
The build of scenario happens in two phases. In first phase the sequences, steps and components call method prepareBuild(). Most often that method uses the default (empty) implementation, but if your component (e.g. custom step) embeds another one (e.g. instance of a Processor) it should call its prepareBuild() method, too. The purpose of this method is mutatation of builders, for example adding extra steps to the scenario or registering handlers elsewhere. We’ll see how to do that later on.
In the second phase build(...) is called. At this point the builders tree must not be mutated further as some components might be already built and the change could not be reflected; this method should validate the builder and return the component.
Mutations of the scenario can be position dependent (e.g. adding one step before or after current step). Each builder that needs to know its position therefore must override two methods, setLocator(Locator) and copy(Locator). The former usually just sets a field storing the Locator and delegates the call to embedded components, the latter performs a deep copy of this builder, storing the provided Locator in the copy.
If you want to add some extra steps elsewhere in the scenario, you can implement the prepareBuild() method this way:
Java
privateLocatorlocator;/* ... */@OverridepublicvoidprepareBuild(){// insert custom step before this step in this sequence
locator.sequence().insertBefore(locator).step(newCustomStep(42));// insert custom step after this step in this sequence, using a builder
locator.sequence().insertAfter(locator).stepBuilder(newCustomStep.Builder().foo(42));// insert a new sequence 'foo' with single custom step to the scenario
locator.scenario().sequence("foo").step(newCustomStep(42));}
Note that when you insert any builders in the prepareBuild() methods it is possible that its prepare phase won’t be executed (if inserting to already prepared sequence), though it might be (if inserting to sequence that is yet to be prepared). It’s up to the calling code to make sure that the inserted component will be prepared.
As mentioned above, components often embed other components. To service-load a component, e.g. an Action you define these methods in the builder:
Java
// This method is not different from regular fluent setter
// and it's useful for programmatic configuration.
publicBuilderonEvent(Action.BuilderonEvent){// you can ensure here that this is called only once
this.onEvent=onEvent;returnthis;}// This is the service-loading method.
publicServiceLoadedBuilderProvider<Action.Builder>onEvent(){returnnewServiceLoadedBuilderProvider<>(Action.Builder.class,locator,this::onEvent);}
The parser instantiates concrete implementation of the Action.Builder, calls its setters and then passes the builder to the consumer method referenced as this::onEvent. Note that the call to ServiceLoadedBuilderProvider constructor requires a Locator parameter, as the embedded Action can mutate the scenario later on.
8 - Controller API
OpenAPI 3 specification of the Hyperfoil controller
As a user you’ll probably interact with the Controller through CLI. However when you set up e.g. regression runs in CI you’ll need to control the test programmatically. Some limited capabilities are already exposed through Ansible Galaxy scripts but to get full control you can use the REST API - the same as the CLI or Ansible scripts connect to.
Hyperfoil Controller API is defined through OpenAPI 3 Specification. Our OpenAPI reference defines some types as free-form objects (e.g. the benchmark definition); refer to source code in these cases.
8.1 - Swagger UI
Hyperfoil controller OpenAPI swagger UI
9 - Architecture
Deep dive into the Hyperfoil architecture
While we have already explained basic concepts in the benchmark and last quickstart shows how to create a custom steps or handlers here we will show how Hyperfoil internally works and give you better idea how to create non-trivial extensions.
Building the scenario
The road from a YAML file to executing the benchmark starts with creating the builder tree. Either the CLI or controller presents this file to the parser (mostly classes from the io.hyperfoil.core.parser package) which reads it token-by-token and invoke methods on the io.hyperfoil.api.config.BenchmarkBuilder instance. Some parts of the parser are hard-coded, but most of them use reflection - that’s why you don’t need to write the parser yourselves.
Generally speaking each mapping (foo: bar) results in invoking method foo() on the builder; if this method accepts an argument (bar) the return value could be ignored - the method mutates the builder and that is all. Other methods do not accept any arguments and return another builder instance - the YAML subtree is then applies to this builder. The builder tree then roughly maps to the YAML tree in the original file.
When the YAML is fully read we execute the first phase of building the benchmark itself. We recursively (depth-first) call prepareBuild() methods on the tree; these methods are allowed to invoke further mutations on the builder tree, such as adding other steps and sequences. An empty (default) implementation of this method is perfectly fine if you don’t need anything complex, but if your extension delegates to children extensions it should recursively call the method on children builders. Make sure to iterate through a shallow copy of any collection of children as the children can mutate its parent, failing the iteration.
Hyperfoil does not track new components created in prepareBuild() and therefore it won’t call prepareBuild() on these if the part of builder tree was already processed - the code creating new components must call the method.
When the first phase completes we invoke the build() method on the builder tree. As the build method is invoked recursively again we end up with the benchmark tree that again mirrors the builder tree. The mapping between builders and benchmark components doesn’t need to be 1:1, e.g. StepBuilder can return several steps, builder can return wrapped components etc. However no mutations are allowed in this phase.
So we end up with the io.hyperfoil.api.config.Benchmark instance that holds this tree. It is important to make sure that this is immutable and serializable as it will be sent over the wire from controller to agents. This means that the components must not reference the builders. A common oversight is using a lambda that uses one of builder’s fields - while you only need the actual value of the field the lambda would capture a reference to the builder; this is quite easy to fix by assigning the field value to a local var, though.
Creating the sessions
Hyperfoil tries to minimize allocations during the benchmark as while Java garbage collector is a good friend of every developer it has a negative effect on the real-time properties of the program. You should use collectors that minimize pause times (such as Shenandoah or ZGC) rather than those that maximize throughput. This is why we allocate all what we could need ahead.
Before the benchmark starts each agent creates all the sessions (based on the maxSessions property in case of open-model phases). When Session.reserve() is called it calls reserve() method on all steps that implement ResourceUtilizer interface. In this method the step must call Session.declareResource() on all the resources it uses.
Note: in previous versions of Hyperfoil it was necessary to also explicitly declare that the step can write into a session variable, and recursively call the ResourceUtilizer.reserve() method on all children components. Since version 0.16 Hyperfoil discovers all ResourceUtilizer implementors in the scenario tree; the recursive invocation is no longer necessary.
Scenario execution
Session execution starts with:
one instance of each sequence declared in the initialSequences list
one instance of the first sequence in orderedSequences list
one instance of the first sequence in the implicit list of sequences (right under the scenario: declaration - this is mutually exclusive to the options above)
The session keeps a list of active sequence instances, each with an index of the step that should execute next. There can be several instances of the same sequence, up to its concurrency (the number in brackets next to sequence name). Whenever the session is notified (using Session.proceed() which schedules Session.call() invocation in its executor) it goes through all active instances and calls Step.invoke() on the current step.
There are two possible results from the invoke() method that returns a boolean:
true: This means that step has been successfuly executed, it’s complete and the sequence can continue with the next step (which it will, immediately) or complete if this was the last step.
false: We say that the step was blocked - it cannot execute immediately. It could be either because the step is short of resources (e.g. httpRequest cannot get any available connection from the pool) but most often this is because the purpose of this step is to wait for certain condition: variable being set/reaching certain value, request being completed etc. The sequence will not progress towards next step and this step’s invoke() method will be called again (when the Session.proceed() is called). An important property is that if the step returns false the step must not have any side-effects - it must not fire a request, set a session variable, simply it was a no-op. If the step acquired a resource from a pool it should return it prior to returning false.
Session variables
The session contains a map of variables the scenario uses. The keys are usually strings but this is not mandated; some steps may e.g. choose to use an unique object as the key. The values in the map are wrapper objects that hold a boolean flag whether this variable is set and the value itself. To avoid boxing and unboxing there’s a different wrapper for integers and other objects - it’s up to step to check the wrapper type and convert the value if necessary.
The map is not manipulated directly - a builder for a step that should work with variable foo should call SessionFactory.readAccess("foo"), SessionFactory.intAccess("foo") or SessionFactory.objectAccess("foo") in its build() method and pass the received Access to the step it creates. The step then operates exclusively using Access methods. And example of this can be found in the getting started: custom steps guide.
Sequence-scoped access
In quickstarts there are examples of sequence-scoped access - variables with [.] suffix, e.g. unset: myVar[.]. This is used when the variable holds an array of variables (wrappers) created using ObjectVar.newArray() or IntVar.newArray() in the ResourcesUtilizer.reserve() method - a common pattern would be
The array is often created in a non-conurrent sequence that starts several concurrent instances of another sequence - the var would be used with the simple access (without the [.] suffix) in the original sequence, and with [.] in the concurrent sequences. Each of the concurrent sequences would get a different SequenceInstance.index() and with sequence-scoped access these would work on the variable on this position in the array. The step does not need to be tailored specifically to work on sequence-scoped variables; when creating the Access instance using SessionFactory.objectAccess() the presence of the suffix is automatically checked and the returned Access will relay the operations to the slot in the array.
Session resources
If a step (or a set of cooperating steps) needs to keep some internal state that is not available to users through arbitrary identifiers as session variables these can use concept called session resources. It is again an immutable map of objects in the session (while the map itself is immutable the values are meant to be mutated).
To make the code type-safe you start with the Resource implementation and matching ResourceKey:
The resource key does not need any methods - it is just an unique marker object that will serve as the key in a map. If the resource will be used exclusively in this very step (action, processor…) you can implement the ResourceKey in there and use this when calling session.getResource():
Both session variables and session resources are declared in the reserve() method and retrieved (and mutated) in the business method (invoke() in the case of a step):
Java
publicclassFooStepimplementsStep,ResourceUtilizer{privatefinalFooResourceKeyresourceKey;// set in constructor
@Overridepublicvoidreserve(Sessionsession){// we are using supplier rather than creating instance directly because
// if this sequence is concurrent we will create N resources, the state
// of concurrent sequences will be isolated by default.
session.declareResource(resourceKey,FooResource::new);}@Overridepublicvoidinvoke(Sessionsession){FooResourceresource=session.getResource(resourceKey);/* work on the resource */}}
Component adapters
There are several types of extension components: steps, actions and processors get extra attention but it is possible to use other interfaces as well. Actions are the simplest of these: these do not require any input (but the session) and do always execute without blocking. Therefore it is possible to use an action on any place where a step or processor would fit. When loading the component by name Hyperfoil automatically wraps the action into an adapter to the target component type.
Thread-local, agent-local and global data
Besides session variables Hyperfoil offers 3 more levels of memory. Neither of those is limited to the currently executing phase: this data is not reset until the run completes.
First level is the thread-local memory: since each session runs using single executor it is possible to share some data between sessions using the same executor without any need for synchronization. Currently this model supports shared counters (see addToSharedCounter and getSharedCounter) and shared map-like objects (see pushSharedMap and pullSharedMap). The latter keeps a pool of maps for each executor; when the map is pulled to a session it is removed from the pool, and it’s up to the user to return it back. This is useful e.g. for simulating stateful virtual users when we don’t want to modify one user concurrently in multiple sessions.
Second level is the agent-local memory. This is intended for caching data that needs to be initialized once and then used throughout the test; the initializing phase should invoke publishAgentData. It’s up to you to make sure that when this is read using readAgentData the data is already available - usually the reading phase should be ordered after the publishing phase using startAfterStrict property. Agent data are identified using keys (names); the data for each key can be published only once and cannot be updated afterwards. This limitation is imposed to minimize synchronization of executors.
Third level is the global mem ory. Again this could be used for distributing initialization data but also for gathering data from other agents and threads. The idea is that each thread or agent produces a reduce-able object; Hyperfoil then combines these objects on the agent level (in arbitrary order), sends it to the controller where data from all agents are combined again and the result is distributed back to all agents. As with the agent data you should strictly order the producing and consuming phases, otherwise the data might not be available yet and the run would fail.
Currently Hyperfoil does not provide any general-use steps/actions to work with global data; you should implement GlobalData.Element in an extension and provide steps to create & publish instances of these.