This section contains practical advices for common things you could want to use in a benchmark.
This is the multi-page printable view of this section. Click here to print.
How To
A collection of practical advices for common things in Hyperfoil
1 - JSON schema support
How to make your life easier while building your own benchmarks
For your convenience we recommend using editor with YAML validation against JSON schema; you can point your editor to docs/schema.json. We can recommend Visual Studio Code
with redhat.vscode-yaml plugin.
You need to edit settings file to map benchmark configuration files (with .hf.yaml extension) to the schema, adding:
"yaml.schemas" : {
"file:///path/to/hyperfoil-distribution/docs/schema.json" : "/*.hf.yaml"
},
Note that you can also directly point to the hosted JSON schema definition:
"yaml.schemas" : {
"https://hyperfoil.io/schema.json" : "/*.hf.yaml"
},
2 - Using credentials
Hyperfoil benchmarks can refer to external files. When you use the upload command in CLI the files are automatically attached the the benchmark YAML (relative paths are resolved relative to this file). Later on when editing the file you can choose to re-upload some of these. Once the benchmark is built the files are loaded to the in-memory representation - Hyperfoil won’t access these files during runtime. With clustered benchmarks these files don’t need to be on the agents either - the controller sends serialized in-memory representation to the agents and that contains everything needed for the actual execution.
When testing a workload you will likely skip user registration and come with a list of username+password keys. A convenient way is to keep these in a CSV file that looks like
"johny","my-5ec12eT_pwd!"
"bob","superlongpassphrasethatnobodywillguess"
We will create a step that selects a random line from such file and stores it in session variables username and password:
- randomCsvRow:
file: credentials.csv # Path relative to the benchmark
columns:
0: username
1: password
The columns property has a mapping of zero-based indices of columns in the CSV file. This way you can use a file with some extra information.
In case you don’t need to split lines into separate variables you can use the randomItem:
- randomItem:
file: usernames.txt
toVar: username
We have selected the credentials, now it’s time to execute the login. When users submit a form on HTML page this usually results in a POST request with content-type: application/x-www-form-urlencoded header and the form contents encoded in the request body. Hyperfoil lets you specify this conveniently using the form body formatter:
- httpRequest:
POST: /login
body:
form:
- name: username
fromVar: username
- name: password
fromVar: password
- name: action
value: log-me-in-please
Successful response from the server carries a token in most cases but the actual method can vary. If the server sets a cookie Hyperfoil automatically records this and sends it on subsequent requests (this can be switched off using ergonomics.repeatCookies). If the server sends the token in a response header, e.g. x-token you can store it using the header handler into the token session variable:
- httpRequest:
# ...
handler:
header:
filter:
header: "x-token"
processor:
store: token
If the server returns this token as a part of JSON object - e.g { "token": "abc123" } you would process response body using the json processor in body handler:
- httpRequest:
# ...
handler:
body:
json:
query: ".token"
toVar: token
Runnable example
Below you can find a runnable example that we have prepared for you and that you can download here.
name: credentials
http:
- host: https://localhost:8084
phases:
- use-cookie:
atOnce:
users: 1
scenario:
# Make sure that without the cookie (before login) the request fails with 401
- before-login:
- httpRequest:
GET: /howto/credentials/secure
handler:
autoRangeCheck: false # Don't fail with 4xx-5xx
status:
range: 401
- login-with-form:
- randomCsvRow:
file: credentials.csv
columns:
0: username
1: password
- httpRequest:
POST: /howto/credentials/login
body:
form:
- name: username
fromVar: username
- name: password
fromVar: password
# Here we already have the cookie set to a httpRequest will succeed
- after-login:
- httpRequest:
GET: /howto/credentials/secure
- use-bearer-token:
atOnce:
users: 1
scenario:
- request-login-page:
# This first request will response with WWW-Authenticate header
- httpRequest:
GET: /howto/credentials/login
handler:
autoRangeCheck: false
status:
range: 401
- login-with-basic-auth:
- randomCsvRow:
file: credentials.csv
columns:
0: username
1: password
- template:
pattern: ${username}:${password}
toVar: concatenated
- httpRequest:
GET: /howto/credentials/login
headers:
# Our example runs over HTTP2 and that mandates lower-case header names
authorization: Basic ${base64encode:concatenated}
# With Authorization header the server will reply with token
# and redirect us to the secured page. `httpRequest` implements
# automatic redirection through `handler.followRedirect` but that
# wouldn't send the token, so we'll do that manually
handler:
header:
- filter:
header: x-token
processor:
- store: token
status:
range: 30x
- after-login:
- httpRequest:
GET: /howto/credentials/secure
headers:
authorization: Bearer ${token}There are two scenarios: use-cookie performs the login using the POST from form as shown above and then keeps the secret token in a cookie; use-bearer-token performs HTTP Basic authentication, receives a token in headers and then uses that for HTTP Bearer authentication.
You should start with running a server with mocked responses in one console:
podman run --rm -p 8084:8084 quay.io/hyperfoil/hyperfoil-examples
In second console start the CLI with in-vm controller (or standalone and open the WebCLI in your browser). We are running in host-network mode to be able to reach localhost from within the container.
podman run --rm -it --net host quay.io/hyperfoil/hyperfoil cli
[hyperfoil]$ start-local
Starting controller in default directory (/tmp/hyperfoil)
Controller started, listening on 127.0.0.1:35243
Connecting to the controller...
Connected to 127.0.0.1:35243!
[hyperfoil@in-vm]$ upload https://hyperfoil.io/benchmarks/credentials.hf.yaml
Loaded benchmark credentials, uploading...
... done.
[hyperfoil@in-vm]$ run
Started run 0000
Run 0000, benchmark credentials
Agents: in-vm[STOPPED]
Started: 2022/10/27 15:48:39.271 Terminated: 2022/10/27 15:48:39.301
NAME STATUS STARTED REMAINING COMPLETED TOTAL DURATION DESCRIPTION</span>
use-bearer-token TERMINATED 15:48:39.271 15:48:39.301 30 ms (exceeded by 31 ms) 1 users at once
use-cookie TERMINATED 15:48:39.271 15:48:39.300 29 ms (exceeded by 30 ms) 1 users at once
[hyperfoil@in-vm]$ stats
Total stats from run 0000
PHASE METRIC THROUGHPUT REQUESTS MEAN p50 p90 p99 p99.9 p99.99 TIMEOUTS ERRORS BLOCKED 2xx 3xx 4xx 5xx CACHE</span>
use-bearer-token after-login 33.33 req/s 1 3.79 ms 3.80 ms 3.80 ms 3.80 ms 3.80 ms 3.80 ms 0 0 0 ns 1 0 0 0 0
use-bearer-token login-with-basic-auth 33.33 req/s 1 4.57 ms 4.59 ms 4.59 ms 4.59 ms 4.59 ms 4.59 ms 0 0 0 ns 0 1 0 0 0
use-bearer-token request-login-page 33.33 req/s 1 7.49 ms 7.50 ms 7.50 ms 7.50 ms 7.50 ms 7.50 ms 0 0 0 ns 0 0 1 0 0
use-cookie after-login 34.48 req/s 1 3.69 ms 3.70 ms 3.70 ms 3.70 ms 3.70 ms 3.70 ms 0 0 0 ns 1 0 0 0 0
use-cookie before-login 34.48 req/s 1 9.60 ms 9.63 ms 9.63 ms 9.63 ms 9.63 ms 9.63 ms 0 0 0 ns 0 0 1 0 0
use-cookie login-with-form 34.48 req/s 1 4.96 ms 4.98 ms 4.98 ms 4.98 ms 4.98 ms 4.98 ms 0 0 0 ns 1 0 0 0 0
The list of possibilities is endless; if your use case does not fit any of the above please check out the reference. You can also have a look at a full example in the second part of the Beginner’s Guide.
3 - Hyperfoil run script
How to quickly run Hyperfoil benchmarks
Starting from release 0.27, Hyperfoil includes an easy-to-use script that simplifies running benchmarks, allowing users to try tests faster when in-vm controller server is acceptable.
This script is particularly beneficial when you need to quickly test, validate or refine your benchmark definitions, ensuring they run as expected without needing to manually orchestrate the controller and agent processes. It also enables seamless integration into automation scripts or CI/CD pipelines, where you can configure benchmarks to run as part of routine testing, with results saved for further analysis. By simplifying the benchmark execution process, this script accelerates your workflow and allows for more streamlined performance testing with Hyperfoil.
Key features
In-vm controller: The script launches an in-VM Hyperfoil controller, so there’s no need for users to set up or manage an external controller.
Benchmark upload & execution: Once the controller is running, the script automatically uploads the benchmark you provide and triggers its execution. This minimizes manual setup, allowing users to focus on their test scenarios.
No CLI interactions: Running the script does not require any CLI interaction, making this scipt suitable for further automation.
Automatic report generation: By adding the
--output <path-to-dir>option, the script will generate and save an HTML report of the test results in the specified directory, making it easy to review performance data immediately after the benchmark completes.
Usage
The syntax of this script is basically a superset of the run command, where the main argument is not the name of the benchmark but the benchmark file itself.
Usage: run [<options>] <benchmark>
Load and start a benchmark on Hyperfoil controller server, the argument can be the benchmark definition directly.
Options:
-o, --output Output destination path for the HTML report
--print-stack-trace
-d, --description Run description
-P, --param Parameters in case the benchmark is a template. Can be set multiple times. Use `-PFOO=` to set the parameter to empty value and `-PFOO` to remove it and use default if available.
-E, --empty-params Template parameters that should be set to empty string.
-r, --reset-params Reset all parameters in context.
Argument:
Benchmark filename.
From the unzipped Hyperfoil distribution, you can simply run the script using the following format:
./distribution/bin/run.sh [-o OUTPUT_DIR] [-PPARAM1=.. -PPARAM2=..] BENCHMARK_FILE
For instance:
./distribution/bin/run.sh -o /tmp/reports /tmp/first-benchmark.yml
A valid output will be something like:
$ ./distribution/target/distribution/bin/run.sh -o /tmp/reports /tmp/first-benchmark.yml
Loaded benchmark first-benchmark, uploading...
... done.
Started run 0021
Monitoring run 0021, benchmark first-benchmark
Started: 2024/09/30 19:19:38.689
Terminated: 2024/09/30 19:19:49.532
Report written to /tmp/reports/0021.html
Alternatively you could also run the same directly using the Hyperfoil docker image:
docker run -it -v /tmp/benchmark/:/benchmarks:Z -v /tmp/reports:/tmp/reports:rw,Z -it --network=host quay.io/hyperfoil/hyperfoil run -o /tmp/reports /benchmarks/first-benchmark.yml
and the output will be the same:
$ docker run -it -v /tmp/benchmarks/:/benchmarks:Z -v /tmp/reports:/tmp/reports:rw,Z -it --network=host quay.io/hyperfoil/hyperfoil run -o /tmp/reports /benchmarks/first-benchmark.yml
Loaded benchmark first-benchmark, uploading...
... done.
Started run 0000
Monitoring run 0000, benchmark first-benchmark
Started: 2024/09/30 17:21:22.484
Terminated: 2024/09/30 17:21:32.490
Report written to /tmp/reports/0000.html