Installation

• 1: Manual startup
• 2: K8s/Openshift deployment
• 3: Manual k8s/Openshift deployment
• 4: Ansible startup

1 - Manual startup

Hyperfoil controller is started with

bin/controller.sh

Any arguments passed to the scripts will be passed as-is to the java process.

By default io.hyperfoil.deployer is set to ssh which means that the controller will deploy agents over SSH, based on the agents configurion. This requires that the user account running the controller must have public-key SSH authorization set up using key $HOME/.ssh/id_rsa. The user also has to be able to copy files to the directory set in agent definition (by default /tmp/hyperfoil) using SCP - Hyperfoil automatically synchronizes library files in this folder with the currently running instance and then executes the agent.

When you don’t intend to run distributed benchmarks you can start the controller in standalone mode:

bin/standalone.sh

This variant won’t deploy any agents remotely and therefore it does not need any agents: section in the benchmark definition; instead it will use single agent started in the same JVM.

Below is the comprehensive list of all the properties Hyperfoil recognizes. All system properties can be replaced by environment variables, uppercasing the letters and replacing dots and dashes with underscores: e.g. io.hyperfoil.controller.host becomes IO_HYPERFOIL_CONTROLLER_HOST.

If io.hyperfoi.trigger.url is set the controller does not start benchmark run right away after hitting /benchmark/my-benchmark/start ; instead it responds with status 301 and header Location set to concatenation of this string and BENCHMARK=my-benchmark&RUN_ID=xxxx. CLI interprets that response as a request to hit CI instance on this URL, assuming that CI will trigger a new job that will eventually call /benchmark/my-benchmark/start?runId=xxxx with header x-trigger-job. This is useful if the the CI has to synchronize Hyperfoil to other benchmarks that don’t use this controller instance.

Security

Since Hyperfoil accepts and invoked any serialized Java objects you must not run it exposed to public to prevent a very simple remote code execution. Even if using HTTPS and password protection (see below) we recommend to limit access and privileges of the process to absolute minimum.

You can get confidential access to the server using TLS encryption, providing the certificate and keys either using Java Keystore mechanism (properties above starting with io.hyperfoil.controller.keystore) or via PEM files (properties starting with io.hyperfoil.controller.pem). These options are mutually exclusive. In the latter case it is possible to use multiple certificate/key files, separated by comma (,).

Authentication uses Basic authentication scheme accepting any string as username. The password is set using io.hyperfoil.controller.password or respective environment variable. If you’re exposing the server using plaintext HTTP you must set -Dio.hyperfoil.controller.secured.via.proxy=true to confirm that this is a desired configuration (e.g. if the TLS is terminated at proxy and the connection from proxy does not require confidentiality).

2 - K8s/Openshift deployment

A convenient alternative to running Hyperfoil on hosts with SSH access is deploying it in Kubernetes or Openshift environment. The recommended way to install it using an operator in your Openshift console - just go to Operators - OperatorHub and search for ‘hyperfoil’, and follow the installation wizard. Alternatively you can deploy the controller manually.

In order to start a Hyperfoil Controller instance in your cluster, create a new namespace hyperfoil: Go to Operators - Installed Operators and open Hyperfoil. In upper left corner select ‘Project: ’ - Create project and fill out the details. Then click on the ‘Hyperfoil’ tab and find the button ‘Create Hyperfoil’.

You should see a YAML definition like this:

apiVersion: hyperfoil.io/v1alpha2
kind: Hyperfoil
metadata:
  name: example-hyperfoil
  namespace: hyperfoil
spec:
  agentDeployTimeout: 60000
  log: myConfigMap/log4j2-superverbose.xml
  route:
    host: hyperfoil.apps.mycloud.example.com
  version: latest

Change the name to just hyperfoil (or whatever you prefer) and delete all the content from the spec section:

apiVersion: hyperfoil.io/v1alpha2
kind: Hyperfoil
metadata:
  name: hyperfoil
  namespace: hyperfoil
spec:

This is a perfectly valid Hyperfoil resource with everything set to default values. You can customize some properties in the spec section further - see the reference.

The operator deploys only the controller; each agent is then started when the run starts as a pod in the same namespace and stopped when the run completes.

When the resource becomes ready (you can check it out through Openshift CLI using oc get hf) the controller pod should be up and running. Now you can open Hyperfoil CLI and connect to the controller. While default Hyperfoil port is 8090, default route setting uses TLS (edge) and therefore Openshift router will expose the service on port 443. If your cluster’s certificate is not recognized (such as when using self-signed certificates) you need to use --insecure (or -k) option.

bin/cli.sh

[hyperfoil]$ connect hyperfoil-hyperfoil.apps.my.cluster.domain:443 --insecure
WARNING: Hyperfoil TLS certificate validity is not checked. Your credentials might get compromised.
Connected!
WARNING: Server time seems to be off by 12124 ms

Now you can upload & run benchmarks as usual - we’re using {% include example_link.md src=‘k8s-hello-world.hf.yaml’ %} in this example. Note that it can take several seconds to spin up containers with agents.

[hyperfoil@hyperfoil-hyperfoil]$ upload examples/k8s-hello-world.hf.yaml
Loaded benchmark k8s-hello-world, uploading...
... done.

[hyperfoil@hyperfoil-hyperfoil]$ run k8s-hello-world
Started run 0000
Run 0000, benchmark k8s-hello-world
Agents: agent-one[STARTING]
Started: 2019/11/18 19:07:36.752    Terminated: 2019/11/18 19:07:41.778
NAME  STATUS      STARTED       REMAINING  COMPLETED     TOTAL DURATION               DESCRIPTION
main  TERMINATED  19:07:36.753             19:07:41.778  5025 ms (exceeded by 25 ms)  5.00 users per second

[hyperfoil@hyperfoil-hyperfoil]$

You can find more details about adjusting the agents in the benchmark format reference.

Running Hyperfoil inside the cluster you are trying to test might skew results due to different network topology compared to driving the load from ‘outside’ (as real users would do). It is your responsibility to validate if your setup and separation between load driver and SUT (system under test) is correct. You have been warned.

Reference

route

auth

3 - Manual k8s/Openshift deployment

If you cannot use the operator or if you’re running vanilla Kubernetes you can define all the resource manually. You deploy only the controller; each agent is then started, when the run starts, as a pod in the same namespace and stopped when the run completes.

Following steps install Hyperfoil controller in Openshift, assuming that you have all the required priviledges. With vanilla Kubernetes you might have to replace the route with an appropriate ingress.

1. Create new namespace for hyperfoil

oc new-project hyperfoil

2. Create required resources

curl -s -L k8s.hyperfoil.io | oc apply -f -

role.rbac.authorization.k8s.io/controller created
serviceaccount/controller created
service/hyperfoil created
rolebinding.rbac.authorization.k8s.io/controller created
deploymentconfig.apps.openshift.io/controller created
route.route.openshift.io/hyperfoil created

The route will use hostname following the format hyperfoil-hyperfoil.apps.my.cluster.domain - feel free to customize the hostname as needed.

3. Wait until the image gets downloaded and the container starts

oc get po

NAME                  READY   STATUS              RESTARTS   AGE
controller-1-pqbvs    1/1     Running             0          57s
controller-1-deploy   0/1     Completed           0          72s

4. Open CLI and connect to the controller

While default Hyperfoil port is 8090, Openshift router will expose the service on port 80.

bin/cli.sh

[hyperfoil]$ connect hyperfoil-hyperfoil.apps.my.cluster.domain -p 80
Connected!
WARNING: Server time seems to be off by 12124 ms

5. Upload and run benchmarks as usual

We’re using k8s-hello-world.hf.yaml in this example.

name: k8s-hello-world
agents:
  agent-one:
http:
  host: https://kubernetes.default.svc.cluster.local
duration: 5s
usersPerSec: 5
scenario:
- test:
  - httpRequest:
      GET: /

Note that it can take several seconds to spin up containers with agents.

[hyperfoil@hyperfoil-hyperfoil]$ upload .../k8s-hello-world.hf.yaml
Loaded benchmark k8s-hello-world, uploading...
... done.

[hyperfoil@hyperfoil-hyperfoil]$ run k8s-hello-world
Started run 0000
Run 0000, benchmark k8s-hello-world
Agents: agent-one[STARTING]
Started: 2019/11/18 19:07:36.752    Terminated: 2019/11/18 19:07:41.778
NAME  STATUS      STARTED       REMAINING  COMPLETED     TOTAL DURATION               DESCRIPTION
main  TERMINATED  19:07:36.753             19:07:41.778  5025 ms (exceeded by 25 ms)  5.00 users per second

[hyperfoil@hyperfoil-hyperfoil]$

You can find more details about adjusting the agents in the benchmark format reference.

Running Hyperfoil inside the cluster you are trying to test might skew results due to different network topology compared to driving the load from ‘outside’ (as real users would do). It is your responsibility to validate if your setup and separation between load driver and SUT (system under test) is correct. You have been warned.

4 - Ansible startup

You can fetch release, distribute and start the cluster using Ansible Galaxy scripts; setup, test, shutdown

First, get the scripts:

ansible-galaxy install hyperfoil.hyperfoil_setup,{{ site.last_release.galaxy_version }}
ansible-galaxy install hyperfoil.hyperfoil_shutdown,{{ site.last_release.galaxy_version }}
ansible-galaxy install hyperfoil.hyperfoil_test,{{ site.last_release.galaxy_version }}

Now, edit your hosts file, it could look like this:

[hyperfoil-controller]
controller ansible_host=localhost

[hyperfoil-agent]
agent-1 ansible_host=localhost

Prepare your playbook; here is a short example that starts the controller, uploads and starts simple benchmark (the templating engine replaces the agents in benchmark script based on Ansible hosts) and waits for its completion. When it confirms number of requests executed it stops the controller.

- hosts: [ hyperfoil-agent, hyperfoil-controller ]
  tasks: [] # This will only gather facts about all nodes
- hosts: hyperfoil-controller
  roles:
  - hyperfoil.hyperfoil_setup
- hosts: 127.0.0.1
  connection: local
  roles:
  - hyperfoil.hyperfoil_test
  vars:
    test_name: example
# Note that due to the way Ansible lookups work this will work only if hyperfoil-controller == localhost
- hosts: 127.0.0.1
  connection: local
  tasks:
  - name: Find number of requests
    set_fact:
      test_requests: "{{ lookup('csvfile', 'example file=/tmp/hyperfoil/workspace/run/' + test_runid + '/stats/total.csv col=2 delimiter=,')}}"
  - name: Print number of requests
    debug:
      msg: "Executed {{ test_requests }} requests."
- hosts:
  - hyperfoil-controller
  roles:
  - hyperfoil.hyperfoil_shutdown

Finally, run the playbook:

ansible-playbook -i hosts example.yml