Getting started: Running the server
Until now we have always started our benchmarks using an embedded controller in the CLI, using the
start-local command. This spawns a server in the CLI JVM. CLI communicates with it using standard REST API, though the server port is randomized and listens on localhost only. All the benchmarks and run results are also stored in
/tmp/hyperfoil/ - you can change the directory as an argument to the
While the embedded controller might be convenient for a quick test or when developing the scenario it’s not something that you’d use for a full-fledged benchmark.
When testing a reasonably performing system you need multiple nodes driving the load - we call them agents. These agents sync up, receive commands and report statistics to a master node, the controller. This node exposes a RESTful API to upload & start the benchmark, watch its progress and download results.
There are two other scripts in the
standalone.shstarts both the controller and (one) agent in a single JVM. This is not too different from the controller embedded in CLI.
controller.shstarts clustered Vert.x and deploys the controller. Agents are started as needed in different nodes. You’ll see this in the next quickstart.
Also note that it is possible to run Hyperfoil in Openshift.
Open two terminals; in one terminal start the standalone server and in second terminal start the CLI. Let’s try to connect to the server (by default running on
http://localhost:8090) and upload the single-request.hf.yaml benchmark:
> bin/cli.sh [hyperfoil@localhost]$ connect Connected! Server has these agents connected: * localhost[REGISTERED] [hyperfoil@localhost]$ upload examples/single-request.hf.yaml Loaded benchmark single-request, uploading... ... done. [hyperfoil@localhost]$ run single-request Started run 0001
When you switch to the second terminal, you can see in the logs that the benchmark definition was stored on the server, the benchmark has been executed and its results have been stored to disk. Hyperfoil by default stores benchmarks in directory
/tmp/hyperfoil/benchmark and data about runs in
/tmp/hyperfoil/run; check it out:
> column -t -s , /tmp/hyperfoil/run/0001/stats/total.csv Phase Name Requests Responses Mean Min p50.0 p90.0 p99.0 p99.9 p99.99 Max MeanSendTime ConnFailure Reset Timeouts 2xx 3xx 4xx 5xx Other Invalid BlockedCount BlockedTime MinSessions MaxSessions example test 1 1 267911168 267386880 268435455 268435455 268435455 268435455 268435455 268435455 2655879 0 0 0 0 1 0 0 0 0 0 0
Reading CSV/JSON files directly is not too comfortable; you can check the details through CLI as well:
[hyperfoil@localhost]$ stats Total stats from run 002D Phase Sequence Requests Mean p50 p90 p99 p99.9 p99.99 2xx 3xx 4xx 5xx Timeouts Errors example: test: 1 267.91 ms 268.44 ms 268.44 ms 268.44 ms 268.44 ms 268.44 ms 0 1 0 0 0 0
By the time you type the
stats command into CLI the benchmark is already completed and the CLI shows stats for the whole run. Let’s try running the eshop-scale.hf.yaml we’ve seen in previous quickstart; this will give us some time to observe on-line statistics as the benchmark is progressing:
> podman run --rm -p 8080:8083 quay.io/hyperfoil/hyperfoil-examples
[hyperfoil@localhost]$ upload examples/eshop-scale.hf.yaml Loaded benchmark eshop-scale, uploading... ... done. [hyperfoil@localhost]$ run eshop-scale Started run 0002 Run 0002, benchmark eshop-scale ...
Here the console would automatically jump into the
status command, displaying the progress of the benchmark online. Press Ctrl+C to cancel that (it won’t stop the benchmark run) and run the
[hyperfoil@localhost]$ stats Recent stats from run 0002 Phase Sequence Requests Mean p50 p90 p99 p99.9 p99.99 2xx 3xx 4xx 5xx Timeouts Errors buyingUserSteady/000: buy: 8 1.64 ms 1.91 ms 3.05 ms 3.05 ms 3.05 ms 3.05 ms 8 0 0 0 0 0 browse: 8 2.13 ms 2.65 ms 3.00 ms 3.00 ms 3.00 ms 3.00 ms 8 0 0 0 0 0 browsingUserSteady/000: browse: 8 2.74 ms 2.69 ms 2.97 ms 2.97 ms 2.97 ms 2.97 ms 8 0 0 0 0 0 Press Ctr+C to stop watching...
You can go back to the run progress using the
status command (hint: use
status --all to display all phases, including those not started or already terminated):
[hyperfoil@localhost]$ status Run 0002, benchmark eshop-scale Agents: localhost[INITIALIZED] Started: 2019/04/15 16:27:24.526browsingUserRampUp/006 RUNNING 16:28:54.565 2477 ms buyingUserRampUp/006 RUNNING 16:28:54.565 2477 ms Press Ctrl+C to stop watching...
Since we are showing this quickstart running the controller and CLI on the same machine it’s easy to fetch results locally from
/tmp/hyperfoil/run/XXXX/.... To save you SSHing into the controller host and finding the directories in a ‘true remote’ case there’s the
export command; This fetches statistics to your computer where you’re running CLI. You can chose between default JSON format (e.g.
export 0002 -f json -d /path/to/dir) and CSV format (
export 0002 -f csv -d /path/to/dir) - the latter packs all CSV files into single ZIP file for your convenience.
When you find out that the benchmark is not going well, you can terminate it prematurely:
[hyperfoil@localhost]$ kill Kill run 0002, benchmark eshop-scale(phases: 2 running, 0 finished, 40 terminated) [y/N]: y Killed.
In the next quickstart we will deal with starting clustered Hyperfoil.