Among the inevitably famous load testing tools such as Gatling, Apache JMeter, HP LoadRunner, ..there is one rising star k6 - is affirming its value as "the best developer experience for load testing". In this post, I will introduce k6, its basic usage and advantages over other tools.
Performance Testing types
Before getting to use k6, let's first understand the definition and the differences of these performance testing types.
Each test types is designed to give you different insights about the system.
Smoke Test's role is to verify that your system can handle minimal load, without any problems.
Load Test is primarily concerned with assessing the performance of your system in terms of concurrent users or requests per second.
Stress Test and Spike Test are concerned with assessing the limits of your system and stability under extreme conditions.
Soak Test tells you about reliability and performance of your system over an extended period of time.
And we want to run specific type of performance test to:
Smoke Test:
Verify that your test script doesn't have errors
Verify that system doesn't throw any errors when under minimal load
Load Test:
Assess the current performance of your system under typical and peak load.
Make sure you continue to meet the performance standards as you make changes to your system (code and infrastructure).
Stress Test/Spike Test:
How your system will behave under extreme conditions.
What the maximum capacity of your system is in terms of users or throughput.
The breaking point of your system and its failure mode.
If your system will recover without manual intervention after the stress test is over.
How your system will perform under a sudden surge of traffic (Spike).
Soak Test:
Verify that your system doesn't suffer from bugs or memory leaks, which result in a crash or restart after several hours of operation.
Verify that expected application restarts don't lose requests.
Find bugs related to race-conditions that appear sporadically.
Make sure your database doesn't exhaust the allotted storage space and stops.
Make sure your logs don't exhaust the allotted disk storage.
Make sure the external services you depend on don't stop working after a certain amount of requests are executed.
Smoke Test: Verify that your system can handle minimal load, without any problems.
Load Test: Assess the current performance of your system under typical and peak load.
Stress Test: Verify the stability and reliability of the system under extreme conditions (determine the limits of system).
Spike Test: Verify the system that being immediately overwhelmed with an extreme surge of load.
Soak Test: Verify uncovers performance and reliability issues stemming from a system being under pressure for an extended period.
The VUs stands for Virtual Users (y axis) and x axis is timestamp.
Getting Started - k6 Open source load testing tool
k6 is an open-source load testing tool that makes performance testing easy and productive for engineering teams. k6 is free, developer-centric, and extensible. It has CLI tool with developer-friendly APIs (similar to ApacheBench) but it is also a scriptable tool using JavaScript ES2015/ES6.
ApacheBench is a non-scriptable tool, easy to use. Just type the command followed by the specified parameters and you are good to go. it's highly suitable for user who do not need to write code but the obvious disadvantage is not able to do anything beyond the provided parameters. Example:
ab -n 100 -c 10 http://www.website.com/ -H "clientId:1234" -H "token:a2ds5mvo93mcnsk3ncsju3nxdl"And Apache JMeter, which can write commands in XML, it limits to the provided XML syntax.
With Checks and Thresholds for goal-oriented, automation-friendly load testing, k6 is much more flexible and powerful over among other competitors in the market.
Installing k6
Linux
Debian/Ubuntu
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6Fedora/CentOS
sudo dnf install https://dl.k6.io/rpm/repo.rpm
sudo dnf install k6MacOS
brew install k6Windows
We can use either Chocolatey package manager
choco install k6or Windows Package Manager
winget install k6Otherwise you can manually download and install the latest official .msi package.
Docker
docker pull loadimpact/k6Running basic k6 test
Let's start by running a simple local test script. Creating new test.js file and paste the following code:
Then run k6 with following command:
k6 run test.jsAnd the result would be:
Some important metrics:
duration: period of time to test.
iterations: the number of times the VUs in the test have executed the JS script
vus: active virtual users.
vus_max: max possible number of virtual users.
iteration_duration: the time it took to complete one full iteration.
data_received: the amount of received data.
data_sent: the amount of sent data.
checks: the rate of successful checks.
http_reqs: the number of HTTP requests generated.
http_req_duration: Total time for the request. It's equal to http_req_sending + http_req_waiting + http_req_receiving.
http_rq_failed: the rate of failed requests.
For more info:
Modifying k6 parameters via command line
Adding more VUs
Set VUs to 10 and duration time to 30 seconds:
k6 run --vus 10 --duration 30s test.js
k6 works with the concept of virtual users (VUs), which run scripts - they're essentially glorified, parallel while(true) loops. Scripts are written using JavaScript, as ES6 modules, which allows you to break larger tests into smaller pieces, or make reusable pieces as you like.
Set up iterations
Set VUs to 10 and iterations to 100 (means 1 VU will execute script 10 times):
k6 run --vus 10 --duration 30s --iterations 100 test.js
More parameters in k6 can be found here: https://k6.io/docs/using-k6/options/
Using options in script to modify parameters
If you want to avoid having to type --vus 10 and --duration 30s all the time, you can include those settings inside your JavaScript file also:
Then we just run the script without those parameters on the command line:
k6 run test.jsMore parameters in k6 can be found here: https://k6.io/docs/using-k6/options/
Ramping up and down VUs to perform testing by stages
We can also have the VU level ramp up and down during the test. The options.stages property allows you to configure ramping behaviour.
It has 3 stages:
First: VUs from 0 to 20 in 30 seconds.
Second: VUs from 20 go down to 10 in 1 min 30 seconds.
Third: VUs from 10 to 0 in 20 seconds.
To make in-depth configurations to how VUs and iterations are scheduled. Scenarios makes it possible to model diverse traffic patterns in load tests. For more about scenarios: https://k6.io/docs/using-k6/scenarios/
Stages play an essential role in simulating performance testing types that serves different purpose.
Load Test:
Stress Test:
Spike Test:
Soak Test:
For the charts about how VUs and time fluctuate, please refer Performance Testing types part.
Check the response of HTTP requests
Checks are like assertions, but differ in that they don't halt the execution. Instead, they store the result of the check, pass or fail, and let the script execution continue.
In this example, I checked status code of HTTP requests/responses, body size and contained text. And here is the result:
For more info: https://k6.io/docs/using-k6/checks/
Using threshold to specify expectations
Thresholds are a pass/fail criteria used to specify the performance expectations of the system under test. Example expectations:
System doesn't produce more than 1% errors.
Response time for 95% of requests should be below 200ms.
Response time for 99% of requests should be below 400ms.
Specific endpoint must always respond within 300ms.
Here is an example where we check the percentage of failed requests should < 0.01 and 95% of requests should below 500ms:
The result:
From the picture, you can see the passed criteria were marked as green check marks.
For more info: https://k6.io/docs/using-k6/thresholds/
k6 Cloud - commercial SaaS product
The k6 Cloud is a premium service to empower your team to manage your load testing efforts:
Scaling your load tests.
Storing and visualizing your test results.
Detecting performance issues.
Correlating results between different tests.
Organizing testing in a central location.
It has some advantages over the local k6, but it's still SaaS product so you still have to pay for scaling up your test, so on. In the scope of this post, I prefer using k6 local. More info on k6 cloud: https://k6.io/docs/cloud
Comparison and summary
After having 2 weeks playing with k6 (I also spent time on building an k6 framework on Github, I will share it in another post later), I found k6 is super powerful with high flexibility in scripting configurations - that serves our diversity in purposes. In the near future, I definitely undoubtedly stick to k6 for my performance testing.
Everything you want to know, are well-structured here: https://k6.io/docs/
For those who want to see how k6 performs compared to other tools, you can check it here: https://k6.io/blog/comparing-best-open-source-load-testing-tools/
See you in the next post about my k6 framework.
Comments