What is load testing?
Load testing is simulating system interactions with n Users, collecting multidimensional metrics from response times over to system metrics like RAM and CPU usage. Load testing helps you to find performance issues that may only occur with 100+ users but then could cause big problems. For our case it was an inefficient query causing the complete application to starve and so frequent azure app service restart and scaling for no reason.
Within load testing there is much more than asking if a server could handle a bunch of concurrent connections, this will never happen in real life so we should not care about that! Maybe it just will happen if you're being DDos attacked, but that's another topic for another post. Moreover, we should care about simulating what users really doing with the system and simulate how complex system user interactions are. Then collecting data by the load tester itself to get information about your requests timing. This Is helpful to find requests that causes problems and further investigate them on your application monitoring tools. NBomber does a great job of simulating requests to find those issues.
Why should I load test my application
The Usability expert Jakob Nielsen defined three response time limits for applications:
- 0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.
- 1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data.
- 10 seconds is about the limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.
You do not want to use an application that takes longer than 10 seconds to perform an action, and so your users will. So performance could make the difference in winning and losing users. Load testing gives us a tool to test the performance of our applications in real conditions without real users.
When you are in a StartUp and you are currently preparing your Application for the release like I do, can you be confident that the application could handle the number of users you wish to have?
For me this was always a big worry, manually you can maybe test that everything will work for 10 people, but will it also work for 100? 1,000? or even more? Can you reproduce these tests multiple times a day, without providing pizza and beer? I don't think so. You can never know since there is no way to test this manually. That's where load testing comes into play. With load testing, you can stress test your application and confidently say that it will work for hundreds of users. This is not just important before launching an application, it is also important before launching a new feature since this could possibly also break your Applications performance and you would not recognize it before shipping it.
Above you can see a user using an application developed without load tests. You do not want that kind of user! Users who were happy before could now be unsatisfied, leave your application, or may cancel subscriptions. For sure, you will lose some points on your user trust scores. So please also integrate some load tests in your CI.
Why NBomber as C# load testing tool?
NBomber is an easy-to-use tool for load testing your application. It provides a clean and easy-to-use API. The load tests themselves are not only fast written, but they are also well-orchestrated and NBomber is collecting lots of useful data for you to really evaluate your tests.
For me, the biggest advantage is the load simulation where you not just making dumb 100 requests to a single endpoint at the time, moreover, they are timed and simulating the behavior of real users using data provided by your API or accessing your application.
Since the tests itself simply perform API calls this is a black box testing process. That means with nBomber you can test every project no matter which programming language!
Basic Nbomber example
Create console application project
dotnet new console -n [project_name] -lang ["F#" or "C#"]
dotnet new console -n NBomberTest -lang "F#"
cd NBomberTestAdd NBomber package
dotnet add package NBomber
Make sure to also add the bomber HTTP package to get an easier HTTP request API.
dotnet add package NBomber.http
Next, let's look at the production-ready example test from the NBomber docs
This test will simulate visiting the NBomber website.
using System;
using NBomber;
using NBomber.Contracts;
using NBomber.CSharp;
using NBomber.Plugins.Http.CSharp;
using NBomber.Plugins.Network.Ping;
namespace NBomberTest
{
class Program
{
static void Main(string[] args)
{
var step = Step.Create("fetch_html_page",
clientFactory: HttpClientFactory.Create(),
execute: context =>
{
var request = Http.CreateRequest("GET", "https://nbomber.com")
.WithHeader("Accept", "text/html");
return Http.Send(request, context);
});
var scenario = ScenarioBuilder
.CreateScenario("simple_http", step)
.WithWarmUpDuration(TimeSpan.FromSeconds(5))
.WithLoadSimulations(
Simulation.InjectPerSec(rate: 100, during: TimeSpan.FromSeconds(30))
);
// creates ping plugin that brings additional reporting data
var pingPluginConfig = PingPluginConfig.CreateDefault(new[] { "nbomber.com" });
var pingPlugin = new PingPlugin(pingPluginConfig);
NBomberRunner
.RegisterScenarios(scenario)
.WithWorkerPlugins(pingPlugin)
.Run();
}
}
}Copy the following code to your Programm.cs and execute
Make sure that you use c#9.0 or later!
If you are using an older version, you can update your c# version in your .csproj file
Simply copy this inside your .csproj file inside your <PropertyGroup> tag.
<LangVersion>10.0</LangVersion>NBomber will then start Bombing your application
You will see this Inside your console:
_ _ ____ _
| \ | | | __ ) ___ _ __ ___ | |__ ___ _ __
| \| | | _ \ / _ \ | '_ ` _ \ | '_ \ / _ \ | '__|
| |\ | | |_) | | (_) | | | | | | | | |_) | | __/ | |
|_| \_| |____/ \___/ |_| |_| |_| |_.__/ \___| |_|
12:24:20 [INF] NBomber '2.1.5' Started a new session:
'2022-03-23_11.24.44_session_787459f0'.
12:24:21 [INF] NBomber started as single node.
12:24:21 [INF] Plugins: no plugins were loaded.
12:24:21 [INF] Reporting sinks: no reporting sinks were loaded.
12:24:21 [INF] Starting init...
12:24:21 [INF] Target scenarios: 'simple_http'.
12:24:21 [INF] Init finished.
12:24:21 [INF] Starting warm up...
simple_http ---------------------------------------- 100% 00:00:30
load: keep_constant, copies: 1
12:24:53 [INF] Starting bombing...
simple_http ---------------------------------------- 100% 00:01:01
load: keep_constant, copies: 1
12:25:54 [INF] Stopping scenarios...
12:25:54 [INF] Calculating final statistics...
────────────────────────────────────────────────────── test info ───────────────────────────────────────────────────────
test suite: 'nbomber_default_test_suite_name'
test name: 'nbomber_default_test_name'
──────────────────────────────────────────────────── scenario stats ────────────────────────────────────────────────────
scenario: 'simple_http'
duration: '00:01:00', ok count: 662, fail count: 0, all data: 0 MB
load simulation: 'keep_constant', copies: 1, during: '00:01:00'
┌────────────────────┬────────────────────────────────────────────────────────┐
│ step │ ok stats │
├────────────────────┼────────────────────────────────────────────────────────┤
│ name │ fetch_html_page │
│ request count │ all = 662, ok = 662, RPS = 11 │
│ latency │ min = 59,66, mean = 90,47, max = 367,5, StdDev = 32,04 │
│ latency percentile │ 50% = 79,61, 75% = 92,48, 95% = 158,98, 99% = 209,15 │
└────────────────────┴────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────── hints ─────────────────────────────────────────────────────────
hint for Scenario 'simple_http':
Step 'fetch_html_page' in scenario 'simple_http' didn't track status code. In order to track status code, you should use
Response.Ok(statusCode: value)
hint for Scenario 'simple_http':
Step 'fetch_html_page' in scenario 'simple_http' didn't track data transfer. In order to track data transfer, you should
use Response.Ok(sizeInBytes: value)
12:25:55 [INF] Reports saved in folder:...So NBomber visited my testing playground and collected some data about it.
To Understand the produced output let's look at the load testing basics.
Load testing basics
Inside the Nbomber docs there is also a good explanation about the load testing basics. I Try to Summarize here:
Important load testing metrics
So at first, we look at the statistics produced by our test case from the last section.
┌────────────────────┬────────────────────────────────────────────────────────┐
│ step │ ok stats │
├────────────────────┼────────────────────────────────────────────────────────┤
│ name │ fetch_html_page │
│ request count │ all = 662, ok = 662, RPS = 11 │
│ latency │ min = 59,66, mean = 90,47, max = 367,5, StdDev = 32,04 │
│ latency percentile │ 50% = 79,61, 75% = 92,48, 95% = 158,98, 99% = 209,15 │
└────────────────────┴────────────────────────────────────────────────────────┘-
Request count
-
RPS (Requests per second)
The number of requests our system is able to handle in a second. This is an important indicator to evaluate the performance of our system.
-
-
Latency
The time between sending the request and receiving the response. This should be in good relation with the RPS. Handling 2k requests with 2 min of latency is as bad as handling only 5 RPS with a latency of 1ms.
-
Latency percentile
How many percent is over a specific time? This tells how many really good or really bad requests we have.
This example has just a few requests with over 200 ms latency but half of it is under 80 ms. This helps you to classify the above data.
Know your system
Within load testing there are two main system types
closed systems: They have a constant number of users, each User will send a query and then wait for a response. This would be an API → Database communication. So a Database is a closed system.
open system: Variable number of clients performing requests. Everything that uses HTTP protocols like websites or APIs should be considered an open system.
Based on this knowledge we will later chose the correct simulation for our system.
For closed system this could also be a constant one but for open system i would recommend using the random one.
How to write an NBomber load test
Each NBomber test consists of three main parts
-
Step
The action a user wants to perform. For example login
-
Scenario
A combination of steps to simulate a user flow inside your system.
For example login, load user data, load users projects, open a project, perform a change to the project
-
NBomberRunner
Registers and runs a test
You define a test name and scenarios to run.
Additionally to these main parts, you could have three more optional parts
-
DataFeed
It injects data into your test. This could be for example loading a list of users from a Json file.
-
ClientFactory
Creates your clients to execute concurrent tests.
So let's start with an example test to describe these parts.
Load testing example on a real API
We are going to test three endpoints of this example API. All these requests will be performed when a user successfully logged in and visits his dashboard, so a good performance is important for them.
-
users/bySignedInUser
returns the User data of the signed-in user
-
companies/bySignedInUser
returns the company details the signedInUser has access to
-
projects/company
returns the active projects of the given company
The API we are going to test returns data only to authenticated users.
Perform authenticated requests with NBomber
public static void Run()
{
var httpFactory = ClientFactory.Create(
name: "http_factory",
clientCount: 5,
// we need to init our client with our API token
initClient: (number, context) =>
{
var client = new HttpClient();
client.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue(
"Bearer",
"<your access token>");
return Task.FromResult(client);
});We are creating 5 clients with the httpFactory. To authenticate the requests in NBomber we must set the Authorization header to our needs.
The next step is to create our steps. (haha)
Creating Authenticated NBomber request steps
var load_user_by_email = Step.Create("load_user_by_email",
clientFactory: httpFactory,
execute: async context =>
{
var response = await context.Client.GetAsync(
baseUrl + "users/bySignedInUser");
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var loadUserCompanies = Step.Create("load_User_companies",
clientFactory: httpFactory,
execute: async context =>
{
var response = await context.Client.GetAsync(
baseUrl + "companies/bySignedInUser");
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var loadCpmpanyProjects = Step.Create("load_company_projects",
clientFactory: httpFactory,
execute: async context =>
{
var response = await context.Client.GetAsync(
baseUrl + "projects/company");
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});Up next we set up our scenario
Setting up Nbomber scenario for out load test
var scenario = ScenarioBuilder.CreateScenario("index_requests",
load_user_by_email, loadUserCompanies, loadCpmpanyProjects)
.WithLoadSimulations(new[]
{
// from the nBomber docs:
// It's to model an open system.
// Injects a random number of scenario copies (threads) per 1 sec
// defined in scenarios per second during a given duration.
// Every single scenario copy will run only once.
// Use it when you want to maintain a random rate of requests
// without being affected by the performance of the system under test.
Simulation.InjectPerSecRandom(minRate: 5, maxRate: 10, during: TimeSpan.FromMinutes(1))
});Last but not least we will register the NBomber runner
Register NBomber Runner
NBomberRunner
.RegisterScenarios(scenario)
.Run();and then do not forget to add your test to the Program.cs to execute it.
class Program
{
static void Main(string[] args)
{
indexLoadTests.Run();
}
}Evaluating NBomber results
After the test has been executed you will immediately see your results inside the command line as shown above.
But that's not everything! You will also see your test results as a beautiful HTML Page, md report, or text file, ready to share with your colleagues.
located under
\bin\Release\net6.0\reportsLooking into the HTML page you will find the evaluated results.
As the first execution, I have produced
You can see a statistic of your failed and successful requests.
When running this test again with successful requests you will see something like this:
The Charts below indicate the latency statistics from the table of the command line in a nice chart. We could also see a cake chart of the failed and succeeded tests. But that's still not everything. On the left-hand side of your menu, you can also navigate through your run scenarios and see each scenario in more detail.
We currently have only built one scenario.
Within this page, you can then see again a table with the information of each step inside this scenario. But what's more interesting here is the chart below.
For each scenario, NBomber sets the load simulation, RPS, and the time in the test in relation.
There is also the same chart indicating the behavior of the latency and the data transfer at the bottom.
As you can see for our System under test, the latency rises just a little for those small loads.
This will be interesting with higher numbers of clients where you then can see the difference of the latency for a high amount of requests.
In the hints section, you can find useful hints for your load tests. Like we should track our response status code to determine if there was just an auth problem or a 500 etc.
Conclusion
I know those charts are nice to look at, but what else can we get out of the load tests?
We can determine where we have slow requests, and where we have real problems that may cause the API to crash!
The next steps after doing the load tests would be visiting your Server statistics pages and also your application monitoring tools like Dynatrace to see where there are bottlenecks in your application.
Then you can dive into those bad requests and use your tools to find the root cause of them. Maybe it is a slow query, some inefficient algorithms, huge mappings, or blocking queries.
Finding those issues would definitely bomb the scope of this article, but there will come articles about application analysis in Azure soon! I’m also planning to review sentry, Dynatrace, and Datadog and analyze performance issues with mini Profiler, so stay tuned!
When you have found them you then could be confident to have a good collection of tests to reproduce those issues and make them gone.
Even if the name of the library sounds a bit silly in those times, I hope you got some new knowledge out of this article. If you want to support me and the work I´m doing you can buy me a coffee . I will donate half of it to Ukraine.
Happy load testing,
Alex