In the actual operation and maintenance process, we need more logs and monitoring to let us have a comprehensive understanding of the running status of our own applications. However, for most developers, what everyone pays attention to is how to implement the business more elegantly, or how to make the application more responsive, and other coding-related technologies, monitoring of the application, may It still stays at the level of the log file, and most of them are trying to locate the problem through the log after the accident is discovered by humans.

The Elastic APM that is described in this article is a set of indicators for monitoring applications, such as system response time, exceptions, SQL records executed by EF, etc., and can organize these records into a traceable link for easy querying. In addition, Elastic APM can also make a very beautiful visual display through Kibana, so that we can locate and find problems.

No more nonsense, we started to fight ~

Introduction to Elastic APM

Elastic APM consists of the following four components, as shown below:

APM Agent

The APM Agent is a Nuget package that is installed into your .NET Core program. It is used to collect various types of data such as performance and errors, and caches the collected data and distributes it to the APM Server. Of course, in addition to the Nuget package used by .NET Core, he can also support many other languages, such as Java, Node.Js, Python, etc.

A list of supported languages ​​can be found here:

APM Server

The APM Server is an application deployed on the server side to receive packets sent by the Agent, and automatically creates documents based on these packets to transfer the data to the Elastic Server.

Elastic Search

I believe everyone is familiar with it. He is a high-performance, distributed full-text search engine based on Lucene for fast, real-time storage, search and analysis of large amounts of data. Here, he provides data storage and search capabilities!


If you are familiar with Elastic Search, then you must know Kibana a little bit. Kibana is an open source analysis and visualization platform. He can work well with Elastic Search to help you quickly visualize the data stored in Elastic Search and make it A variety of beautiful reports, graphics, and more.

Environmental preparation

In this actual combat process, we need the following things:

  • Elastic Search
  • Kibana
  • APM Server
  • A project based on .NET Standard 2.0 +

Elastic Search installation:

Kibana installation:

My environment is Centos 7, so I  installed it according to the official website tutorial

, the whole process is very simple:

  • Download the Kibana RPM package (in this way, because the yum install network speed is too slow, so I use the Thunder download to complete the rpm file and upload it to the Linux machine)
  • Run the rpm –install “downloaded file name” command to install
  • After the installation is complete, add the following configuration to the end of the file in the /etc/kibana/kibana.yml file: your IP
server.port: your port
elasticsearch.hosts: ["ES address"]
logging.dest: /var/log/kibana.log 
  • Install Kibana as a system service and start
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service 
sudo systemctl start kibana.service

Here you must pay attention to the version of Elastic Search and Kibana must match, or will report an error. (My ES was installed some time ago, so there will be this problem. If everyone installs all at once, there should be no problem)

If you are unfortunately having problems, you can view the logs through the path configured in logging.dest in the configuration file.

APM Server installation

The installation of APM Server is similar to the installation of Kibana. The process is as follows:

  • Download the RPM package, find the version you need on this page, and also need to be the same version as ES and Kibana, otherwise you know ~
  • Execute rpm –install “downloaded file name” to install
  • Modify the configuration file apm-server.yml in the /etc/amp-server folder and change the initial host: “localhost:8200” to “” so that he can allow it via ip: The port number is accessed in the way, and the following configuration is added at the end of the configuration:


12output.elasticsearch:    hosts: ["ES Address"]
  • Install apm-server as a system service and start

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable apm-server.service 
sudo systemctl start apm-server.service

After performing the above operations, try to open the server Ip:8200 in the browser . Finally, if there is no problem with the APM Server installation, the browser will print something similar to the following:

{  "build_date": "2019-06-20T14:39:23Z",  
"build_sha": "9a099b63c53eac8c55707df96193143ec66337e9",  
"version": "7.2.0"}

At this point we open Kibana in the browser and click Add APM


Then scroll down the newly opened page and click the Check APM Server Status button. If you have correctly setup APM Server, the installation is complete~


So far, our installation work has been completed. Next, we will try to integrate .NET Core with Elastic APM and continue together~

.NET Core application integration

We created a Demo project to test the various features of APM.

Please refer to GitHub for the address of the project:

Reference dependency package

We need to reference the relevant SDK from Nuget to integrate with our application. In fact, we refer to the APM Agent part we first said. In Nuget, we refer to the Elastic.Apm.NetCoreAll package.

Relying on this package is equivalent to automatically relying on the following three packages, you can also rely on only a part of them as needed.

Here we print the package Elastic.Apm.NetCoreAll directly for the sake of simplicity.

Add Agent to .NET Core

Find the .NET Core StartUp file and add the following code to the Configure method inside:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{    app.UseAllElasticApm(Configuration);}

Then add the following to application.json:

{  "ElasticApm": {    "LogLevel": "Error", // 
"ServerUrls": "http://localhost:8200", 
 "ServiceName" : "MyApp",   }}

At this point we start the project, just refresh it a few times, then go back to Kibana, scroll down on the previous page, select .NET, and then click the Check Agent Status button. If it goes well, it will display “Data successfully received”. From one or more agents”, if you are unfortunately unable to display this sentence, you can track it through the Event in VS Diagnostic Tools to see if there is no configuration pair.




Monitoring data view

At the bottom of Kibana’s Add APM page, find Load Kibana Objects to create the index, then click on the APM dashboard button to go to the APM data view page.

After clicking the APM Dashboard button, the page displayed is as follows:


The functions in this page are divided into two major functional modules, Services and Traces. Let’s take a brief look at the corresponding functions in these two tab pages.


The XiaoDotnetCommunity shown in the following list is actually the ServiceName you configured in the configuration file. Click on the name to enter, you can see the following report, which contains three tab pages: Transactions, Errors, Metrics.




among them

Transactions: an overview of the current application request situation, including request response duration, number of request calls, etc.

Errors: list of exceptions in the program

Metrics: CPU/memory usage of the machine where the application is located

PS: Actually, I feel that I need a memory and CPU value consumed by the current application, but the proxy that looks like the .NET Core version does not implement these functions. I look forward to future updates.


Inside is a view for link tracking, the home page contains a list of names of all transactions, and response time, etc.


Click on the specific transaction, you can see the list of links through this transaction and more detailed response information, so as to help you analyze the bottleneck in the entire link, more details we will discuss below.


Explore more

Elastic APM has many other features, such as link tracking and database call execution. Let’s explore it together~

Monitoring API call link tracking

If you understand the microservices architecture, then you must understand the concept of link tracking. So what is link tracking? Give a chestnut:

There is a service A, he will depend on the service B, C, and the service B will depend on the service D, E, the service C depends on F, G (omits numerous dependencies), and then one day, the service A becomes very slow, then How to locate which service is slow? At this point, the link will eventually come in handy~

Let’s briefly simulate this nested call:

In a WebAPI project Demo1 there is a ConsumerController, which has an API A, which calls the interface B/C/D/E in another WEB API project Demo2. The code is roughly as follows:

Project A:

public class ConsumerController : ControllerBase
{    private readonly IHttpClientFactory _httpClientFactory;    
public ConsumerController(IHttpClientFactory httpClientFactory)   
 {       services.AddHttpClient();       
 _httpClientFactory = httpClientFactory;    }    
 private const string baseUri = "http://localhost:54597";     
 [HttpGet("a")]    public async Task<string> A()   
 {        //HttpClient client = new HttpClient();       
 var client = _httpClientFactory.CreateClient();        
Thread.Sleep(new Random().Next(1, 1500));        
var b = await client.GetStringAsync($"{baseUri}/api/data-source/b");       
 var c = await client.GetStringAsync($"{baseUri}/api/data-source/c");       
 var d = await client.GetStringAsync($"{baseUri}/api/data-source/d");      
  var e = await client.GetStringAsync($"{baseUri}/api/data-source/e");      
  return $"b={b} & c={c} & d={d} & e={e}";    }}

Project B:

public class DataSourceController : ControllerBase{    [HttpGet("b")]
    public async Task<string> B()    
{        Thread.Sleep(new Random().Next(1, 1500));        
return "B";    }     
public async Task<string> C()   
 {        Thread.Sleep(new Random().Next(1, 1500));        return "C";    }    
 [HttpGet("d")]    public async Task<string> D()   
 {        Thread.Sleep(new Random().Next(1, 1500));        return "D";    }    
 [HttpGet("e")]    public async Task<string> E()    
{        Thread.Sleep(new Random().Next(1, 1500));        return "E";    }}

At this point we request API A (xxx/api/consumer/a) in Demo1, then open Traces in APM in Kibana and find the record “GET Consumer/A” (it seems that the default is based on the name of the Controller + Action) Named by), then click to view details.


At the bottom of the details, we find TimeLine and you can see the graph shown below:


We can see that the time we are requesting API A is spent in the time of calling 4 APIs. It can also be seen that it takes longer to call the third API. Click the blue bar to see the details of the request. .

The less good thing here is that the default display name is GET localhost. In fact, what we expect is to display the called api uri right? I have given a pr to them, everyone can pay attention to:

Monitor EF execution records

This does not require too much explanation, that is, when the EF performs DB operations, it is monitored to find performance and other issues. My code is roughly as follows:

public void TestEfCore()
{    using (var db = new ApmDbContext())    
{        var jax = new Person        
{            Name = "xxx",            
Age = 26,            
Remark = "xxxx"        };         
db.Persons.FirstOrDefault(x => x.Id == jax.Id );         
db.Persons.FirstOrDefault(x => x.Name == "xxx");         
jax.Name = ".NET";        
 db.SaveChanges();    }}

When we use Kibana to view this request, TimeLine appears as follows:


We can see clearly and intuitively which SQL statements are executed in this request, which is time consuming, which is quite useful for our request analysis. Click on the specific blue bar, you can also see more detailed data, but unfortunately, the data does not record SQL Params, which is not friendly enough for us to completely reproduce this request~


Self-buried record

Relatively speaking, Elastic APM is not good enough for the current ecosystem. It is still slightly worse than the sky walking. If you want to use Elastic APM, you will have to do some burying records of performance data yourself, or for third-party components and classes. When the library supports it, you also need to do some data burying. Next, we will bury some information we want to record in our request. The sample code is as follows:

public void RecordMyApmData()
{         var transaction = Agent.Tracer.CurrentTransaction;     
var span1 = transaction.StartSpan("Stage 1", "Customize");
      var span2 = transaction.StartSpan("Stage 2", "Customize");
     var span3 = transaction.StartSpan("Stage 3", "Customize");
    Thread.Sleep(500);    span3.End(); 

The final record has the following effects:


Although this demo is very simple to write, I believe that you can probably use the Elastic Apm Agent class to customize some of the monitoring data you need to capture.

Abnormal monitoring

Elastic APM can help you record when something goes wrong with our program. This function is similar to the log, but it may be slightly better than the log. Let’s take a look at it~

The sample code is as follows:

public void TestException()
{    try    
{        throw new Exception("except");    }    
catch (Exception)    {     }     
throw new Exception("none captured except");}

After executing the code, we can see the exception by clicking on the Service Name and then in the Tab page of Errors.


Click on the details and we can see the detailed stack call information:


In addition, we can find the request for the exception in the Trasactions Tab, and then click to view the details, we can also see the exception in the details:



This article describes how to use Elastic APM to collect performance and anomaly data in .NET Core applications and use Kibana for visual analysis. Overall, Elastic APM is still quite powerful, basically enough for performance monitoring, link tracking, and exception monitoring. used.

At present, the components supported by Elastic APM are still limited. For example, the database query only supports EF Core, and does not support more components. Link tracking only supports HTTP request tracking, and it does not support other methods. In addition, personally think that Elastic APM puts the monitor alarm (Watcher) into the X-Pack charge package is also very sad, abnormal monitoring alarm is actually quite critical.

Orignal link: