Learn how to split log data into different tables using Serilog in ASP.NET Core

For most of the application developers, file systems are the primary choice for storing the information generated by the logging providers. One of the main drawbacks of using the files is that it's very difficult for the search for information or to do an analysis of the information written to it over time. Third-party logging providers such as Serilog have facilities to persist the data in database tables instead of the file system. Even then, if you use a single table to write all you errors and other debug information, the size of the table will grow considerably over time which can affect the performance of the whole operation itself.

So, in this post, I will explore the possibility of using multiple tables for storing the logging information using Serilog. If you are new to Serilog, please refer to my previous articles on the same here using the links given below.

• Implementing Logging in a .NET Core Web Application using Serilog
• Rollover log files automatically in an ASP.NET Core Web Application using Serilog
• Write your logs into database in an ASP.NET Core application using Serilog

Step 1: Create a Web Application in .NET Core

To get started we will create a new empty web application using the default template available in Visual Studio. Goto File -> New Project -> ASP.NET Core Web Application

Give a name for the application, leave the rest of the fields with default values and click Create

In the next window, select Web Application as a project template. Before you click on the Create button, make sure that you have selected the desired version of .NET Core in the dropdown shown at the top. Here, for this one, I selected .NET 5.0

You can also do this from .NET CLI using the following command

dotnet new web  --name SerilogMultipleTables

When the command is executed it will scaffold a new application using the MVC structure and then restores the necessary packages needed for the default template.

Writing logs to different files using Serilog in ASP.NET Core Web Application

Application log files play an important role in analyzing the bugs and for troubleshooting issues in an application. It’s worth noting that the log files are also used for writing information about events and other information that occurs when the application is running or serving requests in the case of a web application. Most application developers use a single file to log everything from errors, warnings, debug information, etc. There is no harm in following this approach, but the downside is that it will be harder for you to segregate information from the file easily. We can easily overcome this by maintaining multiple log files depending on the need. In this post, I am going to show how we can achieve this with Serilog

Serilog is a popular third party diagnostic logging library for .NET applications, I have already written some post about it and it’s usage already. If you are new to Serilog, please refer to those posts using the links given below.

• Implementing Logging in a .NET Core Web Application using Serilog
• Rollover log files automatically in an ASP.NET Core Web Application using Serilog
• Write your logs into database in an ASP.NET Core application using Serilog

Step 1 : Create a Web Application in .NET Core

Create a new ASP.NET Core MVC application using the below command

dotnet new mvc  --name MultiLogFileSample

You can also do this from Visual Studio by going into File -> New Project -> ASP .NET Core Web Application

When the command is executed it will scaffold a new application using the MVC structure and then restores the necessary packages needed for the default template.

By default, it will create two json files named, appsettings.json and appsettings.development.json. These are the configuration files for the application and are chosen based on the environment where your application is running. These files will have a default configuration as shown below basically sets the default level for logging.

{
  "Logging":{
    "LogLevel":{
      "Default":Information",
      "Microsoft":Warning",
      "Microsoft.Hosting.Lifetime":Information"
    }
  }
}

By default, it is set as Information, which writes a lot of data to the logs. This setting is very useful while we are developing the application, but we should set it higher severity levels when the application is deployed to higher environments. Since Serilog is not going to reference this section, we can safely remove this from the configuration files

Write your logs into database in an ASP.NET Core application using Serilog

In most scenarios, we normally use a flat-files for writing your logs or exception messages. But what if you write that information into the table in a database instead of a file. To implement this functionality, we can make use of third-party providers such as Serilog to log information in a SQL server database. Even though it's not a recommended approach, one can easily search the logs by executing SQL queries against the table.

Installing Packages

Serilog provides the functionality to write logs to different sources such as files, trace logs, database and the providers for these are called Serilog Sinks. To write logs to a table in a SQL Server database, you will need to add the following NuGet packages

Install-Package Serilog
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.MSSqlServer

The first package contains the core runtime, the second package can read the key under the "Serilog" section from a valid IConfiguration source and the last one is responsible for making the connection to the database and writing information into the log table.

Configuring Serilog 

Modify the appsettings.json file to add a new section called "Serilog". We will set up the connection string to the database, provide the name of the table and instruct Serilog to create the table if not found in the DB

"Serilog": {
    "MinimumLevel": "Error",
    "WriteTo": [
      {
        "Name": "MSSqlServer",
        "Args": {
          "connectionString": "Server=(localdb)\\MSSQLLocalDB;Database=Employee;Trusted_Connection=True;MultipleActiveResultSets=true",
          "tableName": "Logs",
          "autoCreateSqlTable": true
        }
      }
    ]
  },

Rollover log files automatically in an ASP.NET Core Web Application using Serilog

We all implement logging in whatever applications we develop and over time it will grow bigger by each passing day. If we don't control that over time we will run into problems, especially with the size. Most of the logging providers help to overcome this by using rolling log providers which automatically archives the current log file when it reaches specific criteria or a threshold and creates the new file to resume the logging. In this article we will how we can make of the rolling file provider supported by Serilog to implement this functionality.

Step 1: Install Packages

First, install the following packages

Install-Package Serilog
Install-Package Serilog.Extensions.Logging
Install-Package Serilog.Sinks.File

The first package has all the core functionalities of Serilog whereas the second one is a provider for the logging subsystem used by ASP.NET Core(Microsoft.Extensions.Logging). The third package is responsible for writing the log information to the file, manages the rollover and all the related functionalities

Step 2: Configure Serilog

Modify the appsettings.json file to include the path for the log file

 "Logging": {
    "LogPath": "logs//ex.log",
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  }

Then, configure the sink to read this path from the config file and set it up to write to the file by modifying the CreateHostBuilder method in Program.cs file

 
var configSettings = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .Build();

Log.Logger = new LoggerConfiguration()

    .WriteTo.File(configSettings["Logging:LogPath"], rollOnFileSizeLimit:true,fileSizeLimitBytes:10)
    .CreateLogger()

Implementing Logging in a .NET Core Web Application using Serilog

Setting up default logging

Create a new MVC application from Visual Studio. By default, Logging is enabled in the application via the ILogger interface. It has got some built-in providers for writing the log information to the console, event log as well as for third-party providers such as NLog, Serilog, etc.

For example, if you want to write logging information to the console window or event log, you will need to configure it in the CreateDefaultHostBuilder method in the Program.cs file as shown below.

 public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
    .ConfigureLogging(logging =>
    {
        logging.AddConsole();
        logging.AddEventLog();
    })
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup();
    });

In the case of a web app, you will get an ILogger from the DI container and use that object for writing into the configured log providers. Let's see how we can do that in the HomeController

First, create a private variable for the logger interface.

private readonly ILogger<HomeController> _logger;

Then modify the constructor as shown below

public HomeController(ILogger logger)
{
    _logger = logger;
}

Generate Help Pages for your ASP.NET Core Web Api using Swagger

For any Web API developers, documenting your API and its methods is of paramount importance. Because these are intended to be consumed third parties, they will find it hard to incorporate it without any proper documentation. Creating a document is a very tedious and time-consuming job and most of the developers are least worried about it. This is where tools like Swagger which can automatically generate the documentation for you by examining your API code. 

Adding Swagger

To set it up, you will need to add the below package to your project. This is can be done using the NuGet Package Manager in Visual Studio or by executing the following command in the Package Manager Console window

Install-Package Swashbuckle.AspNetCore

This package will add the following components

1. Swashbuckle.AspNetCore.Swagger : Middleware used to expose Swagger Document as a JSON endpoint
2. Swashbuckle.AspNetCore.SwaggerGen : a generator that builds the Swagger Document by looking into your controllers and action methods
3. Swashbuckle.AspNetCore.SwaggerUI : interprets the JSON outputted by the endpoint and builds an interactive UI

Configuring Swagger in your API Project

Adding and configuring Swagger is done in the Program class, first, you will need to import the following namespace

using Microsoft.OpenApi.Models;

Then in the ConfigureServices method, register the Swagger generator and define a document

services.AddSwaggerGen(c =>
 {
     c.SwaggerDoc("v1", new OpenApiInfo { Title = "Swagger Sample API V1", Version = "v1" });
 });

In the Configure method, add the two statements given below. The first one will enable the middleware to serve the generated document and the second one this document to show the interactive UI

Serializing enums as strings using System.Text.Json library in .NET Core 3.0

.NET Core 3.0 uses the System.Text.Json API by default for JSON serialization operations. Prior versions of .NET Core relied on JSON.NET, a third party library developed by Newtonsoft and the framework team decided to create a brand new library that can make use of the latest features in the language and the framework.

The new library has got support for all the new features that got introduced in the latest version of C#. And this was one of the main reasons behind the development of the new library because implementing these changes in JSON.NET meant a significant rewrite.

While serializing an object into JSON using the new library we can control various options such as casing, indentation, etc, but one notable omission is the support for enums by default. If you try to serialize an enum in .NET Core 3.0 with the default library, it will convert it into an integer value instead of the name of the enum.
For example, let consider the following model and see if what happens when we serialize it using the System.Text.Json library

public enum AddressType
{        
    HomeAddress,
    OfficeAddress,
    CommunicationAddress
    }

public class Employee
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public AddressType CommunicationPreference { get; set; }
}

and when you serialize it using the serialize method

List employees = new List{
    new Employee{
        FirstName = "Amal",
        LastName ="Dev",
        CommunicationPreference = AddressType.HomeAddress
    },
    new Employee{
        FirstName = "Dev",
        LastName ="D",
        CommunicationPreference = AddressType.CommunicationAddress
    },
    new Employee{
        FirstName = "Tris",
        LastName ="Tru",
        CommunicationPreference = AddressType.OfficeAddress
    }
}

JsonSerializer.Serialize(employees);

it will produce an output like the one given below

[ 
   { 
      "FirstName":"Amal",
      "LastName":"Dev",
      "CommunicationPreference":0
   },
   { 
      "FirstName":"Dev",
      "LastName":"D",
      "CommunicationPreference":2
   },
   { 
      "FirstName":"Tris",
      "LastName":"Tru",
      "CommunicationPreference":1
   }
]

Avoid conversion errors by using Custom Converters in System.Text.Json API(.NET Core 3.0)

One of the most common error encountered while doing JSON serialization and deserialization is the data type conversion errors. Till now, we were using the Json.NET library from Newtonsoft for performing the serialization and deserialization in .NET/ASP.NET/.NET Core, but in the latest iteration of .NET Core which is currently under preview, they have removed the dependency on Json.NET and introduced a new built-in library for doing the same. Along with that, the all new library that is going to be introduced with .NET Core 3.0 provides you to define custom converters that can be implemented to get rid of this kind of errors. If you are not aware of it, I have already written a couple of posts about it which you can refer to using the following links.

 Serializing and Deserializing Json in .NET Core 3.0 using System.Text.Json API

Step-By-Step Guide to Serialize and Deserialize JSON Using System.Text.Json

The new API is included with the System namespace and you don't need to add any NuGet package to get started with it. Along with the normally used methods for serializing/deserializing JSON, it also includes methods for supporting asynchronous programming. One of the known limitation in the v1 of the API is the limited support for the data types, given below is the list of currently supported types. For more detail, please refer this link Serializer API Document

• Array 
• Boolean
• Byte
• Char (as a JSON string of length 1)
• DateTime 
• DateTimeOffset 
• Dictionary<string, TValue> (currently just primitives in Preview 5)
• Double
• Enum (as integer for now)
• Int16
• Int32
• Int64
• IEnumerable 
• IList 
• Object (polymorhic mode for serialization only)
• Nullable < T >
• SByte
• Single
• String
• UInt16
• UInt32
• UInt64