Very fast C++17 logging library

git clone --recursive https://github.com/ChristianPanov/lwlog

I will cut short on the speed or code simplicity that every other logging library boasts about, and will leave them for later.
The actual importance of the library hides within its meaning to me. This library has served as an amazing journey.
I started it when I knew very little about C++, let alone software engineering, and when I still made no distinction between writing reusable pieces of code and crafting libraries.
I can say that I am proud of what I've made, for it has been a long journey comprised of not giving up and constantly daring to do the currently-impossible for me, according to my knowledge back then.
As you will see later in the documentation, I do claim this library to be very fast and to have very clean code. I do claim to have designed it in a good way. However, even if that weren't objectively true, that would make no difference in the importance of this work to me, I would still be just as proud of it.

No matter what you do or say, there will always be people who will hold a different view. It might always happen that you were in the wrong, or you could have done better. Anybody could take away the joy of your creation. However, no one can take away the hours you've spent crafting it, no one can take away the hours you've spent studying and learning in the process, no one can take away the times when you've felt like a god among men when you've faced the impossible, was stubborn enough to not give up, and actually did it. No one can take away any of that, any of the things which should make you proud. That's what real craftsmanship is - taking pride in what you do with all your heart and being too stubborn to give up on the currently-impossible.

• Clean and descriptive code
• Extremely fast synchronous logging (other logging libraries promise you speed by being asynchronous, which is not always a good idea, because it's hard to identify at what point the program crashed, causes more overhead and you can easily lose important messages, lwlog promises you both speed, as fast as synchronous logging can get, and keeping all your messages)
• High extensibility - very easy to add your own types of sinks and loggers
• Very configurable - lwlog uses policy classes which you can just plug in based on your needs. At the same time, convenient easy-to-use predefined types(convenience aliases) are made for the people who want simplicity without too much configuration. Most of the time you will be just fine with using the predefined types

• Written in modern C++17
• Very fast synchronous logging
• Forward(default, linear) and deferred(non-linear) logging mechanisms
• Verbosity levels
• Various log sinks
  • Console (stdout, stderr)
  • File
  • Custom sinks (it's extremely easy to add your own custom sinks)
• Support for both compile-time and runtime sink configuration
• Custom sink configuration - each sink can have a unique log pattern and verbosity level
• Log formatting according to a custom user-defined pattern
• Source metainformation attributes
• Custom attributes
• Global logger registry

• Binary logging
• Coloring based on log level
• Proper CMake to replace the Premake which is currently used
• Cross-platform colors
• Fmt-like formatting
• Meta-logging (also known as structured logging)

The benchmarks are still limited since there still aren't benchmarks for thread-safe logging, async logging, and loggers which sink to a file (they will be done)
Benchmarked with picobench(https://github.com/iboB/picobench)

===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
            lwlog_bench * |       8 |     0.008 |     950 |      - |  1052631.6
             spdlog_bench |       8 |     0.403 |   50325 | 52.974 |    19870.8
            lwlog_bench * |      64 |     0.071 |    1107 |      - |   902679.8
             spdlog_bench |      64 |     3.232 |   50506 | 45.591 |    19799.5
            lwlog_bench * |     512 |     0.866 |    1691 |      - |   591019.3
             spdlog_bench |     512 |    34.119 |   66638 | 39.385 |    15006.3
            lwlog_bench * |    4096 |     5.150 |    1257 |      - |   795386.1
             spdlog_bench |    4096 |   311.846 |   76134 | 60.556 |    13134.7
            lwlog_bench * |    8192 |     7.365 |     898 |      - |  1112348.3
             spdlog_bench |    8192 |   610.478 |   74521 | 82.894 |    13419.0
===============================================================================

===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
            lwlog_bench * |       8 |     0.010 |    1262 |      - |   792079.2
             spdlog_bench |       8 |     1.875 |  234337 |185.614 |     4267.3
            lwlog_bench * |      64 |     0.078 |    1225 |      - |   816326.5
             spdlog_bench |      64 |    20.370 |  318275 |259.816 |     3141.9
            lwlog_bench * |     512 |     0.691 |    1348 |      - |   741384.3
             spdlog_bench |     512 |   592.973 | 1158149 |858.634 |      863.4
            lwlog_bench * |    4096 |     8.363 |    2041 |      - |   489788.1
             spdlog_bench |    4096 |  3186.113 |  777859 |380.986 |     1285.6
            lwlog_bench * |    8192 |     9.786 |    1194 |      - |   837148.5
             spdlog_bench |    8192 |  2384.691 |  291100 |243.694 |     3435.2
===============================================================================

===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
            lwlog_bench * |       8 |     0.001 |      75 |      - | 13333333.3
             spdlog_bench |       8 |     0.002 |     200 |  2.667 |  5000000.0
            lwlog_bench * |      64 |     0.004 |      68 |      - | 14545454.5
             spdlog_bench |      64 |     0.024 |     368 |  5.364 |  2711864.4
            lwlog_bench * |     512 |     0.034 |      67 |      - | 14883720.9
             spdlog_bench |     512 |     0.117 |     228 |  3.395 |  4383561.6
            lwlog_bench * |    4096 |     0.411 |     100 |      - |  9975645.4
             spdlog_bench |    4096 |     1.345 |     328 |  3.277 |  3044447.7
            lwlog_bench * |    8192 |     1.341 |     163 |      - |  6108874.0
             spdlog_bench |    8192 |     2.273 |     277 |  1.695 |  3604206.1
===============================================================================

===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
   lwlog_deferred_bench * |       8 |     0.001 |     100 |      - | 10000000.0
      lwlog_forward_bench |       8 |     0.010 |    1212 | 12.125 |   824742.3
   lwlog_deferred_bench * |      64 |     0.005 |      79 |      - | 12549019.6
      lwlog_forward_bench |      64 |     0.125 |    1956 | 24.549 |   511182.1
   lwlog_deferred_bench * |     512 |     0.050 |      97 |      - | 10240000.0
      lwlog_forward_bench |     512 |     0.662 |    1292 | 13.238 |   773530.7
   lwlog_deferred_bench * |    4096 |     0.423 |     103 |      - |  9674067.1
      lwlog_forward_bench |    4096 |     6.427 |    1569 | 15.179 |   637331.2
   lwlog_deferred_bench * |    8192 |     2.221 |     271 |      - |  3688262.6
      lwlog_forward_bench |    8192 |     9.877 |    1205 |  4.447 |   829443.6
===============================================================================

Registry
└── Logger
    └── Sink
        └── Record

The architecture of lwlog is very simple, it's divided into three main modules - the registry, the logger, the sinks, and the record.

#include "lwlog.h"

int main()
{
	auto console = std::make_shared<
		lwlog::logger<
			lwlog::default_log_policy,
			lwlog::default_storage_policy,
			lwlog::single_threaded_policy,
			lwlog::sinks::stdout_sink
			>
		>("CONSOLE");
	
	console->set_level_filter(lwlog::level::info | lwlog::level::debug | lwlog::level::critical);
	console->set_pattern("[%T] [%n] [%l]: %v");
	console->critical("First critical message");
	
	return 0;
}

In the file lwlog.h you can see several convenience aliases at your disposal.
They are intended for ease of use, so I encourage you to use them instead of the more complex way of creating loggers directly through the logger class.
They are predefined with default configurations, so unless you need more special functionality, stick to using them.

#include "lwlog.h"

int main()
{
	auto basic = std::make_shared<lwlog::basic_logger<lwlog::sinks::stdout_sink>>("CONSOLE");
	
	auto console = std::make_shared<lwlog::console_logger>("CONSOLE");
	auto file = std::make_shared<lwlog::file_logger>("FILE", "C:/Users/user/Desktop/LogFolder/LOGS.txt");
	
	auto null = std::make_shared<lwlog::null_logger>("LOGGER");
	
	return 0;
}

Both the sinks and the logger classes expect a threading policy as a template parameter, which will determine whether they will be thread-safe or not. However, if you want to use the convenience aliases I mentioned above, you need to keep in mind they are not thread-safe.
And for that reason, all of them have a thread-safe analog with the same name and an _mt suffix.
lwlog::basic_logger_mt, lwlog::console_logger_mt, lwlog::file_logger_mt, lwlog::null_logger_mt

#include "lwlog.h"

int main()
{
	auto console = std::make_shared<
		lwlog::logger<
			lwlog::default_log_policy,
			lwlog::default_storage_policy,
			lwlog::single_threaded_policy,
			lwlog::sinks::stdout_sink
			>
		>("CONSOLE");
	
	return 0;
}

Deferred logging provides extremely low latency, however, it's only applicable when you don't need the logs to be outputted immediately.
The low latency comes from the fact that with deferred logging a log call doesn't sink and doesn't format anything, it only stores data.
This data is sent to a sink and formatted at a later stage, only when needed. There is one problem with it - all log information will be lost if there is an application crash and you haven't sinked the deferred logs.

#include "lwlog.h"

int main()
{
	auto console = std::make_shared<
		lwlog::logger<
			lwlog::deferred_log_policy,
			lwlog::default_storage_policy,
			lwlog::single_threaded_policy,
			lwlog::sinks::stdout_sink
			>
		>("CONSOLE");
	
	console->critical("First critical message");
	console->info("First info message");
	console->debug("First debug message");
	
	console->sink_logs();
	
	return 0;
}

By calling sink_logs() you sink all the logs that are deferred for later use to their respective sinks with their respective patterns.
If sink_logs() is called by a forward logging logger it will emit a warning.

Formatting is handled with a pay for what you need approach.
The user can set a pattern, by which the log messages will be formatted. This pattern is internal to the library language, which is a sequence of formatting flags, alignment specifications(optional), characters, and color flags(optional). It allows flexibility in terms of configuring the output information in the most appropriate for the situation way, allowing as meaningful log output as possible.
How is formatting done? - A pattern is set, and then it gets compiled by the library. Compilation is done in the lwlog::details::pattern class. It first parses the pattern and extracts the formatting flags, which are then used to retrieve only the formatters the pattern will need. It also parses the alignment specifications and extracts all the needed information for the alignments. In the end, all the retrieved formatters are called on the pattern and all formatting flags are replaced with their corresponding values and their corresponding alignment specifications(if any).

lwlog gives you the ability to get source code metainformation in the form of attributes.
One can get the current line on which the log function is called, the file path in which it is called, or the function name in which it is called, and all of that without macros.
It is possible because of compiler intrinsics, which were first introduced in GCC, and now are also implemented in MSVC.
lwlog doesn't use C++20's std::source_location, because I don't want to force users to use the new standard. Instead, the only requirement is to have a newer version of Visual Studio (>= 1927), which implements the needed intrinsics.
If a newer version is not present, the metainformation flags will result into nothing.

Alignment specifications are individual to an attribute, and they contain an alignment side, width, and an optional fill character, which by default, if not specified, is an empty space.

#include "lwlog.h"

int main()
{
	auto console = std::make_shared<lwlog::console_logger>("CONSOLE");
	console->set_pattern("[%T] [%n] [:^12%l]: %v");

	console->info("First info message");
	console->critical("First critical message");
	
	return 0;
}

[19:44:50] [CONSOLE] [    info    ]: First info message
[19:44:50] [CONSOLE] [  critical  ]: First critical message

Color flags are used for coloring a pattern. Each color flag is scoped.

#include "lwlog.h"

int main()
{
	auto console = std::make_shared<console_logger>("CONSOLE");
	console->set_pattern(".br_red([%T] [%n]) .green([%l]): .br_cyan(%v)");

	console->critical("First critical message");
	
	return 0;
}

An attribute is an object, which contains a flag, a value(a reference to some variable) and a callback function.
The callback function plays the role of a string conversion function, therefore, it always needs to return a string.
If a callback function is not provided, the attribute will use the default one, which handles only arithmetic types and strings.
When the attribute needs to be formatted, the callback function is called on the value and it replaces the flag wherever it appears in the formatting pattern.

#include "lwlog.h"

int main()
{
	std::string current_status = "inactive";
	
	auto console = std::make_shared<lwlog::console_logger>("CONSOLE");
	console->add_attribute("{status}", current_status);
	console->set_pattern("{status} --- [%T] [%n] [%l]: %v");
	
	current_status = "active";
	console->info("First info message");
	
	return 0;
}

active --- [19:44:50] [CONSOLE] [info]: First critical message

Currently, an attribute can contain a reference to only a couple of types - int, float, double, std::string and std::string_view.
The reason for this is because more possible types in std::variant create more overhead, so I've tried to select the most probable types a user can use for values.

#include "lwlog.h"

int main()
{
	auto logger = std::make_shared<
		lwlog::logger<
			lwlog::default_log_policy,
			lwlog::default_storage_policy,
			lwlog::single_threaded_policy,
			lwlog::sinks::stdout_sink, 
      			lwlog::sinks::file_sink
			>
		>("LOGGER", "C:/Users/user/Desktop/LogFolder/LOGS.txt");

	// Color attributes will be ignored for the file sink
	logger->set_pattern("[%T] [%n] [%l]: %v");
	// Log message will be distributed to both sinks
	logger->critical("First critical message");
	
	return 0;
}

#include "lwlog.h"

int main()
{
	auto console_sink 	= std::make_shared<lwlog::sinks::stdout_sink<lwlog::single_threaded_policy>>();
	auto file_sink 		= std::make_shared<lwlog::sinks::file_sink<lwlog::single_threaded_policy>>("C:/Users/user/Desktop/LogFolder/LOGS.txt");
	lwlog::sink_list sinks 	= { console_sink, file_sink };

	auto logger_iterator 	= std::make_shared<lwlog::null_logger>("ITERATOR", sinks.begin(), sinks.end());
	auto logger_sink_list 	= std::make_shared<lwlog::null_logger>("SINK_LIST", sinks);
	auto logger_single_sink = std::make_shared<lwlog::null_logger>("SINGLE_SINK", console_sink);
	
	auto logger_combined = std::make_shared<
		lwlog::logger<
			lwlog::default_log_policy,
			lwlog::static_storage_policy,
			lwlog::single_threaded_policy,
			lwlog::sinks::stdout_sink
			>
		>("COMBINED", file_sink);

	return 0;
}

As already mentioned, lwlog is extremely easy to extend. Let's give an example with sinks.
To create your own sink, all you have to do is inherit from lwlog::sinks::sink and implement a sink_it() function, which takes a const details::record& as a parameter. That's it.

namespace lwlog::sinks
{
	template<typename ThreadingPolicy>
	class stdout_sink 
		: public sink<colored_policy, ThreadingPolicy>
		, private details::console_writer
	{
	private:
		using sink_t = sink<colored_policy, ThreadingPolicy>;
		
	public:
		stdout_sink();
		void sink_it(const details::record& record) override;
	};

	template<typename ThreadingPolicy>
	stdout_sink<ThreadingPolicy>::stdout_sink()
		: details::console_writer(stdout)
	{}

	template<typename ThreadingPolicy>
	void stdout_sink<ThreadingPolicy>::sink_it(const details::record& record)
	{
		details::console_writer::write(sink_t::m_pattern.compile(record));
	}
}

Here we inherit from the sink base class and configure it to be colored. Whether it's thread-safe or not is left up to the one using the sink.
The color policy could be either colored(lwlog::colored_policy) or non-colored (lwlog::uncolored_policy).
The non-colored policy will drop the color flags in the pattern instead of processing them, but will not ignore them. Using lwlog::colored_policy is most suitable for console sinks, since it relies on console-specific color codes.
We only need the sink_it() function. It can do whatever you want it to do - write to console, write to file, write to file in some fancy way, write to another application, etc.
Like in the example, you can either use some kind of a writer class, which handles the actual writing, or you can directly handle the writing in the function.
The compiled and formatted message is received with sink_t::m_pattern.compile(record). We access the pattern member from the sink base class and then compile it with the log message.

#include "sink.h"
#include "policy/sink_color_policy.h"

namespace lwlog::sinks
{
	template<typename ThreadingPolicy>
	class new_custom_sink
		: public sink<colored_policy, ThreadingPolicy>
	{
		using sink_t = sink<colored_policy, ThreadingPolicy>;
	public:
		void sink_it(const details::record& record) override
		{
			// sink message to somewhere
		}
	};
}

The default logger is a logger object delivered to you by the library itself.
It's not registered in the logger registry, it's global, it has default configuration and is NOT thread-safe, sinks to stdout.
It's convenient if you just need the logging functionality, but don't want to create loggers by yourself. You can access it from everywhere in your application.\ Before using it, you need to call a function which initializes it.

#include "lwlog.h"

int main()
{
	lwlog::init_default_logger();
	
	lwlog::info("Info message");
	lwlog::warning("Warning message");
	lwlog::error("Error message");
	lwlog::critical("Critical message");
	lwlog::debug("Debug message");

	lwlog::set_level_filter(lwlog::level::debug | lwlog::level::critical);
	lwlog::info("Will not be displayed");

	lwlog::set_pattern("[%T] [%n] [%l]: %v");
	lwlog::debug("Will be displayed according to the new pattern");

	return 0;
}

[22, 20:00:15] [info] [DEFAULT]: Info message
[22, 20:00:15] [warning] [DEFAULT]: Warning message
[22, 20:00:15] [error] [DEFAULT]: Error message
[22, 20:00:15] [critical] [DEFAULT]: Critical message
[22, 20:00:15] [debug] [DEFAULT]: Debug message
[20:00:15] [DEFAULT] [debug]: Will be displayed according to the new pattern

In order to apply a logger function to all loggers present in the registry, you can use the function lwlog::apply_to_all() in such manner

#include "lwlog.h"

int main()
{
	auto console = std::make_shared<lwlog::console_logger>("CONSOLE");
	auto file = std::make_shared<lwlog::file_logger>("FILE", "C:/Users/user/Desktop/LogFolder/LOGS.txt");
	
	//Pattern will be applied to all loggers present in the registry
	lwlog::apply_to_all([](lwlog::logger_ptr logger)
		{
			logger->set_pattern("[%T] [%n] [%l]: %v");
		});
	
	return 0;
}

#include "lwlog.h"

int main()
{
	auto console = std::make_shared<lwlog::console_logger>("CONSOLE");
	
	lwlog::get("CONSOLE")->critical("First critical message");
	
	return 0;
}

If you want to be able to turn off logging completely, you can use the preprocessor directives. These directives use the default logger and are present in the lwlog.h file.
They will log unless you disable logging with LWLOG_DISABLE(should always be at the very top of the file), or you switch off a specific logging level.
Levels can be switched off at runtime as well, just by using the LWLOG_SET_LEVEL_FILTER directive.
If logging is disabled, the directives expand to nothing.

#define LWLOG_DISABLE
#define LWLOG_ERROR_OFF
#include "lwlog.h"

int main()
{
	LWLOG_INIT_DEFAULT_LOGGER();
	LWLOG_SET_PATTERN("[%T] [%n] [%l]: %v");
	LWLOG_SET_LEVEL_FILTER(lwlog::level::error | lwlog::level::critical);
	LWLOG_ERROR("First error message");
	return 0;
}

The tweakme.h file contains macros which serve the purpose of configurations.
Depending on your needs, you can switch on and off certain features. Switching them off will completely remove them from the lwlog code which will increase performance and reduce overhead, because you shouldn't pay for what you don't need.

So how does lwlog achieve this performance? In the following section, I will break down all the performance-enhancing decisions that I've made.

Formatting is usually the bottleneck in logging solutions and for that reason, it's usually handled on a background thread so it doesn't impede performance.
However, because of lwlog's synchronous nature, we cannot take that route, and thus cannot take any liberties in how the compilation process of the pattern is done.
The formatting pattern in question is parsed completely off the log call site, and all that's left for the log call functions is to do the replacement of the flags with their corresponding values. That way we do not burden every log call with doing the extra work of parsing the pattern every time, and it's parsed only once.
The same goes for colors. They are only processed once right after the pattern flags are parsed.
Pattern compilation process:

1. A pattern is set
2. The flags are parsed and only the needed formatters are created and pushed in a storage
3. The alignment specifications are parsed for all the needed information such as alignment side, width, and fill character
4. All color flags are processed
5. When a log function is called, the formatters in the storage are called on the pattern with their appropriate alignment specifications

I/O in logging solutions is the second biggest, if not the biggest, performance bottleneck. Spewing data in the form of a human-readable medium(text) is heavy-duty.
The best that could be done as of now, is manual buffering.
With manual buffering, I manually increase the stream buffering with a size of 2^22 bytes, bigger than the default one(512 bytes), which improves the performance of output to stdout, stderr a lot.

std::setvbuf(stdout, NULL, _IOFBF, 4194304);
std::fwrite("Hello, World!", 14, 1, stdout);

This example here shows how this is achieved. This is as fast as console output can get.

Time is handled in a special way. First off, since std::chrono is not as performant on Windows as it is on Linux, a platform-dependent approach, which proves to be much faster than its standard portable counterpart, is taken for Windows.
It's taken even further. For some reason, getting the local time with std::chrono is faster than getting the UTC, and with the Windows API it's the opposite - getting the gmtime is faster than getting the UTC, so each implementation initially gets the faster of the two, and then arithmetically processes the time to the desired time format(either local time or UTC)

For those of you who happen to not know, heuristics are logical shortcuts, approximate assumptions, which trade optimality, completeness, accuracy, or precision for speed.
One example could be that lwlog does not use exception handling and performs almost no checks. That means that misusing the library could result in undefined behavior, memory leaks, or crashes. Fortunately, the design is simple enough to make it hard to misuse it, but all in all, it will not hold your hand if you do not use it properly.
To further the example, lwlog assumes that the formatting pattern you've written is correct, with no syntax errors, so it doesn't perform any syntax checks which would impact the performance.
These heuristics, which are present in almost any part of the library, benefit the performance greatly.
NOTE: Every library could be misused, some more than others, but if we want speed, we as library crafters must take the responsibility to provide a simple enough interface, as straight-forward and as intuitive as possible, so the client would not misuse the library unless it's intentional.