Utf8 based StreamReader for high performance text processing.

Avoiding unnecessary string allocation is a fundamental aspect of recent .NET performance improvements. Given that most file and network data is in UTF8, features like JsonSerializer and IUtf8SpanParsable, which operate on UTF8-based data, have been added. More recently, methods like Split, which avoids allocations, have also been introduced.

However, for the most common use case of parsing strings delimited by newlines, only the traditional StreamReader is provided, which generates a new String for each line, resulting in a large amount of allocations.

Read simple 1000000 lines text

Incredibly, there is a 240,000 times difference!

While it is possible to process data in UTF8 format using standard classes like PipeReader and SequenceReader, they are generic librardies, so properly handling newline processing requires considerable effort(Handling BOM and Multiple Types of Newline Characters).

Utf8StreamReader provides a familiar API similar to StreamReader, making it easy to use, while its ReadLine-specific implementation maximizes performance.

This library is distributed via NuGet, supporting .NET Standard 2.1, .NET 6(.NET 7) and .NET 8 or above.

PM> Install-Package Utf8StreamReader

The basic API involves using var streamReader = new Utf8StreamReader(stream); and then ReadOnlyMemory<byte> line = await streamReader.ReadLineAsync();. When enumerating all lines, you can choose from three styles:

using Cysharp.IO; // namespace of Utf8StreamReader

public async Task Sample1(Stream stream)
{
    using var reader = new Utf8StreamReader(stream);

    // Most performant style, similar as System.Threading.Channels
    while (await reader.LoadIntoBufferAsync())
    {
        while (reader.TryReadLine(out var line))
        {
            // line is ReadOnlyMemory<byte>, deserialize UTF8 directly.
            _ = JsonSerializer.Deserialize<Foo>(line.Span);
        }
    }
}

public async Task Sample2(Stream stream)
{
    using var reader = new Utf8StreamReader(stream);

    // Classical style, same as StreamReader
    ReadOnlyMemory<byte>? line = null;
    while ((line = await reader.ReadLineAsync()) != null)
    {
        _ = JsonSerializer.Deserialize<Foo>(line.Value.Span);
    }
}

public async Task Sample3(Stream stream)
{
    using var reader = new Utf8StreamReader(stream);

    // Most easiest style, use async streams
    await foreach (var line in reader.ReadAllLinesAsync())
    {
        _ = JsonSerializer.Deserialize<Foo>(line.Span);
    }
}

From a performance perspective, Utf8StreamReader only provides asynchronous APIs.

Theoretically, the highest performance can be achieved by combining LoadIntoBufferAsync and TryReadLine in a double while loop. This is similar to the combination of WaitToReadAsync and TryRead in Channels.

ReadLineAsync, like StreamReader.ReadLine, returns null to indicate that the end has been reached.

ReadAllLinesAsync returns an IAsyncEnumerable<ReadOnlyMemory<byte>>. Although there is a performance difference, it is minimal, so this API is ideal when you want to use it easily.

All asynchronous methods accept a CancellationToken and support cancellation.

For a real-world usage example, refer to StreamMessageReader.cs in Cysharp/Claudia, a C# SDK for Anthropic Claude, which parses server-sent events.

The ReadOnlyMemory<byte> returned from ReadLineAsync or TryReadLine is only valid until the next call to LoadIntoBufferAsync or TryReadLine or ReadLineAsync. Since the data is shared with the internal buffer, it may be overwritten, moved, or returned on the next call, so the safety of the data cannot be guaranteed. The received data must be promptly parsed and converted into a separate object. If you want to keep the data as is, use ToArray() to convert it to a byte[].

This design is similar to System.IO.Pipelines.

You can convert it to a Utf8TextReader that extracts ReadOnlyMemory<char> or string. Although there is a conversion cost, it is still fast and low allocation, so it can be used as an alternative to StreamReader.

After converting with AsTextReader(), all the same methods (TryReadLine, ReadLineAsync, LoadIntoBufferAsync, ReadAllLinexAsync) can be used.

using var sr = new Cysharp.IO.Utf8StreamReader(ms).AsTextReader();
while (await sr.LoadIntoBufferAsync())
{
    while (sr.TryReadLine(out var line))
    {
        // line is ReadOnlyMemory<char>, you can add to StringBuilder or other parsing method.

        // If you neeed string, ReadOnlyMemory<char>.ToString() build string instance
        // string str = line.ToString();
    }
}

You can perform text processing without allocation, such as splitting ReadOnlySpan<char> using MemoryExtensions.Split, and concatenate the results using StringBuilder's Append/AppendLine(ReadOnlySpan<char>). This way, string-based processing can be done with much lower allocation compared to StreamReader.

When a string is needed, you can convert ReadOnlyMemory<char> to a string using ToString(). Even with the added string conversion, the performance is higher than StreamReader, so it can be used as a better alternative.

Similar to StreamReader, Utf8StreamReader has the ability to open a FileStream by accepting a string path.

public Utf8StreamReader(string path, FileOpenMode fileOpenMode = FileOpenMode.Throughput)
public Utf8StreamReader(string path, int bufferSize, FileOpenMode fileOpenMode = FileOpenMode.Throughput)
public Utf8StreamReader(string path, FileStreamOptions options)
public Utf8StreamReader(string path, FileStreamOptions options, int bufferSize)

Unfortunately, the FileStream used by StreamReader is not optimized for modern .NET. For example, when using FileStream with asynchronous methods, it should be opened with useAsync: true for optimal performance. However, since StreamReader has both synchronous and asynchronous methods in its API, false is specified. Additionally, although StreamReader itself has a buffer and FileStream does not require a buffer, the buffer of FileStream is still being utilized.

It is difficult to handle FileStream correctly with high performance. By specifying a string path, the stream is opened with options optimized for Utf8StreamReader, so it is recommended to use this overload rather than opening FileStream yourself. The following is a benchmark of FileStream.

Utf8StreamReader opens FileStream with the following settings:

var useAsync = (fileOpenMode == FileOpenMode.Scalability);
new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize: 1, useAsync: useAsync)

Due to historical reasons, the options for FileStream are odd, but by setting bufferSize to 1, you can avoid the use of internal buffers. FileStream has been significantly revamped in .NET 6, and by controlling the setting of this option and the way Utf8StreamReader is called as a whole, it can function as a thin wrapper around the fast RandomAccess.ReadAsync, allowing you to avoid most of the overhead of FileStream.

FileOpenMode is a proprietary option of Utf8StreamReader.

public enum FileOpenMode
{
    Scalability,
    Throughput
}

In a Windows environment, the table in the IO section of the Performance Improvements in .NET 6 blog shows that throughput decreases when useAsync: true is used.

By setting Utf8StreamReader to FileOpenMode.Scalability, true async I/O is enabled and scalability is prioritized. If set to FileOpenMode.Throughput, it internally becomes sync-over-async and consumes the ThreadPool, but reduces the overhead of asynchronous I/O and improves throughput.

If frequently executed within a server application, setting it to Scalability, and for batch applications, setting it to Throughput will likely yield the best performance characteristics. The default is Throughput.

In Utf8StreamReader, by carefully adjusting the buffer size on the Utf8StreamReader side, the performance difference is minimized. Please refer to the above benchmark results image for specific values.

For overloads that accept FileStreamOptions, the above settings are not reflected, so please adjust them manually.

Utf8StreamReader is a class that supports reuse. By calling Reset(), the Stream and internal state are released. Using Reset(Stream), it can be reused with a new Stream.

The constructor accepts int bufferSize and bool leaveOpen as parameters.

int bufferSize defaults to 65536 and the buffer is rented from ArrayPool<byte>. If the data per line is large, changing the buffer size may improve performance. When the buffer size and the size per line are close, frequent buffer copy operations occur, leading to performance degradation.

bool leaveOpen determines whether the internal Stream is also disposed when the object is disposed. The default is false, which means the Stream is disposed.

Additionally, there are init properties that allow changing the option values for ConfigureAwait and SkipBom.

bool ConfigureAwait { init; } allows you to specify the value for ConfigureAwait(bool continueOnCapturedContext) when awaiting asynchronous methods internally. The default is false.

bool SkipBom { init; } determines whether to identify and skip the BOM (Byte Order Mark) included at the beginning of the data during the first read. The default is true, which means the BOM is skipped.

Currently, this is not an option, but Utf8StreamReader only determines CRLF(\r\n) or LF(\n) as newline characters. Since environments that use CR(\r) are now extremely rare, the CR check is omitted for performance reasons. If you need this functionality, please let us know by creating an Issue. We will consider adding it as an option

Unity, which supports .NET Standard 2.1, can run this library. Since the library is only provided through NuGet, it is recommended to use NuGetForUnity for installation.

For detailed instructions on using NuGet libraries in Unity, please refer to the documentation of Cysharp/R3 and other similar resources.

This library is under the MIT License.