It looks like memory leak in Utf16ValueStringBuilder.Insert

ZString

Zero Allocation StringBuilder for .NET Core and Unity.

Struct StringBuilder to avoid allocation of builder itself
Rent write buffer from ThreadStatic or ArrayPool
All append methods are generics(Append<T>(T value)) and write to buffer directly instead of concatenate value.ToString
T1~T16 AppendFormat(AppendFormat<T1,...,T16>(string format, T1 arg1, ..., T16 arg16) avoids boxing of struct argument
Also T1~T16 Concat(Concat<T1,...,T16>(T1 arg1, ..., T16 arg16)) avoid boxing and value.ToString allocation
Convenient ZString.Format/Concat/Join methods can replace instead of String.Format/Concat/Join
Can build both Utf16(Span<char>) and Utf8(Span<byte>) directly
Can use inner buffer to avoid allocate final string
Integrated with Unity TextMeshPro to avoid string allocation

This graph compares following codes.

"x:" + x + " y:" + y + " z:" + z
ZString.Concat("x:", x, " y:", y, " z:", z)
string.Format("x:{0} y:{1} z:{2}", x, y, z)
ZString.Format("x:{0} y:{1} z:{2}", x, y, z)
new StringBuilder(), Append(), .ToString()
ZString.CreateStringBuilder(), Append(), .ToString()

"x:" + x + " y:" + y + " z:" + z is converted to String.Concat(new []{ "x:", x.ToString(), " y:", y.ToString(), " z:", z.ToString() }) by C# compiler. It has each .ToString allocation and params array allocation. string.Format calls String.Format(string, object, object, object) so each arguments causes int -> object boxing.

All ZString methods only allocate final string. Also, ZString has enabled to access inner buffer so if output target has stringless api(like Unity TextMeshPro's SetCharArray), you can achieve completely zero allocation.

The blog post of detailed explanation by author: medium@neuecc/ZString

Related project for loggin using with ZString, Cysharp/ZLogger - Zero Allocation Text/Strcutured Logger.

Getting Started

For .NET Core, use NuGet.

PM> Install-Package ZString

For Unity, check the releases page, download ZString.Unity.unitypackage.

using Cysharp.Text; // namespace

async void Example(int x, int y, int z)
{
    // same as x + y + z
    _ = ZString.Concat(x, y, z);

    // also can use numeric format strings
    _ = ZString.Format("x:{0}, y:{1:000}, z:{2:P}",x, y, z);

    _ = ZString.Join(',', x, y, z);

    // for Unity, direct write(avoid string allocation completely) to TextMeshPro
    tmp.SetTextFormat("Position: {0}, {1}, {2}", x, y, z);

    // create StringBuilder
    using(var sb = ZString.CreateStringBuilder())
    {
        sb.Append("foo");
        sb.AppendLine(42);
        sb.AppendFormat("{0} {1:.###}", "bar", 123.456789);

        // and build final string
        var str = sb.ToString();

        // for Unity, direct write to TextMeshPro
        tmp.SetText(sb);

        // write to destination buffer
        sb.TryCopyTo(dest, out var written);
    }

    // prepare format, return value should store to field(like RegexOptions.Compile)
    var prepared = ZString.PrepareUtf16<int, int>("x:{0}, y:{1:000}");
    _ = prepared.Format(10, 20);

    // C# 8.0, Using declarations
    // create Utf8 StringBuilder that build Utf8 directly to avoid encoding
    using var sb2 = ZString.CreateUtf8StringBuilder();

    sb2.AppendFormat("foo:{0} bar:{1}", x, y);

    // directly write to steam or dest to avoid allocation
    await sb2.WriteToAsync(stream);
    sb2.CopyTo(bufferWritter);
    sb2.TryCopyTo(dest, out var written);
}

Reference

static class ZString

method	returns	description
CreateStringBuilder()	Utf16ValueStringBuilder	Create the Utf16 string StringBuilder.
CreateStringBuilder(bool notNested)	Utf16ValueStringBuilder	Create the Utf16 string StringBuilder, when true uses thread-static buffer that is faster but must return immediately.
CreateUtf8StringBuilder()	Utf8ValueStringBuilder	Create the Utf8(`Span<byte>`) StringBuilder.
CreateUtf8StringBuilder(bool notNested)	Utf8ValueStringBuilder	Create the Utf8(`Span<byte>`) StringBuilder, when true uses thread-static buffer that is faster but must return immediately.
`Join(char/string, T[]/IE<T>)`	string	Concatenates the elements of an array, using the specified seperator between each element.
`PrepareUtf16<T1,..,T16>(string)`	`Utf16PreparedFormat<T1,...,T16>`	Prepare string format to avoid parse template in each operation.
`PrepareUtf8<T1,..,T16>(string)`	`Utf8PreparedFormat<T1,...,T16>`	Prepare string format to avoid parse template in each operation.
`Concat<T1,..,T16>(T1,..,T16)`	string	Concatenates the string representation of some specified values.
`Format<T1,..,T16>(string, T1,..,T16)`	string	Replaces one or more format items in a string with the string representation of some specified values.
`Utf8Format<T1,..,T16>(IBufferWriter<byte>, T1,..,T16)`	void	Replaces one or more format items in a string with the string representation of some specified values.

struct Utf16ValueStringBuilder : IBufferWriter<char>, IDisposable

method	returns	description
Length	int	Length of written buffer.
AsSpan()	`ReadOnlySpan<char>`	Get the written buffer data.
AsMemory()	`ReadOnlyMemory<char>`	Get the written buffer data.
AsArraySegment()	`ArraySegment<char>`	Get the written buffer data.
Dispose()	void	Return the inner buffer to pool.
`Append<T>(T value)`	void	Appends the string representation of a specified value to this instance.
`Append<T>(T value, string format)`	void	Appends the string representation of a specified value to this instance with numeric format strings.
`AppendJoin(char/string, T[]/IE<T>)`	void	Concatenates and appends the elements of an array, using the specified seperator between each element.
`AppendLine()`	void	Appends the default line terminator to the end of this instance.
`AppendLine<T>(T value)`	void	Appends the string representation of a specified value followed by the default line terminator to the end of this instance.
`AppendLine<T>(T value, string format)`	void	Appends the string representation of a specified value with numeric format strings followed by the default line terminator to the end of this instance.
`AppendFormat<T1,..,T16>(string, T1,..,T16)`	void	Appends the string returned by processing a composite format string, each format item is replaced by the string representation of arguments.
`TryCopyTo(Span<char>, out int)`	bool	Copy inner buffer to the destination span.
ToString()	string	Converts the value of this instance to a System.String.
GetMemory(int sizeHint)	`Memory<char>`	IBufferWriter.GetMemory.
GetSpan(int sizeHint)	`Span<char>`	IBufferWriter.GetSpan.
Advance(int count)	void	IBufferWriter.Advance.
static `RegisterTryFormat<T>(TryFormat<T>)`	void	Register custom formatter.

struct Utf8ValueStringBuilder : IBufferWriter<byte>, IDisposable

method	returns	description
Length	int	Length of written buffer.
AsSpan()	`ReadOnlySpan<char>`	Get the written buffer data.
AsMemory()	`ReadOnlyMemory<char>`	Get the written buffer data.
AsArraySegment()	`ArraySegment<char>`	Get the written buffer data.
Dispose()	void	Return the inner buffer to pool.
`Append<T>(T value)`	void	Appends the string representation of a specified value to this instance.
`Append<T>(T value, StandardFormat format)`	void	Appends the string representation of a specified value to this instance with numeric format strings.
`AppendJoin(char/string, T[]/IE<T>)`	void	Concatenates and appends the elements of an array, using the specified seperator between each element.
`AppendLine()`	void	Appends the default line terminator to the end of this instance.
`AppendLine<T>(T value)`	void	Appends the string representation of a specified value followed by the default line terminator to the end of this instance.
`AppendLine<T>(T value, StandardFormat format)`	void	Appends the string representation of a specified value with numeric format strings followed by the default line terminator to the end of this instance.
`AppendFormat<T1,..,T16>(string, T1,..,T16)`	void	Appends the string returned by processing a composite format string, each format item is replaced by the string representation of arguments.
`CopyTo(IBufferWriter<byte>)`	void	Copy inner buffer to the buffer writer.
`TryCopyTo(Span<byte>, out int)`	bool	Copy inner buffer to the destination span.
WriteToAsync(Stream stream)	Task	Write inner buffer to stream.
ToString()	string	Encode the innner utf8 buffer to a System.String.
GetMemory(int sizeHint)	`Memory<char>`	IBufferWriter.GetMemory.
GetSpan(int sizeHint)	`Span<char>`	IBufferWriter.GetSpan.
Advance(int count)	void	IBufferWriter.Advance.
static `RegisterTryFormat<T>(TryFormat<T>)`	void	Register custom formatter.

class Utf16PreparedFormat<T1,..,T16>

method	returns	description
`Format`	string	Replaces one or more format items in a string with the string representation of some specified values.
`FormatTo<TBufferWriter>(ref TBufferWriter, T1,..,T16)`	void	Replaces one or more format items in a string with the string representation of some specified values.

class Utf8PreparedFormat<T1,..,T16>

method	returns	description
`Format`	string	Replaces one or more format items in a string with the string representation of some specified values.
`FormatTo<TBufferWriter>(ref TBufferWriter, T1,..,T16)`	void	Replaces one or more format items in a string with the string representation of some specified values.

class ZStringWriter : TextWriter

A TextWriter implementation that is backed with Utf16ValueStringBuilder
It's important to make sure the writer is always properly disposed.

static class TextMeshProExtensions(Unity only)

method	returns	description
SetText(Utf16ValueStringBuilder)	void	Set inner buffer to text mesh pro directly to avoid string allocation.
`SetTextFormat<T1,..,T16>(string, T1,..,T16)`	void	Set formatted string without string allocation.

Unity

Install via UPM git URL package or asset package(ZString...*.unitypackage) available in ZString/releases page.

https://github.com/Cysharp/ZString.git?path=src/ZString.Unity/Assets/Scripts/ZString

If you want to set a target version, ZString uses the *.*.* release tag so you can specify a version like #2.4.0. For example https://github.com/Cysharp/ZString.git?path=src/ZString.Unity/Assets/Scripts/ZString#2.4.0.

Supporting minimum Unity version is 2021.3. The dependency managed DLL System.Runtime.CompilerServices.Unsafe/6.0.0 is included with unitypackage. For git references, you will need to add them in another way as they are not included to avoid unnecessary dependencies; either extract the dll from unitypackage or download it from the NuGet page.

TextMeshPro support is automatically enabled when importing the com.unity.textmeshpro package from package manager. (If you do not use the package manager, define the scripting define symbol ZSTRING_TEXTMESHPRO_SUPPORT to enable it.)

Advanced Tips

ZString.CreateStringBuilder(notNested:true) is a special optimized parameter that uses ThreadStatic buffer instead of rent from ArrayPool. It is slightly faster but can not use in nested.

using(var sb = ZString.CreateStringBuilder(true))
{
    sb.Append("foo");

    using var sb2 = ZString.CreateStringBuilder(true); // NG, nested stringbuilder uses conflicted same buffer
    var str = ZString.Concat("x", 100); // NG, ZString.Concat/Join/Format uses threadstatic buffer
}

// OK, return buffer immediately.
using(var sb = ZString.CreateStringBuilder(true))
{
    sb.Append("foo");
    return sb.ToString();
}

ZString.CreateStringBuilder() is same as ZString.CreateStringBuilder(notNested:false).

In default, SByte, Int16, Int32, Int64, Byte, UInt16, UInt32, UInt64, Single, Double, TimeSpan, DateTime, DateTimeOffset, Decimal, Guid, String, Char are used there own formatter to avoid .ToString() allocation, write directly to buffer. If not exists there list type, used .ToString() and copy string data.

If you want to avoid to convert string in custom type, you can register your own formatter.

Utf16ValueStringBuilder.RegisterTryFormat((MyStruct value, Span<char> destination, out int charsWritten, ReadOnlySpan<char> format) =>
{
    // write value to destionation and set size to charsWritten.
    charsWritten = 0;
    return true;
});

Utf8ValueStringBuilder.RegisterTryFormat((MyStruct value, Span<byte> destination, out int written, StandardFormat format) =>
{
    written = 0;
    return true;
});

CreateStringBuilder and CreateUtf8StringBuilder must use with using. Because their builder rent 64K buffer from ArrayPool. If not return buffer, allocate 64K buffer when string builder is created.

Utf16ValueStringBuilder and Utf8ValueStringBuilder are mutable struct, be careful to copy by passing method. Use ref and try-finally.

void Build()
{
    var sb = ZString.CreateStringBuilder();
    try
    {
        BuildHeader(ref sb);
        BuildMessage(ref sb);
    }
    finally
    {
        // when use with `ref`, can not use `using`.
        sb.Dispose();
    }
}


void BuildHeader(ref Utf16ValueStringBuilder builder)
{
    //..
}

void BuildMessage(ref Utf16ValueStringBuilder builder)
{
    //..
}

Utf8Format, Utf8StringBuilder uses Utf8Formatter.TryFormat and there format string is not same as standard format. It uses StandardFormat, combinate of symbol char and precision. Supported format string symbol can find in Utf8Formatter.TryFormat document(For example Int32 supports G, D, N, X and Boolean supports G, I). Precision(zero padding) can pass after symbol like D2. For example sb.AppendFormat("{0:D2}:{1:D2}:{2:D2}", hour, minute, second).

TryFormat(DateTime) and TryFormat(TimeSpan) symbol is too restricted than standard string format. If you want to use custom format, deconstruct there Day, Hour, etc.

Utf8ValueStringBuilder and Utf16ValueStringBuilder implements IBufferWriter so you can pass serializer(such as JsonSerializer of System.Text.Json). But be careful to boxing copy, ValueStringBuilder is mutable struct. For example,

using var sb = ZString.CreateUtf8StringBuilder();
IBufferWriter<byte> boxed = sb;
var writer = new Utf8JsonWriter(boxed);
JsonSerializer.Serialize(writer, ....);

using var unboxed = (Utf8ValueStringBuilder)boxed;
var str = unboxed.ToString();

License

This library is licensed under the MIT License.

.NET Standard 2.0 and Unity version borrows dotnet/runtime conversion methods, there exists under ZString/Number directory. This third-party license follows runtime/LICENSE.TXT.

	int remainLnegth = this.index - index;
	buffer.AsSpan(index, remainLnegth).CopyTo(newBuffer.AsSpan(newBufferIndex));

	buffer = newBuffer;
	this.index = newBufferIndex + remainLnegth;
	}

	buffer.CopyTo(newBuffer, 0);
	if (buffer.Length != ThreadStaticBufferSize)
	{
	ArrayPool<char>.Shared.Return(buffer);
	}

	buffer = newBuffer;
	}

Method	Job	Runtime	Source	Mean	Error	StdDev	Ratio	RatioSD	Gen 0	Allocated
StringJoin	.NET 6.0	.NET 6.0	String[3]	17.68 ns	0.241 ns	0.226 ns	1.00	0.00	0.0019	32 B
ZStringJoin	.NET 6.0	.NET 6.0	String[3]	44.23 ns	0.466 ns	0.436 ns	2.50	0.03	0.0019	32 B
StringJoin	.NET 6.0	.NET 6.0	String[5]	24.33 ns	0.121 ns	0.113 ns	1.38	0.02	0.0024	40 B
ZStringJoin	.NET 6.0	.NET 6.0	String[5]	57.08 ns	0.255 ns	0.226 ns	3.23	0.04	0.0024	40 B

StringJoin	.NET Framework 4.6.1	.NET Framework 4.6.1	String[3]	34.09 ns	0.062 ns	0.058 ns	1.00	0.00	0.0063	40 B
ZStringJoin	.NET Framework 4.6.1	.NET Framework 4.6.1	String[3]	167.40 ns	0.406 ns	0.360 ns	4.91	0.01	0.0062	40 B
StringJoin	.NET Framework 4.6.1	.NET Framework 4.6.1	String[5]	51.88 ns	0.044 ns	0.034 ns	1.52	0.00	0.0076	48 B
ZStringJoin	.NET Framework 4.6.1	.NET Framework 4.6.1	String[5]	203.03 ns	0.318 ns	0.297 ns	5.96	0.01	0.0076	48 B

Method	Mean	Error	Gen 0	Gen 1	Gen 2	Allocated
JoinEmptyList	19.697 ns	NA	0.0048	-	-	40 B
ZJoinEmptyList	11.102 ns	NA	-	-	-	-
JoinList1	29.702 ns	NA	0.0048	-	-	40 B
ZJoinList1	51.796 ns	NA	0.0114	-	-	96 B
JoinArray1	3.841 ns	NA	-	-	-	-
ZJoinArray1	51.001 ns	NA	0.0114	-	-	96 B
JoinEnumerable1	23.128 ns	NA	0.0048	-	-	40 B
ZJoinEnumerable1	81.611 ns	NA	0.0162	-	-	136 B
JoinList2	93.200 ns	NA	0.0248	-	-	208 B
ZJoinList2	97.387 ns	NA	0.0200	-	-	168 B
JoinArray2	25.569 ns	NA	0.0201	-	-	168 B
ZJoinArray2	92.378 ns	NA	0.0200	-	-	168 B
JoinEnumerable2	77.429 ns	NA	0.0248	-	-	208 B
ZJoinEnumerable2	128.068 ns	NA	0.0248	-	-	208 B
JoinList10	526.850 ns	NA	0.2842	0.0019	-	2384 B
ZJoinList10	400.351 ns	NA	0.0906	-	-	760 B
JoinArray10	97.722 ns	NA	0.0908	0.0002	-	760 B
ZJoinArray10	411.841 ns	NA	0.0906	-	-	760 B
JoinEnumerable10	426.296 ns	NA	0.2847	0.0024	-	2384 B
ZJoinEnumerable10	483.942 ns	NA	0.0954	-	-	800 B

cysharp / zstring Goto Github PK

zstring's Introduction

ZString

Table of Contents

Getting Started

Reference

Unity

Advanced Tips

License

zstring's People

Contributors

Stargazers

Watchers

Forkers

zstring's Issues

Unittest failed

Code

Exception

Broken TextMeshPro Extensions

Overview

Steps to reproduce

Expected result

Actual result

Recommend Projects

Recommend Topics

Recommend Org