Git Product home page Git Product logo

stringdb's Introduction

StringDB

StringDB

Build Status Test Status Nuget Version Nuget Downloads

Install-Package StringDB

Introduction

StringDB is a key/value pair store with a friendly API to use as little RAM and space as possible.

Verify the claims for yourself:

Api

Enumerate over a database and it's values, the fastest, by enumerating over it optimally

using var db = new DatabaseBuilder()
	.UseIODatabase(StringDBVersion.Latest, "database.db", out var optimalTokenSource)
	.WithBuffer(1000)
	.WithTranform(StringTransformation.Default, StringTransformation.Default);
	
foreach (var (key, value) in db.EnumerateOptimally(optimalTokenSource))
{
	// do something with the key and value
}

Use fluent extensions to create a database:

using IDatabase<string, string> db = new DatabaseBuilder()
    .UseIODatabase(StringDBVersion.Latest, "database.db")
    .WithBuffer(1000)
    .WithTransform(StringTransformation.Default, StringTransformation.Default);

using IDatabase<int, string> memDb = new DatabaseBuilder()
    .UseMemoryDatabase<int, string>();

Use the IDatabase interface to interface with databases

void InsertGreeting(IDatabase<string, string> database, string user)
{
    database.Insert(user, $"Greetings, {user}!");
}

And inherit BaseDatabase to create your own IDatabases with minimal effort

public class TestDatabase : BaseDatabase<int, string>
{
    private class LazyValue : ILazyLoader<string>
    {
        private readonly string _value;
        public LazyValue(int value) => _value = value.ToString();
        public string Load() => _value;
        public void Dispose() {}
    }
	
	public override void Dispose() {}

    protected override void InsertRange(KeyValuePair<int, string>[] items)
    {
        foreach(var item in items)
        {
            Console.WriteLine($"{item.Key}: {item.Value}");
        }
    }

    protected override IEnumerable<KeyValuePair<int, ILazyLoader<string>>> Evaluate()
    {
        for(var i = 0; i < int.MaxValue)
        {
            yield return KeyValuePair.Create(i, new LazyValue(i));
        }
    }
}

Tiny icon_tiny

StringDB is tiny. Use tiny amounts of RAM, and tiny amounts of space.

Chart

Inserts Size (in KB, 1000 bytes) Absolute Minimum Size Possible StringDB Overhead Percentage
1 1.172 KB 1.152 KB 1.706485%
50 58.208 KB 57.6 KB 1.04453%
100 116.408 KB 115.2 KB 1.037729%

This chart shows the size of a StringDB file after multiple single inserts. Every key is 128 bytes long, and every value is 1024 bytes long. By doing single inserts, file size is dramatically affected due to the additional overhead for the index chain.

Chart

Elements in Insert Range Size (in KB, 1000 bytes) Absolute Minimum Size Possible StringDB Overhead Percentage
1 1.172 KB 1.152 KB 1.706485%
50 57.963 KB 57.6 KB 0.626262%
100 115.913 KB 115.2 KB 0.615117%

This chart shows the size of a StringDB file after a single insert range with the amount of items specified.

Addons

Official addon support will be maintained for these libraries.

Issues welcomed!

Don't be afraid to make an issue about anything and everything!

  • Is there something weird with how databases are created? Submit an issue!
  • Is there something that you wish you could do but can't? Submit an issue!
  • Is this library not suitable for your purposes? Submit an isssue!
  • Want it to do something? Submit an issue!

It's an honour to have you use this library, and feedback is needed to make this the greatest it can be.

Need immediate assistence? Join the discord!

stringdb's People

Contributors

jtone123 avatar lordmike avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stringdb's Issues

Exception on Concurrency Multi Reading and 1 Writeing ?

most time Exception on TryGet Method.

the fllow is the test Code

               StringDBTest.TestFileNameLengthEffects0(); //Test Entry

public class StringDBTest {
        static StringDBTest() {
            try {
                //db = new LiteDatabase("Filename=TestDB.ldb;");//初始化数据库connection=shared
                db = new DatabaseBuilder()
                           .UseIODatabase(StringDBVersion.Latest, "StringDBTest.db", out var optimalTokenSource)
                           .WithBuffer(1000)
                           .WithTransform(StringTransformer.Default, StringTransformer.Default);
            } catch (Exception ex) {
                Console.WriteLine(ex.Message);
            }
        }

        public static IDatabase<string, string> db;


        public static List<CTS1> DataGen(ref int current, int count) {
            List<CTS1> list = new(count);
            for (int i = 0; i < count; i++) {
                var val = (current + i);
                var key = $"0d3deed1234567890abcd{(100000000 - (current + i)):D12}";
                list.Add(new(key, val));
            }
            current = current + count;
            return list;
        }

        static Random r = new Random();

        public static async void SelectTest() {
            while (!IsExit()) {
                await Task.Delay(r.Next(1, 10));//随机触发查询时间
                var sw = Stopwatch.StartNew();
                var rs = r.Next(0, 10000000);
                var key = $"0d3deed1234567890abcd{(100000000 - rs):D12}";
                db.TryGet(key, out string strVal);
                //var rlt = db.Get(key);
                if (sw.Elapsed.Seconds > 1)
                    Console.WriteLine($"查询时间大于1秒-{sw.ElapsedMilliseconds}ms");
                Interlocked.Increment(ref QueryCount);
            }
        }


        public static bool IsExit() {
            return Interlocked.Read(ref exit) == 1;
        }

        static long exit = 0;

        public static long QueryCount;

        public static void TestFileNameLengthEffects0() {
            //using var db1 = new LiteDatabase("Filename=TestDB0.ldb;");
            var sw = Stopwatch.StartNew();
            var swOutput = Stopwatch.StartNew();
            for (int t = 0; t < 50; t++)
                Task.Factory.StartNew(SelectTest);
            int totalIdx = 26860000;
            for (int i = 0; i < 10000; i++) {
                var list = DataGen(ref totalIdx, 20000); //每2w一个批次
                try {
                   db.InsertRange(list.Select(cts=> new KeyValuePair<string,string>(cts.Code, cts.ToString())).ToArray());
                } catch (Exception ex) {
                    Console.WriteLine(ex.Message);
                }
                if (swOutput.Elapsed.TotalSeconds > 8) {
                    Console.WriteLine($" 当前数量:{totalIdx} 平均插入速度:{(totalIdx / sw.Elapsed.TotalSeconds)}op/s , 查询次数{(QueryCount / sw.Elapsed.TotalSeconds)}/s");
                    swOutput.Restart();
                }
            }
            exit = 1;
        }

        /// <summary>
        /// CodeTaskSmall
        /// </summary>
        public class CTS1 {
            public CTS1(string code, int taskId) {
                Code = code;
                TaskId = taskId;
            }
            [LiteDB.BsonId]
            public string Code;
            [LiteDB.BsonField("t")]
            public int TaskId;
        }



    }

Async Databases

a lack of async disturbs me.

currently, the biggest blocker of this feature is that you can't do concurrent "seek & read" operations on a Stream, and for IODatabases this would be an issue.

it is a theoretical possibility to add some kind of DatabaseFactory which produces single threaded IDatabases, and an AsyncDatabase would use this factory to create a database on demand and then read from it and close it, so concurrent operations could be done by having multiple file handles open, though it would be nice if there were alternatives.

additionally, the compatability issues of synchronous code and async is a bit disturbing, it would be nice if code didn't have to be duplicated between synchronous and asynchronous.

Deletable Entries

deleting stuff in a database would be nice.

however, there are multiple ways to do this:

  1. deleting entries from an IODatabase wouldn't actually mean that the space allocated to them is gone, the index of the entry would get replaced by a jump instruction (and perhaps the entries around it) to more indexes. this wouldn't be that easy to do since the jump instruction is only an integer, so if you have files bigger than 2GB you wouldn't be able to jump to an index to the future, and even more far off jumps would have to be made.
  2. move all of the data of the stream over the old data, completely overwriting it and properly deleting it. this would involve changing of lots of positions and relative jump locations, so a delete would prove a costly operation.

moreover, an API for deleting would be complicated. currently, the suggested route is to pass in multiple ILazyLoader[]s to Delete, but then you'd have to manage existing ILazyLoaders to ensure that the relative positions they store get updated as well.

perhaps it'd be better if you'd be forced to close the database, and do separate deletion operations via an entirely different class, forcing developers to adapt to a different, more appropriate style.

StringDB isn't designed to be used to constantly add/remove stuff, and it'd be a more appropriate workflow to be forced to close the database, then modify it, then re-open it.

a current workaround is to and add all the things you want in the new database that aren't the deleted items.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.