Git Product home page Git Product logo

stringdb's Issues

Exception on Concurrency Multi Reading and 1 Writeing ?

most time Exception on TryGet Method.

the fllow is the test Code

               StringDBTest.TestFileNameLengthEffects0(); //Test Entry

public class StringDBTest {
        static StringDBTest() {
            try {
                //db = new LiteDatabase("Filename=TestDB.ldb;");//初始化数据库connection=shared
                db = new DatabaseBuilder()
                           .UseIODatabase(StringDBVersion.Latest, "StringDBTest.db", out var optimalTokenSource)
                           .WithBuffer(1000)
                           .WithTransform(StringTransformer.Default, StringTransformer.Default);
            } catch (Exception ex) {
                Console.WriteLine(ex.Message);
            }
        }

        public static IDatabase<string, string> db;


        public static List<CTS1> DataGen(ref int current, int count) {
            List<CTS1> list = new(count);
            for (int i = 0; i < count; i++) {
                var val = (current + i);
                var key = $"0d3deed1234567890abcd{(100000000 - (current + i)):D12}";
                list.Add(new(key, val));
            }
            current = current + count;
            return list;
        }

        static Random r = new Random();

        public static async void SelectTest() {
            while (!IsExit()) {
                await Task.Delay(r.Next(1, 10));//随机触发查询时间
                var sw = Stopwatch.StartNew();
                var rs = r.Next(0, 10000000);
                var key = $"0d3deed1234567890abcd{(100000000 - rs):D12}";
                db.TryGet(key, out string strVal);
                //var rlt = db.Get(key);
                if (sw.Elapsed.Seconds > 1)
                    Console.WriteLine($"查询时间大于1秒-{sw.ElapsedMilliseconds}ms");
                Interlocked.Increment(ref QueryCount);
            }
        }


        public static bool IsExit() {
            return Interlocked.Read(ref exit) == 1;
        }

        static long exit = 0;

        public static long QueryCount;

        public static void TestFileNameLengthEffects0() {
            //using var db1 = new LiteDatabase("Filename=TestDB0.ldb;");
            var sw = Stopwatch.StartNew();
            var swOutput = Stopwatch.StartNew();
            for (int t = 0; t < 50; t++)
                Task.Factory.StartNew(SelectTest);
            int totalIdx = 26860000;
            for (int i = 0; i < 10000; i++) {
                var list = DataGen(ref totalIdx, 20000); //每2w一个批次
                try {
                   db.InsertRange(list.Select(cts=> new KeyValuePair<string,string>(cts.Code, cts.ToString())).ToArray());
                } catch (Exception ex) {
                    Console.WriteLine(ex.Message);
                }
                if (swOutput.Elapsed.TotalSeconds > 8) {
                    Console.WriteLine($" 当前数量:{totalIdx} 平均插入速度:{(totalIdx / sw.Elapsed.TotalSeconds)}op/s , 查询次数{(QueryCount / sw.Elapsed.TotalSeconds)}/s");
                    swOutput.Restart();
                }
            }
            exit = 1;
        }

        /// <summary>
        /// CodeTaskSmall
        /// </summary>
        public class CTS1 {
            public CTS1(string code, int taskId) {
                Code = code;
                TaskId = taskId;
            }
            [LiteDB.BsonId]
            public string Code;
            [LiteDB.BsonField("t")]
            public int TaskId;
        }



    }

Deletable Entries

deleting stuff in a database would be nice.

however, there are multiple ways to do this:

  1. deleting entries from an IODatabase wouldn't actually mean that the space allocated to them is gone, the index of the entry would get replaced by a jump instruction (and perhaps the entries around it) to more indexes. this wouldn't be that easy to do since the jump instruction is only an integer, so if you have files bigger than 2GB you wouldn't be able to jump to an index to the future, and even more far off jumps would have to be made.
  2. move all of the data of the stream over the old data, completely overwriting it and properly deleting it. this would involve changing of lots of positions and relative jump locations, so a delete would prove a costly operation.

moreover, an API for deleting would be complicated. currently, the suggested route is to pass in multiple ILazyLoader[]s to Delete, but then you'd have to manage existing ILazyLoaders to ensure that the relative positions they store get updated as well.

perhaps it'd be better if you'd be forced to close the database, and do separate deletion operations via an entirely different class, forcing developers to adapt to a different, more appropriate style.

StringDB isn't designed to be used to constantly add/remove stuff, and it'd be a more appropriate workflow to be forced to close the database, then modify it, then re-open it.

a current workaround is to and add all the things you want in the new database that aren't the deleted items.

Async Databases

a lack of async disturbs me.

currently, the biggest blocker of this feature is that you can't do concurrent "seek & read" operations on a Stream, and for IODatabases this would be an issue.

it is a theoretical possibility to add some kind of DatabaseFactory which produces single threaded IDatabases, and an AsyncDatabase would use this factory to create a database on demand and then read from it and close it, so concurrent operations could be done by having multiple file handles open, though it would be nice if there were alternatives.

additionally, the compatability issues of synchronous code and async is a bit disturbing, it would be nice if code didn't have to be duplicated between synchronous and asynchronous.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.