Git Product home page Git Product logo

Comments (1)

apkar avatar apkar commented on September 2, 2024

For the sake of completion pasting the forums post here


It depends on a lot of factors. Some related to your deployment configuration, some related to design traits of Document Layer and in general FoundationDB. I will try to explain top bullets

Write concern

Write concern you are running with will have a huge impact on MongoDB's performance and consistency guarantees.

  • MongoDB: If you don't specify any, defaults would be {w: 1, j: false}. That config only guarantees in-memory writes on standalone or primary. For stronger consistency, you would need something like {w: majority, j: true}. That would make sure the majority of replicas have your write, persistent on disk. It is still eventual consistency. And in case of failures, your writes could be rollbacked. This is how writes work on standard MongoDB.

  • Document Layer: Document Layer, and in general FoundationDB, goal is to provide strong consistency we are used to in SQL world. Document Layer just ignores the write concern provided by the client. Irrespective of the write concern, writes are always done at full consistency. Once the write is acknowledged back to the client, it is guaranteed that it is persisted on disk and replicated. So, your write is never lost in any kind of failures. To achieve this FoundationDB/Document Layer need to do more work than MongoDB. This makes Document Layer slower than standard MongoDB.

Concurrency

As both MongoDB and FoundationDB are distributed they benefit quite a bit by having more concurrency in your application. As @SteavedHams suggested in his response you could have more parallelism in your test either by adding more threads or even better making the test async (with rate limiter). With the test you have above you are pretty much running one request at a time on Document Layer, which runs one transaction at a time on FDB. So, getting limited by the latency of each transaction. But, you are measuring the throughput, how long it takes me to insert 1M documents. Making your test concurrent will probably help both MongoDB and Document Layer. But more for Document Layer, considering it will help hide the latency.

Transactions

Document Layer works more like traditional SQL databases when it comes to transaction management - Everything is a transaction. Either you start a transaction from the client and do your requests part of that transaction, or Document Layer starts a transaction for each and every request. Every transaction has some initiation cost of getting the current database version, getReadVersion(), and commit at the end, which persists the changes made in the transaction. As each transaction has this overhead, just like SQL databases, the best way to hide this overhead is to do more in a transaction.

  • Implicit transactions: You don't manage transactions on the client side, Document Layer creating transactions for each MongoDB request. Do more in each request, this is my first suggestion in the previous post.
  • Explicit transactions: Manage transactions yourself using beginTransaction and commitTranaction and include more business logic there. These are custom commands only work on Document Layer. Sadly, explicit transactions in Document Layer have some limitations regarding connections. We are rewriting them to be compatible with MongoDB v4.0 transactions.

If you compare with Document Layer requests or transactions with MongoDB v4.0 transactions, that's where Document Layer shines a lot. As transactions are primitive operations in FoundationDB.

Project maturity

Having said all that, there are quite a few improvements which will help to bring down the latency even further down. For example, Document Layer manages metadata for every collection with a version. And every request reads that version to make sure collection metadata hasn't changed. There are ways we could cut this read from the critical path, reducing latency. This is one example, there are other improvements which will help the performance. The project is now in initial stages. For us, the initial goal was to have a strongly consistent database. We will have performance improvements coming in, to make these numbers better.

Finally..

To finish this long post, it is fair to say Document Layer is never going to do better if you are measuring the latency of a single write. Even though you are measuring here 1M writes, as they are happening sequentially you are limited by the latency of a single write. Document Layer strengths are better consistency guarantees and has faster transactions, thanks to FoundationDB. As its running on FoundationDB, it inherits point in time backup/restores, fearless DR. To get better performance from Document Layer we will have to follow the same guidelines as FoundationDB applications.

from fdb-document-layer.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.