Comments (1)
For the sake of completion pasting the forums post here
It depends on a lot of factors. Some related to your deployment configuration, some related to design traits of Document Layer and in general FoundationDB. I will try to explain top bullets
Write concern
Write concern you are running with will have a huge impact on MongoDB's performance and consistency guarantees.
-
MongoDB: If you don't specify any, defaults would be
{w: 1, j: false}
. That config only guarantees in-memory writes on standalone or primary. For stronger consistency, you would need something like{w: majority, j: true}
. That would make sure the majority of replicas have your write, persistent on disk. It is still eventual consistency. And in case of failures, your writes could be rollbacked. This is how writes work on standard MongoDB. -
Document Layer: Document Layer, and in general FoundationDB, goal is to provide strong consistency we are used to in SQL world. Document Layer just ignores the write concern provided by the client. Irrespective of the write concern, writes are always done at full consistency. Once the write is acknowledged back to the client, it is guaranteed that it is persisted on disk and replicated. So, your write is never lost in any kind of failures. To achieve this FoundationDB/Document Layer need to do more work than MongoDB. This makes Document Layer slower than standard MongoDB.
Concurrency
As both MongoDB and FoundationDB are distributed they benefit quite a bit by having more concurrency in your application. As @SteavedHams suggested in his response you could have more parallelism in your test either by adding more threads or even better making the test async (with rate limiter). With the test you have above you are pretty much running one request at a time on Document Layer, which runs one transaction at a time on FDB. So, getting limited by the latency of each transaction. But, you are measuring the throughput, how long it takes me to insert 1M documents. Making your test concurrent will probably help both MongoDB and Document Layer. But more for Document Layer, considering it will help hide the latency.
Transactions
Document Layer works more like traditional SQL databases when it comes to transaction management - Everything is a transaction. Either you start a transaction from the client and do your requests part of that transaction, or Document Layer starts a transaction for each and every request. Every transaction has some initiation cost of getting the current database version, getReadVersion()
, and commit at the end, which persists the changes made in the transaction. As each transaction has this overhead, just like SQL databases, the best way to hide this overhead is to do more in a transaction.
- Implicit transactions: You don't manage transactions on the client side, Document Layer creating transactions for each MongoDB request. Do more in each request, this is my first suggestion in the previous post.
- Explicit transactions: Manage transactions yourself using
beginTransaction
andcommitTranaction
and include more business logic there. These are custom commands only work on Document Layer. Sadly, explicit transactions in Document Layer have some limitations regarding connections. We are rewriting them to be compatible with MongoDB v4.0 transactions.
If you compare with Document Layer requests or transactions with MongoDB v4.0 transactions, that's where Document Layer shines a lot. As transactions are primitive operations in FoundationDB.
Project maturity
Having said all that, there are quite a few improvements which will help to bring down the latency even further down. For example, Document Layer manages metadata for every collection with a version. And every request reads that version to make sure collection metadata hasn't changed. There are ways we could cut this read from the critical path, reducing latency. This is one example, there are other improvements which will help the performance. The project is now in initial stages. For us, the initial goal was to have a strongly consistent database. We will have performance improvements coming in, to make these numbers better.
Finally..
To finish this long post, it is fair to say Document Layer is never going to do better if you are measuring the latency of a single write. Even though you are measuring here 1M writes, as they are happening sequentially you are limited by the latency of a single write. Document Layer strengths are better consistency guarantees and has faster transactions, thanks to FoundationDB. As its running on FoundationDB, it inherits point in time backup/restores, fearless DR. To get better performance from Document Layer we will have to follow the same guidelines as FoundationDB applications.
from fdb-document-layer.
Related Issues (20)
- Support hostInfo command HOT 1
- Assign UUID to each incoming message HOT 2
- Re-enable tests for failure cases in 'update' tests
- $addToSet operator is not adding None values correctly HOT 1
- Removing all documents can be optimized HOT 1
- Provide a way to define composite keys
- ExtOperatorUpdate::update() is parsing query for each document HOT 2
- Re-enable None value generation in correctness after #135 got fixed
- Cursors house keeping could get some love
- "transaction_timeout" on long running requests HOT 4
- Store arrays containing null values as sparse arrays across the code
- Publish CPU and memory metrics HOT 1
- Audit all reply messages HOT 1
- Support bitwise query operators
- Documentation link typos
- Use index for $in queries HOT 2
- Memory leak on connections
- Problem with $in and $or filters
- With two indexes (a,b) and (a,b,c), with (a,b,c) unique, document layer allows duplicate values
- Is this repo still actively developed ? compatible with mongo 5.0 ?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fdb-document-layer.