Comments (7)
NFSdb bottleneck is not with disk. Memory Mapped files are perfectly efficient to achieve more than NFSdb does. Speed is a traid off of functionality. Index, partitioning and object reflection take most of the time.
Also making journal not make mapped files and than spill over will be very inefficient.
from questdb.
May be you can option that avoids indexing and partitioning incase you just write and read from the journal as a queue. (With auto commit.) A client can just read it and pipe it into another journals each being a queue based on an allocation methodology and applying a filer. Similarly you can pipe into multiple journals which the clients can read independently.
I am not sure how IO scales with multiple journals but will this be faster? At Least if might be functionally superior by having messaging patterns (#14).
from questdb.
Partitiioning and indexing are fully optional and both are off by default. You can try to see if performance suits your use case. I'll cover auto-commit under issue #9.
BulkWriter
and BulkReader
work at about 95% of performance of non-bulk variants and provided they only do sequential writes/reads they are not affected by your RAM size even. They'll work at constant rate until you run out of disk space. I think they cover the spill over problem in most efficient way possible albeit at a slight overall performance loss.
from questdb.
there is a major new functionality coming in version 3.0, which may have an influence on subject of in-memory only store. Let keep this closed until than to keep backlog clear.
from questdb.
OK. The only part may be an issue.
from questdb.
Do tell more :)
from questdb.
What you run out of memory. Can you spill into disk?
from questdb.
Related Issues (20)
- SQL: Sample by with ASOF join errors with ambiguous column since 7.4.0
- Improvements to feature flagging and handling of breaking changes HOT 3
- Create special function to calculate trade price given L2 Order Book price/quantity pairs HOT 1
- flaky test - RetryIODispatcherTest.testInsertWaitsExceedsRerunProcessingQueueSizeLoop
- Unsupported machine HOT 14
- Support Aggregation Functions with GROUPS Clause in Window Functions
- CASE returning unexpected value for timestamp IS NULL comparison HOT 1
- COALESCE failing for NULL timestamps
- Symbol incorrectly written to table in 7.4.2 when multiple processes sending to QuestDB HOT 6
- Web Console not available after 7.3.7 HOT 8
- JIT bug: COUNT with FLOAT not equals returns wrong results with null values
- split_part returns result from previous row when current row does not have the part HOT 1
- The timestamp was forcibly changed. Procedure HOT 2
- Incorrect result when having NULL with IN operation HOT 1
- HTTPS redirect when using FQDN instead of IP HOT 2
- SAMPLE BY breaks with `last(timestamp)`
- Error while parsing timestamps with times between midnight and 1am formatted in the 12-hour clock convention HOT 2
- Support of Decimal64 type wanted
- Incorrect result when having null in case clause
- Settings pg.net.bind.to=0 Causes Crash on Ubuntu
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from questdb.