orbitdb / orbit-db-store Goto Github PK
View Code? Open in Web Editor NEWBase class for orbit-db data stores
License: MIT License
Base class for orbit-db data stores
License: MIT License
Hello @haadcode, I love orbit-db!
I am implementing a custom store for a specific application, but the index-building operation (updateIndex
) involves some external I/O. I know this makes the index not deterministic, but it should not be an issue for my use case, at least for now.
At this line the updateIndex
call is made, but the call does not use await
(even though the caller is async) so if updateIndex
returns a promise it would be executed concurrently and the replicated
event would get emitted while the index isn't ready yet.
I'd be happy to submit a PR, just wanted to know if I'm missing something important
We ran into situations where an ipfs hash that wasn't pinned for some reason doesn't exist but would be referenced in ipfs-log
. In these cases a store.load()
would be virtually stuck forever as fetches in ipfs don't time out per default.
A solution would be to leverage the timeout that already exists in ipfs-log: https://github.com/orbitdb/ipfs-log/blob/master/src/entry-io.js#L21. We could expose that to the store.load
function as the second argument and/or as a store default (https://github.com/orbitdb/orbit-db-store/blob/master/src/Store.js#L16):
store.load(amount, { fetchHeadTimeout: 5000 })
.
Right now the identity of the store is set when it's opened. It would be great if it was possible to change the identity after the store has been initialized. A use case for this might be if you want to sync the data of the store before the user has been authenticated, if so a temporary identity could be used until that happens.
Add a new method called setIdentity
. This method would change this.identity
of the store to the identity that is passed to the method. It would also update the identity in the ipfs-log so a change is needed there as well.
I use the orbit-db-store
as a dependency of 3box-js
. Compiling it with webpack --mode production
I get the following error:
ERROR in ./node_modules/orbit-db-store/src/Replicator.js
Module not found: Error: Can't resolve 'p-map' in '/dapp/node_modules/orbit-db-store/src'
@ ./node_modules/orbit-db-store/src/Replicator.js 2:13-29
@ ./node_modules/orbit-db-store/src/Store.js
@ ./node_modules/orbit-db-docstore/src/DocumentStore.js
@ ./node_modules/orbit-db/src/OrbitDB.js
@ ./node_modules/3box/lib/3box.js
@ ./src/sagas/communityEntities.js
@ ./src/sagas/index.js
@ ./src/index.js
Looking into dependencies I see that p-map
is missing a dependencies of this package. Adding it might fix that issue ๐
After my PR #103 is merged, there would be only one TODO left in the README.md:
TODO
//TODO
db.loadMoreFrom()
CIDs/multihash can be encoded in many different formats. base58btc, base32, etc. Comparison via normal javascript methods do not work. Use CID.equals(CID2) instead.
https://github.com/orbitdb/orbit-db-store/blob/1b628b6a9035e2b0fda2f68052797c6b0a0a74b4/src/Store.js#L302-L304
E.g. proper timeout useage https://github.com/orbitdb/orbit-db-store/blob/ae16436fd69cf48c6f5fe490893cef9735db9bf4/src/Store.js#L260
This issue is wip, will add more details & fix later
orbit-db: 0.28.4
orbit-db-store: 4.3.3
TypeError: Cannot read properties of null (reading 'length')
at Replicator.onReplicationProgress (...\orbit-db-store\src\Store.js:98:25)
at onProgressCallback (...\orbit-db-store\src\Replicator.js:147:14)
Here is the demo https://github.com/siepra/closeorbitdb
Also I tried enhancing the if statement in Store.js:98
with if (this._oplog &&
but going further results in different error (also reachable in the sample repo I posted above)
.id
property since it's the same as the addressAre there best practices / docs for how we can restore store data from data?
For example, if a peer has no local copy and attempts to restore data from a centralized database, what happens during replication? Does this mean that we will have duplicate documents if another peer comes online and attempts to replicate data?
On a related topic, how would we "pin" a store to a pinning service and retrieve the pinned data?
source.js:17 Uncaught Error: already resolved
at Function.i.resolve (source.js:17)
at index.js:10
at reduce.js:10
at drain.js:20
at async-map.js:28
at map.js:19
at async-map.js:28
at f (index.js:84)
at c (index.js:68)
at s (index.js:29)
at i (async-map.js:27)
at map.js:11
at i (async-map.js:27)
at i (drain.js:16)
at drain.js:29
at async-map.js:39
when call saveSnapshot() ,got the above error
orbit-db: 0.28.3
orbit-db-store: 4.3.3
Hi,
we are using OrbitDb for developing a p2p chat application.
Some time ago we noticed that once in a while users are lacking older messages ("message" - entry in a EventStore). It happens rarely but it already happened at least few times and we were sure that it wasn't a connection issue simply because new messages were arriving and could be sent with no problem.
Right now we are relying on replicate.progress
event to send newly received messages to frontend. After some intensive testing I managed to get to the broken state (missing one message) and gather some logs.
(Notice missing "I received a message but Windows did not start replicating missing messages. Will it trigger now?" on the left side).
What happened was that replicate.progress
event didn't fire for this particular message because none of the conditions in onReplicationProgress
(https://github.com/orbitdb/orbit-db-store/blob/main/src/Store.js#L98) were met.
These are the logs from the application with a broken state. They are a bit messy because I was logging the db snapshot on every 'replicate.progress' to see how oplog is changing.
app1MissingMessage.log
This is the final snapshot that proves that the "missing" entry is in the local db, information about receiving it just wasn't propagated:
app1MissingMessagesFinalSnapshot.log
Looking at the logs of the last 3 messages I noticed that the Replicator received "Yes it did trigger" message before "I received a message but Windows did not start replicating missing messages. Will it trigger now?". I am not sure if this matters but after "Yes it did trigger" the replicationStatus
wasn't recalculated properly thus replicate.progress
didn't happen:
entry.clock.time: 31
onReplicationProgress: I reconnected at 22:49. Will sending a message trigger replication?
onReplicationProgress -> (this._oplog.length + 1 | 43), (this.replicationStatus.progress | 43), {previousProgress | 42}, (this.replicationStatus.max | 44), (previousMax | 44)
entry.clock.time: 44
onReplicationProgress: Yes it did trigger
onReplicationProgress -> (this._oplog.length + 1 | 43), (this.replicationStatus.progress | 44), {previousProgress | 43}, (this.replicationStatus.max | 44), (previousMax | 44)
entry.clock.time: 43
onReplicationProgress: I received a message but Windows did not start replicating missing messages. Will it trigger now?
onReplicationProgress -> (this._oplog.length + 1 | 43), (this.replicationStatus.progress | 44), {previousProgress | 44}, (this.replicationStatus.max | 44), (previousMax | 44)
Unfortunatelly I don't have a working test yet because the path for reproducing the problem is a bit random. Opening and closing apps (aka peers) in some order seems to do the trick. I'll provide a test as soon as I create one.
Do you have any idea what could've happened here?
The Log entry contains the identity of the user that signed the data. This is very useful information.
How can I get this from a given hash? Do I just go to the IPFS CID and the block of data is there?
I was wondering because, to create a ledger chronicling ownership with OrbitDB is technically as simple as having a log with { to }
(and the original owner is the identity). Otherwise, I'd have to store {to, from}
, which would be redundant.
There's some leftover using the old write
set as before inside the sync
function (https://github.com/orbitdb/orbit-db-store/blob/master/src/Store.js#L235), that breaks when an access controller is passed to the store
loadFromSnapshot(hash, onProgressCallback)
Load the state of the database from a snapshot. hash is the IPFS Multihash of the snapshot data. Returns a Promise that resolves when the database has been loaded.
This is a TODO in the README.
Why don't to filter the heads as it's already implemented within the "sync" method, that is called during synchronization with another peer?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.