orbitdb / orbit-db-store Goto Github PK

Hello @haadcode, I love orbit-db!

I am implementing a custom store for a specific application, but the index-building operation (updateIndex) involves some external I/O. I know this makes the index not deterministic, but it should not be an issue for my use case, at least for now.

At this line the updateIndex call is made, but the call does not use await (even though the caller is async) so if updateIndex returns a promise it would be executed concurrently and the replicated event would get emitted while the index isn't ready yet.

I'd be happy to submit a PR, just wanted to know if I'm missing something important

We ran into situations where an ipfs hash that wasn't pinned for some reason doesn't exist but would be referenced in ipfs-log. In these cases a store.load() would be virtually stuck forever as fetches in ipfs don't time out per default.

A solution would be to leverage the timeout that already exists in ipfs-log: https://github.com/orbitdb/ipfs-log/blob/master/src/entry-io.js#L21. We could expose that to the store.load function as the second argument and/or as a store default (https://github.com/orbitdb/orbit-db-store/blob/master/src/Store.js#L16):

store.load(amount, { fetchHeadTimeout: 5000 }).

Feature request

Right now the identity of the store is set when it's opened. It would be great if it was possible to change the identity after the store has been initialized. A use case for this might be if you want to sync the data of the store before the user has been authenticated, if so a temporary identity could be used until that happens.

How it would work

Add a new method called setIdentity. This method would change this.identity of the store to the identity that is passed to the method. It would also update the identity in the ipfs-log so a change is needed there as well.

Things like .nyc_output and the test folder can be added to .npmignore to reduce the package size.

I use the orbit-db-store as a dependency of 3box-js. Compiling it with webpack --mode production I get the following error:

ERROR in ./node_modules/orbit-db-store/src/Replicator.js                                                                         
Module not found: Error: Can't resolve 'p-map' in '/dapp/node_modules/orbit-db-store/src'                                        
 @ ./node_modules/orbit-db-store/src/Replicator.js 2:13-29                                                                       
 @ ./node_modules/orbit-db-store/src/Store.js                                                                                    
 @ ./node_modules/orbit-db-docstore/src/DocumentStore.js                                                                         
 @ ./node_modules/orbit-db/src/OrbitDB.js                                                                                        
 @ ./node_modules/3box/lib/3box.js                                                                                               
 @ ./src/sagas/communityEntities.js                                                                                              
 @ ./src/sagas/index.js                                                                                                          
 @ ./src/index.js      

Looking into dependencies I see that p-map is missing a dependencies of this package. Adding it might fix that issue 😃

After my PR #103 is merged, there would be only one TODO left in the README.md:

loadMoreFrom(amount, entries)

TODO

//TODO
db.loadMoreFrom()

CIDs/multihash can be encoded in many different formats. base58btc, base32, etc. Comparison via normal javascript methods do not work. Use CID.equals(CID2) instead.
https://github.com/orbitdb/orbit-db-store/blob/1b628b6a9035e2b0fda2f68052797c6b0a0a74b4/src/Store.js#L302-L304

https://github.com/orbitdb/orbit-db-store/blob/ae16436fd69cf48c6f5fe490893cef9735db9bf4/src/Replicator.js#L168

E.g. proper timeout useage https://github.com/orbitdb/orbit-db-store/blob/ae16436fd69cf48c6f5fe490893cef9735db9bf4/src/Store.js#L260
This issue is wip, will add more details & fix later

orbit-db: 0.28.4
orbit-db-store: 4.3.3

TypeError: Cannot read properties of null (reading 'length')
    at Replicator.onReplicationProgress (...\orbit-db-store\src\Store.js:98:25)
    at onProgressCallback (...\orbit-db-store\src\Replicator.js:147:14)

Here is the demo https://github.com/siepra/closeorbitdb

Also I tried enhancing the if statement in Store.js:98 with if (this._oplog && but going further results in different error (also reachable in the sample repo I posted above)

TODO

• Move Cache to the store
• Throw errors if a required argument isn't passed to the store
• Remove .id property since it's the same as the address
• Make the Store extend EventEmitter
• Clean up replication event handling
• Move snapshot related code to its own space
• Remove Index from Store
• Should we rename Index to IndexingStrategy?
• Should Cache be named like that since it feels it does more than just caching data?

Are there best practices / docs for how we can restore store data from data?

For example, if a peer has no local copy and attempts to restore data from a centralized database, what happens during replication? Does this mean that we will have duplicate documents if another peer comes online and attempts to replicate data?

On a related topic, how would we "pin" a store to a pinning service and retrieve the pinned data?

source.js:17 Uncaught Error: already resolved
at Function.i.resolve (source.js:17)
at index.js:10
at reduce.js:10
at drain.js:20
at async-map.js:28
at map.js:19
at async-map.js:28
at f (index.js:84)
at c (index.js:68)
at s (index.js:29)
at i (async-map.js:27)
at map.js:11
at i (async-map.js:27)
at i (drain.js:16)
at drain.js:29
at async-map.js:39

when call saveSnapshot() ,got the above error

orbit-db: 0.28.3
orbit-db-store: 4.3.3

Hi,
we are using OrbitDb for developing a p2p chat application.
Some time ago we noticed that once in a while users are lacking older messages ("message" - entry in a EventStore). It happens rarely but it already happened at least few times and we were sure that it wasn't a connection issue simply because new messages were arriving and could be sent with no problem.

Right now we are relying on replicate.progress event to send newly received messages to frontend. After some intensive testing I managed to get to the broken state (missing one message) and gather some logs.

(Notice missing "I received a message but Windows did not start replicating missing messages. Will it trigger now?" on the left side).

What happened was that replicate.progress event didn't fire for this particular message because none of the conditions in onReplicationProgress (https://github.com/orbitdb/orbit-db-store/blob/main/src/Store.js#L98) were met.

These are the logs from the application with a broken state. They are a bit messy because I was logging the db snapshot on every 'replicate.progress' to see how oplog is changing.
app1MissingMessage.log

This is the final snapshot that proves that the "missing" entry is in the local db, information about receiving it just wasn't propagated:
app1MissingMessagesFinalSnapshot.log

Looking at the logs of the last 3 messages I noticed that the Replicator received "Yes it did trigger" message before "I received a message but Windows did not start replicating missing messages. Will it trigger now?". I am not sure if this matters but after "Yes it did trigger" the replicationStatus wasn't recalculated properly thus replicate.progress didn't happen:

entry.clock.time: 31
onReplicationProgress: I reconnected at 22:49. Will sending a message trigger replication?
onReplicationProgress -> (this._oplog.length + 1 | 43), (this.replicationStatus.progress | 43), {previousProgress | 42}, (this.replicationStatus.max | 44), (previousMax | 44)
entry.clock.time: 44
onReplicationProgress: Yes it did trigger
onReplicationProgress -> (this._oplog.length + 1 | 43), (this.replicationStatus.progress | 44), {previousProgress | 43}, (this.replicationStatus.max | 44), (previousMax | 44)
entry.clock.time: 43
onReplicationProgress: I received a message but Windows did not start replicating missing messages. Will it trigger now?
onReplicationProgress -> (this._oplog.length + 1 | 43), (this.replicationStatus.progress | 44), {previousProgress | 44}, (this.replicationStatus.max | 44), (previousMax | 44)

Unfortunatelly I don't have a working test yet because the path for reproducing the problem is a bit random. Opening and closing apps (aka peers) in some order seems to do the trick. I'll provide a test as soon as I create one.

Do you have any idea what could've happened here?

The Log entry contains the identity of the user that signed the data. This is very useful information.

How can I get this from a given hash? Do I just go to the IPFS CID and the block of data is there?

I was wondering because, to create a ledger chronicling ownership with OrbitDB is technically as simple as having a log with { to } (and the original owner is the identity). Otherwise, I'd have to store {to, from}, which would be redundant.

There's some leftover using the old write set as before inside the sync function (https://github.com/orbitdb/orbit-db-store/blob/master/src/Store.js#L235), that breaks when an access controller is passed to the store

https://github.com/orbitdb/orbit-db-store/blob/931c452e5197bcd4a2cb716eecfe675da87e2838/src/Store.js#L305

loadFromSnapshot(hash, onProgressCallback)

Load the state of the database from a snapshot. hash is the IPFS Multihash of the snapshot data. Returns a Promise that resolves when the database has been loaded.

This is a TODO in the README.

https://github.com/orbitdb/orbit-db-store/blob/2ebc0c2cbec70cc98e7cd0f45edc077853098f72/src/Store.js#L241

Why don't to filter the heads as it's already implemented within the "sync" method, that is called during synchronization with another peer?

https://github.com/orbitdb/orbit-db-store/blob/2ebc0c2cbec70cc98e7cd0f45edc077853098f72/src/Store.js#L295