Pinning support about rust-ipfs HOT 17 CLOSED

Would there be guidance available for this? I'd be interested in tackling it

from rust-ipfs.

Related: the discussion in #84 and the ipfs/specs discussion linked from there.

from rust-ipfs.

Great @saresend and welcome! We look forward to it.

from rust-ipfs.

Hey @saresend! 👋

@koivunej was able to take your work and build off of it here: #342 and the remaining work is in a follow-up PR #349. Thank you so much for getting the initial ball rolling.

I bet @koivunej and @ljedrz will have some ideas for things to do once Europe wakes up.

from rust-ipfs.

Either way, here's the list of issues that are marked as "help wanted" or "good first issue", the latest of which was just added :)

from rust-ipfs.

Great to hear that @saresend! My latest batch of big PRs ending in #358 and #360 are an initial, feature-full as far as I know for fs based pinning. As you follow along be welcome to join in any reviews or you can open an issue. The more understandable the code is the better it is for everyone.

Closing as following the PRs #342, #349, #355, #358, #360 are merged. Known unfinished parts are tracked in at least #350, #351, #352.

from rust-ipfs.

I guess the best is to dive in. What sort of guidance did you have in mind?

You probably want to have a table in the datastore Cid -> bool. There's a utility in libipld to find all cids recursively, since supporting recursive pinning is very useful. We might drop rocksdb in the future in favor of sled.

If you have an ipld datastructure you will want to pin the root. After you make a change you want to pin the new root and unpin the old one. We might end up requiring two different types of pinning, as when streaming for example we only the buffered part of the file to be pinned. There we want to manage individual blocks manually.

from rust-ipfs.

I didn't have anything too specific in mind, just wanted to check that you guys would be willing to support the changes since I'm still ramping up on IPFS stuff! I'll take a look and hopefully have a PR up soon

from rust-ipfs.

Have something very tentative up at #117, would be interested in seeing if the direction here is correct!

from rust-ipfs.

@dvc94ch Should we leave this open to capture the remaining work or close and create new issue(s)?

from rust-ipfs.

For posterity: The testing of #90 requires /pin/add so it will likely be added during that, for the conformance tests. If I'll implement it I'll keep it as constrained as possible (for example, non-recursive if allowed by the tests, which I understand only require single block pinning).

from rust-ipfs.

Hey, sorry that I haven't got around to it just yet, expect a PR up in the next couple of days! @koivunej Let me know if you want to take that work, don't want to duplicate if you're already working on it.

from rust-ipfs.

@saresend I'll be soon (hopefully!) moving over to this. I want to handle #238 first though and lets just say it hasn't been so fast moving as I would had hoped. After #238 moving refs over to ipfs for different kinds of pins should start to move forward.

Regarding #152. If you want, we could iron out what exists at the moment and get it merged in. Let me know.

from rust-ipfs.

Ah, so sorry just saw this. I've started work and been very busy. I should have some time later this week however, if there hasn't been too much development on this front since!

from rust-ipfs.

Awesome, sounds good! Sorry I went mia been moving for work and getting settled and haven't had too much time for open source lately. If there's still opportunity for me to help out on this feature, I'd love to re-involve myself!

from rust-ipfs.

Regarding the topic of pinning, my #349 is not going to close the chapter at all. The #349 is "ok" for inmemory version. Once that's done, I'll continue over to a persistent block and data store with a bit different design: instead of "writing" all indirect pins explicitly I'll probably settle for a document per recursive pin, as bigger files are not a huge issue but gazillion small files are. Also, crash safety is easier to work towards with more atomic writes. I might still change the plans if I run into huge issues since we are at the moment on a deadline.

Regardless of how it ends up looking like I am already certain the solution will not perfect and help wanted/enhancement issues should be appearing in the near future :)

from rust-ipfs.

Sounds good, I'm not sure I'm the right person to chime in on architecture planning, but would absolutely be interested to follow along!

from rust-ipfs.