web3-storage / backup Goto Github PK

View Code? Open in Web Editor NEW

This project forked from nftstorage/backup

0.0 0.0 0.0 469 KB

A tool to backup all data for disaster recovery.

JavaScript 94.40% Dockerfile 0.96% Shell 4.64%

backup's People

Contributors

backup's Issues

Some errors are not for modifying

Seeing

2023-02-23T14:30:10 2023-02-23T14:30:10.248Z backup:nft-2.json failed to backup xxx TypeError: Cannot set property code of  which has only a getter
2023-02-23T14:30:10     at withChunkTimeout (file:///home/circleci/app/ipfs-client.js:113:16)
2023-02-23T14:30:10     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-02-23T14:30:10     at async IpfsClient.dagExport (file:///home/circleci/app/ipfs-client.js:54:5)

from

backup/ipfs-client.js

Line 113 in 7af03a2

err.code = 'ERR_TIMEOUT'

as we try to modify the error.code property

Migrate data from IPFS Cluster

We need to migrate data off of IPFS Cluster that is not already in buckets. Rough plan:

Determine list(s) of CIDs to migrate

Consider both upload.backup_urls as well as backup table when determining whether to backup
web3.storage also needs to consider psa_pin_request

Changes needed to this repo

Read an ndjson list of things to backup
Start from index (for restarts)
Replace updating DB with logging success failure
Output results

Process hangs without feedback

When running as a container in ECS, this process has a tendency to lock up. The container is still running, the process is still up, but it stops logging.

I think there are a couple suspects here:

running node as pid 1

Node.js was not designed to run as PID 1 which leads to unexpected behaviour when running inside of Docker. For example, a Node.js process running as PID 1 will not respond to SIGINT (CTRL-C) and similar signals.
– https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals

npm-run-all does not exit when the node script exits, it looks like the ipfs daemon process remains alive, and so the whole process keeps running, even tho it will no longer do the thing.

Ideally backup should exit if the node script exits.

Tasks sometimes fail on start

3/31/2023, 3:47:29 PM | node:internal/deps/undici/undici:11279 | backup
3/31/2023, 3:47:29 PM | fetchParams.controller.controller.error(new TypeError("terminated", { | backup
3/31/2023, 3:47:29 PM | ^ | backup
3/31/2023, 3:47:29 PM | TypeError: terminated | backup
3/31/2023, 3:47:29 PM | at Fetch.onAborted (node:internal/deps/undici/undici:11279:53) | backup
3/31/2023, 3:47:29 PM | at Fetch.emit (node:events:525:35) | backup
3/31/2023, 3:47:29 PM | at Fetch.terminate (node:internal/deps/undici/undici:10534:14) | backup
3/31/2023, 3:47:29 PM | at Object.onError (node:internal/deps/undici/undici:11374:36) | backup
3/31/2023, 3:47:29 PM | at Request.onError (node:internal/deps/undici/undici:8168:31) | backup
3/31/2023, 3:47:29 PM | at errorRequest (node:internal/deps/undici/undici:10220:17) | backup
3/31/2023, 3:47:29 PM | at TLSSocket.onSocketClose (node:internal/deps/undici/undici:9668:9) | backup
3/31/2023, 3:47:29 PM | at TLSSocket.emit (node:events:525:35) | backup
3/31/2023, 3:47:29 PM | at node:net:322:12 | backup
3/31/2023, 3:47:29 PM | at TCP.done (node:_tls_wrap:588:7) { | backup
3/31/2023, 3:47:29 PM | [cause]: SocketError: other side closed | backup
3/31/2023, 3:47:29 PM | at TLSSocket.onSocketEnd (node:internal/deps/undici/undici:9647:26) | backup
3/31/2023, 3:47:29 PM | at TLSSocket.emit (node:events:525:35) | backup
3/31/2023, 3:47:29 PM | at endReadableNT (node:internal/streams/readable:1359:12) | backup
3/31/2023, 3:47:29 PM | at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { | backup
3/31/2023, 3:47:29 PM | code: 'UND_ERR_SOCKET', | backup
3/31/2023, 3:47:29 PM | socket: { | backup
3/31/2023, 3:47:29 PM | localAddress: '10.5.4.173', | backup
3/31/2023, 3:47:29 PM | localPort: 54664, | backup
3/31/2023, 3:47:29 PM | remoteAddress: '104.18.23.52', | backup
3/31/2023, 3:47:29 PM | remotePort: 443, | backup
3/31/2023, 3:47:29 PM | remoteFamily: 'IPv4', | backup
3/31/2023, 3:47:29 PM | timeout: undefined, | backup
3/31/2023, 3:47:29 PM | bytesWritten: 244, | backup
3/31/2023, 3:47:29 PM | bytesRead: 1672024 | backup
3/31/2023, 3:47:29 PM | } | backup
3/31/2023, 3:47:29 PM | } | backup
3/31/2023, 3:47:29 PM | } | backup
3/31/2023, 3:47:29 PM | Node.js v18.15.0

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.