filecoin-project / boost Goto Github PK
View Code? Open in Web Editor NEWBoost is a tool for Filecoin storage providers to manage data storage and retrievals on Filecoin.
License: Other
Boost is a tool for Filecoin storage providers to manage data storage and retrievals on Filecoin.
License: Other
Currently it's difficult to understand how the different bars in the fund manager bar chart relate to each other.
Instead, use a similar design to the storage manager, with multiple bars that sum to a whole.
Fixed by #63
Currently we don't delete shards from the dagstore when deals expire or are slashed. This is currently inoffensive, as we don't have a lot of expiration or slashing activity going on. But it must be resolved.
Destroy shards when deals are slashed or expired, after N confirmations. The main complexity item is dealing with the fact that shard are associated with piece CID, and there could be more than one deal for the same piece CID, but not all of them expire at once. Some form of refcounting may be needed.
We need 2 addresses for making deals:
So in boost init
I suggest we:
Basic Information
When a miner fails 3 times to seal, it will give up and purge the sector. The problem this issue describes is that on the client side there is never any notification that the deal has failed. The client just hangs in a StorageDealAwaitingPreCommit
state forever.
I propose that the miner reports back on a final failure state like this so the client status can change to StorageDealError
or some other error state.
Describe the problem
On the client side (1.7.0
), I'm seeing repeated instances of deals getting stuck in StorageDealAwaitingPreCommit
even days after the miner has failed to seal and given up on the sector.
Version
$ lotus version
Daemon: 1.7.0-dev+mainnet+git.a40fb1e1d+api1.1.0
Local: lotus version 1.7.0-dev+mainnet+git.a40fb1e1d
$ lotus version
Daemon: 1.6.0-rc2+mainnet+git.80224bcb8+api1.1.0
Local: lotus version 1.6.0-rc2+mainnet+git.80224bcb8
# Note: this is after a recent downgrade to 1.6.0; one of the failed deals shown below was made last week when miner was on 1.7.0.
# But this bug report is not about the miner-side issue! It is about the client UX, ie, not telling the user that the deal has failed.
To Reproduce
$ lotus client deal --fast-retrieval --verified-deal bafykbzacebzqkg44jer45btajfibw4hnzjpfoxh6f63eg4ridgfcget2wxbyw f0127896 0.0000002 518400
bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei
StorageDealAwaitingPreCommit
.StorageDealAwaitingPreCommit
status never changes to a final error status.Deal status
The output of lotus client list-deals -v
:
ubuntu@dealbot-mainnet:~$ lotus client list-deals -v
Created DealCid DealId Provider State On Chain? Slashed? PieceCID Size Price Duration TransferChannelID TransferStatus Verified Message
Apr 16 14:39:48 bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy 1742150 f0127896 StorageDealAwaitingPreCommit Y (epoch 0) Y (epoch 0) baga6ea4seaqn2csbrgsdpavbcek5rvj7z52gahph65pta53oanbmqnmw6tgnkoq 1.984 GiB 0.1039372 FIL 519686 12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011385 Completed true
Apr 20 14:33:21 bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei 1770094 f0127896 StorageDealAwaitingPreCommit N N baga6ea4seaqn2csbrgsdpavbcek5rvj7z52gahph65pta53oanbmqnmw6tgnkoq 1.984 GiB 0.1039398 FIL 519699 12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011386 Completed true
lotus-miner storage-deals list -v
(relevant lines only):
$ lotus-miner storage-deals list -v
Apr 16 17:39:48 true bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy 1742150 StorageDealError f1wdxdpqh3hirrhp353i4o6ld7bsw6evh3v7i5jtq 2GiB 0.1039372 FIL 519686 12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011385 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Apr 20 17:33:21 true bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei 1770094 StorageDealAwaitingPreCommit f1wdxdpqh3hirrhp353i4o6ld7bsw6evh3v7i5jtq 2GiB 0.1039398 FIL 519699 12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011386
Lotus daemon and miner logs
This issue is about client UX, not the underlying miner bug(s) that caused these deals to fail miner-side. Nevertheless, here are the relevant miner logs.
This is for the 'jhy
deal started last Friday and where client side is still stuck in StorageDealAwaitingPreCommit
(miner was running 1.7.0
):
$ cat lotus-miner-logs* | grep bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy
2021-04-17T03:21:24.726+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventRestart", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealAwaitingPreCommit", "message": ""}
2021-04-20T12:36:49.398+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventRestart", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealAwaitingPreCommit", "message": ""}
2021-04-20T12:37:14.888+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealFailing", "message": "error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed"}
2021-04-20T12:37:14.890+0300 WARN providerstates providerstates/provider_states.go:536 deal bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy failed: error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
2021-04-20T12:37:14.900+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventFailed", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealError", "message": "error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed"}
This is for the 'dei
deal that was started 4 hours ago and is still in StorageDealAwaitingPreCommit
(miner running 1.6.0`):
$ cat lotus-miner-logs* | grep bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei
2021-04-20T17:33:21.924+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventOpen", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealValidating", "message": ""}
2021-04-20T17:33:21.932+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventDealDeciding", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealAcceptWait", "message": ""}
2021-04-20T17:33:21.942+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventDataRequested", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealWaitingForData", "message": ""}
2021-04-20T17:33:22.070+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventDataTransferInitiated", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealTransferring", "message": ""}
2021-04-20T17:37:55.880+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventDataTransferCompleted", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealVerifyData", "message": ""}
2021-04-20T17:38:14.816+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventVerifiedData", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealReserveProviderFunds", "message": ""}
2021-04-20T17:38:14.822+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventFundsReserved", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealReserveProviderFunds", "message": ""}
2021-04-20T17:38:14.827+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventFunded", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealPublish", "message": ""}
2021-04-20T18:38:14.993+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventDealPublishInitiated", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealPublishing", "message": ""}
2021-04-20T18:44:00.484+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventFundsReleased", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealPublishing", "message": ""}
2021-04-20T18:44:00.489+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventDealPublished", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealStaged", "message": ""}
2021-04-20T18:44:09.211+0300 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventDealHandedOff", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealAwaitingPreCommit", "message": ""}
Right now, on the miner, lotus client list-deals
is is still showing StorageDealAwaitingPreCommit
even though both deals have already failed permanently on the miner:
$ lotus client list-deals
DealCid DealId Provider State On Chain? Slashed? PieceCID Size Price Duration Verified
...xg4s3jhy 1742150 f0127896 StorageDealAwaitingPreCommit Y (epoch 0) Y (epoch 0) ...w6tgnkoq 1.984 GiB 0.1039372 FIL 519686 true
...qsi4gdei 1770094 f0127896 StorageDealAwaitingPreCommit N N ...w6tgnkoq 1.984 GiB 0.1039398 FIL 519699 true
Code modifications
None
An integration test that
Error:
2021-11-24T11:53:25.381+0200 FATAL boost boost/main.go:47 creating node: starting node: getting active deals: scanning deal row: sql: Scan error on column index 1, name "TransferSize": converting NULL to uint64 is unsupported
If there is an active deal waiting to be published and we stop boost
, we can't start it back up:
2021-11-24T11:45:20.428+0200 INFO provider storagemarket/deal_publisher.go:241 waiting publish deals queue period of 1h0m0s before publishing
^C%
โ boost git:(nonsense/improve-readme) โ boost run <<<
Worker: 205; Base: 0; Target: 0 (diff: 0)
State: idle; Current Epoch: 0; Todo: 0
Done!
2021-11-24T11:53:25.106+0200 ERROR alerting alerting/alerts.go:105 alert raised {"type": {"System":"process","Subsystem":"fd-limit"}, "message": {"message":"soft FD limit is low","recommended_min":100000,"soft_limit":256}}
2021-11-24T11:53:25.381+0200 FATAL boost boost/main.go:47 creating node: starting node: getting active deals: scanning deal row: sql: Scan error on column index 1, name "TransferSize": converting NULL to uint64 is unsupported
Consider removing the StagingBlockstore
, potentially ClientDatastore
, etc. These components are a bit tangled, currently inoffensive, and potentially hard to remove because of the blockstore muxing in Graphsync, but we should try to get rid of this as it's functional cruft.
At the moment if you use the devnet
binary, or if you run integration tests, they will create artefacts at various places under the $HOME directory of the user.
Ideally we should be creating artefacts under a temporary directory that devnet
creates and then do a cleanup after we shut the devnet down.
Ideas
.while randomly going through some gas trace os PSD message, we realized that for some PSD message that including deals made with clients that's using bls wallet causes up to ~40% gas usage of PSDs to be verifyingsiguature, which can be significantly saved if the client uses secp256k1.
verifySignature: map[crypto.SigType]int64{
crypto.SigTypeBLS: 16598605,
crypto.SigTypeSecp256k1: 1637292,
},
lotus sp should have an option to say they want to reject such deals so that they can decrease the cost of deal onboarding for clients.
At the moment cancellation of deals is racy - refactor the deal state manager to remove the races
Ideas
.Currently in Lotus, after we add a deal to a sector, we remove it from the markets staging area, and pass it to the sealing subsystem which adds a piece to a sector, and then attempts to seal a sector.
However if a sector fails to seal for some reason, even if some deals are still good in it, there is currently no retry mechanism to try to add them to another sector and seal a new sector with the good deals.
We should figure out how to improve this, some ideas:
Given that we persist the fund manager and storage manager state to the SQL DB, it is possible to end up in consistent state if the Boost process crashes between doing the DB updates. The process should auto-repair this state on startup before resuming all the deals.
Write a resolver that connects the storage manager UX to the implementation
In some places we log with the key "err": err, in other places with "error": err
I went through the UI today and took some notes as I was going through, dumping them here for visibility.
Source wallet tooltip is a bit self descriptive. Maybe we should change to something along the lines of "Move funds from the Source Wallet to Escrow to earmark them for deals". It's not immediately clear why I wouldn't just move all my funds from source to Escrow. Can I also move funds from Source to the publish wallet? I don't see UI for that, is that done via lotus directly?
Escrow tooltip is pretty clear
Publish tooltip is also clear
Perhaps we can add some UX here as a call to action for SPs. Can we get a range of how many deals they have escrow for and maybe add a warning label to recommend they move collateral to escrow ("Escrow only has capacity for ~1 more deal, add funds to escrow before making further deals.")? Effectively I'm thinking about triggers for them too:
Currently there is no good backoff mechanism to allow Storage Providers to rate limit incoming storage deal requests. The key problem is around the associated data transfer, as accepting deals immediately triggers data transfer, and too many in progress deals can starve needed resources.
The ARG team has currently proposed an initial iteration of the ASKv2 protocol to enable Storage Providers to respond to ASK requests with a backoff timing, ie: "Hey, come back in 3 hours and let's do this". This should provide an immediate improvement to the existing deal negotiation that Estuary and other clients are making.
Long term solutions may include order books or deal markets to enable easier discovery of SPs who have bandwidth to accept deals immediately.
Dependencies
Currently we are using github.com/libp2p/go-libp2p-core/event as our pubsub mechanism to publish changes to deal state.
This events system
For our use case each deal state change replaces the last, so in reality we need a pubsub mechanism that
Currently the GQL server uses the default file http.FileServer to serve static files from a directory.
However this file server doesn't sandbox the directory, so a user can request /static/../../some/other/file
To run boost the user needs to have two wallets:
These wallets should be passed as required parameters to boost init
.
We should check that the wallets are distinct.
We should store the wallet addresses in config.toml (rather than in the datastore) so that it's easier for the user to see and update their configuration.
Ensure that all resources for a deal are cleaned up when the deal fails or completes
A request from Murmuration Labs is to expand the set of hook slots for external programs. Currently lotus provides Filter
which runs before storage-deal acceptance, and determines whether to accept the deals bytes based on wallet/peerid/payload-cid/other criteria. Request is to expand these hooks as follows:
StorageNegotiationFilter
: same as current Filter
: receives the JSON of the deal on STDIN and indicates with exit0/1 whether the payment/data for the deal should be acceptedFilter
StorageContentFilter
: when specified, upon receipt of the entire DAG from the client, but before inserting it into a WaitDeals sector, an external program receives the DAG determinisitcally on STDIN as a .car stream, and indicates with exit0/1 whether to proceedRetrievalNegotiationFilter
: same as StorageNegotiationFilter
but for retrieval deals ( allows blocking of known-unwelcome wallets/peerids, or commP/payloadCIDs that became classified as problamtic material post-sealing )RetrievalFilter
RetrievalContentFilter
: same as StorageContentFilter
, the dag is fed to on STDIN prior to graphsyncing the first byte to the client.It is likely that the 4 hooks will invoke portions of the same suite, which will coordinate caching and known-block-list updates out of band.
Ideally when we are on the Storage Space
screen (or any other screen), and we CTRL+R the browser, we should end up on the same screen.
When a deal is executed:
This can cause a race if for example the UX requests to list all deals from the database, then subscribes to updates for the deal, because the deal handler keeps subscription information and it may not have been created when the UX makes the subscription.
When boost starts up, it starts the Provider:
If the graphql server starts before the Provider completes startup, a similar race to above can occur.
Latest release
, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.1.13.1-rc1
Storage deals which are in StorageDealSealing
or StorageDealAwaitingPreCommit
or StorageDealStaged
or StorageDealPublish
turn into StorageDealError
when market service is restarted.
See the following deals;
Nov 6 00:45:58 true bafyreidmksbovp3cbi2a2ttcw6x3p3zo55yt6bf5eec5dpwkcxjn6hn7ti 2717173 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 00:46:19 true bafyreibcw3fcsnn5cpmzdqwtdts5olvgw66nanjlijsljthnmqr4xh5t2q 2717174 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 00:50:19 true bafyreicj7yrokjukncujkjwjkdqgf26ozl6qv26tynvi6fi7aghjauietu 2717175 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 01:05:24 true bafyreibaata2yd7fhghopiyyvgcys3veyejl35wf6iayuf7kii76nfk4pa 2717238 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 02:50:35 true bafyreic7cwb2r4dvgdg4mevai4mnnp6pqw4ls6d56fkdqmp4p43jgsh4ni 2717483 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 02:56:39 true bafyreicintk4hbdzaxxgrq74f6mvq3xedquy6tljt6ux475xp5gpmzzhre 2717484 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 03:50:47 true bafyreiaxo3ba6jfy6iti4oegh2qpn6rerz2jrr7pcmfde3zcnainawzmci 2717676 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262178): failed to look up deal on chain: deal 2717676 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 03:55:45 true bafyreiexg7yuqhks6lktlrx7wji7znqya4tv274yf4po63slasf2xtk43m 2717677 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262178): failed to look up deal on chain: deal 2717677 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 04:06:30 true bafyreiaotmfj5v6ej5zs5ecfkffciu5e7oodktovzwjp6pezikozp4e564 2717729 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262208): failed to look up deal on chain: deal 2717729 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 04:37:16 true bafyreihc2sripy6xz3pf2c6432cxgt4gechmzxv6pq32rtpksk2loyzjry 2717859 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 04:50:59 true bafyreignyfnbq27sscqzk4yinjxvc27bc66ytib27uw4dkechmvqixmkra 2717931 StorageDealActive f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800
Nov 6 05:06:48 true bafyreibejxbpd4anrd3hadvoxm2b3jiwf2xcmyoenmlsvurxcmiwdfgmfq 2718015 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262318): failed to look up deal on chain: deal 2718015 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 05:35:55 true bafyreihtg3cpklevcebr72pxoke5vhlnqsc6t67uxcb5twnpcxqvflotbe 2718088 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262378): failed to look up deal on chain: deal 2718088 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 05:50:56 true bafyreigynvrpqgpahjjsywzu4iri3ygksuur2tzbplbm4ui2oyy7rs4wta 2718124 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262398): failed to look up deal on chain: deal 2718124 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:06:42 true bafyreieca7gnlrj4fq55pimnrbk2jc4cxcykxif7yds4b7r7tvvqcpf324 2718164 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262438): failed to look up deal on chain: deal 2718164 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:07:13 true bafyreidepo53als3yofanwxeqmwv3hc5xaxrwtrh637rtsm73zkuy7cp5e 2718163 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262438): failed to look up deal on chain: deal 2718163 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:11:48 true bafyreie5snk7yw3ptvpl45qup5mvte6tjqmnvbttzezuzoi4otxgtj67m4 2718165 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262438): failed to look up deal on chain: deal 2718165 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:21:03 true bafyreibdt7265sryyfdxhsv4j2eqdhwsrfbqndqhovx67juvcoyovvo6qa 2718196 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262468): failed to look up deal on chain: deal 2718196 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:23:40 true bafyreifhvy4yk2bsf6e2g5o7mxfw5dk3i6nxf2edkdiagi5gua3nqjc4me 2718195 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262468): failed to look up deal on chain: deal 2718195 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:26:31 true bafyreidhnjt57fl2lmbcckfs2clzkdxq5zh5el6dzszh22pkom3kftqo5e 2718197 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262468): failed to look up deal on chain: deal 2718197 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:36:06 true bafyreifrivayunk6sroujq5lr6zadb7tvgddf2bqc75325bbjhpapk7kx4 2718226 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262488): failed to look up deal on chain: deal 2718226 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 06:59:30 true bafyreifreqmqxebcpb43vhcxssrzz4prvgt2iqyu2vbs3er3rkiozh3pei 2718300 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262520): failed to look up deal on chain: deal 2718300 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 07:06:13 true bafyreie7nna5jz6kudgig7evt7k67lmyd7nf6iynnxd3ire5ytns2cbjv4 2718301 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262520): failed to look up deal on chain: deal 2718301 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 07:21:11 true bafyreidixezqfciceongk24addgggjvnliotf3hhbmby2bvuxi6zyjhnca 2718344 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262548): failed to look up deal on chain: deal 2718344 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 07:24:21 true bafyreiacqqgacsrto4oeumxcpnnlwaydfg63qq7m7gpgkkdfuukvctrgti 2718345 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262548): failed to look up deal on chain: deal 2718345 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 07:36:13 true bafyreia5jwkeufwqdcse6cz7za3h4u5sbbydcaiv4btokwywvjfwnmvlza 2718405 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262588): failed to look up deal on chain: deal 2718405 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 07:52:07 true bafyreibrvdwolz7gkea6pjj2gbc3wj7nas26todupzf4in2xv6cdydgsoa 2718489 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262618): failed to look up deal on chain: deal 2718489 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 07:58:12 true bafyreifwyu4vmydhvlvdsmsrafa2r7n5sgenbigh27kgnq7fgsp5ermqla 2718488 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262618): failed to look up deal on chain: deal 2718488 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 08:06:15 true bafyreia3rog4gpisuriuxrheeqy6yweoerc2vuvzjgtp2ma4tw2437icnu 2718525 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262628): failed to look up deal on chain: deal 2718525 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 08:21:18 true bafyreifuptvoaw3vcaxsqgu53emahilja5eq4h3zuj3tknu6uimlx4mre4 2718591 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262649): failed to look up deal on chain: deal 2718591 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 08:52:04 true bafyreia73niu2qltjrttzsmeuqratimcu24xpew4eiyzilbp7w3hd44uhm 2718658 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262698): failed to look up deal on chain: deal 2718658 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 09:06:26 true bafyreicxkqy7cqhb6kgt456jtxont3ht5zdqmvxmo2otndslf6d26woz3u 2718707 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262728): failed to look up deal on chain: deal 2718707 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 09:59:48 true bafyreicpe2m6m46dqwbjzvcce2763m67y3766q7i5xj3r7shi2jfgmvlbm 2718816 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262818): failed to look up deal on chain: deal 2718816 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 10:21:31 true bafyreiavhubln2lsvezrwkhnoorgz5ntdl3juokxt4wlvl6t3vcp42qmaq 2718855 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262848): failed to look up deal on chain: deal 2718855 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 10:51:41 true bafyreibencvi42czj5uiacrzu7cgj6ynhnm35rhupyamclkshueqh4gary 2718932 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262889): failed to look up deal on chain: deal 2718932 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 10:54:54 true bafyreiblhse4maio4pl5i4ooagsdzblxlrty22p2csvuw7mxbjz4g2buye 2718933 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262889): failed to look up deal on chain: deal 2718933 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 10:57:07 true bafyreiejqxqkk2ehltyrckhhtbodvdciue7g47t6u7bqgly5qyggycdqjm 2718962 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262909): failed to look up deal on chain: deal 2718962 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 10:57:38 true bafyreigdog3ln2nwzugej3e2t3l2fgfnovztd5zcherty4xsgmqblmg4wq 2718963 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262909): failed to look up deal on chain: deal 2718963 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 11:06:42 true bafyreihu2s7gtnvvepi7jlgphacr3jjlpn3dzch5mgnc3ckxrjhmkcmyku 2718964 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262909): failed to look up deal on chain: deal 2718964 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 11:21:43 true bafyreidmih4mw4plrd3o2pgtivjvbxfvl7u7k2ys7old7ar4wito34fstm 2719005 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262928): failed to look up deal on chain: deal 2719005 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 11:36:03 true bafyreigttdunmsxrmayhishsn2qm6aii2fe3rurmup46px3agfisgzksya 2719041 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262948): failed to look up deal on chain: deal 2719041 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 11:36:36 true bafyreihpt5attymcqqp35xcodisfqkmignhn36eikt52ade74nf7k7wmum 2719040 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262948): failed to look up deal on chain: deal 2719040 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 11:51:45 true bafyreieyy5e7a5z3xbalera4xztvdl5ujaw4s37ophitwkoaqjlelbmfpa 2719081 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262978): failed to look up deal on chain: deal 2719081 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 12:06:44 true bafyreiepsu6ujowyzlj5xw3iw24gv2vk4u6jt3iujbwbqlvcz5z3aoe524 2719125 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262998): failed to look up deal on chain: deal 2719125 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 12:21:46 true bafyreibgyyt7yzdiuvlqcvurywc3t7hzkdvlgyzgadji7y6qyvrtigziai 2719167 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263018): failed to look up deal on chain: deal 2719167 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 12:36:50 true bafyreidhgj4dfyt2zfqc2lyubprsv2ez6emldrt5p4yle7feste57gtene 2719189 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263048): failed to look up deal on chain: deal 2719189 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 12:39:41 true bafyreiatzghj7jdzmmyjj4nbcf4a4ddkilj67l3cws6ymenfzqibkukpsy 2719190 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263048): failed to look up deal on chain: deal 2719190 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 12:53:38 true bafyreia5d34tqqacxlzeddy4r3o6fpedsybizsl62ti35vnpmxdiiknkgm 2719202 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263059): failed to look up deal on chain: deal 2719202 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 12:57:26 true bafyreifw5pqz3bcau7j35zh33ftplxcew6tlrndunzalcndantopmb37vm 2719203 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263059): failed to look up deal on chain: deal 2719203 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov 6 13:07:38 true bafyreieugjvtjggby67kzro7exyys4x4z3qzpfzpap4wsbqyfvtk7jesje 2719266 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 handing off deal to node: packing piece at path fstmp3156441631: AddPiece failed: piece for deal bafy2bzacebdzdazm3qknj6alu7xd6dqjdwcvny2h2dxnbeusl45d46bp7b2eq already pending
Nov 6 13:16:31 true bafyreihzihk3jj3zf4lnnjmh7wfrdqswuccysnrlreqlhbmbmf2xklyo44 2719267 StorageDealError f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi 32GiB 0 FIL 1468800 handing off deal to node: packing piece at path fstmp2267266342: AddPiece failed: piece for deal bafy2bzacebwyrtc5cc6yeetz7yysnv6cpxrftcxnm2s3tnw57wcpgawqxd35k already pending
I restarted the service at 13:18 and 04:40. The weird thing is that the sectors have sealed correctly, the data is in there, and the deals themselve are perfectly fine on the chain.
The provider of the data actually sees the deals as succesful as well (@jsign).
There seems to be no way for me to reset / recheck the state, and the error state is clearly wrong;
$ lotus state get-deal 2717173
{
"Proposal": {
"PieceCID": {
"/": "baga6ea4seaqcnqpj3mfmyyo7iu465csahbniknmscj75sx324ctgwumz4uua6ia"
},
"PieceSize": 34359738368,
"VerifiedDeal": true,
"Client": "f0743060",
"Provider": "f01392893",
"Label": "mAXASIMNcA3y8yQJ84uqgojU8or16Uyz+bL+0k7zRN/esjhWB",
"StartEpoch": 1263370,
"EndEpoch": 2732170,
"StoragePricePerEpoch": "0",
"ProviderCollateral": "5286449256929280",
"ClientCollateral": "0"
},
"State": {
"SectorStartEpoch": 1262622,
"LastUpdatedEpoch": -1,
"SlashEpoch": -1
}
}
-
To avoid confusion with the deal ID that is stored on chain
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.