Comments (8)
Well! The DefaultMapType
config definitely fixed the original issue here! Pushed to my branch.
I'm not sure if there's other types where this would fail because I'm less familiar with the lexicon than the others in this thread. Seems like a plausible short term solution with a plausible long term follow-up of upgrading to cbor v2.5.0 when it's out.
from indigo.
I will need to look, but off the top of my head:
- this known issue with some posts not being output correctly (probably not a parse issue though)
- a month ago, the repo MST tree would be "complete" based on spidering CID links out of DAG-CBOR records. aka, every target of a CID would be in the repo. during a lexicon refactor, we now make "blobs" (eg, images) contain CID links. the blobs are not "in" the repo, they are stored separately. there is no current way to get a CAR file with repo and all the blobs as blocks. so if this code is still spidering it will fail. the
ipfs
CLI tool chokes in this particular way on newer records
from indigo.
(I haven't looked into what the panic is; just filing before I forget.)
from indigo.
Removing the recover, here's one of the panic stacks:
app.bsky.actor.profile/self
panic: unreachable
goroutine 1 [running]:
github.com/polydawn/refmt/json.(*Encoder).flushValue(0x140002104d0?, 0x101ebddf9?)
github.com/polydawn/[email protected]/json/jsonEncoder.go:218 +0x204
github.com/polydawn/refmt/json.(*Encoder).Step(0x140002104d0, 0x14000562150)
github.com/polydawn/[email protected]/json/jsonEncoder.go:121 +0x89c
github.com/polydawn/refmt/shared.TokenPump.Run({{0x1019c9000?, 0x14000032960?}, {0x1019c9080?, 0x140002104d0?}})
github.com/polydawn/[email protected]/shared/pump.go:35 +0x80
main.cborToJson({0x14000afb824, 0x1da, 0x1da})
github.com/bluesky-social/indigo/cmd/gosky/main.go:763 +0x33c
main.glob..func18.1({0x140000b2e00, 0x1b}, {{0x1400015d620?, 0x0?}})
github.com/bluesky-social/indigo/cmd/gosky/main.go:640 +0x1b0
github.com/bluesky-social/indigo/repo.(*Repo).ForEach.func1({0x1, {0x140000b2e00, 0x1b}, {{0x1400015d620, 0x24}}, 0x0})
github.com/bluesky-social/indigo/repo/repo.go:314 +0x4c
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x14000133178?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
github.com/bluesky-social/indigo/mst/mst.go:936 +0x210
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x140001332e8?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x14000032230?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x1400004c040?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/repo.(*Repo).ForEach(0x140001c8000, {0x1019d1fa8, 0x140001a2008}, {0x0, 0x0}, 0x14000133700)
github.com/bluesky-social/indigo/repo/repo.go:313 +0x148
main.glob..func18(0x14000527340)
github.com/bluesky-social/indigo/cmd/gosky/main.go:623 +0x248
github.com/urfave/cli/v2.(*Command).Run(0x101f2d680, 0x14000527340, {0x14000527380, 0x4, 0x4})
github.com/urfave/cli/[email protected]/command.go:274 +0x75c
github.com/urfave/cli/v2.(*Command).Run(0x140002286e0, 0x14000527140, {0x1400018e050, 0x5, 0x5})
github.com/urfave/cli/[email protected]/command.go:267 +0x97c
github.com/urfave/cli/v2.(*App).RunContext(0x14000564000, {0x1019d1fa8?, 0x140001a2000}, {0x1400018e050, 0x5, 0x5})
github.com/urfave/cli/[email protected]/app.go:332 +0x604
github.com/urfave/cli/v2.(*App).Run(...)
github.com/urfave/cli/[email protected]/app.go:309
github.com/urfave/cli/v2.(*App).RunAndExitOnError(0x14000564000)
github.com/urfave/cli/[email protected]/app.go:373 +0x48
main.run({0x1400006e768?, 0x100ec50e8?, 0x101ecd850?})
github.com/bluesky-social/indigo/cmd/gosky/main.go:91 +0x5fc
main.main()
github.com/bluesky-social/indigo/cmd/gosky/main.go:42 +0x30
exit status 2
from indigo.
oh yeah, this is a bug in the refmt library, it doesnt properly convert certain things. Need to just swap that out i think
from indigo.
I think this might be a generic problem with any CBOR to JSON conversion. For instance, CBOR allows for non-string keys in its maps and JSON libraries typically don't (because the JSON spec says not to). The panic in remft might be caused by that.
As evidence, I did a quick conversion from github.com/polydawn/refmt
in cborToJson
to github.com/fxamacker/cbor/v2
and encoding/json
with a generic map[string]interface{}
in the middle. That errored out because fxamacker/cbor
returns the avatar
and banner
maps as map[interface{}][interface{}]
. Those maps have only string keys, but a generic cbor parser can't assume that. Code in a branch on my fork
I believe there's a few options available:
- Give up on the JSON encoding dream and print the raw Go types returned
- Have an explicit set of types that can be pretty-printed and fallback to raw Go types or the original data when an unknown type is found (That is, parse the CBOR twice. Once to get the top-level
$type
value and the second to parse as the matching Go type for that value. It's unfortunate that the$type
key is at the same level as the object's fields, but whaddaya gonna do) - Walk the generic maps to copy all of them and flip the key types to strings for maps and then error when the conversion is impossible. This assumes the cbor would never contain non-string keys for maps
- Convince a cbor library like
fxamacker/cbor
to provide pretty-print tooling for generic data
from indigo.
FWIW, I'm in the middle of writing up better public specs for our data model (use of JSON and CBOR) and the Lexicon schema definition library. Those should end up in review over at https://github.com/bluesky-social/atproto-website/pulls in the next couple days, and might help nail down the space of data objects that need to be handled (or not).
from indigo.
That errored out because fxamacker/cbor returns the avatar and banner maps as map[interface{}][interface{}]. Those maps have only string keys, but a generic cbor parser can't assume that.
Hi @jmhodges, not sure if it can be used here, but the DefaultMapType
decoding option in fxamacker/cbor
can be useful when converting between CBOR and JSON. It specifies Go map type to create and decode to.
Instead of decoder returning "map[interface {}]interface {}", it will return "map[string]interface {}" like this:
b, _ := hex.DecodeString("a161616141") // {"a": "A"}
// Decoding option to allow user to specify custom default map type.
dm, _ := cbor.DecOptions{
DefaultMapType: reflect.TypeOf(map[string]interface{}(nil)),
}.DecMode()
var v interface{}
_ = dm.Unmarshal(b, &v)
fmt.Printf("decoded: %v (%T)\n", v, v)
// decoded: map[a:A] (map[string]interface {})
https://go.dev/play/p/_og5ySCtqYN
Convince a cbor library like fxamacker/cbor to provide pretty-print tooling for generic data
BTW, it isn't exactly pretty printing, but recently fxamacker/cbor
v2.5.0-beta3 added functions that produce Extended Diagnostic Notation which is defined in Appendix G of RFC 8610: Concise Data Definition Language (CDDL).
Hope this info is useful. FWIW, I found CDDL and EDN to be incredibly helpful in defining CBOR-based data format and creating a codec for it.
from indigo.
Related Issues (20)
- backfill: Events received via the firehose are not buffered correctly HOT 5
- getLatestCommit on Relay returns 500 for (some?) deleted accounts HOT 3
- My Data (.getRepo files) Not Easy HOT 2
- Palomar: Issue with Over-Normalization of Japanese Characters in Search Results HOT 6
- backfill: rev is not loaded from DB HOT 1
- OPENSEARCH_JAVA_OPTS required to build palomar opensearch behind proxy.
- backfill: Impossible to implemented custom store HOT 1
- possible additional search features
- backfill: broken for postgresql HOT 1
- search: profile typeahead not working, full search does, for single test account
- palomar: When the search conditions are set to Japanese and English with |, only Japanese search results are displayed HOT 2
- initial value of perDayLimit (0) causes failures on lots of features.
- search: doing a mention search for an account with invalid handle falls through to "everything" (not an error)
- How to store blobs from another struct type? HOT 5
- Bug on convodefs.go HOT 2
- ModerationQueryStatuses always return nil and missing onlyMuted param HOT 1
- automod/hepa refactors
- identity package improvements
- bug: self-hosting relay raises nil pointer exception on creating account in PDS.
- Relay: getRepoStatus endpoint is not implemented
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from indigo.