KeyError in _t_child from weakref about glom HOT 5 CLOSED

Yeah that's a pretty straightforward case (covered by the tests, etc.), so it's odd that it'd fail. Some idiosyncrasy around Python and PySpark? Maybe just a latent weakref-related bug that's coming out because your data is bigger and more collections are happening.

Thanks for the report, and I'll take a closer look shortly!

from glom.

Best guess is pickling.

Because we store the path itself in a global weakref dict the default pickle behavior leaves it behind.

Need to check if we can override the behavior, or maybe rethink the design of having T instances be "empty".

from glom.

Thanks for looking. Pickling sounds likely. It also turns out it's failing on the first object that hits the worker.

from glom.

Pickle issue reproduced and ostensibly addressed in the commit above. T is pretty special, it definitely needs a smidge of special handling. Kurt and I pickle so little that it slipped our minds, but really who pickles directly these days? It's always these underlying libraries that put us in a sour state ;P

from glom.

To circle back, the fix for this was released v18.3.0 last night. Full changelog here: https://github.com/mahmoud/glom/blob/master/CHANGELOG.md

Thanks again for the report!

from glom.