If dataset shrinks, we fail when updating datapackage about pudl-archiver HOT 8 CLOSED

(It seems like this was happening on main but it looks like the above behavior shouldn't happen on main.... investigating further.)

from pudl-archiver.

This approach seems right to me. It also might be smart to add a check to verify that the files we get after refreshing deposition info matches what we expect from the staged changes, and raise an error if they don't (or perhaps do a retry to see if the deposition hasn't refreshed on zenodo's end).

I am trying to figure out why I didn't run across this before, because I do remember doing some tests where files were deleted, and never ran into a problem. Perhaps this arose from the refactor somehow?

from pudl-archiver.

I thought so, but the ZenodoDepositor isn't being used in main yet, and I was able to repro in main.

from pudl-archiver.

(in main there's code that looks like it tries to fetch the deposition's files link - which should get the latest set)

from pudl-archiver.

Oh, but fetching the latest files doesn't get the deletes because they haven't happened yet. That makes sense to me.

from pudl-archiver.

They should've happened on the draft deposition though, right?

from pudl-archiver.

Oh I guess update_datapackage is happening before apply_changes, so the files we get will just be the original state. I think we just need to swap that order

from pudl-archiver.

Yeah, I have a tiny PR coming in a second.

from pudl-archiver.