MPU parts are removed on storage nodes during the mako-finalize portion of an MPU after being concatenated into a single contiguous file. The metadata shards that hold information about the MPU parts aren't notified that the parts have disappeared from disk.

This complicates things for tools using sharkspotter data to restore object redundancy. There are a couple things sharkspotter can do to work around the current MPU functionality:

1. Make sure that the upload associated with given MPU parts are committed.
2. Ignore all MPU part records

For option 1), If the MPU is committed then the objects found in the manta table can definitely be ignored. If the MPU is not committed then the MPU could be in-flight things are less clear. Maybe the MPU will be finalized and committed between the time sharkspotter is run and the data is analyzed for corruption (meaning HTTP 404s for MPU parts), or maybe the MPU will still be in flight when the data is analyzed for corruption (meaning an HTTP 200 for MPU parts).

As we can see, option 1) doesn't simplify things much. At the end of the day we may have some files that are marked as missing, but were actually part of a committed MPU. So no matter what we'll have to do some post-processing to determine whether or not MPU parts are legitimately unaccounted for.

I suggest sharkspotter implements option 2). We have precedent to take this approach (MANTA-3410). To do this I think we could use a regex that ignores all objects in each user's 'uploads' directory.

Sharkspotter is pretty underdocumented right now. It would be nice for folks wanting to extend sharkspotter if it included some basic docs on how sharkspotter works and why you might want to use it.

Currently sharkspotter can only be pointed at a single Moray shard. This was the quick and easy solution, and led to the addition of kick_off_sharkspotter.sh.

Sharkspotter data is limited in its usefulness without data from multiple (all) shards in a Manta deployment.

We'll also need to handle errors better in sharkspotter if we wanted to implement this change. Currently sharkspotter basically gives up if its Moray connections die. This would be bad behavior if we have 100+ shards and one of them flopping causes all 100 sharkspotter searches to quit. #3 is filed for this.

Adding some more monitoring to sharkspotter would also be necessary. If one Moray backend dies out of 100 it would be time consuming to wade through the noise of log files to discover where in the scan the Moray backend died. #4 is filed for this.

Currently sharkspotter basically gives up if its Moray connections die. So in this case the operator would have to a) notice and b) manually restart sharkspotter from the last successful query chunk (identified by looking in the sharkspotter logs).

Error handling improvements would hopefully include restarting the search for the failed Moray backend when we're able to connect again from the last known successful query chunk.

There's probably a lot more low-hanging fruit to improve error handling too.

Sharkspotter is deployed by creating a tarball from this repository, copying it somewhere, and extracting the tarball. A node binary needs to be shipped with this library too, which is not checked in to this repo.

We should:

• add a Makefile to this repo
• bundle a node binary
• add a linter and style checker

and if we're serious about this tool we should consider bundling/moving it to the mola repository so it can be shipped in the Manta 'ops' zone.

The easiest way to figure out the progress of a sharkspotter process is to run the bundled progress.d script to watch progress through a specified shard.

It would be nice if there were a CLI 'adm'-type command to interact with a running sharkspotter process to:

• get info about the configuration (shard, storage_id, progress through table, number of records discovered, etc.)
• get info about any errors that occurred and how far through the table sharkspotter was when they occurred