Feature Requests / questions on use --> Pipe, Readme about cdxj-indexer HOT 2 OPEN

You can use pipe to cdxj-indexer by passing in - as the input filename, eg: cat ./my-warc.warc.gz | cdxj-indexer -

The --compress and --lines features are designed to generate a compressed index, similar to the one used by CommonCrawl. It compresses every N lines and also produces an outer secondary index.
Yes, the docs need a bit of updating!

If you are using CommonCrawl data, I believe the cdxj indices should also be available for download along with the WARCs so you don't need to reindex the data..

The trick with parallel processing is finding the boundaries of the gzip records, which may not be too bad..
I'll track this issue to update the README, I don't think I'll have time for parallel processing in the near future unfortunately..

from cdxj-indexer.

Thanks for the prompt reply, and the pointer with regard to pipe, using - is obviously new to me.

With regard to CommonCrawl, while I am using that data, I am functionally only interested in website homepages.
My project to date has revolved around sorting through the dataset to extract only homepages, and ideally nothing rated XXX (lol).
As such the existing indexes are of little value to me (this is why I am here, looking to reindex.)

I took a step back and realized that I am in a good position to multiprocess on a per-file basis, as my eventual files match this format / naming convention, which I get the impression will make sense to you:
CC-MAIN-2020-16_cdx-00040.warc.gz

This may be off-topic for this issue, if so, I understand:
At first glance, it seems most straight forward to output to respective CC-MAIN-2020-16_cdx-00040.cdxj indexes.
Is this expected to negatively impact search time during playback? (Having many many small indexes, instead of a single, large, sorted index?

from cdxj-indexer.