Index for WET files? about commoncrawl HOT 5 CLOSED

The Common Crawl index also provides offsets into the WARC file, which could be used to estimate the offsets in the WET file.

Can you please briefly explain this. How do I estimate the offset?

from commoncrawl.

Hi,
the better place for questions would be the Common Crawl user group at
https://groups.google.com/forum/#!forum/common-crawl
You'll probably get a quick answer from users in various time zones.

Unfortunately, we do not provide an index to WET files. It's easy to achieve the location of a WET (or WAT) file given a WARC file:

• replace /warc/ by /wet/in the path
• add .wet before the suffix .gz

s3://commoncrawl/crawl-data/CC-MAIN-2016-18/segments/1461860125175.9/warc/CC-MAIN-20160428161525-00221-ip-10-239-7-51.ec2.internal.warc.gz
s3://commoncrawl/crawl-data/CC-MAIN-2016-18/segments/1461860125175.9/wet/CC-MAIN-20160428161525-00221-ip-10-239-7-51.ec2.internal.warc.wet.gz

The Common Crawl index also provides offsets into the WARC file, which could be used to estimate the offsets in the WET file.

from commoncrawl.

@sebastian-nagel Can you please briefly explain this. How do I estimate the offset? Thank you. Ming

from commoncrawl.

Closing this issue as it does not belong to this repository. Thanks!

from commoncrawl.

Did you find the answer?

In the CDX file, each WARC has its offset, and the length is written next to the URL, Can I find the same information about the WET files?

from commoncrawl.