No output when using index file about hadoop-lzo HOT 6 CLOSED

myInput.lzo.index is not an input file itself. An lzo index file itself exists to help the lzo input to handle the actual lzo file more efficiently.

from hadoop-lzo.

Sorry, thanks for the clarification. A little confusing with the documentation saying "Running MR Jobs over Indexed Files: Now run any job, say wordcount, over the new file".

from hadoop-lzo.

So how does one pass the index file? I run the above, get the .index file and then pass the original file through existing code and get a single split.

from hadoop-lzo.

You don't directly pass the index file. The path of the index file is implicitly assumed to be (lzo file name).index (see https://github.com/twitter/hadoop-lzo/blob/master/src/main/java/com/hadoop/compression/lzo/LzoIndex.java#L41). The lzo file should be large enough to cause multiple splits.

from hadoop-lzo.

If I decompress the file and run it I get multiple splits, keeping it compressed I always get 1 split. This is happening across hundreds of files. Does this mean I'm not creating the index file properly?

from hadoop-lzo.

As long as you use the right input format (e.g. LzoTextInputFormat) and the file is big enough, you should see multiple splits. You might want to play with LzoIndex to see if you can find out more of your file.

from hadoop-lzo.