Move to length-delimited records about oklog HOT 8 OPEN

The main thing that this prohibits again is arbitrary binary records. I understand that's not the primary purpose at this point – it just seems like missing out on use cases oklog would do a great job at.

Of course we could just do escaping when writing to disk. We will be back at having to process every single character though, rather than quickly returning full records without introspection.

I wonder how common the use case of inspecting files directly is. As there are no placement guarantees, going to any given node will just give me a random subset of records. So one would rather go back to query tooling shipped with oklog, which will recover the newlines.

If one does want to interact with files directly, one could imagine an oklog cat command that does the same. It would be one level of indirection of course, i.e. oklog cat myfile | grep .... The question is then whether it's reasonable to assume that the oklog binary is available. On the machine itself most certainly, on local machines it requires a go get ... at least.

from oklog.

Another first-class property of OK Log is that the final segment files on disk are human-readable and greppable. I wouldn't give up this property lightly. I would be much more strongly in favor of using length-delimited records for all inflight operations, and reserializing to \n-delimited on final write. Is this too much of a compromise? Is my goal of plain-greppable log files on disk silly?

from oklog.

It’d be useful to clearly describe some operational (or other) scenarios where the current on-disk plain text encoding is an advantage.

from oklog.

The advantage has always been conceptual: you can still back up, grep, and analyze your logs as files-on-disk with your favorite UNIX tools, independent of the OK Log apparatus on top. You just gotta deal with a weird bunch of letters and numbers at the front of every line.

from oklog.

Another thing to consider is probably that we also touched on a other features (e.g. replica indices) that may require extra metadata. Of course that can also be represented as text, but there could definitely be more things that would move away from the disk format being 100% obvious clear text.

from oklog.

Not sure if it's relevant here, but I wanted to throw in that delimiting records with \n is dangerous in real-life. What would happen when a client logs a stacktrace, where one log entry consists of multiple rows?

from oklog.

OK Log defines log entries (log records) as single-line, as an invariant of the system.

edit: In the context of this issue, the above comment is counterproductive.

from oklog.

I would like to voice my support for having length-delimited records. I also like the suggestion of having an oklog cat command, which reminds me of the zcat command.

from oklog.