Upload SGFs without debug comments about minigo HOT 8 CLOSED

I agree that we only ever look at 0.01% of the SGFs, but the problem is that we don't really know ahead of time which 0.01% we'll look at... so we end up printing out debug info on all of them. Maybe we only really need debug on 1% of them. @amj thoughts?

from minigo.

i'm happy turning off the per-move debug logs and only logging e.g. root-Q, and optionally doing a random.random() < 0.05 check or something to turn them on for only a few percent.

I think realistically, we had them on to check the re-use/c_puct/d-noise effect. At this point, i don't think the individual self-play games are mined through that much. That said, a few hundred GB is not much in the grand scheme of things.

Possibly a better way to approach the problem it is to streamline our process for stripping the comments before hosting the data, or potentially saving two copies: one with and one without the debugging info.

Incidentally, gzipping the files w/ debug info cut the size by ~7/8ths.

from minigo.

I think that the SGFs are actually the data that Go-people are most likely to consume (especially for 19x19s), so making it small will help cut down on our network costs and also make it more user-friendly, since who really wants to DL 100GB of SGFs.

@amj / @brilee -- are you guys actually using the debug info?

from minigo.

Yes, infrequently ;) I think at the point that you find an interesting game, you really want to know what the computer thought. We did all that work already, and bytes are (relatively) cheap, so we might as well store it.

Now bandwidth is less cheap, on all sides of the table, so providing stripped down versions for download makes a lot of sense.

And for the person who wants 100 GBs of SGFs with the full data -- and i'm sure a couple will! -- that'll be possible too.

from minigo.

FWIW LZ handles this by storing the random seed in the sgf file. Games run with 1 thread are reproducible so you can run with debug on afterwards.

from minigo.

LZ also store the training and debug data in two different files, and autogtp upload both file SGF and training data to a database. You could the same: write two files one SGF with and one without comments, store them both and the the user decide what he wants.

from minigo.

I like the solution with two versions of the SGF.

from minigo.

ok i'll set it up to write both versions.

from minigo.