Shakespeare demo broken at save points about prettytensor HOT 8 CLOSED

FWIW this crash seems to be happening in the summary-writer train-steps, not the "vanilla" run iterations.

from prettytensor.

setting save_path to None avoids this crash but results in no checkpoints. :(

from prettytensor.

I've dug into it and it appears that the problem is the inference graph by default creates summaries, which depend on the placeholder.

Default summaries are great for diagnosing the problem, but I think fixing this correctly requires a scoped method to disable summaries and/or put them in a different graph collection so that the training sub-graph doesn't pick up a dependency on the inference sub-graph.

from prettytensor.

I don't really follow this explanation, but if I remove the inference graph and the calls to it, I can indeed make it train with a non-empty save path. But I see no new files showing up in the specified save path, so I wonder if I've missed something.

Is there another workaround to disable the summaries in the inference graph?

from prettytensor.

I don't really follow this explanation, but if I remove the inference graph and the calls to it, I can indeed make it train with a non-empty save path. But I see no new files showing up in the specified save path, so I wonder if I've missed something.

Is there another workaround to disable the summaries in the inference graph?

from prettytensor.

Should I change something in create_model so that if phase==infer it doesn't construct the extra summaries?

from prettytensor.

There wasn't an option before. With the most recent push you can control where summaries are placed (which graph collection has them).

I will package it up later this week for pip.

from prettytensor.

Uploaded package to pip; see CHANGELOG for more details.

from prettytensor.