Support pickling about youtokentome HOT 4 CLOSED

Actually, I solved the problem with multiprocessing. The need for pickling was produced by the pytorch bug when using multiple workers in the data loader.

from youtokentome.

This would be nice to have even if we can do multiprocessing without it.

from youtokentome.

Hi, you can do this to pickle the BPE object or any object that contains a BPE encoder.

import youtokentome as yttm

def bpe_reduce(self):
	return (
		self.__class__,
		(str(MODEL_PATH),),
	)
yttm.BPE.__reduce__ = bpe_reduce

Then pickle.dump any instance of youtokentome.BPE.
This will prevent pickle from saving the BPE object, but it will be able to re-instantiate it given the tuple (class, model_path) when you load your pickle file.

from youtokentome.

Closed in #81

from youtokentome.