Option to manually set random seed globally about dgl-ke HOT 8 CLOSED

The randomness probably comes from DGL sampler. You can try this. https://docs.dgl.ai/en/0.4.x/generated/dgl.random.seed.html
I think it should fix the randomness of the DGL sampler. If not, we'll fix it. Thanks

from dgl-ke.

Thanks for the reply!
After some digging, it seems like OMP_NUM_THREADS also need to be set to 1 to get same outputs from sampler, since the default edge sampler is using multi-threading when creating negative entity list. However, the final output embeddings ares still different. I'm wondering is it even possible to get completetly same embedding from different training process, especially when using multi-process/thread and multi GPU?

from dgl-ke.

If it's multithreading or multiprocessing, I think it's impossible to get it reproducible. I'm not sure if any of the GPU parallel computation can make it non-deterministic as well.

from dgl-ke.

Thanks for your help!
I managed to produce deterministic results on both CPU and GPU. Here's what I've done.

1. Fix random seed of numpy, Pytorch and DGL. For Pytorch on GPU, CUDA random seed and some other CuDNN related options need to be set as well.(https://pytorch.org/docs/stable/notes/randomness.html)
2. Set both --num_thread and --num_proc to 1, and turn off all other multi-thread related options(like --async_update).
3. Set OMP_NUM_THREADS=1. This step is crucial, since even if you set both thread num and procedure num to 1, DGL will still use OpenMP to automatically parallelize other jobs at background, which will introduce randomness.

from dgl-ke.

Good to know. Thanks for showing us how to make the training deterministic. It'll be useful for future users.

from dgl-ke.

3. OMP_NUM_THREAD

Does the second operation matter? And how to set num_thread and num_proc to be 1?

Thank you.

from dgl-ke.

@Megavoxel01, How to set num_threads and num_proc to be 1 and Set OMP_NUM_THREADS=1?

from dgl-ke.

@Megavoxel01 can you please give more detailed information (actual files to modify and code) to get a deterministic model?

Or is there any chance that it has been integrated into dgl-ke since then?

Thanks in advance

from dgl-ke.