dataset downloading. about pytorch-lightning HOT 2 CLOSED

thank you @awaelchli

from pytorch-lightning.

Hey @Esmail-ibraheem

This project, pytorch-lightning, is not the same as Lightning Studio. If you'd like to get help with it, I suggest to post your question in our Discord server in the future: https://discord.com/invite/MWAEvnC5fU

But I can answer your question of course. You are downloading a dataset using the function from huggingface datasets, and this will save the files in a hidden cache folder in your home folder. That's why you don't see it and that's completely normal. This would be the same even on your laptop outside of Studios!

If you'd like to explicitly control where the data is stored, simply pass the cache folder explicitly with the path where you'd like to save:

ds = load_dataset(..., cache_dir="./data")

I'll close the issue now because it's not a bug for pytorch-lightning, but I'll be happy to continue answer your questions here or on Discord.

from pytorch-lightning.