Comments (5)
StefanKarpinski
commented on September 28, 2024
1
It was painful. Hopefully it's actually thread-safe now.
from datasets.jl.
c42f
commented on September 28, 2024
Thanks. Basically we'll need storage drivers for each type of remote data API. So far I haven't had time to make a solid implementation of this, but there was some discussion in #26 related to a Downloads-based backend.
One fairly easy option for cloud storage is to use rclone mount (via Rclone_jll) to mount an S3 prefix. I've got a prototype of that kind of thing lying around separately.
from datasets.jl.
StefanKarpinski
commented on September 28, 2024
Btw, on master Downloads.jl is now thread-safe. I know that was a blocker in various use-cases.
from datasets.jl.
c42f
commented on September 28, 2024
Yes that's great news! I've been following along with some of the Downloads changes but I got a little lost in the list back and forth. It seemed like a difficult fix.
from datasets.jl.
yakir12
commented on September 28, 2024
Any new and exciting development on this front? Some of the online sources would be AWS, FigShare, Zenodo, and growing...
from datasets.jl.
Related Issues (20)
- Why do I need to `open` a dataset twice? HOT 3
- DataSets is hard to use with Distributed.jl HOT 2
- Using data handles with Distributed.jl HOT 4
- JuliaAstro/JuliaSpace use case HOT 1
- New storage backend API based on ResourceContexts
- Creating, deleting and updating datasets HOT 4
- Addressing versioned data
- Refresh documentation
- Large dataset examples
- Rename `BlobTree -> FileTree`
- Iteration of `BlobTree` — `pairs` or `values`? `basename`?
- The road to DataSets 1.0 HOT 1
- DataSets for Testing Use Case HOT 2
- The role of the dataset UUID
- Processing pipeline on dtype level
- Concept of managed datasets for create/update/delete
- API to get the cached path for a blob HOT 1
- Better interoperation with Parquet files
- File materialization functions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datasets.jl.