Comments (10)
I don't see any obvious way to enable this. Perhaps request functions could also be parameterised by their output type. By default, data would be stored in R, but alternatively you could request that it be saved to disk. But maybe this is outside of the scope of httr.
from httr.
I'd love to have this functionality as well. Here's a link that describes the API that's a beta functionality to the API RSiteCatalyst provides access to:
https://marketing-beta.adobe.com/developer/en_US/documentation/analytics-firehose/overview-1
The functionality here will allow for users to stream the collection of their web analytics data being captured by Adobe Analytics (AA is a competitor to Google Analytics) directly via API, rather than wait for it to be collected and processed by Adobe. I'm not sure R is really the right tool for this operations type use case, but I've already written a lot of functionality against their REST API, so it seems natural to extend my package to include this as well.
streamR seems to use a combination of RCurl and writeLines from base to accomplish writing directly to file from the Twitter API.
from httr.
Well, an easier way of doing this would just be using, say, the downloader package, which does what it says on the tin.[0] The problem would be having httr recognise when something is to be downloaded to disk versus read in as a page. Personally I'm not seeing the argument for this to be built into httr - again, there are other libraries that handle it perfectly happily.
[1] I appreciate base R has download.file, but downloader's implementation handles HTTPS by default.
from httr.
@Ironholds how would you solve the issue that i've presented? as i state in my original links, i need to authenticate first and then download a file that is too big for RAM. downloader
and download.file
do not have the authentication functionality but httr
and RCurl
pull everything into RAM. thanks!
from httr.
Ah, point; I missed the authentication-is-necessary bit. Moral of the story: no commenting on things before 2pm ;p. I suspect the answer is "don't use R" :/. As a general principle, R tends not to be a language oriented around streaming - it's oriented around "I have this one big pile of data all at once". StreamR is an outlier for a reason.
from httr.
all of these scripts authenticate, download, import, and clean public-use survey data sets directly into R. the "quick start blocks" at the top of each script make the download automation a cinch. if i require users to run programs in multiple languages, the time-savings is gone: i might as well just ask them to log in and download the files by hand. if you can answer either of my SO posts, i'd appreciate it. thanks!
from httr.
I had an epiphany and realised that this is really pretty simple to implement. Enjoy :)
from httr.
NICE.
from httr.
Looking forward to checking this out, thanks Hadley!
from httr.
excellent, hadley. when i write up the instruction manual to work with international census data in the r language, i will use write_disk()
for those giant microdata files. thank you for making this possible!!
from httr.
Related Issues (20)
- Vectorise functions HOT 1
- Can't get content for FTP directory listing HOT 4
- OAuth callback url is encoded while it shouldn't.
- R CMD check failure on musl-based systems HOT 1
- Release httr 1.4.5
- is.character(headers) is not TRUE HOT 1
- httr & use_proxy vs RCurl behaviour difference HOT 1
- Forbidden with httr, but works with py requests HOT 7
- Release httr 1.4.6
- Make file upload seekable
- Release httr 1.4.7
- Use httpbin mirror
- modify_url() cannot handle API versioning HOT 1
- httr 1.4.7: Option seekfunction (20167) has unknown or unsupported type HOT 4
- Upkeep for httr (2023)
- Bug: Executing `http::GET()` in parallel results in an error when no single core GET request was issued before. HOT 7
- HTTR Error installing HOT 1
- Limit in extracting content from a request ??? HOT 1
- Sonoma 14.6.1 causing issue with httr
- update demo/oauth2-azure.r to oauth2/v2.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from httr.