Git Product home page Git Product logo

Comments (4)

sckott avatar sckott commented on July 30, 2024

Try this:

library(crul)
url = "https://github.com/ycphs/openxlsx/blob/master/inst/extdata/namedRegions.xlsx?raw=true"
cc <- Async$new(url = url)
res <- cc$get()
class(res[[1]]$content)
writeBin(res[[1]]$content, (f=tempfile(fileext=".xlsx")))
openxlsx::read.xlsx(f)

the data in $content is of type raw, you can write it to a file first, then read with openxlsx

let me know if that works for yoyu

from crul.

moldach avatar moldach commented on July 30, 2024

Thank you very much.
Ultimately I'm trying to adapt this from Async -> AsyncQueue and the url example you provided works when I try it for AsyncQueue like this:

> reqlist <- list(
  HttpRequest$new(url = "https://github.com/ycphs/openxlsx/blob/master/inst/extdata/namedRegions.xlsx?raw=true")$get(),
  HttpRequest$new(url = "https://github.com/ycphs/openxlsx/blob/master/inst/extdata/namedRegions.xlsx?raw=true")$get(),
  HttpRequest$new(url = "https://github.com/ycphs/openxlsx/blob/master/inst/extdata/namedRegions.xlsx?raw=true")$get()
)
> out <- AsyncQueue$new(.list = reqlist, req_per_min=3)
> out$request() # make requests
> out$responses()[[1]]
<crul response> 
  url: https://raw.githubusercontent.com/ycphs/openxlsx/master/inst/extdata/namedRegions.xlsx
  request_headers: 
  response_headers: 
    status: HTTP/2 200
    cache-control: max-age=300
    content-security-policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
    content-type: application/octet-stream
    etag: W/"485760db4f84748f4d4cfa4ea6fa6b5711ca08f7adf6424667f42f171301ee35"
    strict-transport-security: max-age=31536000
    x-content-type-options: nosniff
    x-frame-options: deny
    x-xss-protection: 1; mode=block
    x-github-request-id: 7CA6:7A92:39526:ACBFC:60F05459
    accept-ranges: bytes
    date: Thu, 15 Jul 2021 16:12:46 GMT
    via: 1.1 varnish
    x-served-by: cache-sjc10030-SJC
    x-cache: HIT
    x-cache-hits: 1
    x-timer: S1626365567.904596,VS0,VE0
    vary: Authorization,Accept-Encoding
    access-control-allow-origin: *
    x-fastly-request-id: 0e888ce59af5c93de05cbe1bc3dbdd42cf2c1df3
    expires: Thu, 15 Jul 2021 16:17:46 GMT
    source-age: 0
    content-length: 8209
  status: 200
> class(out$content()[[1]])
[1] "raw"
> writeBin(out$content()[[1]], (f=tempfile(fileext=".xlsx")))
> openxlsx::read.xlsx(f)
  This.is.C2     X2
1       Col1   Col2
2     Data11 Data12
3     Data21 Data22
4     Data31 Data32

However, when I try it on my URL(s) only the Async method works but not the AsyncQueue method? (I get a 504?)

Works

> url <- "https://api.targetsafety.info/api/alertcenter/dataset/param?from_date=2021-04-03&to_date=2021-06-02&version=63973&token=[TOKEN]"
> cc <- Async$new(url = url)
> res <- cc$get()
> class(res[[1]]$content)
[1] "raw"
> writeBin(res[[1]]$content, (f=tempfile(fileext=".xlsx")))
> openxlsx::read.xlsx(f)
   Target.ID Target     Action Genetic.Studies/Variants Genetic.Studies/Variants.(literature.term) Uniprot.ID Gene
1          5    ALK Inhibitors                     <NA>                                       <NA>     Q9UM73  ALK
2          5    ALK Inhibitors                     <NA>                                       <NA>     Q9UM73  ALK
...

Doesn't Work

> reqlist <- list(
    HttpRequest$new(url = "https://api.targetsafety.info/api/alertcenter/dataset/param?from_date=2021-04-03&to_date=2021-06-02&version=63973&token=[TOKEN]")$get(),
    HttpRequest$new(url = "https://api.targetsafety.info/api/alertcenter/dataset/param?from_date=2021-02-20&to_date=2021-04-03&version=63973&token=[TOKEN]")$get(),
    HttpRequest$new(url = "https://api.targetsafety.info/api/alertcenter/dataset/param?from_date=2021-01-01&to_date=2021-02-20&version=63973&token=[TOKEN]")$get()
)
> out <- AsyncQueue$new(.list = reqlist, req_per_min=3)
> out$request() # make requests
> out$responses()[[1]]
<crul response> 
  url: https://api.targetsafety.info/api/alertcenter/dataset/param?from_date=2021-04-03&to_date=2021-06-02&version=63973&token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjAwMDE2IiwibmFtZSI6IkdlbmVudGVjaCIsImRlcGFydG1lbnQiOiJHZW5lbnRlY2ggQVBJIn0.p01jxx7PekOUQLOGjvlhR3YWP2GdcD7_r6MM515bYmM
  request_headers: 
  response_headers: 
    status: HTTP/2 504
    server: awselb/2.0
    date: Thu, 15 Jul 2021 16:25:36 GMT
    content-type: text/html
    content-length: 132
  params: 
    from_date: 2021-04-03
    to_date: 2021-06-02
    version: 63973
    token: [TOKEN]
  status: 504
>  class(out$content()[[1]])
[1] "raw"
> writeBin(out$content()[[1]], (f=tempfile(fileext=".xlsx")))
> openxlsx::read.xlsx(f)
Error in file(con, "r") : invalid 'description' argument
In addition: Warning message:
In unzip(xlsxFile, exdir = xmlDir) : error 1 in extracting from zip file

Please let me know if you'd like DM privately to debug so I could share the token 💁🏼

from crul.

sckott avatar sckott commented on July 30, 2024

Sorry for the long delay. Yeah, can you send me the token so I can try to debug it? You can use the email associated with this pkg

from crul.

sckott avatar sckott commented on July 30, 2024

I get those 504 errors, but I do with every request. I think what's happening is intermittent server downtime, resulting in the 504s. I doubt this has anything to do with Async vs. AsyncQueue. Unless you can make a reproducible example.

By the way, you're token is above in #161 (comment), you should get a new one

from crul.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.