Support field type metadata prior to rapidjsonr switch about sergeant HOT 3 CLOSED

@cgivre I noticed that the "metadata" field (and "columns") in the raw JSON out put seems to be on a single line. I'm sure I could answer this q via poking at some Drill Java code but was wondering if you know that said line will always be "a line" (i.e. never wrap). I'm asking b/c I have to do some serious gyrations to support 64-bit integers & would 💙 to be able to line-grep for the "metadata" & "columns" fields, extract it and build the necessary JSON Schema for RapidJSON from said metadata and column names without having to parse the full result set (since it'd mean parsing it twice O_o). The bigrquery package is able to do this as BigQuery's REST API separates query result schema return from query results return and I'm trying to riff as much as possible off of that package.

If not, I may just remove the rapidjsonr compiled bits and try to hack 64-bit integers from conforming mal-parsed (by jsonlite) doubles. i.e. write something like:

is_BIGINT <- function(col) {
  is.numeric(col) & 
    v>-2^53 & v<2^53 & 
    (floor(col) == ceiling(col))
}

and run that on a mal-parsed double columns to validate that it can, in fact, be converted on the R-side w/o running into conversion issues.

(or I could just 🤷‍♂️ and note that folks should use the ODBC driver access to Drill since it does support 64-bit integers)

from sergeant.

@hrbrmstr This is interesting. When I was testing this I used the Cocoa REST API tester and everything seemed to be on new lines (pretty printed).

If I'm understanding what you're trying to do, you'll need both the columns and the metadata arrays from the result. I'm wondering would this work for you?

Moving the metadata to occur after the columns, but before the rows in the response and that way you could just read through it either line by line or via grep until you hit the end of the metadata section. I think this would be pretty easy to do in Drill and not really have any other unanticipated side-effects.

from sergeant.

that def cld work. it'd be super easy to target them, then.

from sergeant.