IntChron parser about c14bazaar HOT 6 OPEN

I think that's the way to go, yeah.

from c14bazaar.

Very cool! - I was not aware of this option.

This indeed sounds like an application for an own package, because the data is not as monolithic as for most of the other "databases" (tables) in c14bazAAR. But writing a parser that simply collects everything may be a good first step towards that direction, as you can ignore the user input for now and nail down the tree merge algorithm first.

A PR would be very welcome! ORAU is extremely juicy.

from c14bazaar.

@nevrome That was my thinking too. I have a rough parser at joeroe/c14bazAAR/tree/intchron. It does seem to be worth it – crawling the full database returns over 11,000 dates, most of which are new for c14bazAAR:

intchron <- get_intchron("https://intchron.org/host")
# Or to save time:
# load("playground/intchron-cache-20201009.Rd")
length(unique(intchron$labcode))
#> [1] 11613

all <- get_c14data("all")
sum(!intchron$labcode %in% all$labnr)
#> [1] 9882

But it's extremely slow. Getting the whole database took about an hour on my fast university connection, because we have to make something like ~2000 separate HTTP requests.

So I'm thinking that splitting this off to its own package is a good idea after all. That way you could provide functions for getting subsets of the full IntChron database (e.g. by host/source, by country) and encourage the user to use that granularity in the c14bazAAR parser. Some sort of caching might also help.

from c14bazaar.

Alright - thanks for testing - excellent work! Downloading the whole thing is not feasible then and an own package for specific queries is clearly the way to go.

Maybe one solution to ensure the interoperability with c14bazAAR would be to use the c14_date_list data format for this new package?

from c14bazaar.

I've split the basic API interaction and querying off into its own package: joeroe/rintchron. I'll rewrite the parser on my intchron branch to use these instead. I also managed to get the time taken to retrieve the whole database down to 7 minutes (joeroe/rintchron#3), so I think we're close to it being viable to use as a normal c14bazAAR database, especially if there are separate parsers for ORAU, NCRF, etc.

from c14bazaar.

Great job! So we could go through intchron to get the data from different individual databases? We could write a parser function get_orau() which calls rintchron::intchron()?

from c14bazaar.