Improve speed of extracting rows about weather-tools HOT 1 CLOSED

Here's an idea that should help speed up the extraction of rows: The extraction of rows should be oriented around coordinates and URIs, not just URIs.

In the BQ pipeline...

The second extract step should consume these pairs (do preprocessing for the rows like filter out variables, etc.). and produce rows:

Other tactics to investigate:

• Move these lines outside of the to_rows loop; instead, perform this operation once on the whole xarray dataset:

  weather-tools/weather_mv/loader_pipeline/bq.py

  Line 181 in ecb194c

  temp_row = row_ds.to_pandas().apply(to_json_serializable_type)

• Determine if XArray's native parallelism capabilities are a good fit to produce rows with multiple threads (https://xarray.pydata.org/en/stable/user-guide/dask.html)
• Investigate if there's a light-weight way to get coordinate information for the first of the two processing steps (e.g. can we just get the coordinates and not the data? Can we open the dataset, but not in memory?)

from weather-tools.