Error in handling grib file about weather-tools HOT 6 CLOSED

What is the (approximate) URI pattern that you used? Is it possible that you are processing a mismatch of real-time data? From a quick search, this appears to be mismatch in row update and the BQ schema: https://stackoverflow.com/a/37356098

from weather-tools.

I tried with only one file only, something like gs://xxx/xxxx/A1Dxxxxx.bz2

from weather-tools.

This is interesting. I want to rule out the possibility of a disagreement between the data and the BQ schema. On this run, were you creating a new table, or writing to an existing table?

Fixing this bug may require that we implement #50.

from weather-tools.

Writing to an existing table

from weather-tools.

I just spoke with @pramodg about this issue. In looking at the real-time grib data, he observed that the schemas varied quite a bit from file to file. His recommendation is not to infer the schema (which is the default option) and instead to manually pass in the variables that you hope to get from the real-time data. This should prevent any schema mismatch errors.

Can you give this a try to see if it addresses the issue?

In addition to this workaround, I plan to document and/or programmatically inform the user of this behavior in an update to #51.

from weather-tools.

I'm going to close this issue. The root cause is that is some invocations of the tool, the schema and rows being written don't match. This is not a fundamental problem with the tool, but rather, how it's invoked.

We may re-open the issue if it persists for users: maybe there is a better way we can structure weather-mv so it is less error prone.

from weather-tools.