Git Product home page Git Product logo

ldm-ingest's Introduction

@chswx LDM Parser

Build Status (test suite in progress, don't read much into this yet!)

A series of scripts to ingest and store NWS watch/warning/advisory products via Unidata’s LDM. Will eventually expand to forecast and METAR processing as well; baby steps for now, though.

New in 2.0

All responsibilities for outputting the results of parsing out products from LDM will be placed on the Alerter going forward. Thus, a lot of what's new in 2.0 has been more code deletion than anything else. Version 2.0, instead of handling the entire lifecycle of a request from receiving it from LDM to sending it out over Twitter, will just dump everything in a pub-sub-aware database (currently RethinkDB) and let other worker processes figure it out. This should improve performance, scalability, and redundancy quite nicely. We store things in JSON; this makes it easy to send the data to virtually anywhere, including directly over a socket into a Web browser. (Wink wink.)

What’s done

  • VTEC (Valid Time Extent Code) awareness
  • JSON output
  • Impact-based warnings
  • Storm-based warnings
  • Pub/sub architecture based on RethinkDB

What’s in the works

  • Support for parsing Special Weather Statements/Significant Weather Advisories (#13)
  • Support for watch probabilities (WWUS40) (#14)
  • Parsing and geocoding Mesoscale Convective/Precip Discussions from SPC and WPC (#15)
  • SPC outlooks (#16)
  • LSRs (#17)
  • Tropical advisories (#18)
  • Climate products (#23)
  • Awareness of conditions via METAR ingest
  • Awareness of forecast updates

Pie in the sky stuff

  • Configurable UI
  • Portable installation

See it in action

The current version of the LDM ingestor works behind the scenes.

To see its predecessor in action, follow @chswx on Twitter during inclement weather in Charleston, SC to see examples of the bot’s output.

Questions?

File issues on GitHub against this project or contact me on Twitter: @jaredwsmith

Licensing, warranty, etc.

Licensing TBD. Not for use in life or death situations; code is getting better but the level of rigor is below what I would expect for mission-critical things.

Many Thanks

Many thanks to @blairblends, @edarc and @chswx\LDMIngest for infrastructure and seeding the initial code for this project.

ldm-ingest's People

Contributors

jaredwsmith avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

ldm-ingest's Issues

Tropical weather-specific parsers

  • Tropical cyclone tracks
  • NHC discussions
  • NHC public advisories
  • HLS products from coastal WFOs

VTEC outlines are already available for tropical weather and will be handled in the interim by the generic parser.

Long-running pipe processes

It's inefficient to open and close pipes constantly. Make it to we can have long-running processes to ingest data. Also, make it to where we can explicitly select the parser that will be used to save a few milliseconds of autoloading because I like complexity.

Support multiple VTEC strings

Currently, the ingestor only respects the first VTEC string in a product. However, multiple VTEC strings are often used in the tropical breakpoint products (and can be used in some situations upgrading watches, etc.)

The parser and associated methods need to be updated to permit multiple VTEC strings.

Database schema

Set up tables for the following:

  • Convective Watches
  • VTEC'd Watches/Warnings/Advisories
  • Special Weather Statements
  • Mesoscale Convective/Precipitation Discussions
  • SPC Outlooks
  • Local Storm Reports
  • Tropical Products
  • River Flood Products
  • Forecast data

Implement LSR parsing

Implement parsing local storm reports.

LSRs should export the type, magnitude, description, time, and location (in GeoJSON coordinates) of the report.

Implement Watch Probabilities parsing

We would want to parse watch probabilities so that we can generate graphics with these probabilities, as well as identify watches as PDS watches.

Impact-based warnings

Implement detection of impact-based warnings and expose that data in events so listeners can take advantage of the additional metadata.

Support H-VTEC

At some point, add support for Hydrologic Valid Time Extent Code for better flooding-related metadata.

Reimplement error logging

Need some basic logging in the beta version beyond just writing to stderr. LDM loses track of that, apparently.

Administrative interface

The bot needs an administrative interface of some kind for easily manipulating configuration.

Backing database

Add a backing store for alerts. Backing store is ideally pubsub-capable.

Initial implementation will be RethinkDB, but this should be abstracted away to allow different engines in a future iteration.

  • Implement storage initialization for first-time use
  • Consider table schema for the DB
  • Implement schema for DB
  • Set up product routing/saving accordingly

Implement SPS parsing

Special Weather Statements and Significant Weather Advisories. NWS Charleston, SC does not issue significant weather advisories by name, but many other offices do, so we'll work a little support in.

SPSes can be polygonal or zone-based depending on how they are issued. We should interpret polygonal SPS products in the “significant weather advisory” column. You can also see SPS products for full forecast zones.

Recently, impact-based bullets have made it to the SPS/SWA, so we can perhaps parse those out, too.

Implement job queuing

As the number of events the system processes increases, we'll almost certainly need some sort of queueing backend to handle the load.

Implement mesoscale convective/precipitation discussions (SWOMCD|FFGMPD)

Parsing the mesoscale discussions from SPC allows us to alert on watch probabilities. These may also be helpful for syndication to other advanced platforms.

Parsing mesoscale precipitation discussions may be less helpful for public consumption. Could be good for alerting for more advanced users.

Support storm motion vector (TIME...MOT...LOC) string

Often times special weather statements and warnings will include a TIME...MOT...LOC string from which the time of the storm fix, the storm motion, and the coordinates of the storm can be derived. Add awareness to the parser to send this data along.

Minimum viable API

Begin work on a small API to allow external sites (such as chswx.com) to display our parsed warnings.

Implement pub/sub pattern

This engine needs to be refactored into a publish/subscribe system, which will allow easy implementation of multiple ways to route products (to the Web, to a XMPP chat room, to e-mail, to multiple Twitter accounts, etc.)

Devise a scheme to allow endpoints to subscribe to events generated by the parser. The NWSChat model is not a bad one to follow.

Todos for this to get us back to some sort of operational parity with the current release version:

  • Debugger for output in debug mode
  • New logging endpoint for overall logging ops
  • New Twitter endpoint
  • New Email endpoint

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.