To support the GitHub pages work we require a set of folders as follows:

- PRUID
    - ERUID
        - dataset

So a folder for each Province containing subfolders for each Economic Region which contain the datasets for that Economic Region.

• Preprocess the newest revised hmaps for all provinces (against newest exposure, nearest neighbor function) to save time on stack build (Will)
• Update changes in opendrr-data-store, model-factory to create table \ copy only for all the hmap_xrefs (Will, Drew)
• Upload/push preprocessed xrefs to https://github.com/OpenDRR/canada-srm2/tree/master/cHazard/output (Will, Murray)

ongoing while gsheet is going revisions.

Create a sandbox environment with pre-processed scenario outputs. This sandbox will be used to generate static GeoJSON files from database views that can be ingested by RiskProfiler and GeoViewer visualizations.

Currently, geopackage exports of the db are generated locally on my win 10 desktop machine. After newest stack build, Drew backups the db onto S3. I restore the DB locally on my end to run my .bat gdal ogr2ogr scripts to generate the geopackage exports. The geopackages are then uploaded to my cloud storage (one drive / google drive) for dissemination. Whole process may take a few hours and upwards of 30+ gbs.
Would be great if able to automate this process and store the data somewhere (could be compressed?).

location of my bat scripts I use to run on local win 10 env.
https://github.com/OpenDRR/opendrr-data-store/tree/master/scripts/Data%20Export

Set default 0 attributes z1pt0, z2pt5 (vs30) to 'NaN'
All previous default 0 gmvs / commented out (sitemesh,gmv) are now being included in the scenario runs so need to go through all the sql and uncomment gmv sections to include them in the indicators.

Need to update changes in model factory as well.
https://github.com/OpenDRR/model-factory/tree/wkhchow_load_all_schemas_new_exposure/scripts

Current shakemap points/grid has asset/sauid points incorporated / interpolated for the settled area centroids. For viewing purposes in ES, check to see if we can filter out those sauid points to only have grid points for shakemap

Please validate the geometry in PostGIS is valid, comparing with raw geometry (from StatsCan???).

Create a entity relationship diagram that we can use for our documentation.

update hardcoded hazard thresholds with LUT

Will notified me of two GitHub Actions error as seen at https://github.com/OpenDRR/opendrr-data-store/actions, in particular:

• https://github.com/OpenDRR/opendrr-data-store/runs/2605540488?check_suite_focus=true#step:16:52
• https://github.com/OpenDRR/opendrr-data-store/runs/2605766714?check_suite_focus=true#step:16:52

It turns out that the Backblaze B2 tool errors out occasionally when it detected the source files being older than what is already on the B2 bucket:

  File "/usr/lib/python3/dist-packages/b2/sync/policy.py", line 99, in files_are_different
    raise DestFileNewer(
b2.exception.DestFileNewer: source file is older than destination: local://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621298707917 cannot be synced to b2://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621299046514, unless --skipNewer or --replaceNewer is provided
ERROR: source file is older than destination: local://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621298707917 cannot be synced to b2://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621299046514, unless --skipNewer or --replaceNewer is provided

Issues:

• The "GeoPackage exports" GitHub workflow, which takes 4 hours or so to run, is being triggered on every single push and pull request, and that is probably too frequent.
• With it being triggered on each push, and each instance taking 3 or 4 hours to run, when there are multiple pushes in close succession, more than one instance might be trying to uploading to B2 bucket at the same time.
• The error avoids a race condition, so it is "good".
• The error message can easily be overridden with --skipNewer or --replaceNewer, but there should be a better way to resolve this issue.

Create a PostGIS database in the AWS sandbox.

Parent task: #12

We need to push the PostGres views to Elasticsearch. We can do this via python script on Lambda or perhaps we can use LogStash.

As per discussion with @tieganh @plesueur @mjourneay, changes to be revised for DSRA (for all building, sauid, b0, r1 if applicable):

• sLm_Asset / sLm_Bldg - divided by replacement value instead of building number (what is replacement value?)
• remove sH_RuppAbbr, resource sH_RupName from sH_RuppAbbr
• sH_GMPE (OpenDRR/model-factory#57) @drotheram
• remove SH_PGV
• add SH_PGA
• Recalculate all displaced household and shelter indicators with updated Et_SFHshld / Et_MFHshl values (#53). Implement new intermediate tables to handle calculation of each portion of the equation to avoid confusion and for ease of pinpointing errors so that the final equation to calculate the displaced household / shelter can be reduced in size.

Complete validation review of existing transformation scripts for converting OpenQuake model outputs into risk profile indicators for 1) physical exposure layer, 2) multi-hazard threat layer, 3) social vulnerability layer, and 4) sample scenario risk layer. Define additional scenario risk profile indicators for A) Shelter requirements, and B) Return on Mitigation Investment

Current workflow is having site level aggregated to building / sauid in the national exposure. Need to change the scripts so that a separate workflow for site/building/sauid for dsra/psra/exposure is generated along with a set for national building exposure model (no aggregation of site data into national data).

See also OpenDRR/opendrr-api#116.

• Create new S3 bucket (no, not opendrr-api-prebuilt-cache-1)
• Create IAM user (AWS Access key ID with write permission, etc.)
• Edit workflow file (see example at OpenDRR/boundaries

But these files are static web contents and need to be publicly accessible IIRC... We may need to stick with Backblaze B2 for the time being? Updates: No worries, the application for public S3 bucket is pending.

write up some preaggregated hexbin geometry examples for comparison (ie ghsl layers).

At a minimum we require:

• Geopackage
• GeoJSON

A nice to have:

• Shapefile

possible solutions:

-in postgis: - create master dsra table that appends relevant information after each dsra scenario is generated in the stack (which indicators tbd, but the basics assetID, scenario name, pga?, all statscan tags DA,CD,CSD,FSA,PR etc). So a end user who wants to know what scenarios are in for example 'Greater Vancouver' can query against the master table to find out which scenarios are involved. They can then look at the specific scenario results views to get further info.

• in es: ??

Include a combined psra national layers (building, sauid) for FGP in postgis db.
Create geopackage exports.

@wkhchow please take a look at the BuildingBase.sql script that I created using deegree-cli-utility. Instructions found at https://github.com/deegree/deegree3/tree/master/deegree-tools/deegree-tools-config

• review hazard threat prioritization excel from @plesueur for BC
• test/implement in database

issue - Current outputs on the scenario outputs are not clear to me and generating the specific 38 indicators looks like it needs to be manipulated in some ways. Need to meetup with Murray, Tiegan and others for clarification.

end goal - create views based views based on indicators from the scenario outputs

Create the tables and views required to support the static scenario output data.

Parent task: #12

• update load source tables
• all_indicators_b, _s
• make changes for revision in model factory

Discussion with @plesueur and released Et_PPH (people per hectare) should be actually people per household. Suggest to fix this indicator to reflect people per household.
Additional indicators to be edited;
Et_Pop_km2, Et_Pop_Ha (new), Et_PPH, Et_Bldg_Km2, Et_Value_Km2

Update social fabric pieces to incorporate new thresholds for each of the 8 SACTYPES. Update changes to model_factory repo.

As per meeting with @plesueur today:

• update indicators E_AreaKm2, E_AreaHa (OpenDRR/openquake-inputs#15)

• update exposure source daypop for 3 CD's (OpenDRR/openquake-inputs#17)

• update indicators Et_BldgArea, Et_BldgAreaRes, Et_BldgAreaComm, Et_BldgAreaInd, Et_BldgAreaCivic, Et_BldgAreaAgr

• rename indicators Et_Pop_Km2, Et_Pop_Ha to Et_PopDay_Km2, Et_PopDay_Ha

• add new indicators Et_PopNight_Km2, Et_PopNight_Ha

• remove indicators E_CensusBldg

• Et_PPH, may need to be revised. Update to population per hectare, (sum(pop)/sauid_ha)

• change Et_PPH to day pop per building, night pop per building

• update Et_SFHshld / Et_MFHshld based on occupancy lut, include into building level indicators

• investigate source exposure model popdu column and see if we can replicate the same values. May have to check with Murray if unable to recalculate column. unable to replicate popdu column after checking and comparing previous exposure data, using SA (proxied) census population, census dwellings, the values are slightly different compared to du. Need to check with Murray how he derived the indicator? confirmed census population / census dwelling unit

test and update scripts to incorporate removal of gmf/sitemesh table, which would be replaced with 1 shakemap table.

Will need to be updated in model factory
https://github.com/OpenDRR/model-factory/tree/wkhchow_load_all_schemas_new_exposure/scripts

Define specifications for transforming OpenQuake outputs of probabilistic risk assessment models (PSRA) into risk profile indicators for the national earthquake risk model.

Add fsauid, dauid, sactype, saccode tags from sauid geometry to all indicators.
Add landuse tag from exposure to all indicators.
Fix changes in model_factory as well.