Git Product home page Git Product logo

rheas's Introduction

RHEAS Build Status

Join the chat at https://gitter.im/nasa/RHEAS

The Regional Hydrologic Extremes Assessment System (RHEAS) is a modular hydrologic modeling framework that has been developed at JPL. At the core of the system lies a hydrologic model that can be run both in nowcasting (i.e. estimation of the current time period) and forecasting (i.e. estimation for future time periods) modes. The nowcasting simulations are constrained by assimilating a suite of Earth Science satellite observations, resulting in optimal estimates of directly and indirectly observed water variables. The latter nowcast estimates can then be used to initialize the 30- to 180-day forecasts. Datasets are automatically fetched from various sources (OpenDAP, FTP etc.) and ingested in a spatially-enabled PostGIS database, allowing for easy dissemination of maps and data.

Documentation for RHEAS can be found at Read the Docs.

A peer-reviewed journal article on RHEAS is also available.

rheas's People

Contributors

adamjduncan avatar daspecster avatar dcrendon avatar gitter-badger avatar jasonduley avatar kandread avatar kmandreadis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rheas's Issues

Severity index -- too much day-to-day variability?

Hi Kostas I'm looking in your article " Twentieth-Century Drought in the Conterminous United States " and feel RHEAS's "severity/SAD" results have higher elasticity than I should be expecting. Here's Severity measured for the first 13 days at five points in the LMR (this is the fifth of ten ensembles, I believe):

capture

Is the equation supposed to lead to these results? Is this index a better indicator of drought if averaged by month? Thank you.

[trmm] requires end date?

[trmm] gives a net-cdf error if I try to fetch anything after Dec 31 2015. Maybe it should instruct the user to specify/move the end date.

DSSAT nowcast error

I was following the steps in the 'Running a nowcast' example on ReadDocs. When I try to run the DSSAT portion of the nowcast, I get the following error:

-----ERROR-----------------

dropgeometrycolumn

basin.agareas.geom effectively removed.
(1 row)

DROP TABLE
BEGIN
CREATE TABLE
ALTER TABLE

addgeometrycolumn

basin.agareas.geom SRID:4326 TYPE:MULTIPOLYGON DIMS:2
(1 row)

INSERT 0 1
CREATE INDEX
COMMIT
ANALYZE
ERROR: column "type" does not exist at character 102
STATEMENT: select st_value(rast,st_geomfromtext('POINT(35.125 1.125)',4326)) as doy from crops.plantstart where type like 'maize' and st_intersects(rast,st_geomfromtext('POINT(35.125 1.125)',4326)) order by doy
Traceback (most recent call last):
File "./rheas", line 16, in
sys.exit(rheas.run())
File "/home/ksriniva/RHEAS/src/rheas.py", line 81, in run
nowcast.execute(dbname, options)
File "/home/ksriniva/RHEAS/src/nowcast.py", line 197, in execute
runDSSAT(dbname, options)
File "/home/ksriniva/RHEAS/src/nowcast.py", line 182, in runDSSAT
model.run(dssatexe)
File "/home/ksriniva/RHEAS/src/dssat.py", line 729, in run
plantdates = self._planting(lat, lon)
File "/home/ksriniva/RHEAS/src/dssat.py", line 581, in _planting
cur.execute(sql)
psycopg2.ProgrammingError: column "type" does not exist
LINE 1: ...1.125)',4326)) as doy from crops.plantstart where type like ...

-----ERROR END --------------

CHIRPS downloading monthly sums, not daily.

Here's a selection from a histogram showing precip (mm) frequency from a CMORPH multiband raster, with each band representing an individual date, July 1 - 31 2015: http://i.imgur.com/paS2T7V.png

Here's the same for CHIRPS: http://i.imgur.com/Pu87XGb.png

It appears the first of each month (Band 1) is a monthly sum, and the rest of the dates (all obfuscated by Band 31) are empty (notwithstanding an error in Yunnan).

I've double-checked by creating individual tables for certain dates and it's a pattern.

This could be an issue on their end.

DSSAT completion

I am trying to run a DSSAT nowcast using the sample files. When I run a nowcast for the 2013 growing season (start and end dates from Feb - September, 2013) using CHIRPS and NCEP as input, the model runs through to completion. However, if I try to run any other year (I tried 2001, 2005, 2009, 2010, 2011, 2012 and 2014, using the same start and end dates as 2013), I get one of 3 different situations depending on the year:

  1. the model terminates at different plant growth stages with an error that says 'Crop failure growth program terminated'
    Or,
  2. The program runs through to different crop growth stages (without reaching harvest) and says 'RUN COMPLETE, SAVING....'. But the model appears to have terminated way before reaching harvest.
    Or,
  3. I get an error "forrtl: severe (64): input conversion error, unit -5, file Internal Formatted Read"
    I get the third error regardless of year when I try to run a forecast.

Also, does the DSSAT nowcast/forecast save the output to the same database? I dont see any of the 2013 results in the rheas database. Does the save option have to be explicitly defined in the nowcast.conf under the dssat parameters (I have a save: db listed under vic, as outlined in the example).

writeForcings - CHIRPS - Maximum area?

Did you ever come across this when doing Africa? Smaller BBoxes work (for both land and ocean... not an issue that CHIRPS gives ocean 'nodata' as opposed to GPM/TRMM/CMORPH). But for the full LMR BBox I get:

  File "/media/adam/UbuData/RHEAS/RHEAS/src/vic/vic.py", line 397, in writeForcings
    prec[i][2], tmax[i][2], tmin[i][2], wind[i][2]))
ValueError: Unknown format code 'f' for object of type 'str'

Last section of the log... error occurs at the 4419th pixel.

writing data_27.875_98.375
writing data_27.875_98.625
writing data_27.875_98.875
writing data_27.875_99.125
writing data_27.875_99.375
writing data_8.125_93.125

It's not that pixel (which is on an island, btw), which passes smoothly in a smaller area. There's nothing out of the ordinary for the precip.chirps rasters in that area. It also runs fine when replacing CHIRPS/NCEP with CMORPH/NCEP.

Since CHIRPS has the highest spatial resolution and this 5-country LMR BBox has the largest extent that I've worked with, I'm tempted to believe there's a limit to writing forcings. Did you ever run into this?

Installling RHEAS in a cluster

Hi Kostas,

Can same installation procedure be used to install RHEAS in a cluster system? Are additional libraries needed (such as MPI)?

Ensembles overwriting the former?

I've run a hindcast on a tributary basin with five ensembles. I expected there would be five outputs for each date. Instead there's only one, all classified as "5" under ensemble. I was expecting to just run 20 ensembles on, say, SMDI, and using the different outputs, classify each pixel as percentile chance of it having a score below -2 next month. Instead the last ensemble has overwritten all the others. Does each ensemble increase the accuracy of the next, so the last is all we need? I thought each ensemble was independent of the others, each fed slightly different initial conditions.

Mac OSX installation issue

I was trying to install RHEAS on a Mac OSX 10.9.5 and received the following error..Can you please look into this?

------ERROR START------

configure: error: could not find geos-config within the current path. You may need to try re-running configure with a --with-geosconfig parameter.
postgis: cmmi failed: /var/folders/5x/8f6hyr9j0gl83w1l5x2171p00000gn/T/tmp_l071ck5buildout-postgis
While:
Installing postgis.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/Users/karthiksrinivasan/RHEAS/eggs/zc.buildout-2.5.1-py3.4.egg/zc/buildout/buildout.py", line 1992, in main
getattr(buildout, command)(args)
File "/Users/karthiksrinivasan/RHEAS/eggs/zc.buildout-2.5.1-py3.4.egg/zc/buildout/buildout.py", line 666, in install
installed_files = self[part]._call(recipe.install)
File "/Users/karthiksrinivasan/RHEAS/eggs/zc.buildout-2.5.1-py3.4.egg/zc/buildout/buildout.py", line 1407, in _call
return f()
File "/Users/karthiksrinivasan/RHEAS/eggs/zc.recipe.cmmi-1.3.6-py3.4.egg/zc/recipe/cmmi/init.py", line 113, in install
self.build()
File "/Users/karthiksrinivasan/RHEAS/eggs/zc.recipe.cmmi-1.3.6-py3.4.egg/zc/recipe/cmmi/init.py", line 196, in build
self.cmmi(dest)
File "/Users/karthiksrinivasan/RHEAS/eggs/zc.recipe.cmmi-1.3.6-py3.4.egg/zc/recipe/cmmi/init.py", line 221, in cmmi
system("%s %s" % (self.configure_cmd, options))
File "/Users/karthiksrinivasan/RHEAS/eggs/zc.recipe.cmmi-1.3.6-py3.4.egg/zc/recipe/cmmi/init.py", line 34, in system
raise SystemError("Failed", c)
SystemError: ('Failed', './configure --prefix="/Users/karthiksrinivasan/RHEAS/parts/postgis" --with-pgconfig=/Users/karthiksrinivasan/RHEAS/bin/pg_config')

-----ERROR END----------

Status of TRMM, CHIRPS, and CMORPH downloads

Just an update with a request to see if something can be done.

CHIRPS comes from UCSB. They appear to be temporarily blocking IP addresses in SE Asia after a set amount of data has been downloaded. This lasts for a week or two. Last time I had to download at a cafe. I contacted the data guy nearly two weeks ago but haven't received a reply. This is too bad as it's the longest continuous stretch of precipitation data we have and is thus useful for drought indices. I'll try the other contact.

CMORPH and TRMM come from IRI/Columbia. They will now only download two dates. Like CHIRPS, this is an issue for both Susantha and I. Unlike CHIRPS, it's not an issue of blocking -- I emailed the data contact and he assured me this wasn't the case.

When I first downloaded the full CMORPH set, Narendra recommended I only download four months at a time as there were issues with the script cutting out when it had to jump servers. I tested for longer and shorter periods and found he was correct. My best guess is that the arrangement of data on IRI's servers has recently changed, causing this new issue (though note that it's always two days -- doesn't matter if the Julian date is odd or even).

Best I can do is make a Python script to continue loading dates into the INI file and running --update on loop. Is there a way to insert a workaround into the program? Thanks.

DSSAT nowcast fails after running a forecast

I ran into a few DSSAT related issues - I will open separate tickets for each one, to make it easier to track - I dont think the issues are related. This installation is on an Ubuntu machine.

I was running nowcasts for the sample basin file for different years (the outcome of each of those runs is one of the issues that I will open a different ticket for). I then went back to 2001 and tried to run a forecast. The VIC forecast seemed to run through, but I had an error with the DSSAT forecast (that was related to crop growth failure - the subject of a separate issue). So, I came back and tried to re-run a nowcast using a config file that I had used right before the forecast. I got the following error:

-----------ERROR START-------------------
Traceback (most recent call last):
File "./bin/rheas", line 14, in
sys.exit(rheas.run())
File "/home/ksriniva/RHEAS/src/rheas.py", line 89, in run
nowcast.execute(dbname, options)
File "/home/ksriniva/RHEAS/src/nowcast.py", line 199, in execute
runDSSAT(dbname, options)
File "/home/ksriniva/RHEAS/src/nowcast.py", line 183, in runDSSAT
model.run(dssatexe)
File "/home/ksriniva/RHEAS/src/dssat.py", line 698, in run
gid, depths)
File "/home/ksriniva/RHEAS/src/dssat.py", line 335, in readVICOutput
gid, depths)
File "/home/ksriniva/RHEAS/src/dssat.py", line 243, in _readVICOutputFromDB
[r[-1] for r in results if r[1] == ens + 1]) for ens in range(vicnens)]
TypeError: range() integer end argument expected, got NoneType.

-----------ERROR END-------------------

bootstrap.py script errors (setuptools issue)

Hi, I'm trying to install RHEAS and I'm getting an error with the buildout script. I've tried using python 2 on MacOSX as well as Ubuntu. I'm greeted with the same error:

`Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-20.10.1.zip
Extracting in /tmp/tmpbavxw72p
Traceback (most recent call last):
File "bootstrap.py", line 98, in
ez'use_setuptools'
File "", line 173, in use_setuptools
File "", line 124, in _do_download
File "", line 64, in _build_egg
File "/usr/lib/python3.4/contextlib.py", line 59, in enter
return next(self.gen)
File "", line 103, in archive_context
File "", line 87, in new
File "/usr/lib/python3.4/zipfile.py", line 937, in init
self._RealGetContents()
File "/usr/lib/python3.4/zipfile.py", line 978, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
ImportError: No module named 'apport'

Original exception was:
Traceback (most recent call last):
File "bootstrap.py", line 98, in
ez'use_setuptools'
File "", line 173, in use_setuptools
File "", line 124, in _do_download
File "", line 64, in _build_egg
File "/usr/lib/python3.4/contextlib.py", line 59, in enter
return next(self.gen)
File "", line 103, in archive_context
File "", line 87, in new
File "/usr/lib/python3.4/zipfile.py", line 937, in init
self._RealGetContents()
File "/usr/lib/python3.4/zipfile.py", line 978, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file`

It looks like there is no file located at https://pypi.python.org/packages/source/s/setuptools/setuptools-20.10.1.zip

I'm not sure how to jump around this issue. Advice on bootstrapping and building is most appreciated!

severity equation

Sorry, 'severity' is pdsi, right? I'm getting the same issue as smdi where the results don't align with outside examples I'm seeing. I know the index won't be very accurate until I go decades back but again, we're talking orders of magnitude.

Example from March 1 2016, 14 months into the run, NCEP/CMORPH, 0.25, entire region: http://i.imgur.com/Mzly8b9.png

Check when data already exist in database

When updating the database and a start date has been defined in the configuration file, the software does not check if that date has been downloaded already.
Possible solution to add a force flag to overwrite the data, after checking if that date already exists.

smdi equation

K, Could you double-check the smdi equation? The results are way too low (orders of magnitude) for every date I've viewed, compared to the examples I'm seeing in papers. Thanks! A

Example: http://i.imgur.com/zOtBAEn.png

Nowcast running error

Hi Kostas,

I got the below error when running a nowcast for a month period with 0.25 res.

Traceback (most recent call last):
File "./rheas", line 17, in
sys.exit(rheas.run())
File "/home/susantha/RHEAS/src/rheas.py", line 81, in run
nowcast.execute(dbname, options)
File "/home/susantha/RHEAS/src/nowcast.py", line 187, in execute
runVIC(dbname, options)
File "/home/susantha/RHEAS/src/nowcast.py", line 29, in runVIC
runDeterministicVIC(dbname, options)
File "/home/susantha/RHEAS/src/nowcast.py", line 76, in runDeterministicVIC
model.writeForcings(prec, tmax, tmin, wind)
File "/home/susantha/RHEAS/src/vic/vic.py", line 380, in writeForcings

tmin) == len(self.lat) * ndays and len(wind) == len(self.lat) * ndays

when I count the days for precip.trmm table is 30.
But precip.trmm_4 table is also 30. As I know it should be 120 (30*4)

there is no issue for temp and wind.

pls let me know your ideas to sort this out.

Best!

Error when downloading IRI

Updating database!
Traceback (most recent call last):
File "./bin/rheas", line 17, in
sys.exit(rheas.run())
File "/home/oware/RHEAS/src/rheas.py", line 80, in run
update(dbname, config_filename)
File "/home/oware/RHEAS/src/rheas.py", line 68, in update
mod.download(dbname, dt, bbox)
File "/home/oware/RHEAS/src/datasets/iri.py", line 71, in download
t0 = np.where(t == ((dt.year - 1960) * 12 + dt.month - 0.5))[0]
AttributeError: 'tuple' object has no attribute 'year'

Ensemble sizes and assertion errors

I've got CHIRPS & NCEP downloaded from 1981-1-1 for the LMB (Edit:LMR). Running forecasts on a single pixel before moving on. I've run nowcasts all the way through to ensure there is no missing data.

Here's my setup:

[forecast]
model: vic
startdate: 1995-1-1
enddate: 1995-3-1
basin: /media/adam/UbuData/RHEAS/RHEAS/data/PP_Bbox/FN_PP.shp
name: ForecastTest3
resolution: 0.25
ensemble size: 10
method: esp

[vic]
precip: chirps
temperature: ncep
wind: ncep
initialize: no
save to: db
save: soil_moist

Where FN_PP is a single pixel in Cambodia.

I know that there's nothing in the _writeForcing section of ensemble.py stating a maximum ensemble sizes, but I'm running into maximums. For example for 1995, an ensemble size of 5 will work, but 10 will return this:

Read meteorological forcing file
Model State Initialization
Running Model
Total Cumulative Water Error for Grid Cell = 0.0000
Total number of fallbacks in Tfoliage: 0
Total number of fallbacks in Tcanopy: 0
Total number of fallbacks in Tsnowsurf: 0
Total number of fallbacks in Tsurf: 0
Total number of fallbacks in soil T profile: 0
writing data_11.125_104.125
writing data_11.125_104.125
writing data_11.125_104.125
writing data_11.125_104.125
Traceback (most recent call last):
  File "./rheas", line 14, in <module>
    sys.exit(rheas.run())
  File "/media/adam/UbuData/RHEAS/RHEAS/src/rheas.py", line 83, in run
    forecast.execute(dbname, options)
  File "/media/adam/UbuData/RHEAS/RHEAS/src/forecast.py", line 92, in execute
    runVIC(dbname, options)
  File "/media/adam/UbuData/RHEAS/RHEAS/src/forecast.py", line 56, in runVIC
    models.writeForcings(method, options)
  File "/media/adam/UbuData/RHEAS/RHEAS/src/ensemble.py", line 138, in writeForcings
    self._ESP(options)
  File "/media/adam/UbuData/RHEAS/RHEAS/src/ensemble.py", line 207, in _ESP
    model.writeForcings(prec, tmax, tmin, wind)
  File "/media/adam/UbuData/RHEAS/RHEAS/src/vic/vic.py", line 382, in writeForcings
    tmin) == len(self.lat) * ndays and len(wind) == len(self.lat) * ndays
AssertionError

When I run the same but 1985-1-1 to 1985-3-1 (with a new schema name), an ensemble size of 10 WILL work! But when I run the exact same thing again, with a new schema name, I get the assertion error. Restarting/vacuuming the db fixes this.

So the max ensemble size appears to decrease the more data we're using... by 2010, the maximum is 3.

Commenting out the 'assert' in vic.py and running the above ini file gives this:

Read meteorological forcing file
Model State Initialization
Running Model
Total Cumulative Water Error for Grid Cell = 0.0000
Total number of fallbacks in Tfoliage: 0
Total number of fallbacks in Tcanopy: 0
Total number of fallbacks in Tsnowsurf: 0
Total number of fallbacks in Tsurf: 0
Total number of fallbacks in soil T profile: 0
writing data_11.125_104.125
writing data_11.125_104.125
writing data_11.125_104.125
writing data_11.125_104.125
writing data_11.125_104.125
writing data_11.125_104.125
writing data_11.125_104.125
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/media/adam/UbuData/RHEAS/RHEAS/bin/rheas in <module>()
     12 
     13 if __name__ == '__main__':
---> 14     sys.exit(rheas.run())

/media/adam/UbuData/RHEAS/RHEAS/src/rheas.pyc in run()
     81             nowcast.execute(dbname, options)
     82         if "forecast" in options:
---> 83             forecast.execute(dbname, options)
     84 
     85 

/media/adam/UbuData/RHEAS/RHEAS/src/forecast.pyc in execute(dbname, options)
     90         if 'vic' in forecast_options['model']:
     91             if 'vic' in options:
---> 92                 runVIC(dbname, options)
     93             else:
     94                 print "ERROR! No configuration options for VIC model."

/media/adam/UbuData/RHEAS/RHEAS/src/forecast.pyc in runVIC(dbname, options)
     54                           saveindb=True, saveto=saveto, saveargs=savevars)
     55     models.writeParamFiles()
---> 56     models.writeForcings(method, options)
     57     models.run(vicexe)
     58     models.setDates(startyear, startmonth, startday, endyear, endmonth, endday)

/media/adam/UbuData/RHEAS/RHEAS/src/ensemble.pyc in writeForcings(self, method, options)
    136         """Write forcings for the ensemble based on method (ESP, BCSD)."""
    137         if method.lower() == "esp":
--> 138             self._ESP(options)
    139         elif method.lower() == "bcsd":
    140             pass

/media/adam/UbuData/RHEAS/RHEAS/src/ensemble.pyc in _ESP(self, options)
    205             model.endyear, model.endmonth, model.endday = t.year, t.month, t.day
    206             prec, tmax, tmin, wind = model.getForcings(options['vic'])
--> 207             model.writeForcings(prec, tmax, tmin, wind)
    208         cur.close()
    209         db.close()

/media/adam/UbuData/RHEAS/RHEAS/src/vic/vic.pyc in writeForcings(self, prec, tmax, tmin, wind, lai)
    395                 print "writing " + filename
    396                 fout.write("{0:f} {1:.2f} {2:.2f} {3:.1f}\n".format(
--> 397                     prec[i][2], tmax[i][2], tmin[i][2], wind[i][2]))
    398             else:
    399                 fout.write("{0:f} {1:.2f} {2:.2f} {3:.1f}\n".format(

IndexError: list index out of range
LOG:  unexpected EOF on client connection with an open transaction

EDIT: When I run it on a 4x4 area the maximums remain the same, so it is forcing files per pixel, not total.

CMORPH specific date issues

I'm having issues with CMORPH for two specific dates: 2009-04-01 & 2013-04-23. The tables ARE populated with rasters that I can see numerically but do not appear in qgis with

create table xxxx.xxxx as (select rid, raster from precip.cmorph where fdate: 'xxxx-xx-xx')

Unlike all other dates. Multiple tries do nothing. Corrupted at source, maybe? I don't think this used to be an issue. Means that running the following nowcast requires manual editing of the rasters for those certain dates. Maybe nodata can be set as 0 in this sort of case? Regards.

[nowcast]
model: vic
startdate: 2013-04-20
enddate: 2013-04-25
name: Test_GapFinding
basin:/home/ubuntu14/RHEAS/bin/bbox/BBox_Cambo.shp
resolution: 0.25

[vic]
precip: cmorph
temperature: ncep
wind: ncep
save to: db
save: rainf
initialize: off

(Area is subset of Lower Mekong Region, which is:)

minlat: 5.5
maxlat: 28.75
minlon: 92.0
maxlon: 109.75

Dryspells order (Low importance)

I'm making an animated map of dryspells and noticed that the table is not sorted by fdate. I just go CREATE TABLE dryspellssorted AS (SELECT * FROM dryspells2015.dryspells ORDER BY fdate ASC); but others in the future may get confused, as the table does not fit the program's standard output style. Cheers.

Cannot fetch GPM data pre-Sept 2015

Every date I've tried post-September has worked. Everything in September and earlier has not. I've tried it with both Kostas's and my account. This is over the entire Mekong basin.

Trouble writing forcings for forecasts

Trouble with most recent update to Master.

ESP forecast runs if the data exists for range between 'startdate' and 'enddate' (ie if it's a hindcast). Extend the enddate past the exiting datarange (ie forecast), and you get the new "Forcing" error. It always raised this issue in the past but just gave a warning. I tried commenting out the assertion as well but it leads to an error now. So forecast appears to require input from the forecast daterange.

Examples:

I have data to Sept 2 for both CMORPH & NCEP.

Works:

[forecast]
model: vic
startdate: 2016-8-1
enddate: 2016-8-31
basin: /home/ubuntu14/RHEAS/bin/bbox/Kayah16.shp
name: ForecastTestKayahA
resolution: 0.25
ensemble size: 20
method: esp

[vic]
precip: cmorph
temperature: ncep
wind: ncep
initialize: off
save to: db
save: soil_moist, prec

Gets ""Missing meteorological data in database for VIC simulation. Exiting...":

[forecast]
model: vic
startdate: 2016-8-1
enddate: 2016-10-31
basin: /home/ubuntu14/RHEAS/bin/bbox/Kayah16.shp
name: ForecastTestKayahA
resolution: 0.25
ensemble size: 20
method: esp

[vic]
precip: cmorph
temperature: ncep
wind: ncep
initialize: off
save to: db
save: soil_moist, prec

Only difference is enddate.

Nowcast test error: relation "raster_resampled" does not exist

Hi,

I'm trying to run the nowcast test from the documentation [http://rheas.readthedocs.org/en/latest/nowcast.html]

Here are the contents of my conf file:
[nowcast]
model: vic, dssat
startdate: 2003-1-1
enddate: 2003-3-31
basin: data/tests/basin.shp
name: basin
resolution: 0.25

[vic]
exe: bin/vicNl
precip: chirps
temperature: ncep
wind: ncep
initialize: yes
initial state: state/vic.state_20030101
save to: db
save: net_short, soil_moist

[dssat]
shapefile: data/tests/basin.shp
exe: bin/dssat.exe
ensemble size: 50

Here is the full error when running the command ./bin/rheas nowcast.conf :
Traceback (most recent call last): File "./bin/rheas", line 15, in <module> sys.exit(rheas.run()) File "/Users/macbookpro/RHEAS/src/rheas.py", line 81, in run nowcast.execute(dbname, options) File "/Users/macbookpro/RHEAS/src/nowcast.py", line 187, in execute runVIC(dbname, options) File "/Users/macbookpro/RHEAS/src/nowcast.py", line 29, in runVIC runDeterministicVIC(dbname, options) File "/Users/macbookpro/RHEAS/src/nowcast.py", line 75, in runDeterministicVIC prec, tmax, tmin, wind = model.getForcings(options['vic']) File "/Users/macbookpro/RHEAS/src/vic/vic.py", line 353, in getForcings rtables[v] = self.createIndexTable("{0}.{1}".format(v, options[v])) File "/Users/macbookpro/RHEAS/src/vic/vic.py", line 294, in createIndexTable "select * from raster_resampled where sname='{0}' and tname like '{1}_%' and resolution={2}".format(sname, tname, self.res)) psycopg2.ProgrammingError: relation "raster_resampled" does not exist LINE 1: select * from raster_resampled where sname='precip' and tnam...

I also tried updating the database with ./bin/rheas -u data.conf

I'm fairly new to pgsql databases so apologies if I am missing something or this is a straightforward fix

IRI and [domain]

Whenever I try downloading IRI data with a [domain] it returns:

mrc@mrc-VPCZ114GX:/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/bin$ ./rheas -u dataPP.conf
Updating database!
Traceback (most recent call last):
  File "./rheas", line 17, in <module>
    sys.exit(rheas.run())
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/rheas.py", line 76, in run
    update(dbname, config_filename)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/rheas.py", line 64, in update
    mod.download(dbname, dt, bbox)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/datasets/iri.py", line 80, in download
    data = pds.variables["prob"][tt, m, i, j, ci]
  File "_netCDF4.pyx", line 3611, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4/_netCDF4.c:31275)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/eggs/netCDF4-1.2.2-py2.7-linux-x86_64.egg/netCDF4/utils.py", line 391, in _out_array_shape
    c = count[..., i].ravel()[0] # All elements should be identical.
IndexError: index out of bounds

It DOES work without a [domain] but hangs after a dozen or so tiles (creating a corrupted precip.iri that can't be dropped but is deleted by autovacuum). This has been the case for many locations, domain sizes, and years.

Does the [domain] need to align with IRI's grid, maybe, or is it something else?

Thanks,
Adam

Current situation of assimilation dataset downloads

(For the inland Kayah State, Myanmar. I'm happy with SMOS alone but would like to use Modis 15 & 16 as well.)

[domain]
minlat: 19
maxlat: 20
minlon: 97
maxlon: 98

SOIL MOISTURE

SMOS

Works.

SMAP

[smap]
startdate: 2015-05-01
enddate: 2015-05-15
WARNING: Missing options for local dataset smap. Nothing ingested!

AMSRE

Ingests data, but extremely slow... I tried running it overnight last week and it only went through a few dates. Guessing it's a server-side issue.

EVAPOTRANSPIRATION

MOD16

Site has been down for a while; seems to be an issue in North America as well. I'll email the providers.

LEAF AREA INDEX

MCD15

Issue with line 67... looking at it today (LAI2.tif is not being created)

proc = subprocess.Popen(["gdalwarp", "-t_srs", "'+proj=latlong +ellps=sphere'", "-tr", str(res), str(-res), "{0}/lai1.tif".format(outpath), "{0}/lai2.tif".format(outpath)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

Make database schema in lower case

Please make the Schema name for rheas database in the config file all in lowercase irrespective of uppercase or lowercase mentioned in the config file.

fetch global chirps

Just adding this here as getting data back to '81 is vital for some of these indices and forecasting in general.

Name of nowcast.conf

Here's a weird one...

This:

[nowcast]
model: vic
startdate: 2005-1-1
enddate: 2005-12-31
name: basin112
basin:  /media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/data/tests/basin.shp
resolution: 0.25

[vic]
precip: chirps
temperature: ncep
wind: ncep
save to: db
save: runoff
initialize: off

If given a new name like 'nowcasttest.conf' runs perfectly. But if I switch the name to 'nowcast.conf' I get this:

mrc@mrc-VPCZ114GX:/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/bin$ ./rheas nowcast.conf
Traceback (most recent call last):
  File "./rheas", line 17, in <module>
    sys.exit(rheas.run())
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/rheas.py", line 81, in run
    nowcast.execute(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 187, in execute
    runVIC(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 29, in runVIC
    runDeterministicVIC(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 76, in runDeterministicVIC
    model.writeForcings(prec, tmax, tmin, wind)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/vic/vic.py", line 380, in writeForcings
    tmin) == len(self.lat) * ndays and len(wind) == len(self.lat) * ndays
AssertionError

It's fine now that I know, but I guess the system has a memory? An exception would be useful.

TRMM issue

Hi, I've been successful with CHIRPS precip data, but not TRMM, so I haven't been able to run the program on the Mekong.

Just to ensure it's not a problem with my Phnom Penh or Mekong basin shapefiles not aligning with the constraints set out in the INI file, I've replicated the problem with the African test basin shapefile and coordinates, both with the original and freshly downloaded data (which appears to commit to precip.trmm without any errors).

Again, CHIRPS works great, but changing precip to TRMM yields:

mrc@mrc-VPCZ114GX:/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/bin$ ./rheas forecasttest.conf
Traceback (most recent call last):
  File "./rheas", line 17, in <module>
    sys.exit(rheas.run())
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/rheas.py", line 81, in run
    nowcast.execute(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 187, in execute
    runVIC(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 29, in runVIC
    runDeterministicVIC(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 75, in runDeterministicVIC
    prec, tmax, tmin, wind = model.getForcings(options['vic'])
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/vic/vic.py", line 355, in getForcings
    for v in ['precip', 'tmax', 'tmin', 'wind']}
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/vic/vic.py", line 355, in <dictcomp>
    for v in ['precip', 'tmax', 'tmin', 'wind']}
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/vic/vic.py", line 314, in _getTiles
    tiles = [int(r[0]) for r in cur.fetchall()]
TypeError: int() argument must be a string or a number, not 'NoneType'

sri equation

OK first off, SPI is fine.

SRI may be, but every time length (1 month, 3 month, etc), throughout the 14-month nowcast, gets negative infinite values for a couple pixels. Oddly, the location of these pixels change for different time lengths... but all occur in a mountainous region in Yunnan. I can just cut it out, but still... an example centroid is 24.125, 100.375.

I'm assuming these are pixels surrounded by higher elevations?

Example: http://i.imgur.com/Ygdnl38.png

RHEAS crashes when running forecast

Error opening "./tmpUjy52V/vic.state_20140101".
Model run-time error...
Unable to open File
...now exiting to system...

[forecast]
model: vic
startdate: 2014-1-1
enddate: 2014-06-30
basin: /home/oware/data/kecounty.shp
name: kenyaFsim1
resolution: 0.25
ensemble size: 10
method: esp

[vic]
precip: chirps
temperature: ncep
wind: ncep
save to: db
save: prec, soil_moist, runoff
initialize: off

save state: /home/kandread/servir

[dssat]
shapefile: /home/oware/data/kecounty.shp
ensemble size: 40

Ingestion of vic.soils in rheas database

Please look into this matter. During buildout the vic.soils data is not getting ingested due to role associated with it, it always come up as, "the kandread role does not exist" and then discontinue the vic.soils ingestion into the rheas database. However, when done separately after buildout, the vic.soils data is successfully ingested.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.