yosukefk / finn_preproc Goto Github PK

Need to also mention docker stop, maybe docker image rm needed, before docker build?

Turned out that there is "Type" field which distinguish four types of thermal anomaly:

0 = presumed vegetation fire
1 = active volcano
2 = other static land source
3 = offshore detection (includes all detections over water)

For our purpose we want to eliminate record type 1 and 2, keeping only biomass burning and near water detections. Type 3 not only covers offshore but a lot of fire near lake/river are included . For now I assume they are fire we want to keep, not artifacts that to be dropped.

VIIRS detection is more sensitive and seems like detection of industrial fire can be an issue. I come up with a code to identify overlapping detection across days, and if that happens very frequently across long period (year), i flag the region as possible fixed source fire (man made flare)

The current method pretty much start over the raster processing if existing one is not good enough for new application. Since I know which hdf file got downloaded, ( i believe i am recording somewhere in the database), i should be able to tell which 10x10 deg tile needs to be refereshed/added. Grab raster needed for those tiles, then update the raster

Update the raster is probably the hard part, i have to come back to PostGIS raster documentation, understand what i should do...

That would be nice

We should add instructions for Windows users to the README.md file about how to link volumes:

From Yo:

...There are two changes needed to use in CMD.exe on windows. $(pwd) needs to be changed to %cd% , ‘finn’, somehow CMD doesn’t like single quote.

Also:

​I need to go to “setting” of docker to allow access to C: drive (my only drive). I didn’t do that, and the docker run command gives windows prompt if you want let this thing to access C: drive (something like that, couldn’t remember exactly). I said yes, but that was not enough. So I believe this should be written down, something like this https://stackoverflow.com/questions/33126271/how-to-use-volume-option-with-docker-toolbox-on-windows/39181789#39181789

Eventually we need to add instructions for users to use active fire shapefiles that exist locally.

Would be nice if i can scrap the docker container/image and db still persists.

Tell user to QA with QGIS, give some instruction (stick with o_32 version until you know exactly which fire to look at)

After long time of processing and VCF importing, it died like below

440 of 443...
441 of 443...
442 of 443...
443 of 443...

terminal has

INSERT 0 1
INSERT 0 1
CREATE INDEX
COMMIT
ANALYZE
psql: FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
psql: FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
psql: FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
psql: FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
Shapefile type: Polygon
Postgis type: MULTIPOLYGON[2]

next command to use matplot lib had following error, no plots made

Got this error: SSL SYSCALL error: EOF detected

Didn't work, use QGIS!

the rest of task didnt run.

Thought that disk may have got full, but there are plenty of space.

So quite didn't know what happened

Hope this is easy...

right now, i am testing database by !psql posstgres -c 'select version();'. That is good, but that still let the first real psql query fail. my older database had different dbname ('gis') and this should be tested, and other similar problems

i should have something like
!psql -c 'select version()'; # test as finn user
!psql -c '\dn'; # see what schema are there
!psql -c '\dn' -c 'set search_path to af_testOTS_092018, public;' -c '\dt' # check again down the road when tag_af is set

We should clarify the section on opening the Jupyter notebook, e.g., copying the output and replacing two parts:

• localhost
• token

So i tried to reuse MODIS 2017 raster, starting wtih testUSA ipynb file. Then told me that i have to download raster manually!! very unpleasant!

So better have one ipynb file, which (1) test if raster for the year exists, (2) if it does, see if it is sufficient (3) if not, go grab some. This setting should server both purpose of testOTS and testUSA. I'd call this as production version of script for local application using MODIS raster. Recommend user to use this new script to do any real work.

It would be good to add instructions for users to delete hdf files from the filesystem.

Hi Yo,
I can no longer build the finn docker as it fails with the following error:

Reading package lists...
W: Conflicting distribution: http://deb.debian.org/debian stable InRelease (expected stretch but got buster)
W: Conflicting distribution: http://deb.debian.org/debian stable-updates InRelease (expected stretch-updates but got buster-updates)
E: Could not open file /var/lib/apt/lists/deb.debian.org_debian_dists_stable-updates_main_binary-amd64_Packages.diff_Index - open (2: No such file or directory)
W: Conflicting distribution: http://security.debian.org/debian-security stable/updates InRelease (expected stretch but got buster)

From Yo:

​Somehow, the .sh scripts are changed to dos format in my local clone of the finn_preprocessor I got with GitHub desktop. And in the script --password XXX was the last argument in the line. So I believe that my password is appeneded with ‘\r’ and authentication was failing. I had to test by “docker exec -it finn bash” to manually type the command, test with $EARTHDATAPW etc, and I think that’s the problem. Easiest work around is to move the -P downloads to the end of line, guiding the password in the middle of line surrounded by whitespace.

Read the raster, convert burned area pixel into quadilateral (one shape per one pixel). No need to break up, ready to cross with raster. It is kind of stupid that both burned area and land cover starts with raster of same projection.

(It would make more sense to process as is. But land cover raster is already unprojected to geographic, per request by Christine. Better design would be to never mess with raster land cover and always project polygon to match with raster. but need approval from Christine.)

This is meant to save disk space, as described in section 8 of README

It is very confusing now where to have downloaded data, how that got into database etc. Organize things so that

• Ones schema for one raster dataset?
• Need example of user provided raster (use FCCS?). How they should be named, what the schema name would be...?
• Dir for active fire download becomes schema for fire
• Better create a directory to work with AF data set, have notebook there, code separate dir. Intermediates/logs dumped to the work dir.

Test, and support the use case where you want to process both NRT and archive in the same run. Say, if you want to process last 12 month , latest 6 month data is only available as NRT. Older days are replaced with archived version (better QA'd). Does current code supports this mix? If not, let it to work.

May related to #7

Add step-by-step instruction with screen shots to use QGIS

MODIS land tile does not cover entire earth and some tile over the ocean do not have tile. Somehow fire ooccasionally detect on ocean. The current code tries to get raster for that, meaning user cannot bypass the raster download part although they should.

FIX is to prepare MODIS tile's wireframe, and have attributes if LCT/VCF exists for the tile. If fire is on the tile where LCT/VCF does not exists, dont count it as a fire which require download raster

I am wondering if it is a good idea to integrate calcuation of emission into this tool. Currently it is designed to do it in finnemit. But it is written in python. We may be able to wrap it into plpython function. Then use the function with UPDATE query for each line of out_XXX table.

I often do

• stop and start docker container
• restart docker
• remove container and start new one.

Also

• Tell that database is not affected by above.
• Wiping the finn_postproc dir won't "wipe everything and start over", cuz that won't wipe docker containers/images. Need instuction to wipe them. (maybe cocker rm volume to wipe db?)

https://e4ftl01.cr.usgs.gov is not responding now, and hence raster downloader is not working. Will see if this is permanent issue, and if so come up with alternative URL or something.

at the end of main.ipynb, tell user where the file is

This file is now out of date, and can be removed.

It has been difficult to use bind-mounted storage for PosrgreSQL database ($PGDATA directory), because windows filesystem doesnt support chown. Want to know if it goes easy on Linux

We should explain in the README.md file what the EARTHDATAUSER and EARTHDATAPW are, and how to get an EarthData username and password, linking to this page: https://urs.earthdata.nasa.gov/

By default i got only one cpu. What shall i do to use more? And what would be recommended setting, should i match what's available?

With docker, outdb raster may make more sense than loading all the raster into database. I did this earlier on TACC's machine for some reason, but stopped doing that later.
https://postgis.net/docs/manual-dev/postgis_enable_outdb_rasters.html

@mbjoseph Do you think it is possible to write a notebook which visits each work_* directory, run the main_*.ipynb, and repeat the for different directories?

I tried this with two ways, and neither behave the way i wanted. First, my notebook had cells like

%cd work_mod_na_2012
!ls
%run main_mod_na_2012.ipynb
%cd ..

and so on. It runs each notebook, but all the output comes to this batch notebook. Wondering if it is possible to get them to each notebook, so that it is like a log.

Another way I tried was based on this google find.
https://nbconvert.readthedocs.io/en/latest/execute_api.html#executing-notebooks-from-the-command-line
My batch notebook has cell like below.

%cd work_mod_na_2012
!jupyter nbconvert --to notebook --execute main_mod_na_2012.ipynb
%cd ..

I think behavior is the same, output goes to the calling notebook.

Need intstuction for three levels of wiping

1. Rebuild all the docker things ( this will keep the downloaded data etc)
2. Wipe the finn_preproc dir (database is still intact)
3. Remove docker volume "pg_data" as well ($HOME/pg_data if user is on Linux)

Some kind of SQL error, saying that

ERROR from 'step1_prep':
 psql:../code_anaconda/step1_prep_v7m.sql:8: ERROR:  syntax error at or near " "

Not in the original use case, but we have done that before, e.g. using FCCS and CDL combined land cover raster. User provide tif, it got imported to database, and the rest is the same. Hopefully this is easy...

I got disk full error, and needed to see if it is database. Needed to login as bash, df, Would be nice to have such analysis as python script that can be started somewhere.

Dockerfile has conda install gdal, then it test if install was success. Getting following error.

The same error with earlier version of Dockerfile , where @mbjoseph has extra specifics for version/repository of anaconda python, or the latest version.

Step 9/15 : RUN python -c 'from osgeo import gdal'
---> Running in d01d3f8f9339
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.7/site-packages/osgeo/init.py", line 21, in
_gdal = swig_import_helper()
File "/opt/conda/lib/python3.7/site-packages/osgeo/init.py", line 17, in swig_import_helper
_mod = imp.load_module('_gdal', fp, pathname, description)
File "/opt/conda/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/opt/conda/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /opt/conda/lib/python3.7/site-packages/osgeo/../../.././libkea.so.1.4: undefined symbol: _ZNK2H510H5Location15getObjnameByIdxB5cxx11Ey
The command '/bin/sh -c python -c 'from osgeo import gdal'' returned a non-zero code: 1

We should add some information to the README to point users to https://urs.earthdata.nasa.gov/ to get an EarthData account if they do not already have one.

Alex Urqhart is good reference for QGIS access. Still need brief explanation to how to do for QA the data. -e ALLOW_IP_RANGE=0.0.0.0/0 -p 5432:5432 is what needed and unlike what Alex says i can just use localhost as hostname with the latest docker desktop (not toolbox that he used.

Kartoza has instruction for use with psql https://hub.docker.com/r/kartoza/postgis . Basically postgres being installed on host machine and then access through port 5432? Need to test this.

ALternatively I can docker exec -it postgis_novolume /bin/bash and then sudo -u postgres psql -U postgres -W -d gis with password postgres to have psql session

We should add instructions for starting and stopping the container in the README:

​To stop the container, you can either to ^C from the terminal that is actively printing the output from the Jupyter notebook server, or from another terminal use:

docker stop finn

and then to start it again, you can use​:

docker start finn

​Then, once the container is running, you can start the notebook server via the same command that started it originally:

docker exec -it finn jupyter notebook --ip=0.0.0.0 --port=8888 --allow-root --notebook-dir /home/finn​

Currently users may stop the container when they don't need to. Let's put these stop and start instructions toward the bottom of the README instead.

Seems like my docker (Docker toolbox on Windows, having Oracle VM as backend) default to ~19GB, either for a container, or entire sets of docker container. For FINN global application that is probably too little. Need instruction to either (1) secure more diskspace up front (2) expand diskspace when it runs out or (3) store database in host filesystem so that database won't out grow the virtual storage. Most prefered (3), then (2) then (1).