sydney-informatics-hub / dataharvester Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 3.0 4.11 MB

Home Page: https://sydney-informatics-hub.github.io/dataharvester/

License: GNU General Public License v3.0

JavaScript 4.80% Python 82.58% R 12.62%

dataharvester's People

Contributors

Stargazers

Forkers

datalearns dschnei5 januarharianto

dataharvester's Issues

PackagesNotFoundError: The following packages are not available from current channels: - google-cloud-sdk

From https://github.com/Sydney-Informatics-Hub/AgReFed-DataHarvester/issues/49

Local RStudio setup EE error

── Welcome to AgReFed Data-Harvester ────────────────────────────────────────────

ℹ Checking if dataharvesteR has been set up appropriately
ℹ Check Python/conda install
✔ conda binary: /Users/seb/mambaforge/bin/conda
✔ conda env: geopy

ℹ Checking package versions
✔ all packages validated
ℹ Set up Earth Engine API access
Error in py_module_import(module, convert = convert) :
TypeError: 'numpy._DTypeMeta' object is not subscriptable

Line 8 harvest() failure

In Rstudio cloud, I ran:

# This script is for package testing in RStudio Cloud
pacman::p_load_current_gh("sydney-informatics-hub/dataharvester")
initialise_harvester("r-reticulate", 
                     earthengine = TRUE, 
                     auth_mode = "rstudiocloud")

And followed all the setup steps, and it worked.

I then ran the line 8:

> # test harvest a single source
> harvest("assets/basic_config.yaml", plot = TRUE)

and it failed with the following message:

`TypeError: 'NoneType' object is not iterable` if `spectral` not included in `preprocess_ee()`

This won't work:

img <- collect_ee(
  collection = "LANDSAT/LC09/C02/T1_L2",
  coords = c(-11.4011, 21.1229),
  date = "2021-06-01",
  end_date = "2022-06-01",
  buffer = 60000,
  bound = TRUE)
img <- preprocess_ee(
  object = img)

But this does:

img <- collect_ee(
  collection = "LANDSAT/LC09/C02/T1_L2",
  coords = c(-11.4011, 21.1229),
  date = "2021-06-01",
  end_date = "2022-06-01",
  buffer = 60000,
  bound = TRUE)
img <- preprocess_ee(
  object = img,
  spectral = "NDVI")

Since it is not necessary to include a spectral index the top code block should work - likely a validation step that didn't work out too well (and needs to be fixed)

Install instructions are not complete for Windows users

In general, Windows users must be instructed to manually install the following:

GDAL: https://gdal.org/download.html -- needed for manipulation of geospatial data
Google Cloud CLI: https://cloud.google.com/sdk/docs/install#windows -- needed to authenticate to Earth Engine API

Line 16: rasterio not recognising a tif as a supported file type

Ran line 16:

harvest("assets/full_config.yaml", plot = TRUE)

Appeared to work and files downloaded, however an error occurred in python, with rasterio not recognising a tif as a supported file type:

Extracting data points for Pointdata_Llara.csv  -----
⊙ • ee_LAN_20210101_20211231_NDVISRB2SRB3SRB4_median_100m | pixel size: (145, 201) 0.6s                        
⊙ • landsat_barest_earth | pixel size: (77, 107) 0.0s                                                          
⊙ • ga_ls_fc_pc_cyear_3_2021-1-1 | pixel size: (77, 107) 0.0s

Error in py_call_impl(callable, dots$args, dots$keywords) :
rasterio.errors.RasterioIOError: 'downloads/DEM_SRTM_1_Second_Hydro_Enforced_2022_10_21.tif' not recognized as a supported file format.

Error: Error creating conda environment 'r-reticulate' [exit code 1] in Windows 11

Looks like OpenSSL is needed but not installed by default on Windows 10 machines, causing Conda to fail to communicate with the host via HTTPS

CondaSSLError: OpenSSL appears to be unavailable on this machine. OpenSSL is required to
download and install packages.

Exception: HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /conda
-forge/win-64/current_repodata.json (Caused by SSLError("Can't connect to HTTPS URL because the SSL module 
is not available."))

Some interesting warnings when installing on Windows 10

Package installs fine but these warnings should be checked.

Info on PC:

> Sys.info()
       sysname        release        version       nodename        machine 
     "Windows"       "10 x64"  "build 22000"   "MESHIFY-PC"       "x86-64" 
         login           user effective_user 
         "uno"          "uno"          "uno" 
> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252   
[3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C                      
[5] LC_TIME=English_Australia.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] dataharvester_0.0.7

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.9        lattice_0.20-45   lubridate_1.8.0   png_0.1-7        
 [5] ps_1.7.1          prettyunits_1.1.1 crayon_1.5.1      withr_2.5.0      
 [9] rprojroot_2.0.3   grid_4.1.2        R6_2.5.1          jsonlite_1.8.0   
[13] pacman_0.5.1      cli_3.4.1         curl_4.3.2        remotes_2.4.2    
[17] rstudioapi_0.14   callr_3.7.2       Matrix_1.3-4      reticulate_1.26  
[21] generics_0.1.3    tools_4.1.2       yaml_2.3.5        compiler_4.1.2   
[25] processx_3.7.0    pkgbuild_1.2.0

Log:

* installing *source* package 'dataharvester' ...
** using staged installation
** R
Warning: unable to re-encode 'initialise.R' lines 11, 18, 21, 33, 72, 76, 197, 199
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'dataharvester'
    finding HTML links ... done
    Llara                                   html  
    aggregate_ee                            html  
    finding level-2 HTML links ... done

Rd warning: C:/Users/uno/AppData/Local/Temp/RtmpKu4BU5/R.INSTALL265041ab75ec/dataharvester/man/aggregate_ee.Rd:10: missing link 'collect_e'
    authenticate_ee                         html  
    collect_ee                              html  
    config_bbox                             html  
    config_crs                              html  
    config_dates                            html  
    config_dea                              html  
    config_dem                              html  
    config_paths                            html  
    config_res                              html  
    config_silo                             html  
    config_slga                             html  
    config_xy_names                         html  
    download_dea                            html  
    download_dem                            html  
    download_ee                             html  
Rd warning: C:/Users/uno/AppData/Local/Temp/RtmpKu4BU5/R.INSTALL265041ab75ec/dataharvester/man/download_ee.Rd:17: missing link 'collect_e'
    download_landscape                      html  
    download_radiometric                    html  
    download_silo                           html  
    download_slga                           html  
    harvest                                 html  
Rd warning: C:/Users/uno/AppData/Local/Temp/RtmpKu4BU5/R.INSTALL265041ab75ec/dataharvester/man/harvest.Rd:19: missing link 'create_yaml'
    init_logtable                           html  
    initialise_harvester                    html  
    load_config                             html  
    load_settings                           html  
    map_ee                                  html  
Rd warning: C:/Users/uno/AppData/Local/Temp/RtmpKu4BU5/R.INSTALL265041ab75ec/dataharvester/man/map_ee.Rd:10: missing link 'collect_e'
    preprocess_ee                           html  
    print.harvester                         html  
    update_logtable                         html  
Rd warning: C:/Users/uno/AppData/Local/Temp/RtmpKu4BU5/R.INSTALL265041ab75ec/dataharvester/man/update_logtable.Rd:20: missing link 'init_logfile'
    validate_conda                          html  
    wrap_config                             html  
    write_config                            html  
*** copying figures
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
** testing if installed package can be loaded from final location
*** arch - i386
*** arch - x64
** testing if installed package keeps a record of temporary installation path
* DONE (dataharvester)

Local install issues - with geopy environment which existed before, gdal definitely works for other projects

> pacman::p_load(dataharvester)
> initialise_harvester(earthengine = FALSE)

── Welcome to AgReFed Data-Harvester ────────────────────

ℹ Checking if dataharvesteR has been set up appropriately
ℹ Check Python/conda install
✔ conda binary: /Users/darya/opt/miniconda3/bin/conda
✔ conda env: geopy

ℹ Checking package versions
✔ all packages validated
> initialise_harvester(earthengine = TRUE)

── Welcome to AgReFed Data-Harvester ────────────────────

ℹ Checking if dataharvesteR has been set up appropriately
ℹ Check Python/conda install
✔ conda binary: /Users/darya/opt/miniconda3/bin/conda
✔ conda env: geopy

ℹ Checking package versions
✔ all packages validated
ℹ Set up Earth Engine API access
Error in py_module_import(module, convert = convert) : 
  ImportError: dlopen(/Users/darya/miniconda3/envs/geopy/lib/python3.9/site-packages/rasterio/crs.cpython-39-darwin.so, 0x0002): Library not loaded: '@rpath/libgdal.31.dylib'
  Referenced from: '/Users/darya/miniconda3/envs/geopy/lib/python3.9/site-packages/rasterio/crs.cpython-39-darwin.so'
  Reason: tried: '/Users/darya/miniconda3/envs/geopy/lib/libgdal.31.dylib' (no such file), '/Users/darya/miniconda3/envs/geopy/lib/libgdal.31.dylib' (no such file), '/Users/darya/miniconda3/envs/geopy/lib/libgdal.31.dylib' (no such file), '/Users/darya/miniconda3/envs/geopy/lib/libgdal.31.dylib' (no such file), '/Users/darya/miniconda3/envs/geopy/lib/libgdal.31.dylib' (no such file), '/usr/local/lib/libgdal.31.dylib' (no such file), '/usr/lib/libgdal.31.dylib' (no such file)

Install errors - on RStudio cloud

> initialise_harvester(earthengine = FALSE)
• Checkking python/conda install...
✔ Conda binary: /cloud/home/r199168/.local/share/r-miniconda/bin/conda
• Verifying python configuration...
✔ Using conda environment 'r-reticulate'
> # note: will need to restart dataharvester more than once
> # a third check will also happen to finalise dependencies
> initialise_harvester(earthengine = TRUE)
• Checkking python/conda install...
✔ Conda binary: /cloud/home/r199168/.local/share/r-miniconda/bin/conda
• Verifying python configuration...
✔ Using conda environment 'r-reticulate'
• Starting Earth Engine authetication...
Matplotlib is building the font cache; this may take a moment.
Error in py_module_import(module, convert = convert) : 
  ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /cloud/home/r199168/.local/share/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/rasterio/../../../libgdal.so.31)
> # note: will need to restart dataharvester more than once
> # a third check will also happen to finalise dependencies
> initialise_harvester(earthengine = TRUE)
• Checkking python/conda install...
✔ Conda binary: /cloud/home/r199168/.local/share/r-miniconda/bin/conda
• Verifying python configuration...
✔ Using conda environment 'r-reticulate'
• Starting Earth Engine authetication...
Error in py_module_import(module, convert = convert) : 
  AttributeError: partially initialized module 'rasterio' has no attribute '_loading' (most likely due to a circular import)

Install errors - Local machine

My code:

DataHarvesteR

Install dataharvester

install.packages("remotes") # uncomment and run this line if necessary
install.packages("pacman")
remotes::install_github("januarharianto/dataharvester")

Load library

pacman::p_load(dataharvester)
initialise_harvester(earthengine=TRUE)

Last part of the standard output:

Restarting R session...

> knitr::opts_chunk$set(echo = TRUE)
> pacman::p_load(dataharvester)
> initialise_harvester(earthengine=TRUE)

-- Welcome to AgReFed Data-Harvester ------------------------------------------------------------------

<U+2139> Checking if dataharvesteR has been set up appropriately
<U+2139> Check Python/conda install
<e2><9c><94> conda binary: C:/Users/tche7417/AppData/Local/r-miniconda/condabin/conda.bat
<e2><9c><94> conda env: geopy

<U+2139> Checking package versions
Cannot validate required Python packages. Attempting to reinstall all packages to be safe...
Warning in system2(conda, shQuote(args), stdout = TRUE) :
  running command '"C:/Users/tche7417/AppData/Local/r-miniconda/condabin/conda.bat" "list" "--prefix" "C:/Users/tche7417/AppData/Local/r-miniconda/envs" "--no-pip" "--json"' had status 1
Something went wrong here.
<U+2139> Set up Earth Engine API access
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'dateutil'

Modifying the `test_install.R` to avoid student confusion

install.packages("pacman")

This is in the test_install.R script - may need to be replaced with an if statement or just commented out as it's already installed on RStudio Cloud + causes complaints/issues when testing