Add README to docs landing page

Switch date format to yyyy/mm/dd HH:MM for consistency with AEMO and NEMOSIS formats

• Change validators and converters
• Change README examples
• Change docstrings

Thanks @nick-gorman and @dec-heim

• Implement method
• Write docstring
• Options in compile_raw_data
• Add example in quick start

Running the example code below on a windows machine leads to a permission error.

Example:

import nemseer
nemseer.download_raw_data(
    forecast_start = "01/01/2020 00:00", 
    forecast_end = "02/01/2020 00:00",
    forecasted_start = "02/01/2020 00:00",
    forecasted_end = "02/01/2020 00:00",
    forecast_type = "P5MIN",
    tables = "REGIONSOLUTION",
    raw_cache = "/my/raw/cache/",
    keep_csv = False
)

Error:

INFO: Created directory at C:\cache
INFO: Downloading and unzipping REGIONSOLUTION for 1/2020
PUBLIC_DVD_P5MIN_REGIONSOLUTION_202001010000.zip: 100%|██████████| 32.9M/32.9M [00:09<00:00, 3.70MB/s]
Traceback (most recent call last):
  File "C:/Users/nick/PycharmProjects/nemseer_test/main.py", line 2, in <module>
    nemseer.download_raw_data(
  File "C:\Users\nick\PycharmProjects\nemseer_test\venv\lib\site-packages\nemseer\nemseer.py", line 46, in download_raw_data
    downloader.download_csv()
  File "C:\Users\nick\PycharmProjects\nemseer_test\venv\lib\site-packages\nemseer\downloader.py", line 362, in download_csv
    get_unzipped_csv(url, self.raw_cache)
  File "C:\Users\nick\PycharmProjects\nemseer_test\venv\lib\site-packages\nemseer\downloader.py", line 296, in get_unzipped_csv
    Path(file_path).unlink()
  File "C:\Users\nick\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1324, in unlink
    self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '\\cache\\PUBLIC_DVD_P5MIN_REGIONSOLUTION_202001010000.zip'

Process finished with exit code 1

• netCDF with metadata
• parquet with metadata
• Check processed cache before initiating download actions
• Compile from processed cache
• Change names and docstrings of core funcs - e.g. compile_raw_data to compile_data
• Update quick start in docs

Examples of:

• Compile raw data
• Generate runtimes

Run:

• Doctests

This table is not in the MMS Data Historical SQL Loader.

• Add new URL in data
• Query will need to handle this request. Validation will need to be reconfigured
  • One solution is a wrapper function for validation
• Downloader will need to handle this request
  • downloader module may need refactoring or additional functions

• Add to glossary
• Add Dylan McConnell to acknowledgements

OS specific issues seems to be popping up on WIndows

Would be good to add testing on Windows to the GitHub Actions Workflow

Some downloadable files are not being tested because the download size is large and processing to parquet requires a lot of memory.

This can mean some key functionality is not being tested, e.g. handling enumerated PREDISP_ALL_DATA requests.

Potential solutions:

1. Test download anyway, and make these tests optional
2. Test up to request, and check that request has MB Content-Length

requirements.txt in docs/ is based on dependencies from master. This is obtained by manually running poetry export

However, ReadtheDocs has a solution, which needs to be implemented in the YAML.

• Change YAML
• Remove requirements.txt and remove config to load this on ReadtheDocs

User will likely know what forecasted times they want to look at

Generate maximum range of run times based on this

nemseer currently only accepts datetimes down to minutes.

It should be able to accept datetimes down to seconds, with seconds set as 0. This will improve usability with NEMOSIS

Functionality of Loader better matches that of a Query

Thanks @nick-gorman

• Core functionality function docstrings should have, or be linked to examples

• Xarray, including example in quick start
• Note or section on memory management
• Deprecated tables
• Update development in README

E.g. data from LOAD1 and LOAD2 should be mapped to LOAD, not to LOAD1 and LOAD2

Forecasted end was previously assessed based on increment from forecasted_start.

Instead, it should be validated based on its on "latest forecast time"

rrule may skip months if the start date is the end of the month. This is apparently expected rrule behaviour, see dateutil/dateutil#1079

A fix can be implemented by using datetime.relativedelta to calculate the delta months, then use relative delta to get intervening dates on a monthly basis

I'm trying to download the predispatch price forecast data over the time range:

cache_start_time = "2021/01/01 00:00:00"
cache_end_time = "2023/10/31 00:00:00"

This range should be available, according to the output of:

nemseer.get_data_daterange()

However, when I run the download (see attached jupyter notebook), I get the following error message:

HTTPError: 404 Client Error: Not Found for url: http://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/2022/MMSDM_2022_10/MMSDM_Historical_Data_SQLLoader/PREDISP_ALL_DATA/PUBLIC_DVD_PREDISPATCHPRICE_202210010000.zip

It's expecting a file with basename:

PUBLIC_DVD_PREDISPATCHPRICE_202210010000.zip

but the basename that actually exists in that directory is:

PUBLIC_DVD_PREDISPATCHCONSTRAINT1_202210010000.zip

Do you have any suggestions?

mkFeats_ERROR_REPRO_is_really_ipynb.txt

• Set up PyPI + API token via secrets
• Set up workflow to use poetry for build
  • Dependency on linting+autoformatting and testing

Validate datetimes based on forecast horizons

• Create validators
• Test validators
• Implement in DataCompiler

An issue arises where a user might cache nemseer data in the same dir as csvs (e.g. NEMOSIS csvs). `nemseer will then proceed to convert all csvs.

A more targeted/smarter conversion to csv is required

Thanks @dec-heim

Enumerated tables mapped to returned data

Expected behaviour is to map to parent table - e.g. LOAD1 and LOAD2 should be mapped to LOAD - since this is how the user requests data.

See prakaa/mms-monthly-cli#15. Many of these changes should be implemented in NEMSEER

1. Catch exception and continue downloading other files in query
2. Write faulty file to a hidden file
3. ForecastTypeDownloader behaviours based on faulty zips/csvs in hidden file
4. Modify DataCompiler behaviour based on faulty zips/csvs in hidden file

• Implement step 1
• Implement step 2
• Implement step 3
• Implement step 4

See:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary

• Create glossary
• Reference glossary terms across docs

Once released (check here)

• Update pyproject.toml with new poetry-core version (non-beta) and re-lock
• Update Contributing

For the meantime, progressively upgrade as pre-releases come out

keep_csv=True will result in a parquet and csv being cached.

The convert_to_parquet method should only convert csvs when the parquet does not exist

• Add parquet exists check to convert_to_parquet method

Thanks @nick-gorman

1. Given a set of run_times, generate some dummy forecasted and initiate download
2. Given a set of forecasted_times use runtime generators to generate relevant run_times

Flow could be as follows:

• Make all times optional params with None as default

• Raise error if start and end not provided for run, and/or for forecasted time pairs

• Depending on which pair is provided, do 1 or 2

• Implement code changes

• Write tests

• Modify docstrings

• Modify Quick Start in docs