unsw-ceem / nemed Goto Github PK

Issue when no cache is provided or the file location doesn't exist from running example_2.ipynb

download_generators_info returns the error: UserInputError: The raw_data_location provided does not exist.

In read_json_to_df:

Problem:

• Current method is to initialise empty DataFrame, concatenate to this df in for loop for each iteration

This can be slow, see tip 5 in this article as an example: https://towardsdatascience.com/pandas-concat-tricks-you-should-know-to-speed-up-your-data-analysis-cd3d4fdfe6dd

Solution:

• Initialise empty list, append each DataFrame to this list in the for loop. Then call pd.concat once on this list after the for loop

Suggested Improvements to Marginal Emissions Methodology (specifically how data is treated from price setter files).

To be reviewed....

Current Issue:
The price setter file 'Increase' column is indicative of MLFs not the actual MW increase in power from a marginal generator. Current method however weights marginal emissions intensity for a 5-min DI by the increase factors. This results in overcounting (quite high emissions factors >> 1tCO2/MWh for some DIs) while also undercounting in other intervals.

Proposed Method:

1. Cleaning for ‘Increase’ amounts only over 0.05MW (to represent significant influences on the margin, price setter files includes some minor curtailed/varied generation)
2. Resampling using the mean of the ‘Intensity_Index’ in that Time-Region
3. ‘CO2E_ENERGY_SOURCE’ as multiple unit energy sources for that Time-Region (e.g. Hydro, Natural Gas (Pipeline) rather than a single technology.

Additional changes to consider:

• Optional setting to consider Hydro/Storages in step 2 (mean of 'Intensity_Index') or not. For example if both Hydro (0tCO2/MWh) and Gas (~0.6tCO2/MWh) are marginal generators in the same DI/region. Instead of averaging their intensities which would yield
  0.3tCO2/MWh (a factor half of the actual gas emissions intensity), this setting would simply adopt the mean emissions intensities excluding hydro (therefore 0.6tCO2/MWh). Where hydro is the only generator on the margin in a single DI/region, the emissions factors should remain that of the hydro plant (e.g. 0tCO2/MWh).

Note the above additional filter is more a makeshift workaround to mitigate 0tCO2/MWh factors from distorting emissions observations but does not intend to completely offer a solution for substituting different emissions factors where hydro/storage is on the marginal.

Credits
Credit to Louise Bardwell for picking up this issue and contributing proposed solution.

For reference:

https://wattclarity.com.au/articles/2019/02/a-preliminary-intermediate-guide-to-how-prices-are-set-in-the-nem/
https://wattclarity.com.au/articles/2019/03/price-setting-concepts-an-explainer/

Dead link, should be example_1.html

You could also avoid replicating README.md and create a dummy readme page in docs/source/.
I did this with nemseer. There is a page called readme.md, which can "source" the README in your repo root:
https://github.com/UNSW-CEEM/NEMSEER/blob/master/docs/source/readme.md

Recent v0.3.3 has resolved not reading in some files from AEMO correctly. This issue rather relates to some intervals being filtered out in the function which reads the json files which contain raw price setter data from AEMO xml files which filters on ENOF.

• Update marginal emissions request for timeseries does not match due to price setter files containing data from 04:05:00 to 04:00:00 only, not the full day

Currently average emissions method uses total emissions from dispatched generators divided by sent-out generation of registered generators. An alternative argument that can be raised is:

• ('True') Average Emissions Intensity should consider exported rooftop PV as part of the generation. In practice the average emissions intensity may be lower than currently found, if there is a high volume of rooftop PV. e.g. the total emissions are further spread across 'grid' energy.
• Average Emissions Intensity should not consider registered renewable generators which have signed bundled PPAs for LGCs. e.g. the emissions intensity of the grid is theoretically higher than currently, since a portion of clean energy isn't available for energy users consuming grid energy but rather directly sourced by those contracted to bundled ppas.

Interesting points for a discussion, feel free to contribute ideas below

• Update tech_name function of process.py to resolve 'Grid' field in Fuel Source - Primary as 'Battery Discharge'