openego / etrago Goto Github PK

For the new export2db feature of ego.powerflow everything is prepared.
For testing check out the following branches:
https://github.com/openego/eTraGo/tree/feature/add-export2db
https://github.com/openego/ego.io/tree/features/add-pf-result-tables
https://github.com/openego/ego.powerflow/tree/features/add-export2db
https://github.com/openego/data_processing/tree/features/add-pf-result2db

The new result tables are not yet in the oedb, will do that in a moment.

Preparation:

• dis-aggregate EHV dispatch results to HV buses
• prepare HV LOPF with pre-defined dispatch from EHV LOPF (using p_min_pu and p_max_pu)
• prepare HV LPF and/or PF with pre-defined dispatch from EHV LOPF (pq-sets)

Testing with SH test scenarios:

• LOPF for HV grid with grid operator regions, refer to #7
• LOPF for HV grid with shortest-path clusters of HV grids
• LOPF for complete grid with predefined dispatch
• LPF/PF for complete grid with predefined dispatch

@mariusves The adjustment of the load shedding function does not seem to work fine.
I think this should fix the problem. Partly it does (now the id seems to be unique) but apparently it still does not work fine.
Can you please check this out?

I tried to run the snapshot clustering, but there is still the following error:

Traceback (most recent call last):
  File "snaphot_clustering_app.py", line 159, in <module>
    network, snapshots = create_powerflow_problem(timerange, components)
  File "/home/simon/uni/repos/snapshot_clustering/eTraGo/venv/src/ego.powerflow/egopowerflow/tools/io.py", line 172, in create_powerflow_problem
    component)
  File "/home/simon/uni/repos/snapshot_clustering/eTraGo/venv/src/pypsa/pypsa/io.py", line 187, in import_components_from_dataframe
    attrs = network.components[cls_name]["attrs"]
KeyError: 'Storage'

I guess there is still something wrong with the table names and derived component names (should
resolve to StorageUnit and not Storage I guess)

Attaching load shedding generators by something like

network.add("Carrier", "load")
      network.import_components_from_dataframe(
          pd.DataFrame(
              dict(marginal_cost=1e4,
                   # intersect between macroeconomic and surveybased
                   # willingness to pay
                   # http://journal.frontiersin.org/article/10.3389/fenrg.2015.00055/full
                   p_nom=1e6,
                   carrier='load',
                   bus=network.buses.index),
              index=network.buses.index + ' load'),
          "Generator"
      )

and then looking at where and when there is load shedding in order to hunt infeasibility locations

• test it
• write a little load shedding function

I will work on this branch

Currently, the optimization of storage takes place in one LOPF run.
An alternative would be to abstract the storage_unit inputs in order to determine the flexibility needs and define storages / units based on these needs. The defined units can then be applied in a separate LOPF for validation.

We decided to get rid of the package ego.powerflow and include our functionalities into eTraGo.

It seems that this equals that. Can we delete the first mentioned lines of code?

goal:

• prevent optima plateau
• maybe approximate line loss effects?!

for more information see here

Apparently, the extendable storages are clustered to the EHV nodes as well. I'm not sure, if this makes sense and if we should do so, what do you think? The parameters of the storages (efficiency, max_hours, standing_loss) are obviously touched by this as they are averaged and not really comparable any longer. An alternative could be to just disregard all HV storages and just consider the ones at the EHV nodes.

Please state your opinion here, we could then discuss if we want to change the current setting.

Came up in #33 by @ulfmueller

Hi,
I receive the following error when setting networkclustering = true with data from the versioned schema. It seems to be irrelevant which scenario or spatial dimension I use (SH or DE). Any clues?

network = etrago(args)
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
Busmap does not exist and will be created.

cpu_cores (default 4): 4
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
Traceback (most recent call last):
File "", line 1, in
network = etrago(args)
File "", line 54, in etrago
network = cluster_on_extra_high_voltage(network, busmap, with_time=True)
File "/srv/ES2050/open_eGo/AP3/github/eTraGo/etrago/cluster/networkclustering.py", line 54, in cluster_on_extra_high_voltage
new_df, new_pnl = aggregategenerators(network, busmap, with_time)
File "/srv/ES2050/open_eGo/AP3/github/eTraGo/src/pypsa/pypsa/networkclustering.py", line 68, in aggregategenerators
new_df = generators.groupby(grouper, axis=0).agg(strategies)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/groupby.py", line 3690, in aggregate
return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/groupby.py", line 3179, in aggregate
result, how = self._aggregate(arg, _level=_level, *args, **kwargs)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/base.py", line 549, in _aggregate
result = _agg(arg, _agg_1dim)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/base.py", line 500, in _agg
result[fname] = func(fname, agg_how)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/base.py", line 483, in _agg_1dim
return colg.aggregate(how, _level=(_level or 0) + 1)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/groupby.py", line 2653, in aggregate
return self._python_agg_general(func_or_funcs, *args, **kwargs)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/groupby.py", line 820, in _python_agg_general
result, counts = self.grouper.agg_series(obj, f)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/groupby.py", line 1939, in agg_series
return self._aggregate_series_pure_python(obj, func)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/groupby.py", line 1969, in _aggregate_series_pure_python
res = func(group)
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/groupby.py", line 814, in
f = lambda x: func(x, *args, **kwargs)
File "/srv/ES2050/open_eGo/AP3/github/eTraGo/src/pypsa/pypsa/networkclustering.py", line 50, in _consense
assert ((x == v).all() or x.isnull().all())
AssertionError

Moved from #10

• test the PF results: are there feasible results for DE and does the consideration of Q and losses lead to line loading > 100 %?

Depending on the results of load shedding analyses in #19 allow the respective lines (those that connect nodes with increased load shedding) to be extendable. Then test this functionality and its potential to replace load shedding in a new run.

• update requirements file
• merge dev into master
• release repository (possibly with pypi)

Add username to result table to be to identify the person who uploaded the results.
Use this solution from the DP:

https://github.com/openego/data_processing/blob/release/v0.3/dataprocessing/python_scripts/functions/ego_scenario_log.py#L55-L64

Running snapshot clustering I got the following warning:

WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
WARNING:pypsa.io:'p_max_pu_fixed' for generators is deprecated, use static 'p_max_pu' instead.
WARNING:pypsa.io:'p_min_pu_fixed' for generators is deprecated, use static 'p_min_pu' instead.
WARNING:pypsa.io:'p_max_pu_fixed' for generators is deprecated, use static 'p_max_pu' instead.
WARNING:pypsa.io:'p_min_pu_fixed' for generators is deprecated, use static 'p_min_pu' instead.
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)

Not a bug when creating the model but sould be fixed I guess

It seems that the current (data) setting of eTraGo can only be solved by gurobi solver. Open source alternatives like GLPK and CBC should also do the task.

Due to this bda27bc changes in ego.io and ego.powerflow are relevant.
I would suggest we have small new releases in ego.io and ego.pf in order to hint to them in the setup.py. Otherwise we would have to link to the dev branches of those repos.

@lukasol aka release-master What do you think?

After this commit the problem is still pending.
I know @S3PP is already working on it. Just to have this pending task transparent here. Discussion might help as well...

Taken from @ulfmueller (issue in ego.powerflow):

Check out how the session maker in ego.eTraGo works with the adjusted session maker in ego.io.

• Check if we can get rid of the function here in ego.eTraGo and do that only in ego.io centrally.
• may be create a functionality that it is also possible to insert the session parameters manually

work on https://github.com/openego/eTraGo/tree/features/load-shedding-analysis

• one-year simulation in order to find all nodes with load shedding
• discuss solutions: could we improve the subgrid connection if this is the reason

• finish the sql script implementation
• test the sql implementation
• perspectively write a function in python which does the same but within the pypsa network and not on the pgsql database as a new scenario

Similarly to network clustering, I also suggest to merge snapshot clustering into dev...

• develop a functionality to couple lopf and pf in order to simulate Q and line losses after having done the linear optimization.
• implement the cosphi assumptions based on the lopf results
• implement and consider slack bus
• test the PF results: are there feasible results for DE and does the consideration of Q and losses lead to line loading > 100 %?
• (calculation of additional costs should be covered by #23 )
• work on a feature branch
• define a function

Apart from default 'angles' method other methods like 'kirchhoff', 'cycles' or 'ptdf' should be tested in order to find ways to reduce computation time.

Once the clustering approaches are implemented, test their respective performance with regards to the results (optimization results, total costs, ...) and their computation time.

created this branch to work on this.

to cite Jonas Hoersch:
"Code is in pypsa/networkclustering.py. The code idea is to build a
series termed busmap that maps the old buses index to the new buses
index and then call get_clustering_from_busmap to generate the new
network."

For this a branch was created. And the current development is going on here

See #10 for the implementation of the PF.
Especially with regards to the eGo tool, we need full costs for grid losses.

• Calculate the grid losses for all lines using the results of the PF
• assume average costs to cover these losses

We have merged the line-loading-minimization into the dev branch and closed that issue (see #8).

First test have shown that computation time is essentially slowed down by this method (roughly factor 2). Therefore a new idea popped up. We could perform the line loading minimization once for all scenarios in order to define a marginal cost noise (see #4 ) which is 'inspired' by that minimal line loading result and which can reproduce that setting without increase of computation time.
One aspect to think about is that the line loading minimization could produce different settings from snapshot to snapshot. And we want only one marginal cost setting set once for all snapshots.

Currently, dispatch is presented as a total system with all neighbouring countries being one node.

• differentiate between German and non-German load and dispatch
• present the difference as import or export
• goal: get one network that only presents German load and dispatch and all import and export separately
• adjust the plotting functions, i.e. stacked_gen accordingly

It might make sense to assign H2 storages to EHV nodes and batteries to HV nodes only. Among other things due to the fact that storage results are transferred to the MV grids.

Additionally, it might make sense to set a minimal capacity for H2 storages in order to prevent small storage units.

Implement and test this.

the task was to

1. be able to connect not only to the model_draft schema but also to the versioned grid schema.
2. write a translater from the database structure which is based on the PyPSA 0.6 model to the PyPSA 0.8 world - without changing the database
3. get rid of io functionalities within the application files and shift that to the io in ego.powerflow
4. find an easy way for working with just one appl.py in order to avoid redundancies and chaos

all this can be found on this branch !

it is still lacking documentation. even though it should be quite intuitive it is not 100% self explanatory. The goal would be to merge this into dev soon (e.g. next week) and have it as the new standard.

The extra_high_voltage_appl.py and extra_high_voltage_SH_appl.py was already 'merged' into the appl.py. For the

• k-means clustering and
• snapshot clustering
  it is still a pending task

Do you like it? Do you agree? Objections?

Running eTraGo with solver=glpk and otherwise default settings on the most recent oedb data yields
ValueError: Expecting 's' row after 'c' rows

Traceback (most recent call last):
  File "[path]/eTraGo/etrago/appl.py", line 170, in <module>
    network = etrago(args)
  File "[path]/eTraGo/etrago/appl.py", line 136, in etrago
    network.lopf(scenario.timeindex, solver_name=args['solver'], extra_functionality=extra_functionality)
  File "[path]/eTraGo/venv/src/pypsa/pypsa/opf.py", line 1099, in network_lopf
    keepfiles=keep_files, options=solver_options)
  File "[path]/eTraGo/venv/lib/python3.5/site-packages/pyomo/opt/base/solvers.py", line 610, in solve
    result = self._postsolve()
  File "[path]/eTraGo/venv/lib/python3.5/site-packages/pyomo/opt/solver/shellcmd.py", line 268, in _postsolve
    results = self.process_output(self._rc)
  File "[path]/eTraGo/venv/lib/python3.5/site-packages/pyomo/opt/solver/shellcmd.py", line 330, in process_output
    self.process_soln_file(results)
  File "[path]/eTraGo/venv/lib/python3.5/site-packages/pyomo/solvers/plugins/solvers/GLPK.py", line 362, in process_soln_file
    raise ValueError(msg)
ValueError: Error parsing solution data file, line 

It seems to fail at the very first line of the *.glpk.glp-file, which is:
p lp min 5641 4090 12292

Additionally, I receive the following warnings:

No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
[path]/eTraGo/etrago/extras/utilities.py:300: RuntimeWarning: divide by zero encountered in reciprocal
  'r_pu': lambda x: np.reciprocal(np.sum(np.reciprocal(x))),
[path]/eTraGo/etrago/extras/utilities.py:314: RuntimeWarning: divide by zero encountered in reciprocal
  'x_pu': lambda x: np.reciprocal(np.sum(np.reciprocal(x))),
[path]/eTraGo/etrago/extras/utilities.py:299: RuntimeWarning: divide by zero encountered in reciprocal
  'r': lambda x: np.reciprocal(np.sum(np.reciprocal(x))),

The goal is to implement a new function which plots a heatmap showing the voltage at each bus to identify areas with high voltages.

Do you also encounter subnetworks when using networkclustering? Currently, I am working with NEP 2035 data on version v0.2.11 and receive 7 subnetworks...

To be able to install eTraGo a setup.py is needed. Also certain parts in the modules as well as the appl.py have to be adjusted.

implement s.th. similar to this for calculation time intensive simulations such as the one in #19

We have talked about it already, I think.
Load shedding makes the lopf for the 'Status Quo' scenario feasible. Turning load shedding off and instead setting storage extendable on the other hand does not lead to a feasible result. To me, that makes no sense. Any ideas for reasons or solutions?

for reproducibility, take e.g. the following setting:
'gridversion':'v0.2.11', snapshot: 2320, 'scn_name': 'Status Quo'

See #6

• add functionality to save the resulting cluster for reproducibility

In my opinion it always leads to confusion when the bar indicating the line loading is scaled differently.
What do you think of just strictly scaling it to 0% - 100%?

Obviously, the plots would often not be that nice and colorful, but that is also a message...

I tried to use the attribute 'lpfile': in order to save my results but following Error appeared:

...
INFO:pypsa.pf:Slack bus for sub-network 0 is 25569
INFO:pypsa.opf:Performed preliminary steps
INFO:pypsa.opf:Building pyomo model using `angles` formulation
INFO:pypsa.opf:Solving model using gurobi
INFO:pypsa.opf:Optimization successful
Traceback (most recent call last):
  File "k_means_testing.py", line 136, in <module>
    network = etrago(args)
  File "k_means_testing.py", line 128, in etrago
    True})
  File "/eTraGo/lib/python3.5/site-packages/pyomo/core/base/block.py", line 1646, in write
    io_options)
TypeError: __call__() takes from 1 to 3 positional arguments but 5 were given

I took a look into the etrago() function:

# write lpfile to path
     if not args['lpfile'] == False:
         network.model.write(args['lpfile'], io_options={'symbolic_solver_labels':
                                                      True})

For me, It seems to be an Error in the network class -> from egopowerflow.tools.io import NetworkScenario, or?

Any ideas, how to solve this error?

• technical implementation of read the docs to eTraGo
• fill documentation with at least description of args

as an idea / nice-to-have we could think about realizing the simplification in the dp within eTraGo and leave the data as it was before...

@lukasol do you have an opinion on that?

Right now depending on choosing model_draft or the grid version for the args 'gridversion' we have to consequently change the 'ormcls_prefix': 'EgoPfHv' or 'EgoGridPfHv'.
As a default I would like that this change in ormcls_prefix would be realized automatically.

Start: Monday, 17.7.
Finish: Thursday, 20.7.

Consider all Tickets with Milestone 0.2 and the update of higher resoluted weather data and wind power curves (redmine ticket 1070, 431)

For the lopf, does the grouping of lines changes runtime? Can you do some comparison?

see #17