Based on #1

Timezone is not computed based on config_general time info but hard-coded (see code snippet below). Change so that it uses config_general instead.

weather.index = dtime.tz_convert("Europe/Berlin")

Additional information

Currently, the aggregator is included in the config file but not yet implemented. For the implementation it needs to be checked how the config needs to look like to actually implement it, it needs to be considered in the scenario creation and execution. And of course the analysis once this exists.

This issue contains all the forecasting methods and to which plants they apply:

(TBD)

the path separator is now hard-coded in creator so the code is not working when running on other OS, should be adjusted correspondingly

Currently the forecaster class inits the forecaster to train all methods. However, it does so only for the plants and not the markets. This needs to be added.

Furthermore, the class requires a function that creates all the forecasts for each plant and market. I have added it to the class with some comments but feel free to change it however you like. I also added more variable to the init. Just let me know if you don't need some of them.

def make_forecasts(self):
    """Make forecasts for all plants."""
    # TODO: @Jiahe, please implement this method

    # Loop through all forecast columns (probably best to sort this by plant ID and market type)

    # Check if the model needs to be re-fitted

    # Make the forecast

    # Return the forecasts dataframe (i.e. self.forecasts)
    return self.forecasts

Currently the task list contains the type of coupling, e.g. above_l. This needs to be replaced with just above when it is appropriate and in other instances with None. This way the market knows when coupling is necessary and when not without having to check it for itself. It only retrieves the other bids and offers if the row of the coupling column contains some other than None.

Currently, the tool uses concurrent.futures thread-functionality to run processes in parallel. However, this does not speed up the process as the GIL hinders the processes to actually be executed in parallel. Therefore another approach needs to be used.

Bug in:
setup.py: __execute_agents_parallel() and __execute_agents()

Line where bug is found:

        # Get the data of the agents that are part of the tasklist
        agents = self.database.get_agent_data(region=tasklist[0, 'region'])

Issue
The following line returns a dict with the keys being the agent IDs. The dict should contain as keys the agent types and as subkeys the agent IDs.

All entries are added twice to the market transactions.

Some of the columns are still using the old naming format in the creator.

Change all column names accordingly, e.g. "cop_file" to "file_add".

Implement chp. Lemlab's implementation can be used for inspiration.

This is lemlab's config setup for chp:

"chp_fraction": 0                         # fraction of prosumers with chp

 "chp_sizing_power": 1                     # thermal sizing of chp plant
                                           # 1 = thermal power of chp equals ceiled max heat demand
                                           # note: if value <1 without other heating devices, the simulation will crash

 "chp_heat_elec_ratio": 2                  # ratio of heating to electric power (p_th/p_el)

 "chp_efficiency": 0.9                     # efficiency of chp plant: (e_th + e_el)/e_tot

 "chp_capacity": [2, 3]                    # storage capacity in hours of chp power
                                           # note: value randomly chosen from list

 "chp_soc_init": 0                         # initial soc of the storage (0-1)

 "chp_tes_efficiency": 0.98                # charging and discharging efficiency of thermal energy storage

 "chp_fcast": "perfect"                    # heating demand forecasting technique
                                           #   "perfect" - perfect forecast of heating demand
                                           #   "nn" - neural network based on the included weather data
                                           #   "mr" - multiple regression based on the included weather data

 "chp_fcast_retraining_period": 86400      # models should be periodically retrained based on new data
                                           # how many seconds between retraining periods?

 "chp_fcast_update_period": 3600           # forecasts are periodically updated and saved to file where the
                                           # most current forecast can be retrieved by model predictive controllers
                                           # and market agents. How many seconds between updates?

When running the Executor, following progress bars appears:

But actually there should be only one progress bar. If I delete from hamlet.creator.setup import Creator in HAMLET\hamlet\__init__.py, I got the right version of the progress bar:

My assumption is that it's because the progress bar in class Creator is initialized outside of the __init__() function of the Creator:

But if I move progress_bar = tqdm() into the def __init__() as self.progress_bar = tqdm(), I got the following error while creating the scenario:

If the input format of the demand/generation in the pandapower Excel file is too long, it cannot be casted as integer. Correct the line for all agent types and devices. The line looks most often like this:
old:
self.df[f"{key}/sizing/power_{num}"] = (self.df.index.map(df['power']) * 1e6).astype('Int64')

new (add round()):
self.df[f"{key}/sizing/power_{num}"] = (self.df.index.map(round(df['demand'] * 1e6))).astype('Int64')

Each forecast method for each device gets its own entry in the config file.

Change the creator so that it can work with the new feature.

Example:

    fcast: 
      method: naive_average                     # household load forecasting method
                                                # options:
                                                #   - perfect:       perfect knowledge of the future
                                                #   - naive:         today will be the same as yesterday
                                                #   - naive_average: today will be the same as the average the last n days
                                                #                    defined in naive_average_days
                                                #   - smoothed:      prediction value is a moving mean of the future values 
                                                #                    with the window width defined in smoothed_timesteps
                                                #   - sarma:         full sarma model with the order defined in sarma_order

      perfect:

      naive:


      naive_average:
        days: 2

      smoothed:
        timesteps: 9

      sarma:
        order: [2, 0, 2, 2, 0, 0, 96, 2, 0, 0, 672]
                                                # order of double seasonal arma model 
                                                # order: [s2_ar, 0, s2_ma, s2]

I got the following error when creating the single market scenario:

I opened the debugger to figure out how to fix the error. I noticed at line 507 in setup.py (where the error occurs) self.grids is an empty dict. Then I went to lines 488-502 in setup.py, the following code defines self.grids:

Here, path_grids is defined as 'HAMLET/04 - scenarios/example_single_market/grids'. But this folder doesn't exist at all. So at line 495, when using os.walk(path_grids) to define self.grids, we will get an empty dict for self.grids because nothing was looped through.

Let me know if this error while creating grids also occurs on your laptop or if I misunderstood something.

Include the functionality to create the generation time series for PV and wind using the spec json files. The function to be used are labeled with @jiahe and are found in creator/agents/agents.py class Agents.

The database should contain methods that allow to obtain the entire general info, the weather as well as the market dict from the respective region (see #23 for the structure). The entire general info can then also be used to obtain the weather data as it is a subdict.

def get_general_data() -> dict:
    # code
    return data
    
def get_weather_data() -> dict:
    # code
    return data

def get_market_data(region: str) -> dict:
    # code
    return data

Tool does not contain an analyzer yet. Therefore, an implementation starting with an analyzer architecture is necessary.

I'm not sure if the error is caused by MacOS but I got the following error while creating scenario (both multi-market and single-market):

Let me know if you get the same error while calling 01 - examples/01_creating_scenarios.py @TUM-Doepfert . If not, I'll check if it's a MacOS problem.

The database needs to be set up in the structure shown in the attached image and discussed on 25.05.2023:

Include the creation of a cop and/or power timeseries based on the specifications json file. The function should return a pandas dataframe with the index as unix timestamp and the columns named after what they contain, e.g. 'cop'.

The function is in the Agents class named '__timeseries_from_specs_hp'.

All auxiliary functions should also be included in this class.

Create separate files for the subclasses of grids, agents and markets in the creator.

Agents with only inflexible load will get in the executor this error message in the executor : (hope this is not only for me the case)
File "C:\Users\claud\PycharmProjects\HAMLET-develop\hamlet\executor\utilities\database\database.py", line 195, in get_bids_offers
bids_offers_table = self.filter_market_data(market=bids_offers_table, by=by, values=values, inclusive=True)
File "C:\Users\claud\PycharmProjects\HAMLET-develop\hamlet\executor\utilities\database\database.py", line 288, in filter_market_data
dtype = datetime_index.dtypes[0]
File "C:\Users\claud\anaconda3\envs\hamlet\lib\site-packages\polars\lazyframe\frame.py", line 672, in dtypes
return self._ldf.dtypes()
exceptions.ColumnNotFoundError: timestep

They are only applied to the current timestamp settlement but previous market clearings for the same timestep need to be also considered and feed and levies only applied to the net energy amount.

Bug in:
setup.py: __execute_agents_parallel() and __execute_agents()

Line where bug is found:

    # Get the data of the agents that are part of the tasklist
    agents = self.database.get_agent_data(region=tasklist[0, 'region'])

Issue
The following line returns a dict with AgentDB objects. However, I cannot inspect the objects and see what information they contain. When calling something like agents['sfh']['B0gUAuzRUlrkye1'] I would expect to see a dict of dicts instead of <hamlet.executor.utilities.database.agent_db.AgentDB object at 0x000002A4C0241300>

Sells the energy that it bought previously on the market as balancing energy and uses flexibility instead.

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of HAMLET.

Reproducible example

import sys
sys.path.append("..")  # Add the parent directory to the Python path for execution outside of an IDE
sys.path.append("./")  # Add the current directory to the Python path for execution in VSCode
from hamlet import Creator
from hamlet import functions as f
import os

# Path to the scenario folder (relative or absolute)
path = "../02_config/scenario_pv_60_hp_60_ev_60_transition"

# Create the creator object
sim = Creator(path=path)

# Create the scenario
sim.new_scenario_from_grids()

# get all agents in this scenario
sfh_agents = f.get_all_subdirectories("../04_scenarios/scenario_pv_60_hp_60_ev_60_transition/agents/sfh")

# now find the agent at grid bus 20136, since it has 3 EVs with different time series files according to electricity.xlsx
for sfh_agent in sfh_agents:    # iterate each agent to find agent at bus 20136, it has multiple EVs
    agent_path = os.path.join("../04_scenarios/scenario_pv_60_hp_60_ev_60_transition/agents/sfh", sfh_agent)
    agent_config = f.load_file(os.path.join(agent_path, "account.json"))    # load agent config to get the agent's bus

    if agent_config['general']['bus'] == 20136:     # find agent at bus 20136, print config for EVs of this agent
        print("Agent at bus 20136 has agent id " + sfh_agent + ", it has multiple EVs: \n")

        agent_plants = f.load_file(os.path.join(agent_path, "plants.json"))     # load plants config for this agent

        for plant in agent_plants.values():     # loop plants of this agent to find EVs
            if plant['type'] == 'ev':
                print(plant)

        break

Issue description

If an agent has multiple plants of the same type (e.g. EV), the time series files for all these EV plants are always the same as the first plant when creating a scenario from grid, even though they have different time series files according to the grid file (electricity.xlsx).

Expected behavior

The time series files should be individual for each plant as defined in the grid file (electricity.xlsx).

Screenshots

No response

Additional information

No response

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of HAMLET.

Reproducible example

Issue description

Meters doesnt show the power consumption of the Heat Pump. The Meters.ft file which is one of the Results form the Executor doesnt provide the power consumption of the heat pump

Expected behavior

Meters show the power consumption of the Heat Pump.

Screenshots

No response

Additional information

No response

Currently all timeseries files have timestamps instead of datetime dtypes. This decreases the functionality of the timeseries and requires to write more complicated and less standard code. To right more standardized code, the files need to be changed.

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of HAMLET.

Reproducible example

# Make all plants controllable
self.df[f"{key}/sizing/controllable_{num}"] = True

Issue description

All plants are made controllable in the Creator in function def _pv_grid at the sfh file

Expected behavior

Not All plants are made controllable in the Creator

Screenshots

No response

Additional information

No response

Some of the parameters only add unnecessary complexity which are not required in a simulation environment. Therefore, the functionality is not needed and the parameters should be taken out.

The parameters to take out are:

• has_submeter
• market_agent/preference_quality
• market_agent/premium_preference_quality
• meter (all subparameters)

The tables that are defined in the scenario creator still need to be assigned their dtype as currently they are still floats and objects while they can easily be turned into ints.