Meta reinforcement learning benchmark with building control environments. An attempt to help scale greener building controllers to more buildings.
Home Page: https://bauwerk.readthedocs.io
License: MIT License
Relevant literature:
SolarBatteryHouse is misspelled in line 79 of bauwerk/docs/envs/solar_battery_house.pynb.
Garage implements its own wrapper system that appears to be very involved and given the maintenance status of garage is likely not worth developing for. Instead it might be a good idea to give building distributions internally the power to add wrappers to train and test environments. This way almost no garage-specific wrapper code would have to be written.
One of the most popular implementation of meta-RL algorithms is garage. We would like to make Bauwerk a drop-in replacement for meta world benchmarks in the garage examples.
Currently, the discharging at consistent rate over a certain time with different step sizes, whilst the rest of the environment is identical, appears to lead to different results. The test_changing_step_size test in the dev/fix-align-discharge-rate-varying-stepsizes branch fails because of this. Note that charging and taking no action does not appear to have similar issue. This may have to do with the behaviour of the battery simulation when reaching (dis-)charging limits.
See docstring of EnvPlotter class.
The maintenance of OpenAI gym has moved to its fork Gymnasium. Bauwerk should depend on the maintained fork.
Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.
Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.
I ran the experiment script with the command config below, and got an error message.
command:
/opt/conda/bin/bauwerk-exp -m train_steps_per_task=2500 env_cfg.battery_size=5.0 infeasible_control_penalty=True wandb_project=bauwerk_exp08_benchmark penalty_factor=0.1 env_mode=benchmark add_param_obs=True num_train_tasks=20 benchmark="BuildDistC"
Error:
Traceback (most recent call last):
File "/bauwerk/bauwerk/exp/core.py", line 221, in run
model.learn(
File "/opt/conda/lib/python3.9/site-packages/stable_baselines3/sac/sac.py", line 309, in learn
return super().learn(
File "/opt/conda/lib/python3.9/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 353, in learn
callback.on_training_start(locals(), globals())
File "/opt/conda/lib/python3.9/site-packages/stable_baselines3/common/callbacks.py", line 70, in on_training_start
self._on_training_start()
File "/opt/conda/lib/python3.9/site-packages/stable_baselines3/common/callbacks.py", line 194, in _on_training_start
callback.on_training_start(self.locals, self.globals)
File "/opt/conda/lib/python3.9/site-packages/stable_baselines3/common/callbacks.py", line 70, in on_training_start
self._on_training_start()
File "/opt/conda/lib/python3.9/site-packages/stable_baselines3/common/callbacks.py", line 194, in _on_training_start
callback.on_training_start(self.locals, self.globals)
File "/opt/conda/lib/python3.9/site-packages/stable_baselines3/common/callbacks.py", line 70, in on_training_start
self._on_training_start()
File "/bauwerk/bauwerk/utils/sb3.py", line 360, in _on_training_start
self._log_image()
File "/bauwerk/bauwerk/utils/sb3.py", line 382, in _log_image
img_data = np.frombuffer(plotter.fig.canvas.tostring_rgb(), dtype=np.uint8)
File "/opt/conda/lib/python3.9/site-packages/matplotlib/backends/backend_agg.py", line 460, in tostring_rgb
return self.renderer.tostring_rgb()
AttributeError: 'FigureCanvasAgg' object has no attribute 'renderer'This addresses #18.
This should add support for Gym standard rendering of actions.
Sometimes the shape is a nested array, sometimes not. This leads to warning/errors checking whether observation is inside observation space.
{'load': array([0.], dtype=float32),
'pv_gen': array([0.], dtype=float32),
'battery_cont': array([[0.]], dtype=float32),
'time_step': 1}
and
{'load': array([0.], dtype=float32),
'pv_gen': array([0.], dtype=float32),
'battery_cont': array([0.], dtype=float32),
'time_step': 1}
This is likely due to an error inside the bauwerk.envs.components.battery module.
I'm finding that pv_gen varies across tasks which is unexpected as I presumed it be task-independent.
build_dist_b = bauwerk.benchmarks.BuildDistB()
tasks = build_dist_b.train_tasks
for i, task in enumerate(tasks):
env = build_dist_b.make_env()
env.set_task(task)
actions = bauwerk.solve(env)
task_obs = []
env = ObsWrapper(env)
obs = env.reset()
for action in actions[0]:
obs, reward, done, info = env.step(np.array(action, dtype=np.float32))
task_obs.append(obs)
task_obs = np.concatenate(task_obs).reshape(len(actions[0]),-1)
print('task {} max pv_gen: {:.2f}'.format(i, np.max(task_obs[:, 2])))
------------------------
task 0 max pv_gen: 7.50
task 1 max pv_gen: 7.46
task 2 max pv_gen: 5.13
task 3 max pv_gen: 7.50
task 4 max pv_gen: 7.50
task 5 max pv_gen: 7.50
task 6 max pv_gen: 7.50
task 7 max pv_gen: 7.50
task 8 max pv_gen: 7.49
task 9 max pv_gen: 7.50
task 10 max pv_gen: 7.50
task 11 max pv_gen: 7.50
task 12 max pv_gen: 7.38
task 13 max pv_gen: 7.50
task 14 max pv_gen: 7.43
task 15 max pv_gen: 7.50
task 16 max pv_gen: 5.58
task 17 max pv_gen: 6.00
task 18 max pv_gen: 7.46
task 19 max pv_gen: 6.81
For reference my ObsWrapper is taking the dictionary values form obs and flattening into an array:
class ObsWrapper(gym.Wrapper):
def __init__(self, env):
super(ObsWrapper, self).__init__(env)
self.env = env
def step(self, action):
"""
Steps simulator given action and flattens observation.
:param action: array of shape [act_dim, ]
:return:
"""
obs_dict, reward, done, info = self.env.step(action)
# modify obs
vals = []
for _, value in obs_dict.items():
vals.append(value)
obs_array = np.concatenate(vals, axis=0, dtype=np.float32)
return obs_array, reward, done, info
Afaik, gym environments usually allow you to call env.observation_space.shape to access the the obs_space shape. At the moment, Bauwerk's observation_space is stored as a Dict so this can't be done. This functionality would be helpful, but maybe you've already implemented it? If so, apologies for the redundant request!
This issue collects improvements to be added in v0.2.0 and beyond.
Currently, it has been reported that cfg parameter does not work when using gym.make() to create bauwerk.HouseEnv.
Add the cfg parameter to the gym registration.
Currently there is a large amount of parallel code doing plotting, that also needs to be maintained in a similar way. This should be merged if possible.
Bauwerk will throw an error when you pass an action array of dtype np.float64. I believe this is because you hardcode the env.action_space to be float32 dtype.
action = np.array([0.16], dtype=np.float64)
obs, reward, _, _ = env.step(action)
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Input In [37], in <cell line: 1>()
----> 1 mean_reward, model_actions = rollout_with_prompt(model=model, env=env, eval_steps=24*30)
Input In [36], in rollout_with_prompt(model, env, eval_steps)
34 tokens, obs_mask, act_mask = tokenizer.update_sequences(tokens, obs_mask, act_mask, out_seq[:, -1], action=True)
37 action = np.array([0.16], dtype=np.float64)
---> 38 obs, reward, _, _ = env.step(action)
40 obs_tokens = tokenizer.tokenize(obs)
42 print(obs_tokens)
File ~/phd/research/elizabeth-homes/exp/enjeeneer/rl-exp2/utils/utils.py:14, in ObsWrapper.step(self, action)
13 def step(self, action):
---> 14 obs_dict, reward, done, info = self.env.step(action)
16 # modify obs
17 vals = []
File ~/miniforge3/envs/odes/lib/python3.8/site-packages/gym/wrappers/order_enforcing.py:11, in OrderEnforcing.step(self, action)
9 def step(self, action):
10 assert self._has_reset, "Cannot call env.step() before calling reset()"
---> 11 observation, reward, done, info = self.env.step(action)
12 return observation, reward, done, info
File ~/miniforge3/envs/odes/lib/python3.8/site-packages/bauwerk/utils/gym.py:48, in make_old_gym_api_compatible.<locals>.GymCompatEnv.step(self, action)
46 """Run one timestep of the environment's dynamics."""
47 if not GYM_NEW_STEP_API_ACTIVE:
---> 48 obs, reward, terminated, truncated, info = super().step(action)
49 done = terminated or truncated
50 return obs, reward, done, info
File ~/miniforge3/envs/odes/lib/python3.8/site-packages/bauwerk/envs/solar_battery_house.py:234, in SolarBatteryHouseCoreEnv.step(self, action)
216 def step(self, action: object) -> Tuple[object, float, bool, dict]:
217 """Run one timestep of the environment's dynamics.
218
219 When end of episode is reached, you are responsible for calling `reset()`
(...)
232 (helpful for debugging, and sometimes learning)
233 """
--> 234 assert self.action_space.contains(action), f"{action} ({type(action)}) invalid"
236 info = {}
238 action = float(action) # getting the float value
AssertionError: [0.16] (<class 'numpy.ndarray'>) invalid
The hard-coded upper limit on env.observation_space is unrealistically high for the load and pv_gen variables (dims 1 and 2 of the array printed below) i.e.
obs_high = []
for key, _ in env.observation_space.items():
obs_high.append(env.observation_space[key].high)
print(np.concatenate(obs_high))
[7.500e+00 3.403e+38 3.403e+38 1.000e+00 1.000e+00]
This becomes problematic when one attempts to normalise the observation space using the floor env.observation_space.low and ceiling env.observation_space.high.
Looking through the repo, it seems the max value might be the load_scaling_factor and solar_scaling_factor for the load and pv_gen variables respectively? I see that default_load_data.txt and default_solar_data.txt are in [0, 1], and the load and pv_gen observations are the values from the .txt file multiplied by the scaling factor. Thoughts?
Currently in the dev branch sometimes the tests fail
--sb3 flag is added to pytest and all tests are run
traitlets.traitlets.TraitError: The 'center' trait of an AppLayout instance expected a Widget or None, not the FigureCanvasAgg at '0xffff63828be0'.
[...]
=========================================== short test summary info ============================================
FAILED tests/test_widget.py::test_basic_widget_functionality - traitlets.traitlets.TraitError: The 'center' t...
================================= 1 failed, 53 passed, 4654 warnings in 58.70s =================================pytest tests/test_widget.py it passes, or if I run pytest it passes.Minimal example:
vscode ➜ /bauwerk (dev/general ✗) $ pytest --sb3 tests/test_exp_script.py tests/test_widget.py
============================================= test session starts ==============================================
platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0
rootdir: /bauwerk, configfile: setup.cfg
plugins: anyio-3.5.0, hydra-core-1.2.0
collected 2 items
tests/test_exp_script.py . [ 50%]
tests/test_widget.py F [100%]
=================================================== FAILURES ===================================================
_______________________________________ test_basic_widget_functionality ________________________________________
def test_basic_widget_functionality():
> widget = bauwerk.widget.core.Game(step_time=0.001)
tests/test_widget.py:24:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
bauwerk/widget/core.py:117: in __init__
self.game_lower_part = widgets.AppLayout(
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget_templates.py:157: in __init__
super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget_box.py:64: in __init__
super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget.py:443: in __init__
super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget_templates.py:80: in __init__
super().__init__(**kwargs)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:1079: in __init__
setattr(self, key, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:606: in __set__
self.set(obj, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:580: in set
new_value = self._validate(obj, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:612: in _validate
value = self.validate(obj, value)
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:1851: in validate
self.error(obj, value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <traitlets.traitlets.Instance object at 0xffff90325730>, obj = AppLayout()
value = <matplotlib.backends.backend_agg.FigureCanvasAgg object at 0xffff70f0a790>, error = None, info = None
def error(self, obj, value, error=None, info=None):
"""Raise a TraitError
Parameters
----------
obj : HasTraits or None
The instance which owns the trait. If not
object is given, then an object agnostic
error will be raised.
value : any
The value that caused the error.
error : Exception (default: None)
An error that was raised by a child trait.
The arguments of this exception should be
of the form ``(value, info, *traits)``.
Where the ``value`` and ``info`` are the
problem value, and string describing the
expected value. The ``traits`` are a series
of :class:`TraitType` instances that are
"children" of this one (the first being
the deepest).
info : str (default: None)
A description of the expected value. By
default this is infered from this trait's
``info`` method.
"""
if error is not None:
# handle nested error
error.args += (self,)
if self.name is not None:
# this is the root trait that must format the final message
chain = " of ".join(describe("a", t) for t in error.args[2:])
if obj is not None:
error.args = ("The '%s' trait of %s instance contains %s which "
"expected %s, not %s." % (self.name, describe("an", obj),
chain, error.args[1], describe("the", error.args[0])),)
else:
error.args = ("The '%s' trait contains %s which "
"expected %s, not %s." % (self.name, chain,
error.args[1], describe("the", error.args[0])),)
raise error
else:
# this trait caused an error
if self.name is None:
# this is not the root trait
raise TraitError(value, info or self.info(), self)
else:
# this is the root trait
if obj is not None:
e = "The '%s' trait of %s instance expected %s, not %s." % (
self.name, class_of(obj), self.info(), describe("the", value))
else:
e = "The '%s' trait expected %s, not %s." % (
self.name, self.info(), describe("the", value))
> raise TraitError(e)
E traitlets.traitlets.TraitError: The 'center' trait of an AppLayout instance expected a Widget or None, not the FigureCanvasAgg at '0xffff70f0a790'.
/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py:692: TraitError
=============================================== warnings summary ===============================================
../opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146
/opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.4
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
../opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252
/opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252: PytestConfigWarning: Unknown config option: collect_ignore
self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
tests/test_exp_script.py::test_default_run
/opt/conda/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, "__version__") or LooseVersion(
tests/test_exp_script.py::test_default_run
/opt/conda/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
) < LooseVersion("1.15"):
tests/test_exp_script.py::test_default_run
/opt/conda/lib/python3.9/site-packages/botocore/httpsession.py:34: DeprecationWarning: 'urllib3.contrib.pyopenssl' module is deprecated and will be removed in a future release of urllib3 2.x. Read more in this issue: https://github.com/urllib3/urllib3/issues/2680
from urllib3.contrib.pyopenssl import orig_util_SSLContext as SSLContext
tests/test_exp_script.py: 2268 warnings
/opt/conda/lib/python3.9/site-packages/cvxpy/interface/numpy_interface/ndarray_interface.py:49: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. To silence this warning, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
if result.dtype in [numpy.complex, numpy.float64]:
tests/test_exp_script.py::test_default_run
tests/test_exp_script.py::test_default_run
/opt/conda/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:1369: DeprecationWarning: setting an array element with a sequence. This was supported in some cases where the elements are arrays with a single element. For example `np.array([1, np.array([2])], dtype=int)`. In the future this will raise the same ValueError as `np.array([1, [2]], dtype=int)`.
return np.asarray(x, float)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================== short test summary info ============================================
FAILED tests/test_widget.py::test_basic_widget_functionality - traitlets.traitlets.TraitError: The 'center' t...
================================= 1 failed, 1 passed, 2275 warnings in 22.27s ==================================And when leaving out experiment script:
vscode ➜ /bauwerk (dev/general ✗) $ pytest --sb3 tests/test_widget.py
============================================= test session starts ==============================================
platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0
rootdir: /bauwerk, configfile: setup.cfg
plugins: anyio-3.5.0, hydra-core-1.2.0
collected 1 item
tests/test_widget.py . [100%]
=============================================== warnings summary ===============================================
../opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146
/opt/conda/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.4
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
../opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252
/opt/conda/lib/python3.9/site-packages/_pytest/config/__init__.py:1252: PytestConfigWarning: Unknown config option: collect_ignore
self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
tests/test_widget.py::test_basic_widget_functionality
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget.py:443: DeprecationWarning: Passing unrecognized arguments to super(Toolbar).__init__().
__init__() missing 1 required positional argument: 'canvas'
This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
super().__init__(**kwargs)
tests/test_widget.py::test_basic_widget_functionality
/opt/conda/lib/python3.9/site-packages/ipywidgets/widgets/widget.py:621: DeprecationWarning: Deprecated in traitlets 4.1, use the instance .metadata dictionary directly, like x.metadata[key] or x.metadata.get(key, default)
if trait.get_metadata('sync'):
tests/test_widget.py::test_basic_widget_functionality
/opt/conda/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:1369: DeprecationWarning: setting an array element with a sequence. This was supported in some cases where the elements are arrays with a single element. For example `np.array([1, np.array([2])], dtype=int)`. In the future this will raise the same ValueError as `np.array([1, [2]], dtype=int)`.
return np.asarray(x, float)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================== 1 passed, 5 warnings in 1.06s =========================================This issue tracks a preliminary roadmap of bauwerk. Not all features here will likely be implemented. The order of list elements indicates the priority (for each list, not between lists).
When I pass a seed to env.reset(seed=seed) I receive an error stating Bauwerk was not expected more that one argument to env.reset(). At first glance, the error may be in line 491 of bauwerk/envs/solar_battery_house.py where the gym-compatible env is not expecting a seed arg. I may be wrong though.
As a wider point, do we really want set a seed using .reset() rather than .make()? Should the seed not be a property of the env upon instantiation, rather than changed when we reset the env to receive the first obs?
env = gym.make("bauwerk/BuildDistB-v0")
obs = env.reset(seed=123)
File ~/miniforge3/envs/odes/lib/python3.8/site-packages/gym/wrappers/order_enforcing.py:16, in OrderEnforcing.reset(self, **kwargs)
14 def reset(self, **kwargs):
15 self._has_reset = True
---> 16 return self.env.reset(**kwargs)
TypeError: reset() got an unexpected keyword argument 'seed'
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.