... or links to their definition on this page.

Right now we address assets a lot by their name (possibly because that is more user-friendly). This happens in our data model (aggregation class Resource is initialised with the name of an asset or an asset group), the UI (session variable "resource") and maybe the API (validator assets_required, which also works for weather sensors and markets, expects asset information encoded in the entity addresses, and that the ID for assets but the name for markets and sensors, so it's unclear what effort is required).

The problems with this are that:

• the asset name might not be url-safe (see #19)
• we have both display name and asset name, which can be set independently, and that can lead to confusing states (they can be quite different which would not be good and also the name needs to be unique if it is being used to identify assets, so we need good logic to ensure that).

In #20 I already attempted to make sure the display name is derived from the asset name (as I believed the asset name is the crucial functional field, the display name is for showing) but stopped when I realised how much code I would have to change.

We need a new policy here. Options:

1. Assets (and sensors and markets) will only have an ID and a name. The name can be freely chosen but is just for display. Simplifies the model, but requires UI work (always longer than you think), breaking API changes and the solution for aggregated asset levels is not clear.
2. Keep using asset.name but make it strictly dependent on display name (parameterised, so url-friendly), i.e. no user can edit it. Maybe rename it to ident_name or so.

In all cases, assets, sensors and markets should be modelled and treated in the same way (right now, markets and sensors have no id field for instance, but assets do).

https://github.com/Flask-Middleware/flask-security/releases/tag/4.0.0

We already made a major upgrade (User.fs_uniquifier which was available from 3.3. on), so likely we should just add a few dependencies (e.g. Flask-Mail) which are not explicit anymore if we actually require them.

But I just glanced over the release notes, let's read carefully.

Only the v2.0 page has this problem (for all endpoints, new and copied), not the earlier versions. So the difference is probably in some new approach we did there.

This ticket could also give a new tag to data endpoints which now have "User".

We deliver some scheduling algorithms out-of-the-box, with the core FlexMeasures code. Currently, this includes batteries and EV charging stations.

But certainly some algorithms will be more elaborate, more specific or private.

We need an interface to connect algorithms (as a file) to a FlexMeasures instance, so that FlexMeasures picks it up. This should happen from within a plugin.

\requirements\app.in
line 23 and line 24

should be changed to
rq-win; os_name=='win'
redis; os_name=='win'

when installing on Windows.
Maybe make mention of this in Developing.md?

We need a simple UI for data upload (think version 1 of something that can grow).

The use case here is a (potential) customer who wants to try FlexMeasures on their data for a simulation or for just seeing the forecasts.

They'll probably have a spreadsheet available, so that's what we'll support with this. We can make certain demands in this version (e.g. we'll only load two columns ― datetime and value).

A stretch goal would be to show a spinner after the upload, which spins until the forecasting job for this data (if one is enough) is finished. Or an email if that is the case (for which we'd need a monitoring task).

They are not showing now.

Also, they should not be hosted in the repo, so if we need to update them, maybe we can begin storing them in a sister repo (like Nexcloud does).

We already have this behaviour, but for historical reasons we activate it for FLEXMEASURES_MODE="play".

Here is a typical snippet:

    try:
        save_to_session(ts_value_schedule)
    except IntegrityError as e:

        current_app.logger.warning(e)
        click.echo("Rolling back due to IntegrityError")
        db.session.rollback()

        if current_app.config.get("FLEXMEASURES_MODE", "") == "play":
            click.echo("Saving again, with overwrite=True")
            save_to_session(ts_value_schedule, overwrite=True)

    db.session.commit()

It should really be happening due to a setting FLEXMEASURES_ALLOW_DATA_OVERWRITE.

TODO: add this setting (in config_defaults.py and in the documentation, then find these places in the code and adapt them to react to the new setting.

CSRF is a tricky security subject, but there is some due diligence we can do to improve security.

Flask-Security provides good advice.

From the pro-tips at the end, I already see that tips 1,2,3,4 and 6 are relevant for our non-form token-based API.

It should be possible to pip-install FlexMeasures. We need to

• make the path to the config file parametrisable (to app.create)
• host on Pypi
• find a good solution for FLASK_SECRET (use os.urandom(24))
• see if .flaskenv needs special treatment (this is important for running flask CLI commands)
• Document quickstart: pip install, create config file, add least necessary settings to run the server and CLI commands.

We want to guide people with different intentions to different parts of the docs ― for people interesting in the purpose and benefits of FlexMeasures or for developers looking for installation and setup advice.

Both can use high quality content like tutorials, but for now we are looking for the right structure.

To give more help to users, we need tutorials. We already have an Installation tutorial.

To create more will be ongoing work, but there is are some content sources in documentation/api/simulation.rst where we wrote a tutorial for researchers interacting with FlexMeasures per Python, as well as in the HTML page we made for demonstrating recent API features per JavaScript.

With a little work, these can be split up and become the first set of tutorials which shows how to use FlexMeasures.

0. Installation
1. First steps for admins ― create users, import existing data (that's a TODO, but the API can be used as well, see point 2)
2. Posting data (weather, price, power)
3. Read data: user, assets, meter data
4. Load charts to display in your app
5. Energy flexibility (get prognoses, post flex constraints, get control signals)

From the top of my head, we are missing the following tutorial content:

• user creation per CLI (this exists)
• running forecasting and scheduling jobs (this exists)

We want to base fm.Asset, fm.Market and fm.WeatherSensor on tb.SensorDBMixin, as a prerequisite to base our fm.TimedValue time series values on tb.TimedBelief, but also to take immediate advantage of the concept of knowledge horizons (and knowledge times) built into Timely Beliefs.

The concept of knowledge horizons allows us to standardize the notion of a forecast horizon for power measurements and prices. Power and price forecasts are distinct in terms of when their true values could have been known. Power values can be known after the deliver period (when power flow has been measured), while prices can be known well in advance of the delivery period to which the price pertains (for example, at the gate closure of some call auction, or on the publication date of some tariff).

I propose to:

• set the knowledge time for tariffs to their publication date
• set the knowledge time for auctions to their gate closure
• set the knowledge time for power and weather sensors to right after the fact

We can then ensure that the forecast horizon is interpreted as relative to the knowledge time, the time at which the true outcome could have been known. At least two todo items (specifically, in data/queries/utils.py and api/common/utils/validators.py) are linked to this issue, and the horizon of existing auction prices and tariffs in the database may have to be updated to reflect the change in anchoring the horizon to their relevant knowledge time rather than to their event end.

DarkSky will discontinue service end of 2021. Alternatives:

• Climacell

• OpenWeatherMap (with migration tutorial)

• Yr

• Some more

• And more

The class should get a name more indicative of what it is for. Probably becomes AggregatedAsset.

However, this still needs some more discussion, because we actually use Resource in both ways. For example:

• As an AggregatedAsset: when we set up a Resource called “wind turbines” to include all of a user’s assets of type “wind”.
• As a GeneralizedAsset: when we set up a Resource called “renewables” to include all of a user’s assets of type “solar” or “wind”.

Specifically, on the portfolio page, we only use the Resource class as an AggregatedAsset. But on the dashboard and analytics pages, we use the Resource class in both ways. And I believe “renewables” is currently our only use of a GeneralizedAsset.

The mapping of the names (like “renewables”) is handled in services/resource.py, which is good. For clarity, we could use a variable to distinct this, e.g. Resource.type=["aggregated", "generalized"].

“AggregatedAsset” seems to be a better description for both cases (generalization can be seen as a special case) and we use it more often. If a better name comes along, we're open (e.g. “AssetGroup”).

We document the API endpoints via Sphinx, but it would be much more developer-friendly to do that with the OpenAPI standard.

Our preferred way is to use ApiSpec to generate the endpoint specs (via YAML), as that might be saving a lot of work in describing what goes in and out (via Marshmallow). We've started using Marshmallow since API version 3 (and earlier versions are deprecated by now).

In any case, the existing endpoints docstrings should be straightforward to turn into OpenAPI YAML as they are now (without specific in-and-out specs), which ApiSpec will probably also eat.

So via ApiSpec for all endpoints we should get to a swagger.json file, which we'd host as a static resource.

Finally, we can use Flask-Swagger-UI so that each FlexMeasures instance hosts its own API documentation.

My open architectural questions are:

• Ideally, we would still include endpoint documentation in the official FlexMeasures API. Shpinx supports OpenAPI, so that should not be a huge problem.
• does every FM API version host their own OpenAPI documentation via a Swagger UI? I assume yes, to support every third-party developer. It could also be only the latest, with a note for each endpoint since when it is supported, but that might leave out implementation details of earlier versions. The FlexMeasures UI might need to link to these API docs (as it should also link to our Sphinx docs).
• We can look into using an API framework instead of bare-bones MethodView. Then this one and this one would be interesting. We already use the important ingredients (Marshmallow, webargs, and ApiSpec is also by the same group of people). But using one of these could also lead to other troubles, shoehorning what we do into their way. The first is my current favorite.

What is the error?

The /assets/new/ page is not showing a map. The console says: "tileLayer unknown"

When does it happen?

Every time that page is visited.

I believe we forgot to update the map logic for this template in BB-PR124

This is only a nuisance, but probably an easy fix. Clearing the session logs you out and brings you to this page (in this case when running the server locally):

http://localhost:5000/login?next=%2F%3Fclear-session%3D1

After logging in you are immediately logged out. Ad infinitum.

Tag the code and list what we did since 25 January

To empower third-party app building on top of FlexMeasures, we'll make it possible to load our figures through the FlexMeasures API.

This ticket is about the most important figure (power data). It establishes the new practice that the ui package also offers API endpoints, something which we will build out more powerful in the future (see below).

The new endpoint

Here is how the endpoint will look:

GET /charts/data/power

with a JSON body including details about which dataset to load:

{
    "resource": ....,
    "start_time": ...,
    "end_time": ....,
    "resolution": ...,
    "consumption_as_positive": true
}

The endpoint returns custom Javascript and HTML data (TODO: figure out how to send that safely through HTTP). Developers will then only need to include Bokeh CSS and JS in their HTML and the figure will show.

The new endpoint will use data.services.analytics.get_power_data and ui.views.analytics.make_power_figure.

It will be added to the API routes. I checked, and this is compatible with our plans to document all API endpoints with OpenAPI using the ApiSpec package (see #17).

Future work: more powerful chart API and more responsive figures

For the record, we have ongoing efforts for a more powerful charts API, like this:

GET /chart/<data_structure>/<chart_type>/<dataset_name>/<agg_demand_unit>

where developers can configure charts aspects of the charts. It will be based on Vega-Lite instead of Bokeh.

Data will then be loaded with a second endpoint, which allows for more interactive apps. See the charts-api branch.

Which empties it afterwards. Luckily we have flexmeasures db-restore now.

I believe it has to do with the developer's actual config file overwriting the SQLALCHEMY_URL setting from the testing defaults. Before, we could assume that there was no dedicated testing config file, so this default wasn't going to be overwritten. I believe we should not read in any user config when testing.

This page (which index.html links to at the bottom) says it documents all internal modules, but in fact only documents a few.

In ui.utils.plotting_utils.get_latest_power_as_plot, we query for the latest state (the latest non-prognosed data point) for an asset.

• It would be cleaner if the query would be on the Asset class
• The code which checks for "demo" mode (to adjust the year to the current year should actually use the new setting FLEXMEASURES_DEMO_YEAR or simply data.services.time_series.convert_query_window_for_demo.

On the control page (hidden per default), we indicate one idea how schedules can be compared (w.r.t. to economic expectations) and then ordered.

We made some mock images for this, which are expected on file, see flexmeasures/ui/templates/views/control.html and flexmeasures/ui/static/js/flexmeasures.js.

This causes an extra step in deployment of our demo. These pictures should simply live in our new screenshot repo.

We have prose content in Readmes, which should all live in the Sphinx doc (one place to look for ):

• Installation.md
• Developing.md
• flexmeasures/Readme.md
• flexmeasures/data/Readme.md
• flexmeasures/api/Readme.md

The majority of this content is directed at FlexMeasures developers, so it should get its own place. Installation gets a prominent place next to change log.

I found a tool called M2R which can help the translation.

Retire the use of the rolling parameter entirely.

• Replace the fixed (rolling=False) “horizon” JSON parameter with a “prior” JSON parameter for POST endpoints.
• Update docs: no more need to show users how to set repeating time intervals.
• New API version:
  • A user that would specify {“horizon”: “R/PT1H”} in api v1.3 could just as well specify {“horizon”: “PT1H”} in api v2.0 (both horizons would be interpreted as a 1-hour rolling horizon, which is technically a non-breaking change).
  • A user that would specify {“horizon”: “PT1H”} in api v1.3 would need to specify {“prior”: <start+duration-1H>} in api v2.0 (this is the breaking change).

Relevant discussion on Bitbucket.

FlexMeasures isn't licensed in any way. This ticket will change that.

We are considering Apache2.0.

The license needs to be added and properly mentioned. Code which the license holder (Seita) didn't write needs to be excluded. Contributors need to be aware that their code will live under the license.

Our roadmap for this comprises:

• Return BeliefsDataFrame from database query results
• Base fm.DataSource on tb.BeliefSourceDBMixin
• Base fm.Asset, fm.Market and fm.WeatherSensor on tb.SensorDBMixin (#12)
• Base fm.Power, fm.Price and fm.Weather on tb.TimedBelief

Connected to this roadmap are:

• Correct knowledge horizon functions for day-ahead markets
• Split fm.Asset into fm.Sensor and fm.Asset
• Unify fm.Power, fm.Price and fm.Weather to fm.TimedValue

Also open user data / actions via REST API as far as possible (as Flask-Security allows, it might be fine that some things only work in our UI).

The reason is so that apps based on FlexMeasures can include some user management. If we

This is what I believe should be possible:

• List users (GET)
• Get a user (GET)
• Edit a user (PATCH)
• Deactivate a user (GET)
• Trigger password reset (GET)

Not possible through the API:

• Adding a user ― this is not possible in the UI either atm.
• Deleting a user (and their data) ― that should not be available through the API.
• Login
• Get an access token

Idea: Use Flask-Smorest this time, trying to understand if that works well (marshmallow integration and generating an open API documentation).

We should host our Sphinx documentation on ReadTheDocs.

• First, we'll need an account on RTD and learn how it works.
• Maybe it's worth trying out on timely-beliefs first. Its documentation just consists of a set of markdown files with some links between them.
• We do have a custom theme, but that does not seem necessary, so we should simply use a standard one without including its code (Felix checked git history for all files in our custom sphinx theme, and we changed nothing, only the name changed from flexmeasures, and init.py got changed once by running black). We did add the file _static/css/custom.css next to the _themes folder (to support two-column layout in tables of content), but I believe we might be able to add that if we need to. For now, we added a description of content licensed by others in NOTICE but that should change.
• The API documentation will move to OpenAPI
• We'll need to improve the text we have.

Move code for offering the CLI functionality to flexmeasures.cli (now lives in flexmeasures.data.scripts.cli_tasks).

The idea is to make this code a first-class citizen.

The scripts directory should maybe not contain anything of real importance after this refactoring, only test scripts.

If I chose a different market than the one assigned to the asset (in the db), a page reload (e.g. by changing the date range) goes back to the assigned market.

Background:

The market is an asset attribute, and this is what it resets to. I'm not sure if the market also lives as a session variable, but I don't think it should. Avoiding the page refresh upon changing the date will remedy this issue to a great extent (which is what we are already working on). I consider this less a bug than a UX challenge.

Within the context of an analytics page it has some value to see how an asset would perform within another price regime, but the market should pop back to the real price regime at some point (e.g. when leaving the analytics page). Maybe we could show a warning (similar to our timezone warning) when a market is selected that is not the relevant market for the asset, with a button to pop back?

They need to load from CDN since #21 (probably were forgotten)

NB: this ticket is work in progress. I don't agree with my own analysis yet, but I couldn't afford to linger on this. In general we don't support scheduling under heterogeneous sensor resolutions, and there may be several issues at play here, like:

• Arrival times falling between interval boundaries of the power sensor.
• Missing resampling of prices to the resolution of the power sensor.

What is the problem?

Scheduling an asset with a sensor resolution of 5 minutes against market prices with a resolution of 15 minutes crashes the scheduler.

Analysis

The length of the price data is less than the length of the power schedule.

Speculation

The time range for the price query doesn't return all the prices that are needed. For example: scheduling per 5 minutes from 13:50 to 16:05 right now obtains prices per 15 minutes from 14:00 to 16:00. After resampling, len(prices) == 24, but len(schedule) == 27.

Suggested fix for speculated problem

The query should be filtered using start and end differently. Right now we have:

query = query.filter((cls.datetime > start - asset_class.event_resolution))  # dt > 13:35
query = query.filter((cls.datetime < end))  # dt < 16:05

Actually this seems correct already.

PR #65 made first progress in clearing up how we talk about FlexMeasures' concepts of flexibility and its role.

However, we can add more clarity. Here are a few terms which need better definition. Once we agreed on these and written down, the next step is to look through the documentation and use them throughout (replace alternative wordings)

Main concepts:

• flexibility
• flexibility opportunities
• activation of flexibility (opportunities)

Crucial inputs and outputs (this list needs to be shorter):

• forecasts / baseline
• schedules
• flexible schedules
• forecasted schedule / suggested schedule (vs all possible fllexible schedules)
• scheduled deviation

Steps to reproduce

1. Clone the repository
2. Read and follow the installation guide
3. First step is make install

Expected result

The install script will install the application, without affecting the existing user installed packages or operating system.

Actual result

The install script tries to uninstall existing user packages. Because this is not permitted, this leads to an error.

pip-sync requirements/app.txt
Found existing installation: ansible 2.9.6
Can't uninstall 'ansible'. No files were found to uninstall.
Found existing installation: apt-clone 0.2.1
ERROR: Cannot uninstall 'apt-clone'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Possible solutions

1. Check if the installation is run in a virtual environment and stop if it is not. Even better, have the application installed in a container by building a docker image during the installation.
2. In the developer guide, a virtual environment is set up. This step is omitted in the installation guide.

We want to be able to add custom views to be shown in FlexMeasure's UI.

Also, we want to configure which views actually get shown.

Right now we offer that for charge points (see flexmeasures.data.services.resources.get_asset_group_queries), but we could offer that easily for all assets of the same type on the same location.

Customers probably want to compare their assets this way, and it would make usage of our comparison UI more likely.

Probably we need to build a list of locations, go through the list checking if there is more than one asset, and if so, offer a new location.

Problem: Make it possible to get started with running FlexMeasures from scratch.

Similar to flexmeasures new-user, we should have flexmeasures new-asset with parameters:

• name
• asset_type_name
• unit
• latitude
• longitude
• owner_id
• market_id

Also, we need flexmeasures new-weather-sensor with parameters:

• name
• weather_sensor_type_name
• latitude
• longitude

Maybe these new functionality could make use of the asset schema (for validation). That currently resides in api/v2_0/implementations/assets.py but could move to data/models/assets.py (it's even derived from there). That's basically what the API endpoint is doing. But let's keep it simple for now.

Also, a new asset needs a market. This functionality should be possible without an existing market.

Two ideas are to always have a dummy market (with populate structure) or to allow for asset.market to be None (with unknown consequences for UI and API functionality). We tend to the former.

Preferably checking CDN performance.

At the very least we need to set the versions of external CSS and JS dependencies in one place. One example is Leaflet, which currently gets loaded individually for the following templates: dashboard.html, asset.html and asset_new.html. This makes version management more cumbersome than it should be. I suggest moving the version requirements into the app, and using Jinja to inject the right scripts in the right place within our templates. This would be similar to how bokeh handles this already:

In the view:

bokeh_html_embedded = ""
for css in CDN.css_files:
    bokeh_html_embedded += (
        """<link href="%s" rel="stylesheet" type="text/css">\n""" % css
    )
for js in CDN.js_files:
    bokeh_html_embedded += """<script src="%s"></script>\n""" % js

In the html:

{{ bokeh_html_embedded | safe }}

The map icons can be hard to discern against the new mapbox tiles. I propose to put them on top of a marker by yielding some more css magic.

Various improvements we might consider and implement:

• We might want to add a namespace to our ea schemes, to make parsing more straightforward and to maybe denote flexmeasures ea scheme versions. E.g. ea1.2021-01.io.flexmeasures:fm1.40:30
• Also, we could add keys in there for more readable entity addresses, e.g. ea1.2021-01.io.flexmeasures:fm1.owner_id:40-asset_id:30

Careful: sensor coordinates can include "-" and ".".

• Enforce full EA usage and code hygiene: Right now, sending an asset name is supported in tests, as there is code (e.g. api_utils.message_replace_name_with_ea) which turn an asset name into its ID. That code should not live in non-test modules, and I want to study why tests are succeeding now when we don't accept addresses like "CS 1" anymore (scheme and authority are needed in parse_entity_address)

Flask-Security offers a handy function for this.

The password reset route we are offering is meant for the case that users lost a password or one isn't sure if the password is safe any more. Then it seems like a good idea to reset their cookies and tokens, as well.

Also, deactivated users should stop having access immediately.

This happens on current main and can be observed in the demo data with the demo user who should have 5 assets, which also shows when you click on the zero.

What is the error?

IndexError : "list index out of range"

When does it happen?

In two places, when clicking on "Analyse this asset", on the assets page and on the asset edit page. They lead to the analytics page where the error is then shown.

Upon further analysis, the asset_name seems to be url encoded twice, making a space turn into %2520 rather than %20. The asset can still be viewed in the analytics view by selecting from the list of assets there.

We want to support adding datasets (CSV data) through the CLI functions (for admins starting a new FlexMeasures instance), which should in the future also allow users to upload their own datasets through the UI.

Research: This data service should probably draw on some functionality we are using in the API endpoints which allow adding data (e.g. validation).

Currently, posted data is checked for its resolution, but not for whether events start exactly an integer number of resolutions past midnight (for example, 0, 15, 30 or 45 minutes past the hour, assuming a 15 min resolution). Data sent for a timeslot of 3 past midnight to 18 past midnight is accepted without problems for a sensor with a 15-minute resolution, but doesn't conform to the sensor period in the way that data is handled downstream. We should deal with this.

One straightforward way forward is to disallow such data input, by having the API return an appropriate error message that lays the (arguably small) burden of resampling with zero offset on the API user.

Less straightforward alternatives are:

1. to allow such data for a sensor configured to measure events with a non-zero offset.

In case the sensor would both have some data with and without offset, this would lead to data gaps that are unresolvable, because the gaps have a duration less than the sensor resolution. However, if all data for a given sensor has the same (non-zero) offset, this could still be considered a valid time series with a fixed resolution and could be handled downstream (although requiring additional inference logic).

2. to adjust the sensor to a more fine-grained resolution and to (asynchronously) resample all existing data.

In the example above, 3 minutes would be the greatest common denominator between the offset of 3 minutes and the original resolution of 15 minutes.

E.g. using a comma in the longitude field leads to the attached error message which clearly is about something else: