gauthig / greenbutton-influxdb Goto Github PK

Dashboard with

• Time series that shows both Revered power from SCE and Generated to SCE
• 7 Day sums of Delivered
• 7 Day Sums of Generated

As each utility can format data differently, will add am argument to state which format to parse.

Add SSL support to influx

Create first python script to generate parsed SCE Green Button data into a CSV format to check and validate the parsing puls the file correctly.

Hi This is not an issue, Just trying to reach out to you for help. I am not a programmer but I found your while looking for a way to read from SCE Green button data. Your code provide excellent information and might be useful for a lot of user. But not being a programmer I couldn't understand how to use it. I have a Docker installed on my synology and am also using Grafana to get data from Teslamate. Can you help me with using your tool. I can then write detailed instructions for general community. Thanks.

Program sets event time as end of the interval, but the utility daily tottals are derived form the start of each event. Changing to use Start interval for the time event.

Get copy of non-solar custom SCE csv and be able to parse.
Setup new parameter
FileType
sce-nem = SCE CSV in Net Metering format for solar, has both generated and delivered sections
sce-res = SCE CSV in Residential non-nem format, only shows delivered energy.

After changes to timezone conversion of pytz library, normal datetime functions do not work which are implemented in the fileoutput function.

Datetime being sent split out by segment and was spending influxdb's cpu/prog wait time to ingest.

need to convert to string before loading into dictionary for data import

with open("myfile.txt", 'w') as f:
for key, value in details.items():
f.write('%s:%s\n' % (key, value))

SCE data is local time but influxdb is supposed to be UTC. So any reports in a tool like grafana which converts to your local time will be off by -8.

Add time zone conversion
default to Los Angeles time zone since the program is primary for SCE customers.

Thanks for your efforts on this project! Attempting to set it up with my local utility provider, Pepco, the format of their residential net metering data is notably different than the SCE example provided. I've attached an example in case you might be able to add support in a future release.
example_pep_electric_interval_data_Service 1_2021-12-06_to_2022-12-06.csv

Currently only accepts command line parameters.
Desire is to have a parameter file to pull all options and parameters for database for quick reuse.

Since influxdb does not group by real month, need a tag for month.

Cleanup code to match PEP 8

Windows 11
Latest version of python installed Python 3.11.5
Requirements satisfied

C:\Temp\SCEPython>python greenbutton_import.py -f SCE-Nemfile.sample --nodb
Traceback (most recent call last):
File "C:\Temp\SCEPython\greenbutton_import.py", line 165, in
(delivered, generated, metricsout) = parse_data(args.file, args.verbose,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Temp\SCEPython\utilparse.py", line 27, in parse_data
sce_tou_parse(csv_reader, verbose, metricsout)
File "C:\Temp\SCEPython\utilparse.py", line 54, in sce_tou_parse
dt_local = datetime.strptime(util_timestamp[0], '%Y-%m-%d %H:%M:%S')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib_strptime.py", line 352, in _strptime
raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: Â

Setup first pass to sending influxdb data the parsed results.
Will be non-ssl, no password influxdb - private lab environment only due to no security

Create version of dashboard to use FLUX

• Add policy to json parm file
  -use in createdb like
  client.create_retention_policy('awesome_policy', '3d', 3, default=True)
  -add policy name to write_points

Listing as bug as default policy may delete data

Add functionality to push to Influxdb v2 with token security