gauthig / greenbutton-influxdb Goto Github PK
View Code? Open in Web Editor NEWParse SCE Green Button download files and creates import file or sends to influxdb. Grafana dashboard is included.
License: GNU General Public License v3.0
Parse SCE Green Button download files and creates import file or sends to influxdb. Grafana dashboard is included.
License: GNU General Public License v3.0
Dashboard with
As each utility can format data differently, will add am argument to state which format to parse.
Add SSL support to influx
Create first python script to generate parsed SCE Green Button data into a CSV format to check and validate the parsing puls the file correctly.
Hi This is not an issue, Just trying to reach out to you for help. I am not a programmer but I found your while looking for a way to read from SCE Green button data. Your code provide excellent information and might be useful for a lot of user. But not being a programmer I couldn't understand how to use it. I have a Docker installed on my synology and am also using Grafana to get data from Teslamate. Can you help me with using your tool. I can then write detailed instructions for general community. Thanks.
Program sets event time as end of the interval, but the utility daily tottals are derived form the start of each event. Changing to use Start interval for the time event.
Get copy of non-solar custom SCE csv and be able to parse.
Setup new parameter
FileType
sce-nem = SCE CSV in Net Metering format for solar, has both generated and delivered sections
sce-res = SCE CSV in Residential non-nem format, only shows delivered energy.
After changes to timezone conversion of pytz library, normal datetime functions do not work which are implemented in the fileoutput function.
Datetime being sent split out by segment and was spending influxdb's cpu/prog wait time to ingest.
need to convert to string before loading into dictionary for data import
with open("myfile.txt", 'w') as f:
for key, value in details.items():
f.write('%s:%s\n' % (key, value))
SCE data is local time but influxdb is supposed to be UTC. So any reports in a tool like grafana which converts to your local time will be off by -8.
Add time zone conversion
default to Los Angeles time zone since the program is primary for SCE customers.
Thanks for your efforts on this project! Attempting to set it up with my local utility provider, Pepco, the format of their residential net metering data is notably different than the SCE example provided. I've attached an example in case you might be able to add support in a future release.
example_pep_electric_interval_data_Service 1_2021-12-06_to_2022-12-06.csv
Currently only accepts command line parameters.
Desire is to have a parameter file to pull all options and parameters for database for quick reuse.
Since influxdb does not group by real month, need a tag for month.
Cleanup code to match PEP 8
Windows 11
Latest version of python installed Python 3.11.5
Requirements satisfied
C:\Temp\SCEPython>python greenbutton_import.py -f SCE-Nemfile.sample --nodb
Traceback (most recent call last):
File "C:\Temp\SCEPython\greenbutton_import.py", line 165, in
(delivered, generated, metricsout) = parse_data(args.file, args.verbose,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Temp\SCEPython\utilparse.py", line 27, in parse_data
sce_tou_parse(csv_reader, verbose, metricsout)
File "C:\Temp\SCEPython\utilparse.py", line 54, in sce_tou_parse
dt_local = datetime.strptime(util_timestamp[0], '%Y-%m-%d %H:%M:%S')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib_strptime.py", line 352, in _strptime
raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: ร
Setup first pass to sending influxdb data the parsed results.
Will be non-ssl, no password influxdb - private lab environment only due to no security
Create version of dashboard to use FLUX
Listing as bug as default policy may delete data
Add functionality to push to Influxdb v2 with token security
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.