ssenart / pygazpar Goto Github PK
View Code? Open in Web Editor NEWPython library to download gas consumption from a Grdf account
License: MIT License
Python library to download gas consumption from a Grdf account
License: MIT License
Hi,
I'm using the latest pygazpar version and since, it seems to be broken :
Here is my script : https://github.com/Jeoffreybauvin/pygazpar_to_influxdb
Here is the error : pygazpar.webdriverwrapper (WARNING): Not any data file has been found in '/tmp' directory
Full log :
pygazpar.webdriverwrapper (DEBUG): find_element_by_xpath('//button[contains(.,'Télécharger')]'): Download button...
selenium.webdriver.remote.remote_connection (DEBUG): POST http://127.0.0.1:46769/session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element {"using": "xpath", "value": "//button[contains(.,'T\u00e9l\u00e9charger')]"}
urllib3.connectionpool (DEBUG): http://127.0.0.1:46769 "POST /session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element HTTP/1.1" 200 88
selenium.webdriver.remote.remote_connection (DEBUG): Finished Request
pygazpar.webdriverwrapper (DEBUG): find_element_by_xpath('//button[contains(.,'Télécharger')]'): Download button -> Ok
pygazpar.webelementwrapper (DEBUG): click(): Download button...
selenium.webdriver.remote.remote_connection (DEBUG): POST http://127.0.0.1:46769/session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element/e8b78b92-2ebe-4ada-97ad-5919606e311b/click {"id": "e8b78b92-2ebe-4ada-97ad-5919606e311b"}
urllib3.connectionpool (DEBUG): http://127.0.0.1:46769 "POST /session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element/e8b78b92-2ebe-4ada-97ad-5919606e311b/click HTTP/1.1" 200 14
selenium.webdriver.remote.remote_connection (DEBUG): Finished Request
pygazpar.webelementwrapper (DEBUG): click() -> Ok
pygazpar.webdriverwrapper (WARNING): Not any data file has been found in '/tmp' directory
pygazpar.client (DEBUG): The data update terminates normally
pygazpar.webdriverwrapper (DEBUG): quit()...
selenium.webdriver.remote.remote_connection (DEBUG): DELETE http://127.0.0.1:46769/session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e {}
urllib3.connectionpool (DEBUG): http://127.0.0.1:46769 "DELETE /session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e HTTP/1.1" 200 14
selenium.webdriver.remote.remote_connection (DEBUG): Finished Request
pygazpar.webdriverwrapper (DEBUG): quit() -> Ok
root (DEBUG): End update pygazpar data
Is it any change I'm missing ?
Regards,
PermissionError: [WinError 32] Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus: '....\tmp\Donnees_informatives_22423299474865_2023-05-09_2024-05-08_663b5831638cb.xlsx'
Can't retrieve data from my Gazpar account. I get a:
File "/Users/martindaniel/opt/miniconda3/lib/python3.9/site-packages/pygazpar/datasource.py", line 92, in _login
raise Exception(loginData["error"])
Exception: CAPTCHA_FAIL
Since more 1 week, GrDF web site is half broken and Télécharger button does not work any more.
PyGazpar relies on the same URL behind the Télécharger button. We now have a 404 error on this URL.
GrDF is aware of that issue and is working on it.
Please, be patient !
Bonjour
J’ai cette erreur dans les logs:
`
File "/usr/local/lib/python3.9/dist-packages/pygazpar/client.py", line 57, in update
self.__updateLiveMode()
File "/usr/local/lib/python3.9/dist-packages/pygazpar/client.py", line 108, in __updateLiveMode
startDate = endDate + datetime.timedelta(days=-self.__lastNDays)
TypeError: bad operand type for unary -: 'str' `
Merci d’avance
Philippe
I hope they are not introducing captcha
requests.exceptions.HTTPError: 500 Server Error: for url: https://login.monespace.grdf.fr/sofit-account-api/api/v1/auth
First, thanks for your very useful script to provide data for home automation.
When launching a script using pygazpar from 2020-05-29, I got this error:
/usr/local/lib/python2.7/dist-packages/openpyxl/styles/stylesheet.py:214: UserWarning: Workbook contains no default style, apply openpyxl's default
warn("Workbook contains no default style, apply openpyxl's default")
Traceback (most recent call last):
File "./test_json.py", line 17, in <module>
client.update()
File "/usr/local/lib/python2.7/dist-packages/pygazpar/client.py", line 127, in update
ws = wb['Historique par jour']
File "/usr/local/lib/python2.7/dist-packages/openpyxl/workbook/workbook.py", line 290, in __getitem__
raise KeyError("Worksheet {0} does not exist.".format(key))
KeyError: 'Worksheet Historique par jour does not exist.'
Could it be possible to provide a fix for this? Thanks :)
With JsonWebDataSource, it should be possible to load all frequencies with only one query to GrDF site. It permits to save resource on server side.
Bonjour,
J'ai un problème de connexion, j'ai essayé ce code minimal pour débugger:
import requests, datetime
EMAIL = "..." # a remplir
PASSWORD ="..."
PCE ="..."
session = requests.Session()
session.post('https://login.monespace.grdf.fr/sofit-account-api/api/v1/auth', data={
'email': EMAIL,
'password': PASSWORD,
'capp': 'meg',
'goto': 'https://sofa-connexion.grdf.fr:443/openam/oauth2/externeGrdf/authorize?response_type=code&scope=openid%20profile%20email%20infotravaux%20%2Fv1%2Faccreditation%20%2Fv1%2Faccreditations%20%2Fdigiconso%2Fv1%20%2Fdigiconso%2Fv1%2Fconsommations%20new_meg&client_id=prod_espaceclient&state=0&redirect_uri=https%3A%2F%2Fmonespace.grdf.fr%2F_codexch&nonce=skywsNPCVa-AeKo1Rps0HjMVRNbUqA46j7XYA4tImeI&by_pass_okta=1&capp=meg'
})
dateFormat = "%Y-%m-%d"
endDate = datetime.date.today()
startDate = endDate + datetime.timedelta(days=-7)
url = f"https://monespace.grdf.fr/api/e-conso/pce/consommation/informatives?dateDebut={startDate.strftime(dateFormat)}&dateFin={endDate.strftime(dateFormat)}&pceList%5B%5D={PCE}"
response = session.get(url)
print(response.text)
Mais la réponse ne contient rien d'utile:
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Mire de connexion</title>
<base href="/mire/">
<script type="text/javascript">
var onloadCallback = function() {
setTimeout(onloadCallback, 500)
};
</script>
<meta name="viewport" content="width=device-height, initial-scale=1, maximum-scale=0.42">
<link rel="icon" type="image/x-icon" href="favicon.ico">
<!--<script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit" async></script>-->
</head>
<body>
<app-root></app-root>
<script src="runtime.e7df24b5d39b2102df5c.js" defer></script><script src="polyfills.35a5ca1855eb057f016a.js" defer></script><script src="styles.b81fc9aa724cfcc4a2de.js" defer></script><script src="main.f69e72e0f4844b0bf934.js" defer></script></body>
</html>
Y a-t-il une connexion initiale à faire avec OAuth pour autoriser l'accès à l'API?
I use the sample code https://github.com/ssenart/PyGazpar#library and it works fine.
The only problem is that I do not get any "temperature_degC" field.
I get:
[{'time_period': '15/01/2022',
'start_index_m3': 86.0,
'end_index_m3': 87.0,
'volume_m3': 0.0,
'energy_kwh': 5.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '16/01/2022',
'start_index_m3': 87.0,
'end_index_m3': 87.0,
'volume_m3': 0.0,
'energy_kwh': 4.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '17/01/2022',
'start_index_m3': 87.0,
'end_index_m3': 87.0,
'volume_m3': 0.0,
'energy_kwh': 3.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '18/01/2022',
'volume_m3': 0.0,
'energy_kwh': 0.0,
'type': 'Absence de Données',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '19/01/2022',
'end_index_m3': 88.0,
'volume_m3': 0.0,
'energy_kwh': 4.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '20/01/2022',
'start_index_m3': 88.0,
'end_index_m3': 88.0,
'volume_m3': 0.0,
'energy_kwh': 4.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '21/01/2022',
'start_index_m3': 88.0,
'end_index_m3': 89.0,
'volume_m3': 0.0,
'energy_kwh': 4.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '22/01/2022',
'start_index_m3': 89.0,
'end_index_m3': 89.0,
'volume_m3': 0.0,
'energy_kwh': 4.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'},
{'time_period': '23/01/2022',
'start_index_m3': 89.0,
'end_index_m3': 89.0,
'volume_m3': 0.0,
'energy_kwh': 3.0,
'converter_factor_kwh/m3': 11.18,
'type': 'Mesuré',
'timestamp': '2022-01-25T21:13:48.897760'}]
I modified the code to comment the line os.remove(filename)
and open the fetched .xlsx
file in LibreOffice.
To my surprise the column 8 labelled "Température locale (°C)" has no temperature values in the .xlsx
file.
If I login to https://monespace.grdf.fr/ using a web browser and download the data I get an .xlsx
file with correct values for "Température locale (°C)".
Did I miss something?
Depuis quelques jours, pygazpar ne fonctionne plus. Message d'erreur:
Unexpected error occured : Message: Element is not clickable at point (918,569) because another element
obscures it
Peut-être un changement du site ?
Bonjour,
a chaque fois que je tente de recuperer une plage de données (20j par exemple) le programme ne tente de recuperer que la donnée 20 jours avant uniquement.
y a t il un pb avec le procole gazpar ?
cdlt
Philippe
The new GRDF web site now permits using simple requests to get the data.
Currently, only daily data are read from GrDF.
Be able to get weekly and monthly data as well.
Bonsoir,
Je tente de faire marcher ce package Python.
Je fais mon test sur mon Ubuntu 20.10, Python 3.8 , Geckodriver 0.29.1 et Firefox 81.0.2+build1-0ubuntu1.
Lorsque je lance le script, il tourne assez longtemps, pour au final ne rien renvoyer :
(venv) jeo@bud:~/dev$ pygazpar -u '[email protected]' -p 'pouet' -w geckodriver -s 30 -t tmp/ -l 0
[]
Je semble bien avoir des consommations sur mon compte GRDF :
Les logs de geckodriver :
1618521671214 geckodriver INFO Listening on 127.0.0.1:57303
1618521671227 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "-headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofileL5Xmt8"
*** You are running in headless mode.
console.error: SearchCache: "_readCacheFile: Error reading cache file:" (new Error("", "(unknown module)"))
1618521680829 Marionette INFO Listening on port 38081
1618521680904 Marionette WARN TLS certificate errors will be ignored for this session
JavaScript error: resource://gre/modules/URIFixup.jsm, line 271: NS_ERROR_FAILURE: Should pass a non-null uri
1618521693174 Marionette WARN TimedPromise timed out after 500 ms: stacktrace:
TimedPromise/<@chrome://marionette/content/sync.js:245:13
TimedPromise@chrome://marionette/content/sync.js:230:10
interaction.flushEventLoop@chrome://marionette/content/interaction.js:416:10
webdriverClickElement@chrome://marionette/content/interaction.js:182:31
(firefox:18593): GLib-GObject-CRITICAL **: 23:24:10.655: g_object_get: assertion 'G_IS_OBJECT (object)' failed
1618521884487 Marionette INFO Stopped listening on port 38081
Une idée :( ?
Hello
When trying to use the sample code, I get
Traceback (most recent call last):
File "C:\Users\nicol\Documents\dev\grdf\grdf-import.py", line 7, in
meterReadingFrequency=Frequency.DAILY,
NameError: name 'Frequency' is not defined
If I addthe line below, the error is gone
from pygazpar.enum import Frequency
Nicolas
Bonjour,
Est-il possible d'installer PyGazpar sur Home Assistant OS ?
J'ai bien vu comment installer le custom_components, mais il requiert (ce qui est logique) cette librairie PyGazpar.
Il est possible de l'installer facilement sous HomeAssistant OS ? Merci d'avance :)
It seems a new problem occurs preventing pygazpar to work since a few days (probably a new « help » button on grdf web site that I do not remember existing before)
Any way to solve this ?
Thanks
Bonjour,
J’ai finalement réussi à faire fonctionner le programme après installation.
Pour y arriver je fais:
cd /home/PyGazpar. < ——- répertoire d’installation
source venv/bin/activate < —— visiblement nécessaire sinon on n’entre pas dans l’environnement virtuel
pygazpar -u 'xxxxxx@xxxx .fr' -p 'Trucmuche' -w '/usr/local/bin/geckodriver' -s 30 -t /home/PyGazpar
A la fin du process, j’ai bien l’affichage de toutes les consommations dans ma session ssh.
J’ai pu voir aussi un fichier Excel apparaître dans le répertoire /home/PyGazpar avec le meme nom que ce qu’on peut récupérer sur le site grdf.
Mais à la fin du process, ce fichier disparaît
Comment faire pour le garder ?
je suis sous Ubuntu 20.10
Merci d’avance
Cdlt
Philippe
It applies also for month and year period.
Script Version : 1.2.2 - Ubuntu Focal - AMD64
I'm trying this script to get the data from the GRDF web site downloaded in Excel format.
The script runs fine, the output is displayed, the log file is generated in the temporary folder but the Excel file is not generated...
Here is the command line:
pygazpar -u '<user>' -p '<password>' -c 'PCE' -t '/tmp' -d 365 --datasource 'excel'
Here are the last lines of the log file:
2024-06-01 15:18:15,670 DEBUG [urllib3.connectionpool] Starting new HTTPS connection (1): monespace.grdf.fr:443
2024-06-01 15:18:16,561 DEBUG [urllib3.connectionpool] https://monespace.grdf.fr:443 "GET /api/e-conso/pce/consommation/informatives/telecharger?dateDebut=2023-06-02&dateFin=2024-06-01&frequence=Journalier&pceList%5B%5D=<PCE> HTTP/1.1" 200 48592
2024-06-01 15:18:16,579 DEBUG [pygazpar.excelparser] Loading Excel data file '/tmp/Donnees_informatives_<PCE>_2023-06-02_2024-06-01_665b1f97b279d.xlsx'...
2024-06-01 15:18:16,619 DEBUG [pygazpar.excelparser] Daily data read successfully between row #10 and row #371
2024-06-01 15:18:16,620 DEBUG [pygazpar.datasource] The data update terminates normally
2024-06-01 15:18:16,620 DEBUG [pygazpar.client] The data load terminates normally
Am'I missing something?
The readme says that this project exists for an homeassistant integrationbut i cant find the said module sources.
Do you mind to upload even if it is still wip ?
I might be interested in helping you in this project.
Bonjour,
Je viens d'installer la nouvelle version 1.0.1 et lorsque je lance le script j'obtient l'erreur suivante :
An unexpected error occured while updating the data Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/pygazpar/client.py", line 255, in __updateLiveMode self.__data = DataFileParser.parse(filename, self.__meterReadingFrequency, self.__lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 40, in parse res = parseByFrequency[dataReadingFrequency](worksheet, lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 82, in __parseDaily DataFileParser.__fillRow(row, PropertyName.START_INDEX.value, worksheet.cell(column=3, row=rownum), True) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 54, in __fillRow row[propertyName] = float(cell.value.replace(',', '.')) ValueError: could not convert string to float: 'Index de début de période (m3)' Traceback (most recent call last): File "/home/pi/script_gazpar_msql.py", line 11, in <module> client.update() File "/usr/local/lib/python3.7/dist-packages/pygazpar/client.py", line 121, in update self.__updateLiveMode() File "/usr/local/lib/python3.7/dist-packages/pygazpar/client.py", line 255, in __updateLiveMode self.__data = DataFileParser.parse(filename, self.__meterReadingFrequency, self.__lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 40, in parse res = parseByFrequency[dataReadingFrequency](worksheet, lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 82, in __parseDaily DataFileParser.__fillRow(row, PropertyName.START_INDEX.value, worksheet.cell(column=3, row=rownum), True) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 54, in __fillRow row[propertyName] = float(cell.value.replace(',', '.')) ValueError: could not convert string to float: 'Index de début de période (m3)'
Comment je peux résoudre le problème svp ?
Hello
I'm trying to use your module and I get the following error
An unexpected error occured while updating the data
Traceback (most recent call last):
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 115, in __updateLiveMode
self.__downloadFile(session, downloadUrl, self.__tmpDirectory)
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 155, in __downloadFile
response.raise_for_status()
File "D:\Logiciels\Python\Python39\lib\site-packages\requests\models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://monespace.grdf.fr/api/e-conso/pce/consommation/informatives/telecharger?frequence=Journalier&dateDebut=2021-12-09&dateFin=2021-12-19&pceList%5B%5D=your%20PCE%20identifier
Traceback (most recent call last):
File "C:\Users\nicol\Documents\dev\grdf\grdf-import.py", line 12, in
client.update()
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 57, in update
self.__updateLiveMode()
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 115, in __updateLiveMode
self.__downloadFile(session, downloadUrl, self.__tmpDirectory)
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 155, in __downloadFile
response.raise_for_status()
File "D:\Logiciels\Python\Python39\lib\site-packages\requests\models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://monespace.grdf.fr/api/e-conso/pce/consommation/informatives/telecharger?frequence=Journalier&dateDebut=2021-12-09&dateFin=2021-12-19&pceList%5B%5D=your%20PCE%20identifier
Nicolas
hi
It was working but now I have several lines of errors including:
requests.exceptions.HTTPError: 500 Server Error: for url: https://login.monespace.grdf.fr/sofit-account-api/api/v1/auth
any idea ?
Phil
Hi
I use your engine version 1.1.6. I had to change my password in the GRDF site .
I have verified that my new password works
I can't login anymore and get those errors:
Exception: LOGIN_INVALID_ATTEMPTS
Traceback (most recent call last):
File "/data/./pygazpar_to_influxdb.py", line 55, in
client.update()
File "/usr/local/lib/python3.9/site-packages/pygazpar/client.py", line 54, in update
self.__updateLiveMode()
File "/usr/local/lib/python3.9/site-packages/pygazpar/client.py", line 96, in __updateLiveMode
self.__login(session)
File "/usr/local/lib/python3.9/site-packages/pygazpar/client.py", line 160, in __login
raise Exception(loginData["error"])
Exception: LOGIN_INVALID_ATTEMPTS
Phil
An error occured while querying PyGazpar library : %s Traceback (most recent call last):
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/main.py", line 86, in main
client.update()
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/client.py", line 118, in update
self.__updateLiveMode()
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/client.py", line 239, in __updateLiveMode
self.__data = DataFileParser.parse(filename, self.__meterReadingFrequency, self.__lastNRows)
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/datafileparser.py", line 37, in parse
worksheet = workbook[worksheetNameByFrequency[dataReadingFrequency]]
File "/home/PyGazpar/venv/lib/python3.8/site-packages/openpyxl/workbook/workbook.py", line 290, in getitem
raise KeyError("Worksheet {0} does not exist.".format(key))
KeyError: 'Worksheet Historique par jour does not exist.'
The objective is the gain time while retrieving all the data at once.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.