ssenart / pygazpar Goto Github PK

Hi,

I'm using the latest pygazpar version and since, it seems to be broken :

Here is my script : https://github.com/Jeoffreybauvin/pygazpar_to_influxdb

Here is the error : pygazpar.webdriverwrapper (WARNING): Not any data file has been found in '/tmp' directory

Full log :

pygazpar.webdriverwrapper (DEBUG): find_element_by_xpath('//button[contains(.,'Télécharger')]'): Download button...
selenium.webdriver.remote.remote_connection (DEBUG): POST http://127.0.0.1:46769/session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element {"using": "xpath", "value": "//button[contains(.,'T\u00e9l\u00e9charger')]"}
urllib3.connectionpool (DEBUG): http://127.0.0.1:46769 "POST /session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element HTTP/1.1" 200 88
selenium.webdriver.remote.remote_connection (DEBUG): Finished Request
pygazpar.webdriverwrapper (DEBUG): find_element_by_xpath('//button[contains(.,'Télécharger')]'): Download button -> Ok
pygazpar.webelementwrapper (DEBUG): click(): Download button...
selenium.webdriver.remote.remote_connection (DEBUG): POST http://127.0.0.1:46769/session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element/e8b78b92-2ebe-4ada-97ad-5919606e311b/click {"id": "e8b78b92-2ebe-4ada-97ad-5919606e311b"}
urllib3.connectionpool (DEBUG): http://127.0.0.1:46769 "POST /session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e/element/e8b78b92-2ebe-4ada-97ad-5919606e311b/click HTTP/1.1" 200 14
selenium.webdriver.remote.remote_connection (DEBUG): Finished Request
pygazpar.webelementwrapper (DEBUG): click() -> Ok
pygazpar.webdriverwrapper (WARNING): Not any data file has been found in '/tmp' directory
pygazpar.client (DEBUG): The data update terminates normally
pygazpar.webdriverwrapper (DEBUG): quit()...
selenium.webdriver.remote.remote_connection (DEBUG): DELETE http://127.0.0.1:46769/session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e {}
urllib3.connectionpool (DEBUG): http://127.0.0.1:46769 "DELETE /session/46887c7d-0a20-46c7-9dc1-f0309fa4b05e HTTP/1.1" 200 14
selenium.webdriver.remote.remote_connection (DEBUG): Finished Request
pygazpar.webdriverwrapper (DEBUG): quit() -> Ok
root (DEBUG): End update pygazpar data

Is it any change I'm missing ?

Regards,

PermissionError: [WinError 32] Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus: '....\tmp\Donnees_informatives_22423299474865_2023-05-09_2024-05-08_663b5831638cb.xlsx'

Can't retrieve data from my Gazpar account. I get a:

  File "/Users/martindaniel/opt/miniconda3/lib/python3.9/site-packages/pygazpar/datasource.py", line 92, in _login
    raise Exception(loginData["error"])
Exception: CAPTCHA_FAIL

Since more 1 week, GrDF web site is half broken and Télécharger button does not work any more.

PyGazpar relies on the same URL behind the Télécharger button. We now have a 404 error on this URL.

GrDF is aware of that issue and is working on it.

Please, be patient !

Bonjour
J’ai cette erreur dans les logs:

`
File "/usr/local/lib/python3.9/dist-packages/pygazpar/client.py", line 57, in update

self.__updateLiveMode()

File "/usr/local/lib/python3.9/dist-packages/pygazpar/client.py", line 108, in __updateLiveMode

startDate = endDate + datetime.timedelta(days=-self.__lastNDays)

TypeError: bad operand type for unary -: 'str' `
Merci d’avance
Philippe

I hope they are not introducing captcha

requests.exceptions.HTTPError: 500 Server Error:  for url: https://login.monespace.grdf.fr/sofit-account-api/api/v1/auth

First, thanks for your very useful script to provide data for home automation.

When launching a script using pygazpar from 2020-05-29, I got this error:

/usr/local/lib/python2.7/dist-packages/openpyxl/styles/stylesheet.py:214: UserWarning: Workbook contains no default style, apply openpyxl's default
  warn("Workbook contains no default style, apply openpyxl's default")
Traceback (most recent call last):
  File "./test_json.py", line 17, in <module>
    client.update()
  File "/usr/local/lib/python2.7/dist-packages/pygazpar/client.py", line 127, in update
    ws = wb['Historique par jour']
  File "/usr/local/lib/python2.7/dist-packages/openpyxl/workbook/workbook.py", line 290, in __getitem__
    raise KeyError("Worksheet {0} does not exist.".format(key))
KeyError: 'Worksheet Historique par jour does not exist.'

Could it be possible to provide a fix for this? Thanks :)

With JsonWebDataSource, it should be possible to load all frequencies with only one query to GrDF site. It permits to save resource on server side.

Bonjour,
J'ai un problème de connexion, j'ai essayé ce code minimal pour débugger:

import requests, datetime
EMAIL = "..."  # a remplir
PASSWORD ="..."
PCE ="..."
session = requests.Session()
session.post('https://login.monespace.grdf.fr/sofit-account-api/api/v1/auth', data={
    'email': EMAIL,
    'password': PASSWORD,
    'capp': 'meg',
    'goto': 'https://sofa-connexion.grdf.fr:443/openam/oauth2/externeGrdf/authorize?response_type=code&scope=openid%20profile%20email%20infotravaux%20%2Fv1%2Faccreditation%20%2Fv1%2Faccreditations%20%2Fdigiconso%2Fv1%20%2Fdigiconso%2Fv1%2Fconsommations%20new_meg&client_id=prod_espaceclient&state=0&redirect_uri=https%3A%2F%2Fmonespace.grdf.fr%2F_codexch&nonce=skywsNPCVa-AeKo1Rps0HjMVRNbUqA46j7XYA4tImeI&by_pass_okta=1&capp=meg'
})
dateFormat = "%Y-%m-%d"
endDate = datetime.date.today()
startDate = endDate + datetime.timedelta(days=-7)
url = f"https://monespace.grdf.fr/api/e-conso/pce/consommation/informatives?dateDebut={startDate.strftime(dateFormat)}&dateFin={endDate.strftime(dateFormat)}&pceList%5B%5D={PCE}"
response = session.get(url)
print(response.text)

Mais la réponse ne contient rien d'utile:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Mire de connexion</title>
  <base href="/mire/">
  <script type="text/javascript">
    var onloadCallback = function() {
      setTimeout(onloadCallback, 500)
    };
  </script>
  <meta name="viewport" content="width=device-height, initial-scale=1, maximum-scale=0.42">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
  <!--<script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit" async></script>-->
</head>
<body>
  <app-root></app-root>
<script src="runtime.e7df24b5d39b2102df5c.js" defer></script><script src="polyfills.35a5ca1855eb057f016a.js" defer></script><script src="styles.b81fc9aa724cfcc4a2de.js" defer></script><script src="main.f69e72e0f4844b0bf934.js" defer></script></body>
</html>

Y a-t-il une connexion initiale à faire avec OAuth pour autoriser l'accès à l'API?

I use the sample code https://github.com/ssenart/PyGazpar#library and it works fine.
The only problem is that I do not get any "temperature_degC" field.

I get:

[{'time_period': '15/01/2022',
  'start_index_m3': 86.0,
  'end_index_m3': 87.0,
  'volume_m3': 0.0,
  'energy_kwh': 5.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '16/01/2022',
  'start_index_m3': 87.0,
  'end_index_m3': 87.0,
  'volume_m3': 0.0,
  'energy_kwh': 4.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '17/01/2022',
  'start_index_m3': 87.0,
  'end_index_m3': 87.0,
  'volume_m3': 0.0,
  'energy_kwh': 3.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '18/01/2022',
  'volume_m3': 0.0,
  'energy_kwh': 0.0,
  'type': 'Absence de Données',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '19/01/2022',
  'end_index_m3': 88.0,
  'volume_m3': 0.0,
  'energy_kwh': 4.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '20/01/2022',
  'start_index_m3': 88.0,
  'end_index_m3': 88.0,
  'volume_m3': 0.0,
  'energy_kwh': 4.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '21/01/2022',
  'start_index_m3': 88.0,
  'end_index_m3': 89.0,
  'volume_m3': 0.0,
  'energy_kwh': 4.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '22/01/2022',
  'start_index_m3': 89.0,
  'end_index_m3': 89.0,
  'volume_m3': 0.0,
  'energy_kwh': 4.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'},
 {'time_period': '23/01/2022',
  'start_index_m3': 89.0,
  'end_index_m3': 89.0,
  'volume_m3': 0.0,
  'energy_kwh': 3.0,
  'converter_factor_kwh/m3': 11.18,
  'type': 'Mesuré',
  'timestamp': '2022-01-25T21:13:48.897760'}]

I modified the code to comment the line os.remove(filename) and open the fetched .xlsx file in LibreOffice.
To my surprise the column 8 labelled "Température locale (°C)" has no temperature values in the .xlsx file.

If I login to https://monespace.grdf.fr/ using a web browser and download the data I get an .xlsx file with correct values for "Température locale (°C)".

Did I miss something?

Depuis quelques jours, pygazpar ne fonctionne plus. Message d'erreur:

Unexpected error occured : Message: Element is not clickable at point (918,569) because another element

obscures it

Peut-être un changement du site ?

Bonjour,
a chaque fois que je tente de recuperer une plage de données (20j par exemple) le programme ne tente de recuperer que la donnée 20 jours avant uniquement.
y a t il un pb avec le procole gazpar ?
cdlt
Philippe

The new GRDF web site now permits using simple requests to get the data.

Currently, only daily data are read from GrDF.

Be able to get weekly and monthly data as well.

Bonsoir,

Je tente de faire marcher ce package Python.

Je fais mon test sur mon Ubuntu 20.10, Python 3.8 , Geckodriver 0.29.1 et Firefox 81.0.2+build1-0ubuntu1.

Lorsque je lance le script, il tourne assez longtemps, pour au final ne rien renvoyer :

(venv) jeo@bud:~/dev$ pygazpar -u '[email protected]' -p 'pouet'  -w geckodriver -s 30 -t tmp/ -l 0
[]

Je semble bien avoir des consommations sur mon compte GRDF :

Les logs de geckodriver :

1618521671214	geckodriver	INFO	Listening on 127.0.0.1:57303
1618521671227	mozrunner::runner	INFO	Running command: "/usr/bin/firefox" "--marionette" "-headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofileL5Xmt8"
*** You are running in headless mode.
console.error: SearchCache: "_readCacheFile: Error reading cache file:" (new Error("", "(unknown module)"))
1618521680829	Marionette	INFO	Listening on port 38081
1618521680904	Marionette	WARN	TLS certificate errors will be ignored for this session
JavaScript error: resource://gre/modules/URIFixup.jsm, line 271: NS_ERROR_FAILURE: Should pass a non-null uri
1618521693174	Marionette	WARN	TimedPromise timed out after 500 ms: stacktrace:
TimedPromise/<@chrome://marionette/content/sync.js:245:13
TimedPromise@chrome://marionette/content/sync.js:230:10
interaction.flushEventLoop@chrome://marionette/content/interaction.js:416:10
webdriverClickElement@chrome://marionette/content/interaction.js:182:31

(firefox:18593): GLib-GObject-CRITICAL **: 23:24:10.655: g_object_get: assertion 'G_IS_OBJECT (object)' failed
1618521884487	Marionette	INFO	Stopped listening on port 38081

Une idée :( ?

Hello

When trying to use the sample code, I get

Traceback (most recent call last):
File "C:\Users\nicol\Documents\dev\grdf\grdf-import.py", line 7, in
meterReadingFrequency=Frequency.DAILY,
NameError: name 'Frequency' is not defined

If I addthe line below, the error is gone
from pygazpar.enum import Frequency

Nicolas

Bonjour,
Est-il possible d'installer PyGazpar sur Home Assistant OS ?
J'ai bien vu comment installer le custom_components, mais il requiert (ce qui est logique) cette librairie PyGazpar.
Il est possible de l'installer facilement sous HomeAssistant OS ? Merci d'avance :)

Looks like grdf added the « questions client » part just in front of the button...

Any idea on how to solve this?

Just to give context, it’s on the download_button_element.click()

It seems a new problem occurs preventing pygazpar to work since a few days (probably a new « help » button on grdf web site that I do not remember existing before)
Any way to solve this ?
Thanks

Bonjour,
J’ai finalement réussi à faire fonctionner le programme après installation.
Pour y arriver je fais:
cd /home/PyGazpar. < ——- répertoire d’installation
source venv/bin/activate < —— visiblement nécessaire sinon on n’entre pas dans l’environnement virtuel

pygazpar -u 'xxxxxx@xxxx .fr' -p 'Trucmuche' -w '/usr/local/bin/geckodriver' -s 30 -t /home/PyGazpar
A la fin du process, j’ai bien l’affichage de toutes les consommations dans ma session ssh.
J’ai pu voir aussi un fichier Excel apparaître dans le répertoire /home/PyGazpar avec le meme nom que ce qu’on peut récupérer sur le site grdf.
Mais à la fin du process, ce fichier disparaît
Comment faire pour le garder ?

je suis sous Ubuntu 20.10
Merci d’avance
Cdlt
Philippe

It applies also for month and year period.

Script Version : 1.2.2 - Ubuntu Focal - AMD64

I'm trying this script to get the data from the GRDF web site downloaded in Excel format.
The script runs fine, the output is displayed, the log file is generated in the temporary folder but the Excel file is not generated...

Here is the command line:
pygazpar -u '<user>' -p '<password>' -c 'PCE' -t '/tmp' -d 365 --datasource 'excel'

Here are the last lines of the log file:
2024-06-01 15:18:15,670 DEBUG [urllib3.connectionpool] Starting new HTTPS connection (1): monespace.grdf.fr:443
2024-06-01 15:18:16,561 DEBUG [urllib3.connectionpool] https://monespace.grdf.fr:443 "GET /api/e-conso/pce/consommation/informatives/telecharger?dateDebut=2023-06-02&dateFin=2024-06-01&frequence=Journalier&pceList%5B%5D=<PCE> HTTP/1.1" 200 48592
2024-06-01 15:18:16,579 DEBUG [pygazpar.excelparser] Loading Excel data file '/tmp/Donnees_informatives_<PCE>_2023-06-02_2024-06-01_665b1f97b279d.xlsx'...
2024-06-01 15:18:16,619 DEBUG [pygazpar.excelparser] Daily data read successfully between row #10 and row #371
2024-06-01 15:18:16,620 DEBUG [pygazpar.datasource] The data update terminates normally
2024-06-01 15:18:16,620 DEBUG [pygazpar.client] The data load terminates normally

Am'I missing something?

The readme says that this project exists for an homeassistant integrationbut i cant find the said module sources.
Do you mind to upload even if it is still wip ?
I might be interested in helping you in this project.

Bonjour,

Je viens d'installer la nouvelle version 1.0.1 et lorsque je lance le script j'obtient l'erreur suivante :

An unexpected error occured while updating the data Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/pygazpar/client.py", line 255, in __updateLiveMode self.__data = DataFileParser.parse(filename, self.__meterReadingFrequency, self.__lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 40, in parse res = parseByFrequency[dataReadingFrequency](worksheet, lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 82, in __parseDaily DataFileParser.__fillRow(row, PropertyName.START_INDEX.value, worksheet.cell(column=3, row=rownum), True) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 54, in __fillRow row[propertyName] = float(cell.value.replace(',', '.')) ValueError: could not convert string to float: 'Index de début de période (m3)' Traceback (most recent call last): File "/home/pi/script_gazpar_msql.py", line 11, in <module> client.update() File "/usr/local/lib/python3.7/dist-packages/pygazpar/client.py", line 121, in update self.__updateLiveMode() File "/usr/local/lib/python3.7/dist-packages/pygazpar/client.py", line 255, in __updateLiveMode self.__data = DataFileParser.parse(filename, self.__meterReadingFrequency, self.__lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 40, in parse res = parseByFrequency[dataReadingFrequency](worksheet, lastNRows) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 82, in __parseDaily DataFileParser.__fillRow(row, PropertyName.START_INDEX.value, worksheet.cell(column=3, row=rownum), True) File "/usr/local/lib/python3.7/dist-packages/pygazpar/datafileparser.py", line 54, in __fillRow row[propertyName] = float(cell.value.replace(',', '.')) ValueError: could not convert string to float: 'Index de début de période (m3)'

Comment je peux résoudre le problème svp ?

Hello

I'm trying to use your module and I get the following error

An unexpected error occured while updating the data
Traceback (most recent call last):
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 115, in __updateLiveMode
self.__downloadFile(session, downloadUrl, self.__tmpDirectory)
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 155, in __downloadFile
response.raise_for_status()
File "D:\Logiciels\Python\Python39\lib\site-packages\requests\models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://monespace.grdf.fr/api/e-conso/pce/consommation/informatives/telecharger?frequence=Journalier&dateDebut=2021-12-09&dateFin=2021-12-19&pceList%5B%5D=your%20PCE%20identifier
Traceback (most recent call last):
File "C:\Users\nicol\Documents\dev\grdf\grdf-import.py", line 12, in
client.update()
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 57, in update
self.__updateLiveMode()
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 115, in __updateLiveMode
self.__downloadFile(session, downloadUrl, self.__tmpDirectory)
File "D:\Logiciels\Python\Python39\lib\site-packages\pygazpar\client.py", line 155, in __downloadFile
response.raise_for_status()
File "D:\Logiciels\Python\Python39\lib\site-packages\requests\models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://monespace.grdf.fr/api/e-conso/pce/consommation/informatives/telecharger?frequence=Journalier&dateDebut=2021-12-09&dateFin=2021-12-19&pceList%5B%5D=your%20PCE%20identifier

Nicolas

hi
It was working but now I have several lines of errors including:

requests.exceptions.HTTPError: 500 Server Error: for url: https://login.monespace.grdf.fr/sofit-account-api/api/v1/auth
any idea ?
Phil

Hi
I use your engine version 1.1.6. I had to change my password in the GRDF site .
I have verified that my new password works
I can't login anymore and get those errors:

Exception: LOGIN_INVALID_ATTEMPTS
Traceback (most recent call last):
File "/data/./pygazpar_to_influxdb.py", line 55, in
client.update()
File "/usr/local/lib/python3.9/site-packages/pygazpar/client.py", line 54, in update
self.__updateLiveMode()
File "/usr/local/lib/python3.9/site-packages/pygazpar/client.py", line 96, in __updateLiveMode
self.__login(session)
File "/usr/local/lib/python3.9/site-packages/pygazpar/client.py", line 160, in __login
raise Exception(loginData["error"])
Exception: LOGIN_INVALID_ATTEMPTS

Phil

An error occured while querying PyGazpar library : %s Traceback (most recent call last):
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/main.py", line 86, in main
client.update()
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/client.py", line 118, in update
self.__updateLiveMode()
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/client.py", line 239, in __updateLiveMode
self.__data = DataFileParser.parse(filename, self.__meterReadingFrequency, self.__lastNRows)
File "/home/PyGazpar/venv/lib/python3.8/site-packages/pygazpar/datafileparser.py", line 37, in parse
worksheet = workbook[worksheetNameByFrequency[dataReadingFrequency]]
File "/home/PyGazpar/venv/lib/python3.8/site-packages/openpyxl/workbook/workbook.py", line 290, in getitem
raise KeyError("Worksheet {0} does not exist.".format(key))
KeyError: 'Worksheet Historique par jour does not exist.'

The objective is the gain time while retrieving all the data at once.