open-power-system-data / weather_data Goto Github PK

I would suggest to split up the weather data package to multiple files. This could be one "country-level" file weather_data_countries.csv and one NUTS2-level file weather_data_regions.csv. Of course people can also use the filter, but that is much slower and people might not notice the possibility to filter.

Seems a SSL connection can not be made in step "4.1 Get wind data".

WARNING:opendap_download:<urlopen error [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:645)>

Does someone has a suggestion how to fix this?

I was using the notebook: download_merra2.ipynb and in the section: Setting up the DataFrame the following function raised an error:

 xr.open_mfdataset(file_path, concat_dim='date', preprocess=extract_date)

raised:

xarray ValueError: Could not find any dimension coordinates to use to order the datasets for concatenation

I solved it (for Germany) by changing the code in the following way:

def extract_date(data_set):
    """
    Extracts the date from the filename before merging the datasets. 
    """
    try:
        # The attribute name changed during the development of this script
        # from HDF5_Global.Filename to Filename. 
        if 'HDF5_GLOBAL.Filename' in data_set.attrs:
            f_name = data_set.attrs['HDF5_GLOBAL.Filename']
        elif 'Filename' in data_set.attrs:
            f_name = data_set.attrs['Filename']
        else: 
            raise AttributeError('The attribute name has changed again!')

        # find a match between "." and ".nc4" that does not have "." .
        exp = r'(?<=\.)[^\.]*(?=\.nc4)'
        res = re.search(exp, f_name).group(0)
        # Extract the date. 
        y, m, d = res[0:4], res[4:6], res[6:8]
        date_str = ('%s-%s-%s' % (y, m, d))
        data_set = data_set.assign(date=date_str)
         data_set = data_set.expand_dims("date") 
         data_set.coords["lat"] = [47.5, 48.0, 48.5, 49.0, 49.5, 50.0, 50.5, 51.0, 51.5, 52.0, 52.5, 53.0, 53.5, 54.0, 54.5, 55.0] 
         data_set.coords["lon"] = [5.625, 6.25, 6.875, 7.5, 8.125, 8.75, 9.375, 10.0, 10.625, 11.25, 11.875, 12.5, 13.125, 13.75, 14.375, 15.0] 
         data_set.coords["time"] = list(range(24)) 


        return data_set

    except KeyError:
        # The last dataset is the one all the other sets will be merged into. 
        # Therefore, no date can be extracted.
         data_set.coords["lat"] = [47.5, 48.0, 48.5, 49.0, 49.5, 50.0, 50.5, 51.0, 51.5, 52.0, 52.5, 53.0, 53.5, 54.0, 54.5, 55.0] 
         data_set.coords["lon"] = [5.625, 6.25, 6.875, 7.5, 8.125, 8.75, 9.375, 10.0, 10.625, 11.25, 11.875, 12.5, 13.125, 13.75, 14.375, 15.0] 
         data_set.coords["time"] = list(range(24)) 
        return data_set

and by commenting in the following cell:

df.drop('DISPH', axis=1, inplace=True)
df.drop(['time', 'date'], axis=1, inplace=True)
df.drop(['U2M', 'U10M', 'U50M', 'V2M', 'V10M', 'V50M'], axis=1, inplace=True)

# df['lat'] = df['lat'].apply(lambda x: lat_array[int(x)])
# df['lon'] = df['lon'].apply(lambda x: lon_array[int(x)])

I could not check whether the same error occurred on another machine.

For the rest thank you for writing this awesome notebook!

For the actual solar generation time series for Spain 2015 (ES_solar_generation_actual, hourly based time series, from 2015-01-01 to 2015-12-31) there are non-zero loads for every hour of the year. I would expect the loads to be zero between sunset and sunrise.

I would be very interested in finding out why this is the case.

Currently, I can't call the function extract_date(data_set). If I run the current version of the script I get this error:

If I type in combine='nested' after concat_dim='date' I get this error:

The dependencies 'netcdf4' and 'h5netcdf' are already installed. I am using Python 3.10 on Windows 11. The version of 'netcdf4' is 1.6.3; for 'h5netcdf' I am using version 1.10. For xarray am using the version 2023.3.0.

How do I fix this problem?

Thanks in advance!

• add support for solar radiation data
• add support for pressure data
• add support for temperature data
• enable flexible timeframes (less or more than one year)
• implement extension of spatial boundaries to next available data point
• enable import of manually downloaded raw data for processing
• add calculation of windspeed af user-given heigth

The timestamp column seems to be be in Central European (CET) time, not UTC as claimed, i.e. there is a "missing" hour in March and a "duplicated" hour in October, ie.e in 2013:

Hi，
The resolution of the data provided by MERRA2 is 0.5x0.625. I tried to make the following changes to in the download_merra2.ipynb file to reduce the resolution, but in the final output csv file, the latitude and longitude values are negative numbers.

Does anyone know how to reduce the resolution of the data while downloading it?

Hi,

trying to connect I received an IOSError after providing username and password. After some trial and error I found that the issue is the url:
old value: BASE_URL = 'https://goldsmr4.sci.gsfc.nasa.gov/opendap/MERRA2/M2T1NXSLV.5.12.4/'
new value: BASE_URL = 'https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2/M2T1NXSLV.5.12.4/'
This needs to be updated in the "download_script.ipynb" (in the "subsetting data" section of the Jupyter notebook").

Also note that for the temperature, density, etc the url is starting with "http://", this should be changed into "https://" (add the 's' after 'http').

Please update this in the code and all subsequent base_url (e.g. for roughness, pressure, temperature, etc).

Kind regards,
Pieter