awslabs / open-data-registry Goto Github PK

This is being addressed here cytodata/cytodata-hackathon-2018#10

Can someone pls expand the short names or what attributes/parameters these folders stand for? I could not find any documentation on NOAA website.

Attempting to crawl this location with Glue pointed at the ARN produces an Access Denied issue. What's the correct way to configure this?

I tried giving Glue more permissions (all the way to Admin) to no avail.

The HRRR has plans to extended their forecast lead times form 18h to 48h for the 00,06,18,24 initializations. Their website states june 2020 for the move to v4 (https://rapidrefresh.noaa.gov/hrrr/) but I already see the extended lead times on a number of plots (weathermodels.com)

Can this extended lead time be added to the opendata registry?

https://github.com/awslabs/open-data-registry/blob/master/datasets/noaa-hrrr-pds.yaml

Would it be possible to have an SNS topic setup for the noaa-ghe data (https://registry.opendata.aws/noaa-ghe/)?

Thank you!

Handle similar to tags

Hi,

I have pulled a request out #536 in order to add PMSP LiDAR 3D Yaml Dataset file.

Thanks,

The csv prefix is missing from bucket dataforgood-fb-data. I cannot load partition from Athena.
I created the table by using:

CREATE EXTERNAL TABLE IF NOT EXISTS default.hrsl (
  `latitude` double,
  `longitude` double,
  `population` double 
) PARTITIONED BY (
  month string,
  country string,
  type string 
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
  'serialization.format' = '\t',
  'field.delim' = '\t'
) LOCATION 's3://dataforgood-fb-data/csv/'
TBLPROPERTIES ('has_encrypted_data'='false', 'skip.header.line.count'='1');

Then load partitions by using:
MSCK REPAIR TABLE hrsl;

But I got this error message:
Tables missing on filesystem: hrsl

While doing some analysis on 990 filings, I noticed a discrepancy between the number of filings in the 2019 CSV and JSON index files. It appears that the CSV index file has 416,880 while the JSON has 396,217. The CSV file also looks to have been updated much more recently than the JSON file (4/2020 vs 12/2019). I have not checked the index files for other years, though there may be conflicting counts there as well.

Wasn't sure if this is the best place to report it, but the ~20K difference seemed pretty significant. I haven't done any additional analysis yet to rule out something like duplicate records - figured I'd start here. Happy to lend a hand if I can help in any way.

https://github.com/awslabs/open-data-registry/blob/master/_scripts/deploy.sh

We use cp which will not remove pages if they no longer exist.

I would like to suggest adding the NPPES NPI registry. It is a national registry for prescribers publicly available and incredibly useful/critical to healthcare/medical organizations.

https://npiregistry.cms.hhs.gov/

How can I download only 5% of cse-cic-ids2018 dataset since the storage in my local mac machine is not available to download all this 220 GiB dataset?

I am requesting to add the ERA5 Land data set to the registry.

Although the ERA5 is already in the registry, the Land component is significnalty higher resolution (9km vs 32km) so is more suited for land based use cases.

I am currently trying to download the open data (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-land?tab=overview) to my local to push to S3. I thought I would reach out here to see if the Open data team could help push it into the S3 so others could use as well.

Thanks!

I might be mistaken, but based on this comment and the fact that the download instructions on the usaspending site have changed, it seems that the snapshot is no longer available.

given that, it would be nice if the documentation here reflected that.

This would make it more clear who manages the ingest process when not being done by the data source (or a group affiliated with them).

@borenstein - I'm looking at the entry for the 990 data and it seems that http://990.charitynavigator.org may be stale at this point. Should we replace that with a pointer to https://github.com/CharityNavigator/990_long?

If you think so, please feel free to create a pull request or let me know if you'd like me to.

@jedsundwall Was trying to find contact info for the IRS folks to alert them of an error in their indexes and noticed the existing link at the registry site is throwing a 404.

The best link I could find was this one?

In case the fine folks at the IRS read this, the 2018 indexes are currently out of sync (e.g. the 2018 CSV index shows 457,509 records while the 2018 JSON index shows just 432,801).

Speedtest by Ookla Global Fixed and Mobile Network Performance Map Tiles

Request to add the following entries and additional tags related to the Speedtest Global Performance Maps dataset.

Dataset entry

speedtest-global-performance.yaml

Add tags

broadband, global, telecommunications, tiles

Has anyone looked into adding Landsat 5 and 7 to AWS?

Landsat provides the longest running continuous earth observation system in history. Although Landat 8 is great, it only provides data from 2013 and on. Most studies involving long term trends like climate change require decades of satellite imagery.

What would be involved in getting Landsat 5 and 7 on AWS and who would I talk to?

Hi! I have uploaded the digitalcorpora corpus to s3://digitalcorpora/ and it is live!

% aws s3 ls s3://digitalcorpora/corpora/
                           PRE bin/
                           PRE drives/
                           PRE drives_bulk_extractor/
                           PRE drives_dfxml/
                           PRE files/
                           PRE hashes/
                           PRE mobile/
                           PRE packets/
                           PRE ram/
                           PRE scenarios/
                           PRE sql/
2020-11-21 10:56:19         43 README.txt
2020-11-21 10:56:20    1783404 digitalcorpora.org-hashdeep-2020-04-01.csv
2020-11-21 10:56:19    1787101 digitalcorpora.org-hashdeep-2020-05-01.csv
2020-11-21 10:56:19    1794086 digitalcorpora.org-hashdeep-2020-06-01.csv
2020-11-21 10:56:19    1794914 digitalcorpora.org-hashdeep-2020-07-01.csv
2020-11-21 10:56:20    1796103 digitalcorpora.org-hashdeep-2020-08-01.csv
2020-11-21 10:56:20    1796275 digitalcorpora.org-hashdeep-2020-09-01.csv
2020-11-21 10:56:20    1796447 digitalcorpora.org-hashdeep-2020-10-01.csv
2020-11-21 10:56:20    1796619 digitalcorpora.org-hashdeep-2020-11-01.csv
%

I will send a pull request.

Sorry, the issue was sent wrong.

Dear @zflamig,
Could you please help us?
I do not know the reason why the Explorer link (under Resources on AWS section on the right) is empty. The noaa-ncn.yaml has this information:

Resources:

• Description: "NCN Data and Products"
  ARN: arn:aws:s3:::noaa-cors-pds
  Region: us-east-1
  Type: S3 Bucket
  Explore: "Browse NOAA-NCN Bucket"

Thank you very much for your time and help.
Sincerely,
Ira.Sellars

Should fail if region is not of form us-west-2.

We've observed that currently the latest GHE data available in S3 is s3://noaa-ghe-pds/rain_rate/2020/04/20/NPR.GEO.GHE.v1.S202004201230.nc.gz. NOAA currently has https://satepsanone.nesdis.noaa.gov/pub/HydroEst/GHE/NPR.GEO.GHE.v1.S202004201430.nc.gz available.

Is there perhaps an issue with the ingestion to S3?

Thank you.

I'm opening this issue to accompany the forthcoming pull request to add the SOREL-20M dataset to the registry.

Thanks,
Rich

Hi there,
I was wondering if it would be possible to add CORS settings to the genome-in-a-bottle dataset

genome-in-a-bottle/giab_latest_release#10 (comment)

It is very helpful to test our demo datasets like GIAB, but it requires CORS

I think something like this would be ideal, I had talked previously with an AWS admin when settings like this were added to the 1000genomes s3 bucket

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedHeader>Range</AllowedHeader>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <ExposeHeader>Accept-Ranges</ExposeHeader>
        <ExposeHeader>Content-Range</ExposeHeader>
        <ExposeHeader>Content-Encoding</ExposeHeader>
        <ExposeHeader>Content-Length</ExposeHeader>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Hi,
I encountered below errors when fetching data for one of: GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/SCL.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B01.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B02.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B03.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B04.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B05.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B06.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B07.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B08.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B09.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B11.tif), GDALRasterSource(s3://s22s-test-geotiffs/luray_snp/B12.tif):
"
FAILURE(3) CPLE_AWSInvalidCredentials(15) "AWSInvalidCredentials." AWS_SECRET_ACCESS_KEY and AWS_NO_SIGN_REQUEST configuration options not defined, and /root/.aws/credentials not filled

py4j.protocol.Py4JJavaError: An error occurred while calling o142.collectToPython.
"
After my searching on the Internet, I run the command to confirm whether the issue related about "AWS account".

Some programmers suggest to "try setting AWS_NO_SIGN_REQUEST=YES", but I am so confused how to set "AWS_NO_SIGN_REQUEST=YES". So, could you pls give me some suggestions? Thanks!

https://registry.opendata.aws/schweizer-haltestellen-oev/

aws s3 ls s3://data.geo.admin.ch/ch.bav.haltestellen-oev/ --no-sign-request

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Each dataset lists where the data resides e.g. taking NOAA GOES https://registry.opendata.aws/noaa-goes/

It's lists the location of the data e.g. 'noaa-goes16'.

It would be great if that link is hyper-linked to the AWS S3 Explorer so you can easily see what the files look like e.g. hyperlink to https://noaa-goes16.s3.amazonaws.com/index.html

I have done a quick random check (I have not checked all folders) but it appears as all the files related to subset of fields above 700mb are missing for the D>10 days forlders.

In the documentation it says:

For most grib2 files, the data are provided on a grid with a 0.25-degree grid spacing,
archived every 3 hours for the first 10 days of the forecast; beyond 10 days, 0.50 degrees grid
spacing is used. For pressure-level data above 700 hPa, even during the first 10 days of the
forecast, data are saved at 0.5-degree grid spacing in order to conserve space. The grid
proceeds from 90°N to 90°S and from 0E to 359.75 E.

I initially thought that because of the similar lat/lon grid they had one grib file for the whole Day 1-16/35 period, but nope, the file in the Day 1-10 folder has only lead times for 3 to 240 hours.

The file names are

hgt_pres_abv700mb_YYYYMMDDHH_ens.grib2
spfh_pres_abv700mb_YYYYMMDDHH_ens.grib2
tmp_pres_abv700mb_YYYYMMDDHH_ens.grib2
ugrd_pres_abv700mb_YYYYMMDDHH_ens.grib2
vgrd_pres_abv700mb_YYYYMMDDHH_ens.grib2
vvel_pres_abv700mb_YYYYMMDDHH_ens.grib2

where for ens i mean the ensemble members, c00 and p##

The same issue seems to be present for those days with 11 members and forecast times up to Day 35.

Sondehub provides users with a AWS IoT MQTT websocket stream to make use realtime streaming data.

We provide this via a presigned URL (from api gateway / lambda) : https://github.com/projecthorus/sondehub-infra/wiki/API-(Beta)#-get--sondeswebsocket and also via our python SDK.

Would having this as a resource be useful for this project? I'm not exactly sure how it would be expressed in the schema.

When running tests, I see the following locally. It looks like this should be handled at the top of ext.py so not sure why it's still firing. @zflamig could you look into this?

/Library/Python/2.7/site-packages/requests-2.13.0-py2.7.egg/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

Although the data has been restored in the bucket on #545, but "latitude", "longitude" and "population" columns are all empty now. Could you please help to fix it?

Thanks

There are two links to documentation on the Sentinel-2 dataset, and if you click on it, it will attempt to open this URL:

https://roda.sentinel-hub.com/sentinel-s2-l1c/readme.html,https://roda.sentinel-hub.com/sentinel-s2-l2a/readme.html

Obviously, this doesn't resolve. How do we accommodate multiple links to documentation? Could we allow links or Markdown in the Documentation field?

At https://registry.opendata.aws/giab/
and in the giab.yaml file at
https://github.com/awslabs/open-data-registry/tree/master/datasets
the AWS region for bucket=giab is listed as us-east-1.

However, when I try to access the bucket via the AWS Java API or at
https://giab.s3.us-east-1.amazonaws.com/
AWS returns "The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2' " (or similar from the URL).

If I try https://giab.s3.us-west-2.amazonaws.com/
it succeeds.

Can you please investigate and perhaps change the yaml file so the region is listed as us-west-2?

Thank you!

It would be useful to have searchable/indexable data on the characteristics of a dataset:

• total dataset size
• AVG file size
• file type
• etc.

On http://spacenet-dataset.s3.amazonaws.com/ , public s3 bucket is arn:aws:s3:::spacenet-dataset.
However when I try to access the bucket via https
http://spacenet-dataset.s3.amazonaws.com/, it is giving following error

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>80AA03A4A2B74B07</RequestId>
<HostId>
vpeLzorRhAVQAc5WIy7aC9atGuyrgnBw+xdRR8GklvasUI4jgQ+Bd8Yl6Jo9QU5rQnbcMOkwxME=
</HostId>
</Error>

Is this dataset no longer public ?

Create lists under DataAtWork for Publications, Tools & Applications, Tutorials

Thank you for providing the Amazon customer reviews public data set.
https://s3.amazonaws.com/amazon-reviews-pds/readme.html

I was comparing the TSV and Parquet file sizes of the above dataset from S3 Console and found that: TSV size was 32.2 GB and Parquet Size was 47.4 GB

Could you kindly provide insight into why the Parquet size is bigger? I was expecting the same or better due to binary+compressed nature of Parquet.

Dear All,

The latest mentions on the events folder are all from april 16. I thought this is updated daily, anything not working?

Same thing for the SNS.

2019-04-16 13:03:11 808.2 KiB v2/events/20190416130000.export.csv
2019-04-16 13:19:11 749.9 KiB v2/events/20190416131500.export.csv
2019-04-16 13:34:10 848.7 KiB v2/events/20190416133000.export.csv
2019-04-16 13:49:13 806.1 KiB v2/events/20190416134500.export.csv
2019-04-16 14:03:11 893.0 KiB v2/events/20190416140000.export.csv
2019-04-16 14:19:11 697.3 KiB v2/events/20190416141500.export.csv
2019-04-16 14:34:10 775.7 KiB v2/events/20190416143000.export.csv
2019-04-16 14:49:11 720.8 KiB v2/events/20190416144500.export.csv
2019-04-16 15:03:11 909.5 KiB v2/events/20190416150000.export.csv
2019-04-16 15:19:10 700.7 KiB v2/events/20190416151500.export.csv

I got this by running

aws s3 ls s3://gdelt-open-data/v2/events --recursive --human-readable --summarize

from a terminal window

Regards

Can you upload lo given a direct link to download the dataset of IDS2018 set instead of AWS while downloading with AWS it was disconnecting in the middle so it became an issue to download the Dataset.

Files should be '.yaml', so we should warn when we see a '.yml'.

This issue is for the addition of the terrafusion dataset yaml file.

For example, with GOES data, we have two different buckets with different formats. In this case, the documentation is slightly different, but the docs link is currently a top level dataset object.

I am trying to download SpaceNet Buildings Dataset v1

I ran the suggested command in a Sagemaker notebook, and I got this error.

Command:

$ aws s3 cp s3://spacenet-dataset/SpaceNet_Roads_Competition/AOI_2_Vegas_Roads_Train.tar.gz 

Error:
fatal error: An error occurred (404) when calling the HeadObject operation: Key "SpaceNet_Roads_Competition/AOI_2_Vegas_Roads_Train.tar.gz" does not exist

I tried this with Paris data as well, and I got the same error. I found this repo, but it has achieved by the owner, so I put there any issue.

How can I download this dataset properly?

I'm unable to access the license / citation information in the GIAB registry, per the AWS giab registry:

There are no restrictions on the use of this data. More information on citation is available here.

The here is https://sites.stanford.edu/abms/content/giab-reference-materials-and-data which is blocked by a sign in page. I know it was a while ago @jflasher, but do you have any information about this?

hi, is there a way to get a list of catalog IDs for the imagery released for a Digital Globe Open Data event? (Cyclone Idai)

There are over 100 imagery strips listed for this event on https://www.digitalglobe.com/ecosystem/open-data/cyclone_idai and I would like to save some time by not parsing them.

getting some basic metadata with the ids as well would be a plus (date, sensor, cloud cover, etc), but I think catalog IDs are a good start.

Actually, it would be a plus plus if we could get the imagery extents as well (geojson or shapefile).