sociepy / covid19-vaccination-subnational Goto Github PK

counts in Slovakia are not cumsum

• Move API folder within data folder
• Add folder with templates to generateu updated docs, e.g. _templates.

Working on a simple minimal dashboard. Plotly based.

Hi @lucasrodes, the data for India is not updating after 20/4/2021.
However the data at the source is til 22/4/2021 -> state_timeline.csv

Implement some tests to verify the reliability of data.

Some ideas:

• Check that fields total_vaccinations, people_vaccinatedand people_fully_vaccinated are monotonically increasing.

I have observed from comparing India.csv and state_timeline.csv that the script is leaving the date with zero vaccinations (I've checked and its seems it is the case for all other countries also).
For example : On 20th January 202, the union territory of AN in India had zero vaccination does administered so that date is not present in India.csv.

I'm relatively new in this stuff, so please don't mind if I'm wrong here. Will not this create issues when using the API to directly plot any visualizations or using the data for analysis directly ?

source: https://es.wikipedia.org/wiki/Departamentos_del_Per%C3%BA

Map department ids (https://es.wikipedia.org/wiki/Departamentos_del_Per%C3%BA) to ISO-3166-2 codes.

Official data source: https://emvolio.gov.gr/vaccinationtracker
Script example: https://github.com/balikasg/covid19-vaccine-greece/blob/main/export_csv.py (another project)

Denmark provides data in PDF format. To read it, we use tabula-py. It appears that depending on the day, tables move around the document.

OWID uses same source, follow their approach

Currently the country-wise latest and all API have the following structure :

{
    "country": "India",
    "country_iso": "IN",
    "last_update": "2021-02-10",
    "source_url": "https://india-covid19vaccine.github.io",
    "data": [
    ]
}

Two more fields can be added : total_vaccinations and population as such:

• total_vaccination :

  • One can loop over the data for all the regions of a country and get cumulative, but I think it will be better if it can be provided pre-calculated.
  • Another reason for this is some countries (I'm only aware of India in this case currently, but it is fairly possible that it maybe the case somewhere else also), are adding some vaccinations under the heading of Miscellaneous, so this can't be accounted to any region and should be reflected in the national total.

• population :
  will be helpful in normalizing data as per capita. (I'm not sure about what source should be used here, maybe https://www.worldometers.info/world-population/)

The updates structure as :

{
    "country": "India",
    "country_iso": "IN",
    "total_vaccinations":7017114,
    "population":1371360350,
    "last_update": "2021-02-10",
    "source_url": "https://india-covid19vaccine.github.io",
    "data": [
    ]
}

source: https://monitor.uruguaysevacuna.gub.uy/

JSON: https://msp.gxportal.net/data-vaccine.json

Data from 2021-01-25 to 2021-01-26 goes up and down

Add data from russia.

source: gogov.ru

Documentation should be generated separately from data updates.

Colombia gov publishes data in an image.

See proposal in owid/covid-19-data#633

Number of vaccines in chile not monotonically increasing. Check!

Same dates are occurring twice for states such as IN-BR, IN-GJ and IN-KA in API for India. (not sure about other countries)

Example

{
            "region_iso": "IN-KA",
            "region_name": "Karnataka",
            "data": [
                {
                    "date": "2021-01-16",
                    "total_vaccinations": 13594,
                    "total_vaccinations_per_100": 0.02
                },
                {
                    "date": "2021-01-16",
                    "total_vaccinations": 13594,
                    "total_vaccinations_per_100": 0.02
                },
                {
                    "date": "2021-01-17",
                    "total_vaccinations": 29504,
                    "total_vaccinations_per_100": 0.05
                },
                {
                    "date": "2021-01-17",
                    "total_vaccinations": 29504,
                    "total_vaccinations_per_100": 0.05
                },
                {
                    "date": "2021-01-18",
                    "total_vaccinations": 66392,
                    "total_vaccinations_per_100": 0.11
                },
                {
                    "date": "2021-01-18",
                    "total_vaccinations": 66392,
                    "total_vaccinations_per_100": 0.11
                }
            ]
}

Add table in API website, to country API file links (for all and latest)

Some countries started second dose administration. This implies adding new columns people_vaccinated and people_fully_vaccinated

Current 2-dose reportings

• Argentina
• Austria
• Belgium
• Brazil
• Bulgaria
• Canada
• Chile
• Czechia
• Denmark
• France
• Germany
• Italy
• Liechtenstein
• Norway
• Poland
• Slovakia
• Spain
• Sweden
• Switzerland
• United Kingdom
• United States

This issue is to keep track of those regions that are not present in our data, either in the CSV or API files.

How to report?

• Make sure that you are using the latest version of the repo.
• Share the country and the region missing.
• Specify if it is missing in country csv files, vaccinations csv file or in the API.

This was initially reported by a user, in particular, the regions were:

• Puerto Rico (https://github.com/youyanggu/covid19-cdc-vaccination-data/blob/main/aggregated.csv)
• Few Indian states, including Chhattisgarh, Odisha, Telangana, Uttarakhand, and Ladakh (https://github.com/india-covid19vaccine/india-covid19vaccine.github.io/blob/main/csv/state.csv)

which were missing across all CSV and API files

Check dates in data. Check if they are referring to same date (e.g. execution date or report date)

In an attempt to simplify the logic and remove redundancy, now scraping logic is located in library
module.

Current status:

• Austria
• Belgium
• Brazil
• Canada
• Chile
• Czechia
• Germany
• India
• Italy
• Slovakia
• Spain
• Sweden
• Switzerland
• Liechtenstein
• United Kingdom
• United States
• Argentina
• Bulgaria
• Denmark
• France
• Norway
• Poland

No updates for Poland for ~7 days. May want to review data source

Data for Australia is available, as reported in owid/covid-19-data#230 (comment) by @ulankford

Hi - Taking Canada as an example in the shot below: the timeseries is not continuously increasing as one would expect from a cumulative timeseries. It does not correspond neither to a daily (or "new") number as the latest figures are pretty close to the total vaccinations (just below 1 million people).

I observed this for a lot of countries: USA, Italy, Czechia, ...

source: http://ncv.kdca.go.kr/

Sweden can be easily added, regional data is available here.

OWID uses this source, their script with the corresponding attribution could be used.

Denmark provides data in PDF format. To read it, we use tabula-py. It appears that depending on the day, tables move around the document.

OWID uses same source, follow their approach

xref youyanggu/covid19-cdc-vaccination-data#3

Seems like there might be an issue in its obtention/computation. Should be monotonically increasing (as it depicts cumulative sum). However, some cases for Italy and Spain (there might be others) do not follow this.

Long time no update, check with source

The website for Norway has slightly changed, now including 2 dose information. The scraping script now fails to execute.

May want to use underlying API instead of selenium:

• 1 dose
• 2 dose

No updates for some days, check data source

no updates, Check data source

The initial source is discontinued and current data is found here: https://www.data.gouv.fr/fr/datasets/donnees-relatives-aux-personnes-vaccinees-contre-la-covid-19-1/#_

Specifically:

• 1st dose: https://www.data.gouv.fr/fr/datasets/r/735b0df8-51b4-4dd2-8a2d-8e46d77d60d8
• 2nd dose: https://www.data.gouv.fr/fr/datasets/r/96db2c1a-8c0c-413c-9a07-f6f62f3d4daf

Watch out for the column names: they are the same, but the descriptions of the files mention 1st and 2nd.

Current sourcehas been discontinued otherwise.

Excellent

Shouldnt we try to use same as instead of just text strings

See how Wikidata can help table Table_of_administrative_divisions_by_country

• Introduction to Wikidata

I think it will be much easier if we have not just text strings but same as Wikidata

We changed the country reporting to using ISO 3 lettercode

Turkey now provides information at the regional level. Second dose information available.

link: https://covid19asi.saglik.gov.tr/

I am building out a GraphQL Server with StepZen and the query https://sociepy.org/covid19-vaccination-subnational/data/api/v1/latest/country_by_iso/FR.json breaks due to a NaN value rather than null value in total_vaccinations_per_100.

// Response from RestAPI

        {
            "region_name": "Corse",
            "region_iso": "FR-COR",
            "date": "2021-02-22",
            "total_vaccinations": 27059,
            "total_vaccinations_per_100": NaN
        },

GraphQL Query

  vaccines(countryISO: "FR") {
  country
  }

Response Error

{
  "data": {
    "vaccines": null
  },
  "errors": [
    {
      "message": "Invalid json content.: invalid character 'N' looking for beginning of value",
      "locations": [
        {
          "line": 58,
          "column": 3
        }
      ],
      "path": [
        "vaccines"
      ]
    }
  ]
}

Seems it might be soon available in current source.

xref rsalzer/COVID_19_VACC#6

Create a simple API, exposing the data as JSON files. Could take inspiration from https://github.com/india-covid19vaccine/.

• tutorial how to add scraper: https://github.com/sociepy/covid19-vaccination-subnational/blob/main/docs/add-new-scraper.ipynb
• data source: https://piikki-api.lab.juiciness.io/administrations/

xref MergHQ/piikki#2

No update, check data source

Vaccination data for Lebanon available: https://impact.cib.gov.lb/home/dashboard/vaccine

At kadaa level, might need to map to regions

I have made the changes as mentioned here. You can make the changes to the project w.r.t this.

I have made both state_timeline.json and state_timeline.csv as per the new structure, you can use whatever best suits.

In the case of Uruguay, you can take the information from the following repository:
https://github.com/3dgiordano/covid-19-uy-vacc-data/blob/main/data/Uruguay.csv

The same repository that is used in OWID for its data.
I added columns to reflect the sub-national data in the last few weeks.

If you want to migrate, the actual column format is total_[sub_id] where [sub_id] is the identification of the ISO format region in lowercase.
Example: total_ar is total_vaccinations of UY-AR

In the next week I hope to resolve the data for people_vaccinated and people_fully_vaccinated.
I will notify you when it is available.

Source data for Brazil in use now provides data related to the second dose vaccination

no updates, Check data source

Now source excel file for Belgium includes 2nd dose.

TODO: change incremental to batch update.

Review which vaccines are being administered in each region and take the following actions:

• Track these in file data/country_info.csv
• Pay special attention to countries using 1-dose vaccines, as computation of some fields might be different than with 2-dose vaccines.

Add a github-action release job:

• https://github.com/actions/create-release.
• Run every day, use date as tag.
• Create DOI with zenodo
• Add badges