Git Product home page Git Product logo

Comments (14)

konradkalemba avatar konradkalemba commented on June 17, 2024 2

@mhajder There is also an option to stay with manually updated data, but we would have to create a team responsible for data updating, because right now I'm not available all the time. With more people, delays in data update would be lower.

@mulawamichal I see - that's good, we wouldn't have to deal with showing "powiat" on map.

In both approaches there are some trade-offs:

  • In manual data update with more people someone would have to be online, which isn't the problem with big team. But the problem is to assemble such team.

  • In automatic scraping we would lose source of individual cases and of course there is some time to implement such scraper.

from korona.ws.

mhajder avatar mhajder commented on June 17, 2024 1

For me, the idea of scraping official data is a good one. But is it sense to scrape them to create the same map? I would stay with the official account of the MZ twitter.

I like the data with cities more. They are more readable.

You would also need to prepare the same map as it is on gov.pl, or use coordinate location by place name.

JSON data from gov.pl 
[
   {
      "Województwo":"Cała Polska",
      "Powiat/Miasto":"Łącznie",
      "Liczba":"156",
      "Liczba zgonów":"3",
      "Id":"t0000"
   },
   {
      "Województwo":"łódzkie",
      "Powiat/Miasto":"bełchatowski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1001"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"będziński",
      "Liczba":"2",
      "Liczba zgonów":"",
      "Id":"t2401"
   },
   {
      "Województwo":"lubelskie",
      "Powiat/Miasto":"biłgorajski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0602"
   },
   {
      "Województwo":"dolnośląskie",
      "Powiat/Miasto":"bolesławiecki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0201"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"Chorzów",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t2463"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"cieszyński",
      "Liczba":"4",
      "Liczba zgonów":"",
      "Id":"t2403"
   },
   {
      "Województwo":"pomorskie",
      "Powiat/Miasto":"Gdańsk",
      "Liczba":"2",
      "Liczba zgonów":"",
      "Id":"t2261"
   },
   {
      "Województwo":"lubuskie",
      "Powiat/Miasto":"Gorzów Wielkopolski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0861"
   },
   {
      "Województwo":"lubelskie",
      "Powiat/Miasto":"janowski",
      "Liczba":"5",
      "Liczba zgonów":"",
      "Id":"t0605"
   },
   {
      "Województwo":"opolskie",
      "Powiat/Miasto":"kędzierzyńsko-kozielski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1603"
   },
   {
      "Województwo":"świętokrzyskie",
      "Powiat/Miasto":"Kielce",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t2661"
   },
   {
      "Województwo":"małopolskie",
      "Powiat/Miasto":"Kraków",
      "Liczba":"2",
      "Liczba zgonów":"",
      "Id":"t1261"
   },
   {
      "Województwo":"opolskie",
      "Powiat/Miasto":"krapkowicki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1605"
   },
   {
      "Województwo":"dolnośląskie",
      "Powiat/Miasto":"Legnica",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0262"
   },
   {
      "Województwo":"dolnośląskie",
      "Powiat/Miasto":"legnicki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0209"
   },
   {
      "Województwo":"podkarpackie",
      "Powiat/Miasto":"leżajski",
      "Liczba":"7",
      "Liczba zgonów":"",
      "Id":"t1808"
   },
   {
      "Województwo":"lubelskie",
      "Powiat/Miasto":"lubartowski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0608"
   },
   {
      "Województwo":"lubelskie",
      "Powiat/Miasto":"Lublin",
      "Liczba":"9",
      "Liczba zgonów":"1",
      "Id":"t0663"
   },
   {
      "Województwo":"łódzkie",
      "Powiat/Miasto":"łódzki wschodni",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1006"
   },
   {
      "Województwo":"łódzkie",
      "Powiat/Miasto":"Łódź",
      "Liczba":"14",
      "Liczba zgonów":"",
      "Id":"t1061"
   },
   {
      "Województwo":"dolnośląskie",
      "Powiat/Miasto":"oleśnicki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0214"
   },
   {
      "Województwo":"warmińsko-mazurskie",
      "Powiat/Miasto":"Olsztyn",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t2862"
   },
   {
      "Województwo":"świętokrzyskie",
      "Powiat/Miasto":"ostrowiecki",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t2607"
   },
   {
      "Województwo":"warmińsko-mazurskie",
      "Powiat/Miasto":"ostródzki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t2815"
   },
   {
      "Województwo":"łódzkie",
      "Powiat/Miasto":"pabianicki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1008"
   },
   {
      "Województwo":"mazowieckie",
      "Powiat/Miasto":"piaseczyński",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t1418"
   },
   {
      "Województwo":"łódzkie",
      "Powiat/Miasto":"poddębicki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1011"
   },
   {
      "Województwo":"zachodniopomorskie",
      "Powiat/Miasto":"policki",
      "Liczba":"2",
      "Liczba zgonów":"",
      "Id":"t3211"
   },
   {
      "Województwo":"wielkopolskie",
      "Powiat/Miasto":"Poznań",
      "Liczba":"7",
      "Liczba zgonów":"1",
      "Id":"t3064"
   },
   {
      "Województwo":"wielkopolskie",
      "Powiat/Miasto":"poznański",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t3021"
   },
   {
      "Województwo":"mazowieckie",
      "Powiat/Miasto":"pruszkowski",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t1421"
   },
   {
      "Województwo":"mazowieckie",
      "Powiat/Miasto":"Radom",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1463"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"Rybnik",
      "Liczba":"5",
      "Liczba zgonów":"",
      "Id":"t2473"
   },
   {
      "Województwo":"podkarpackie",
      "Powiat/Miasto":"Rzeszów",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1863"
   },
   {
      "Województwo":"lubuskie",
      "Powiat/Miasto":"słubicki",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0805"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"Sosnowiec",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t2475"
   },
   {
      "Województwo":"zachodniopomorskie",
      "Powiat/Miasto":"stargardzki",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t3214"
   },
   {
      "Województwo":"opolskie",
      "Powiat/Miasto":"strzelecki",
      "Liczba":"4",
      "Liczba zgonów":"",
      "Id":"t1611"
   },
   {
      "Województwo":"zachodniopomorskie",
      "Powiat/Miasto":"Szczecin",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t3262"
   },
   {
      "Województwo":"warmińsko-mazurskie",
      "Powiat/Miasto":"szczycieński",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t2817"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"tarnogórski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t2413"
   },
   {
      "Województwo":"lubelskie",
      "Powiat/Miasto":"tomaszowski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0618"
   },
   {
      "Województwo":"łódzkie",
      "Powiat/Miasto":"tomaszowski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t1016"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"Tychy",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t2477"
   },
   {
      "Województwo":"dolnośląskie",
      "Powiat/Miasto":"wałbrzyski",
      "Liczba":"1",
      "Liczba zgonów":"",
      "Id":"t0221"
   },
   {
      "Województwo":"mazowieckie",
      "Powiat/Miasto":"Warszawa",
      "Liczba":"24",
      "Liczba zgonów":"",
      "Id":"t1465"
   },
   {
      "Województwo":"dolnośląskie",
      "Powiat/Miasto":"Wrocław",
      "Liczba":"18",
      "Liczba zgonów":"1",
      "Id":"t0264"
   },
   {
      "Województwo":"śląskie",
      "Powiat/Miasto":"zawierciański",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t2416"
   },
   {
      "Województwo":"łódzkie",
      "Powiat/Miasto":"zgierski",
      "Liczba":"3",
      "Liczba zgonów":"",
      "Id":"t1020"
   }
]

from korona.ws.

konradkalemba avatar konradkalemba commented on June 17, 2024 1

Okay, for the time being we are staying with the data updated manually. Official MZ's website was outdated for at least 1 hour after the latest confirmation.

from korona.ws.

mhajder avatar mhajder commented on June 17, 2024 1

Okay, for the time being we are staying with the data updated manually. Official MZ's website was outdated for at least 1 hour after the latest confirmation.

Probably the change are only during working hour 😆

from korona.ws.

konradkalemba avatar konradkalemba commented on June 17, 2024 1

There is an another problem - MZ's Twitter doesn't specify cities where new cases are, only voivodeships now...

from korona.ws.

mhajder avatar mhajder commented on June 17, 2024 1

Also correct the case that the patient recovered, as it is written on Twitter this is the first patient. So it is from Zielona Góra.

from korona.ws.

mulawamichal avatar mulawamichal commented on June 17, 2024

there are cities specified in some cases, in others there are "powiaty".

sometimes powiat is named same as city. theese are "miasta na parawach powiatu". see https://pl.wikipedia.org/wiki/Lista_powiat%C3%B3w_w_Polsce or TERYT database (http://eteryt.stat.gov.pl/eTeryt/rejestr_teryt/udostepnianie_danych/baza_teryt/uzytkownicy_indywidualni/wyszukiwanie/wyszukiwanie.aspx?contrast=default)

from korona.ws.

mhajder avatar mhajder commented on June 17, 2024

@konradkalemba I can help with adding data.

Implementing such a scraper is very simple. It's best to do a simple python script that will be run in cron. Also, page scraping is not very ethical. It generates a lot of traffic.

from korona.ws.

konradkalemba avatar konradkalemba commented on June 17, 2024

@mhajder I know that isn't the best way ethical-wise, but it wouldn't generate a lot of traffic though. Running the script every 5 minutes wouldn't hurt the server very much.

An another problem with this data source is that I'm not sure if it's updated regularly

from korona.ws.

mulawamichal avatar mulawamichal commented on June 17, 2024

geoportal now has "koronawirus" layer: https://mapy.geoportal.gov.pl/imap/Imgp_2.html?locale=en&gui=new&sessionID=4955220

from korona.ws.

mhajder avatar mhajder commented on June 17, 2024

geoportal now has "koronawirus" layer: https://mapy.geoportal.gov.pl/imap/Imgp_2.html?locale=en&gui=new&sessionID=4955220

But it is extremely difficult to scrape it.

For gov.pl, all you need is:

import json

import requests
from bs4 import BeautifulSoup

response = requests.get('https://www.gov.pl/web/koronawirus/wykaz-zarazen-koronawirusem-sars-cov-2')

soup = BeautifulSoup(response.text, 'html.parser')

json_data = json.loads((soup.find(id='registerData')).text)
print(json_data['parsedData'])

from korona.ws.

konradkalemba avatar konradkalemba commented on June 17, 2024

@mulawamichal @mhajder Guys, we have a big problem - their official twitter account lists only voivodeships now as I wrote above, but I thought maybe their website will be more precise... but they limited both map and table to voivodeships.

from korona.ws.

mhajder avatar mhajder commented on June 17, 2024

@konradkalemba Gov.pl also now only provides voivodships.

from korona.ws.

konradkalemba avatar konradkalemba commented on June 17, 2024

@mhajder Yes... I think we also have to do so. Because trying to find where every new case is will be very time-consuming.

from korona.ws.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.