code4romania / catalog-data-chambers Goto Github PK

We need to figure out how they have moved from party to party across the years, we should scrape their belonging to a parliamentary group.

The endpoint used to scrape data about MEPs is not up anymore and the information is not being retrieved.
The scraper for members of european parliament needs to be rewritten, the information should be scraped from this endpoint: http://www.europarl.europa.eu/meps/en/full-list/xml/a

To discuss data structure and how it fits in the current architecture.

We currently have this data generated by some older Java code, we'd like to have it as part of our scrape queues. Will provide sample JSON for data structure as we need this to be consistent with what is being implemented in the core DB.

Master scraper will be a fork of CZL canonical scraper, see this folder.

Initial data scrapers tags will be:

1. laws - law inititiatives (@Owlree this will be your start-up scraper)
2. voting - voting activity across the chambers (follow-up with @costibleotu when master scraper is ready)
3. activity - toate datele (we will need to rewrite this as the JSON file we initially got is written in Java, I need to chase for the code if we'll need it, it's not in our repos)
4. more to be added as we move along

All data scrapers must work together to gather data when updates are needed. We discussed using Celery early on so that we can easily scale up and add/remove scrapers as we move along.

All individuals must be taken into account. Please provide a relational structured output.

{
    individual: "some name or id", // check technical document for specifications
    initiatives: [ 5234, 12345, 1234, 12365443]
}

{
    initiativeId: 5234,
    title: "Propunere legislativă pentru abrogarea Ordonanţei de urgenţă nr.13/2017 pentru modificarea şi completarea Legii nr.286/2009 privind Codul penal şi a Legii nr.135/2010 privind Codul de procedură penală"
}

Will provide credentials to DB sometime this week.

Tocmai am setat si rulat proiectul pe windows. dacă sunt info de interes, le pot adăuga.

Besides the number of times the character has spoken in the plenary, we want to see how much time he has spoken.