lhaze / data-science-playground Goto Github PK

Prepare an architecture & a skeleton code of models:

• load/save some instances of models to DB
• load some instances of models from configuration files (YAML)
• prepare test process (doctests/pytest)
• instance register for the model instances loaded from the configuration
• supply some validation for models
• prepare an entry point for library (all models loaded, main code loaded)

Prepare & code Event structure. Some of model instances can be concerned as an Event (Game, Season, Battle, aso).

Expectations:

• Event/EventTree models
• Event.resolve
• EventResolver
• registration of events
• dispatching an event

Example Event structure:

Game #foo
↳ … setup
↳ Season #Spring
    ↳ … political phase
    ↳ war phase
        ↳ start of the war phase
        ↳ battle #1 Osaka
             ↳ all players involved bidding blindly
                 ↳ waiting for Dragonfly bid

• population of alive people

• natural growth (mind that it is the natural growth on the other side of the Shroud)

• population of wraiths per DoD

• population of wraiths en masse

• population of shades

• migration of wraiths

• alive population pyramid per year

• population pyramid per year

• Handlers' module can be considered an entry point to the application.
• Let's assume that there exists some kind of request (GUI, HTTP, game socket, agent command or from any other source).
• The process of processing the request should start from extracting a command from the request and lead to building a response, based on the result of the processing, describing a state of the game (it may be a new state, but not necessarily).
• If the command is a read command, the state described should be the current state.
• If the command is some kind of submit, the state is gonna be some new state of the game (returned in full details or some kind of diff of states).

Task that translates medi PDF with PDFX service.
Defined by #4

• Timeline can provide regular & ephemeral timepoints of the property
• Interpolations can be build basing on regular timepoints
• Interpolations can build dictionaries of ephemeral values
• Interpolations can be build regarding both regular & ephemeral timepoints

• use pyDatalog to use programming in logic in models
• narrow down pyDatalogMixin feature of "every attribute is a term" strategy
• example of programming in logic

Proposing processing pipeline (based on masking medical data processing at Roche):

1. Finders
   • regex: some transformation triggers are trivial and context-independent,
   • NLP classifiers: some rules ARE language- and context-dependent, so much of the total number of finders should be classifiers,
   • DNN classifiers: we may try an approach with deep neural network classification,
   • rule-based finders: some transformation triggers are obvious after gathering some experience.
2. Filters: some candidates may be false-positives; we may not be able or don't want to change/retrain finders (especially external NLP classifiers) to gain more confidence; some false-positive reduction rules may be arbitrarily declared by the domain expert.
3. Merging: some triggers may overlay each other; ask the domain expert/explore/arbitrarily decide about transformation prioritization.
4. Transform - execute non-conflicting transformations.

• survivability rates (as a function of time)
• populations of alive people...
• ...vs wraiths
• ... vs shades (as a function of time)
• population pyramids at representative years

Part 1:

• alive population
• mortality & additional deaths
• enfant & senior oblivion rates
• pass away rate (decorpsing, leaving Shadowlands permanently & ascension)
• imigration rate

Part 2:

• alive population pyramid for representative years

• utils.functools.extended_to_sequence_of_inputs -> np.vectorize
• a bunch of np.arrays -> pandas.DataFrame

How many Restless are there in the Necropolis of Warsaw?

Actually, there are two questions:

AoD) What is the age of death of how many wraiths across the population of Necropolis? AoD alters demeanor & nature of wraiths' psyche & shadow.

DoD) What is the date of death of how many wraiths? DoD defines culture context of a character and certain aspects of his/her social position.

Both questions share the same demographic model of getting to the Shadowlands, migrations & Ascension/Oblivion/decorpsing and differs in entry data.

Sources:

1. Demography of Warsaw (Wikipedia)
2. Historical demography of Poland (academic pdf)
3. Births & mortality (educational materials pdf)

Explore list of processing challenges:

• remove split words
• gather language blocks: English and Polish language blocks
• remove page headers & footers
• extract figure images
• JATS tagging

Let's do a PoC with Spark.