vizabi / reader-ddfcsv Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 3.0 114.06 MB

License: GNU General Public License v3.0

JavaScript 3.33% TypeScript 96.67%

reader-ddfcsv's Introduction

What is Vizabi?

Gapminder Vizabi is a framework for developing visual exploration tools.

The main features are:

Embeddable tools with customization
Responsive layout
Support for UI Controls & Interaction
Translation & Localization
Unified data-schema for multi dimensional statistics.

You can use Vizabi to create a new visualization based on your data or modify our set of visualization tools.

See examples in action: Vizabi Examples, and DOCS here: Vizabi Docs

Setup & Quickstart

See contributing guide

License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

reader-ddfcsv's People

Contributors

Stargazers

Watchers

Forkers

antonolkhovskyi korel-san buchslava

reader-ddfcsv's Issues

avoid unneeded translation requests

Should ónly try reading translation files of languages in translations in datapackage.json. language is not necessary. now it tries reading files in /lang/en/ if en is language.

implement custom aggregation functions on ddf schema queries

here is an example:
{
select: {
key: ["key", "value"],
value: ["min(value)", "max(value)"]
},
from: "datapoints.schema"
}

Important: general schema queries should be implemented too!

allow to use datapackage without translation section

incorrect conditions processing

dataset: https://github.com/open-numbers/ddf--gapminder--systema_globalis
query:

{
        select: {
          key: ['geo', 'time'],
          value: [
            'life_expectancy_years', 'population_total'
          ]
        },
        from: 'datapoints',
        where: {
          $and: [
            {geo: '$geo'},
            {time: '$time'},
            {
              $or: [
                {population_total: {$gt: 10000}},
                {life_expectancy_years: {$gt: 30, $lt: 70}}
              ]
            }
          ]
        },
        join: {
          $geo: {
            key: 'geo',
            where: {
              $and: [
                {'is--country': true},
                {latitude: {$lte: 0}}
              ]
            }
          },
          $time: {
            key: 'time',
            where: {$and: [{time: {$gt: '1990', $lte: '2015'}}]}
          }
        },
        order_by: ['time', 'geo']
      }

current result: countries akr_a_dhe and 'ant' were selected

expected result: countries akr_a_dhe and 'ant' should not be selected in accordance with {latitude: {$lte: 0}}

reader can't consume files with many indicators in different columns

the attached file contains two datasets
data.zip

the below code reads the data. in case of dataset ddf--bubbles-2 it's fine
in case of dataset ddf--bubbles-3 reader returns datapoints for only one column at a time

if need to resolve dependencies have a look here
https://github.com/Gapminder/csv-conf-2017-talk/tree/master/lib

see screenshots below

<!doctype html>
<html>

<head>
  <link rel="stylesheet" href="lib/vizabi/vizabi.css">
  <link rel="stylesheet" href="lib/vizabi/bubblechart.css">


  <script src="lib/d3.min.js"></script>
  <script src="lib/vizabi/vizabi.js"></script>
  <script src="lib/vizabi/vizabi-ddfcsv-reader.js"></script>
  <script src="lib/vizabi/bubblechart.js"></script>

</head>

<body>
  <div id="placeholder" style="width:100%; height:800px;"></div>

  <script>
    var ddfReader = new DDFCsvReader.getDDFCsvReaderObject()
    Vizabi.Reader.extend("ddf", ddfReader);

    ConfigBubbleChart = {};
    ConfigBubbleChart.locale = {
      "id": "en",
      "filePath": "assets/translation/"
    };
    ConfigBubbleChart.data = {
      "reader": "ddf",
      "path": "data/ddf--bubbles-2"
    };

    var viz = Vizabi("BubbleChart", document.getElementById("placeholder"), ConfigBubbleChart);

  </script>

</body>

</html>

Entity values null in response

reader version: 0.13.0
dataset: https://github.com/Gapminder/ddf--pcbs--census/tree/features/flattened
query:

{
  "language": "en",
  "from": "datapoints",
  "animatable": "year",
  "select": {
    "key": [
      "governorate",
      "year"
    ],
    "value": [
      "economic_establishments",
      "unemployment_male_total",
      "illiteracy_both_sexes_55_59"
    ]
  },
  "where": {},
  "join": {},
  "order_by": [
    "year"
  ]
}

response:

0:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 99120, unemployment_male_total: 94243, illiteracy_both_sexes_55_59: 20883}
1:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 69400, unemployment_male_total: 49578, illiteracy_both_sexes_55_59: 13636}
2:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 29720, unemployment_male_total: 44665, illiteracy_both_sexes_55_59: 7247}
3:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 9634, unemployment_male_total: 7816, illiteracy_both_sexes_55_59: 1966}
4:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 858, unemployment_male_total: 1093, illiteracy_both_sexes_55_59: 418}
5:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 6269, unemployment_male_total: 4291, illiteracy_both_sexes_55_59: 1262}
6:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 2900, unemployment_male_total: 2519, illiteracy_both_sexes_55_59: 583}
7:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 1442, unemployment_male_total: 1527, illiteracy_both_sexes_55_59: 452}
8:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 13934, unemployment_male_total: 7777, illiteracy_both_sexes_55_59: 1926}
9:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 9091, unemployment_male_total: 5671, illiteracy_both_sexes_55_59: 1745}
10:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 3609, unemployment_male_total: 3411, illiteracy_both_sexes_55_59: 791}
11:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 1069, unemployment_male_total: 576, illiteracy_both_sexes_55_59: 306}
12:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 5700, unemployment_male_total: 4160, illiteracy_both_sexes_55_59: 990}
13:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 14894, unemployment_male_total: 10737, illiteracy_both_sexes_55_59: 3197}
14:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 4124, unemployment_male_total: 6349, illiteracy_both_sexes_55_59: 1230}
15:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 12500, unemployment_male_total: 13045, illiteracy_both_sexes_55_59: 2248}
16:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 3725, unemployment_male_total: 7347, illiteracy_both_sexes_55_59: 1255}
17:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 5883, unemployment_male_total: 11239, illiteracy_both_sexes_55_59: 1474}
18:{governorate: undefined, year: Wed Jan 01 1997 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 3488, unemployment_male_total: 6685, illiteracy_both_sexes_55_59: 1040}
19:{governorate: undefined, year: Mon Jan 01 2007 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 132874, unemployment_male_total: 157634, illiteracy_both_sexes_55_59: 10827}
20:{governorate: undefined, year: Mon Jan 01 2007 01:00:00 GMT+0100 (W. Europe Standard Time), economic_establishments: 94205, unemployment_male_total: 59769, illiteracy_both_sexes_55_59: 7338}
...

expected response:
governorate contains governorate entity id's instead of undefined (as in https://github.com/Gapminder/ddf--pcbs--census/blob/features/flattened/ddf--entities--geo--governorate.csv)

fs in ./~/vizabi-ddfcsv-reader/dist/vizabi-ddfcsv-reader-node.js

And

https://www.npmjs.com/package/fs

This package name is not currently in use.

how to fix it?

incorrect result when query contains string based indicator

fix entity query based on doman

Add test: Incorrect result on entities query (possibly special tautology case)

Dataset: https://github.com/Gapminder/ddf--pcbs--census/tree/master
Query:

{
  "language": "en",
  "from": "entities",
  "animatable": "year",
  "select": {
    "key": [
      "region"
    ],
    "value": [
      "name"
    ]
  },
  "where": {},
  "join": {},
  "order_by": [
    "rank"
  ]
}

Expected response:

0:{region: "1", name: "West Bank"}
1:{region: "2", name: "Gaza Strip"}

Actual Response:

0:{region: "pse", name: "Palestine"}
1:{region: "1", name: "West Bank"}
2:{region: "2", name: "Gaza Strip"}
3:{region: "1", name: "Jenin"}
4:{region: "1", name: "Tubas"}
5:{region: "1", name: "Tulkarm"}
6:{region: "1", name: "Qalqilya"}
7:{region: "1", name: "Salfit"}
8:{region: "1", name: "Nablus"}
9:{region: "1", name: "Ramallah & Al-Bireh"}
10:{region: "1", name: "Jerusalem"}
11:{region: "1", name: "Jericho"}
12:{region: "1", name: "Bethlehem"}
13:{region: "1", name: "Hebron"}
14:{region: "2", name: "North Gaza"}
15:{region: "2", name: "Gaza"}
16:{region: "2", name: "Deir Al - Balah"}
17:{region: "2", name: "Khanyunis"}
18:{region: "2", name: "Rafah"}
19:{region: "isr", name: "israel"}
20:{region: "not_stated", name: "Not Stated"}
21:{region: "other_countries", name: "Other Countries"}
22:{region: "1948_areas", name: "1948 areas"}

Possible reason:
Seeing region column in ddf--entities--geo.csv as key column containing entity id's, which it's not (as stated in datapackage.json resources table)

Adapt reader's functionality to new ddfShema

Adapt reader's functionality to new ddfShema. It should be refactoring and code simplifying.

unexpected result for country based request

request is:

{
select: {
key: ['country', 'time'],
value: ['population_total', 'life_expectancy_years']
},
from: 'datapoints',
where: {
$and: [
{time: '$time'}
]
},
join: {
$time: {
key: 'time',
where: {
time: {
$gte: '1993',
$lte: '2015'
}
}
}
},
order_by: ['time'],
language: 'en',
}

add test for assets

translation behavior should be fixed

web version does not work under ie11

es5 build should be provided

improve build system

avoid webpack processing for backend version

Filtering on property of entity property doesn't work (but does in ws-reader)

So i changed vizabi config to get "un_state" instead of "countries"
while un_state is a property on countries

WS-reader and WS were able to handle it perfectly
ddfcsv reader couldn't

this query when being run against systema globalis

 {
  "language": "en",
  "from": "entities",
  "animatable": "time",
  "select": {
    "key": [
      "geo"
    ],
    "value": [
      "name",
      "world_4region"
    ]
  },
  "where": {
    "$and": [
      {
        "un_state": true
      }
    ]
  },
  "join": {},
  "order_by": [
    "rank"
  ]
}

should return countries from this file, where un_state = true

also, this query doesn't work either

 {
  "language": "en",
  "from": "entities",
  "animatable": "time",
  "select": {
    "key": [
      "country"
    ],
    "value": [
      "name",
      "world_4region"
    ]
  },
  "where": {
    "$and": [
      {
        "un_state": true
      }
    ]
  },
  "join": {},
  "order_by": [
    "rank"
  ]
}

fix datapoints behavior

empty data when:

"entities": {
  "dim": "geo",
  "show": { "is--xxxxxxxxxxx": true }
},

DDFcsv reader returning country while geo is in query

Go to http://static.gapminderdev.org/vizabi/bug/2204-ddfcsv_reader/preview/
Open console

Result:
See that query which requests geo gets a response with country instead of the requested geo.

some entity queries are working incorrectly

for example, this one:

{
  "language": "en",
  "from": "entities",
  "animatable": false,
  "select": {
    "key": [
      "phase_from"
    ],
    "value": [
      "name"
    ]
  },
  "where": {},
  "join": {},
  "order_by": [
    "rank"
  ]
}

wrong datapoints behavoir

key column is missing

DDFCSV reader can not handle filtering by plain entity properties

DDFCSV reader can not handle filtering by plain entity properties
the code below #exposes a bug or a lack of feature in DDFCSV reader

steps to reproduce

run the demo on a local server
try to switch Countries to UN states using waffle server — works fine.
try to do the same with DDFCSV reader

Here countries is an entity set and un_state is a property on entities.

Expected behavior

Taiwan disappears because it's not a UN state. Chart doesn't crash or give errors.

Observed behavior

Taiwan is still there. Bubbles are broken. Errors in console. Reader for data source 'data' returned empty array for query, so all the data was removed from the response.

Reader for data source 'data' returned empty array for query: {
  "language": "en",
  "from": "entities",
  "animatable": "time",
  "select": {"key": ["geo"],"value": ["name","world_4region"]},
  "where": {"$and": [ {"un_state": true} ]},
  "join": {},
  "order_by": ["rank"]
}

screencast: http://recordit.co/4pInsCApfb

Demo code

<!doctype html>
<html>

<head>
  <link rel="stylesheet" href="lib/vizabi/vizabi.css">
  <link rel="stylesheet" href="lib/vizabi/bubblechart.css">


  <script src="lib/d3.min.js"></script>
  <script src="lib/vizabi/vizabi.js"></script>
  <script src="lib/vizabi/vizabi-ddfcsv-reader.js"></script>
  <script src="lib/vizabi/vizabi-ws-reader.js"></script>
  <script src="lib/vizabi/bubblechart.js"></script>
  <script src="lib/vizabi/ConfigBubbleChart.js"></script>

</head>

<body>
  <div id="placeholder" style="width:100%; height:800px;"></div>
  <p> 
    <a style="cursor: pointer; color: blue; text-decoration: underline" onclick="setModel('is--country')">Countries</a>
    <a style="cursor: pointer; color: blue; text-decoration: underline" onclick="setModel('un_state')">UN states</a>
    <span id="modelOutput"></span>
  </p>
  

  <script>
    var ddfReader = new DDFCsvReader.getDDFCsvReaderObject()
    Vizabi.Reader.extend("ddf", ddfReader);
    
    //WS reader integration
    var wsReader = new WSReader.WSReader().getReader();
    Vizabi.Reader.extend("waffle", wsReader);

    ConfigBubbleChart.locale = {
      "id": "en",
      "filePath": "assets/translation/"
    };
    ConfigBubbleChart.data = {
      "reader": "waffle", //ddf
      "path": 'https://waffle-server-dev.gapminderdev.org/api/ddf' //path-to-systema-globalis
    };

    var viz = Vizabi("BubbleChart", document.getElementById("placeholder"), ConfigBubbleChart);
    

    function setModel(arg) {
      var show = {};
      show[arg] = true;
      viz.model.state.entities.show = show;
      updateOutput();
    }
    function updateOutput(){
      var mdl = Vizabi.utils.deepClone(viz.model.state.entities.getPlainObject())
      delete mdl.autogenerate;
      delete mdl.skipFilter;
      d3.select("#modelOutput").text(JSON.stringify(mdl))
    };
    updateOutput();
    
    
  </script>

</body>

</html>

support translations

joins supporting

support datapackage.json

also, remove index.csv functionality

verbosity data regarding query passage should be provided

some kind of query returns empty result (SG)

{
      "language": "en",
      "from": "datapoints",
      "animatable": "time",
      "select": {
        "key": [
          "geo",
          "time"
        ],
        "value": [
          "population_total"
        ]
      },
      "where": {
        "$and": [
          {
            "geo": "$geo"
          }
        ]
      },
      "join": {
        "$geo": {
          "key": "geo",
          "where": {
            "world_4region": {
              "$in": [
                "americas",
                "asia"
              ]
            }
          }
        }
      },
      "order_by": [
        "time"
      ]
    }

ability to read DDF dataset from github directly

publish new ddfcsv-reader version with S3 supporting

provide the reader as oop solution

optimize slow query based on $in clause

example is here (DS=population)

{
      language: 'en',
      from: 'datapoints',
      animatable: 'year',
      select: {
        key: [
          'geo',
          'year',
          'gender',
          'age'
        ],
        value: [
          'population'
        ]
      },
      where: {
        $and: [
          {
            geo: '$geo'
          },
          {
            year: '$year'
          },
          {
            gender: '$gender'
          },
          {
            age: '$age'
          }
        ]
      },
      join: {
        $geo: {
          key: 'geo',
          where: {
            $and: [
              {
                $or: [
                  {
                    un_state: true
                  },
                  {
                    'is--global': true
                  },
                  {
                    'is--world_4region': true
                  }
                ]
              },
              {
                geo: {
                  $in: [
                    'world'
                  ]
                }
              }
            ]
          }
        },
        $year: {
          key: 'year',
          where: {
            year: '2018'
          }
        },
        $gender: {
          key: 'gender',
          where: {
            gender: {
              $in: [
                'female'
              ]
            }
          }
        },
        $age: {
          key: 'age',
          where: {
            age: {
              $nin: [
                '80plus',
                '100plus'
              ]
            }
          }
        }
      },
      order_by: [
        'year'
      ]
    }

no need to read all of the datapoint files, we need to read global based files based on

clause

apply different kind of time

regarding this: https://docs.google.com/document/d/1Cd2kEH5w3SRJYaDcu-M4dU5SY8No84T3g-QlNSW6pIE/edit#heading=h.oafc7aswaafy

Response contains subset of expected response for schema-queries

Reader: 0.13.0
Dataset: https://github.com/Gapminder/ddf--pcbs--census/tree/features/flattened
Query:

{
   "select":{
      "key":[
         "key",
         "value"
      ],
      "value":[ ]
   },
   "from":"datapoints.schema"
}

Expected: returns all availability as given in datapackage.json of dataset

Actual result: returns subset of availability as given in datapackage.json of dataset

Tested case: population_both_sexes
In datapackage.json availability for this concept with key geo,year and migrate_to,year is defined here and here.
However, DDFcsv reader does not include geo,year and migrato_to,year availability in its response:

Additional question: How can this happen? This seems suuuuuper easy logic. Just read datapackage.json and return whatever is in it.

provide an appropriate result format for WS

Current format:

Expected format:

Local file reader does not correctly read UTF8-BOM

Take a dataset and save one of its files as UTF8-BOM (instead of normal UTF8) and see how DDFcsv reader (the one shipped with Gapminder Tools Offline 4.0.0) fails.

implement schema queries

... for concepts, entities and datapoints

improve productivity

ds: https://github.com/open-numbers/ddf--gapminder--population
branch new_recipe

queries:

{
  "language": "en",
  "from": "datapoints",
  "animatable": "year",
  "select": {
    "key": [
      "geo",
      "year",
      "age"
    ],
    "value": [
      "population"
    ]
  },
  "where": {
    "$and": [
      {
        "geo": "$geo"
      },
      {
        "year": "$year"
      },
      {
        "age": "$age"
      }
    ]
  },
  "join": {
    "$geo": {
      "key": "geo",
      "where": {
        "geo": {
          "$in": [
            "world"
          ]
        }
      }
    },
    "$year": {
      "key": "year",
      "where": {
        "year": "2017"
      }
    },
    "$age": {
      "key": "age",
      "where": {
        "age": {
          "$nin": [
            "80plus",
            "100plus"
          ]
        }
      }
    }
  },
  "order_by": [
    "year"
  ]
}

{
  "language": "en",
  "from": "entities",
  "animatable": false,
  "select": {
    "key": [
      "geo"
    ],
    "value": [
      "name",
      "world_4region"
    ]
  },
  "where": {
    "$and": [
      {
        "$or": [
          {
            "is--country": true
          },
          {
            "is--global": true
          },
          {
            "is--world_4region": true
          }
        ]
      }
    ]
  },
  "join": {},
  "order_by": [
    "rank"
  ]
}

{
  "language": "en",
  "from": "datapoints",
  "animatable": "year",
  "select": {
    "key": [
      "geo",
      "year",
      "age"
    ],
    "value": [
      "population"
    ]
  },
  "where": {
    "$and": [
      {
        "geo": "$geo"
      },
      {
        "age": "$age"
      }
    ]
  },
  "join": {
    "$geo": {
      "key": "geo",
      "where": {
        "geo": {
          "$in": [
            "world"
          ]
        }
      }
    },
    "$age": {
      "key": "age",
      "where": {
        "age": {
          "$nin": [
            "80plus",
            "100plus"
          ]
        }
      }
    }
  },
  "order_by": [
    "year"
  ]
}

{
  "language": "en",
  "from": "entities",
  "animatable": false,
  "select": {
    "key": [
      "geo"
    ],
    "value": [
      "name",
      "world_4region"
    ]
  },
  "where": {
    "$and": [
      {
        "$or": [
          {
            "is--country": true
          },
          {
            "is--global": true
          },
          {
            "is--world_4region": true
          }
        ]
      }
    ]
  },
  "join": {},
  "order_by": [
    "rank"
  ]
}

new version based on Vizabi 0.15.2-3

Clarify documentation

I am trying to run the ddfCsvReader on the server per the documentation:

const Vizabi = require('vizabi');
const ddfCsvReader = require('vizabi-ddfcsv-reader');
const readerObject = ddfCsvReader.getDDFCsvReaderObject();
Vizabi.Reader.extend('ddf-csv-reader', readerObject);

This gives me "d3 not found". So I simplify to:

const ddfCsvReader = require('vizabi-ddfcsv-reader');
const readerObject = ddfCsvReader.getDDFCsvReaderObject();

This works after building node_modules/vizabi-ddfcsv-reader.
(Note: The build works but gives some errors "ERROR in Entry module not found: Error: Can't resolve './src'")

Would be super helpful if you could update the example and the built version.

Thank you :)

provide new reader functionality

implement asset functionality

error format unification

export some tests data

export reader init data and queries from

test/definition/concepts-definition.spec.ts
and
test/definition/entities-definition.spec.ts

the goal is to use this data in other tests

Same query, two outcomes, depending on when called from Vizabi

Run vizabi branch feature/remove_gapminder-vizabi.js or visit http://static.gapminderdev.org/vizabi/feature/remove_gapminder-vizabi.js/preview/mountainchart.html
Let it load completely
Put a breakpoint at src/base/data.js at line 143 with query.from == 'datapoints'
Resize Vizabi a bit and click the 'show' button that appeared
OR
Click options and then show
Click Angola in show dialog
Breakpoint should be triggered now
Note the query which is sent to the reader (maybe save it through JSON.stringify(query, null, 2))
Remove the breakpoint
Resume script execution
You should see the data for Angola is loaded correctly
Now refresh the page (f5)
See how in console error is shown for reader returning empty data
Compare the query with the previous query.

Result:
Queries are the same, but first query returns data while second returns empty.

Expected result:
Both queries return requested data for Angola.

wrong records quantity

example.zip
wrong records quantity for query

{
      "language": "en",
      "from": "datapoints",
      "animatable": "year",
      "select": {
        "key": [
          "country_code",
          "year",
          "gender",
          "age"
        ],
        "value": [
          "population"
        ]
      },
      "where": {
        "$and": [
          {
            "country_code": "$country_code"
          }
        ]
      },
      "join": {
        "$country_code": {
          "key": "country_code",
          "where": {
            "country_code": {
              "$in": [
                "900"
              ]
            }
          }
        }
      },
      "order_by": [
        "year"
      ]
    }

current quantity==151 but it's too small set of data for this kind of query and dataset

{
      from: 'entities',
      animatable: 'time',
      select: {
        key: ['geo'],
        value: ['name', 'world_4region', 'latitude', 'longitude']
      },
      where: {'is--country': true},
      grouping: {},
      orderBy: null
    }

In this case, all of geo-files will be processed. But only geo-country-files are expected in this case in accordance with related record in datapackage schema:

{
                "primaryKey": [
                    "geo"
                ],
                "value": "is--country",
                "resources": [
                    "ddf--entities--geo--country"
                ]
            },

So, we should process only ddf--entities--geo--country instead

ddf--entities--geo--global,
ddf--entities--geo--g77_and_oecd_countries,
ddf--entities--geo--landlocked,
ddf--entities--geo--main_religion_2008,
ddf--entities--geo--world_4region,
ddf--entities--geo--income_groups,
ddf--entities--geo--country,
ddf--entities--geo--world_6region

multidimentional dataset reading returns wrong values (DANGEROUS)

start the file
demo-05-popbyage.html

from this repo
https://github.com/Gapminder/csv-conf-2017-talk

everything looks fine

so this is data from waffle server

now switch to local dataset and ddfcsv reader

you will see population of 0 year olds going to 66 million in 2017

compare it with the actual data in the files

https://github.com/Gapminder/csv-conf-2017-talk/blob/master/data/ddf--gapminder--population/ddf--datapoints--population--by--global--year--age.csv#L5969

the real values are roughly twice as large

this is veeeery dangerous that change numbers invisibly