@mharrower

Idea for a quick 'chart explorer' demo using Cedar + Vega

1. Data can be link to Feature Layer, AGO ID for Service or Array of values
2. Style can be URL to Vega JSON, AGO ID for 'Chart', or inline JSON
3. Sum/Group are dynamically created based on Vega extensions for the required input values. Array of Fields like Feature Layer
4. Save to Online pushes the styling as an Application with Data as the JSON
5. Load from Online loads the instantiated Chart. but links to Style - so changing style, saving style and then loading from online shows the updated rendering.

/cc @dbouwman

Histograms are almost like bar charts except X-axis will show numeric intervals with some predefined breaks (equal intervals, natural breaks, etc.).

The implementation could probably use getHistogram method from new JS API.

hoping to get some data from @olearydw and create a sample Punch card visualization

Examples

)

chartist.js

/cc @cooney @dbouwman

Add documentation example of continuous updating (e.g. traffic or sensor datasets)

var chart = new Cedar();
setInterval(function(){ chart.update(); }, 3000);

Look at the Vega v2 Examples for how brushing will come into Vega.

Also read their paper on interactive vega

To address #13 and other requirements where the cedar chart should hold state, we need to re-work it to be an object, not just an object literal with functions.

ask @mharrower

requirements from @jpeterson

Something happened during the refactor related to #31 that broke the timeline example

Should look like

but looks like

Options

• http://bl.ocks.org/dbuezas/9306799
• http://bl.ocks.org/Guerino1/2295263
• http://d3pie.org/
• https://blog.safaribooksonline.com/2014/03/11/solving-d3-label-placement-constraint-relaxing/

Requirement from @thejones

There is a select method, but there is no way to clear selection. Something like clearSelection method would be very useful here. As a dirty workaround, I have to call update and then select new items via setTimeout(..., 200)

Nuno and I got together to put some thoughts down as use cases for charting. Our context includes product and projects such as Living Atlas (demographics, urban systems, landscape) and Urban Observatory.

Use case 1: “Map, talk to me” a.k.a. “extent popup"
I zoomed the map to this area. While it is useful to use the popup to query parts of the map, I really want the chart to instantly tell me something about the data using this map’s extent.

Example: in Urban Observatory, I see three population density maps side by side, but I really want to know how many people are represented, in total, for each map’s extent:

Bar Chart underneath each map showing
124,000 people 3.6 million people 231,000 people

The bar chart updates as map extent changes (zoom or pan)
If user pans away from content, chart shows something friendly, or disappears after showing a friendly message. No “NaN” or other cryptic messages.

Authoring note: The bar chart uses the popup’s chart definition to “clue in” to what the map author already knows is important. This is not the only way to define new charting options, but it is critical we leverage knowledge already expressed in good popups.

Use case 2: Add Context by Comparing
User is looking at a map of Utah. They have never been to Utah. They click on the map. They get back a comparative chart showing median household income for multiple goegraphies, in this case: the country, state, county, and ZIP code where they clicked.

This explicitly puts geographies into the chart, a key distinguishing characteristic of Esri software.

Note: it is also useful sometimes to enter a “fixed” value that appears in the chart as a reference point. Examples: a national average, 98.6 degrees, index = 100, or other “anchor” value.

Good example: Census Reporter http://censusreporter.org/ and its embeddable charts as in this news story: http://yorkandfig.com/

Use case 3: a map tip chart
For a defined chart, the user can move the mouse along the map and the chart self-updates, like a tooltip or maptip, based on the chart’s geographic definition (e.g. 1 mile ring, 10 minute drive, nearest facility, find similar). If no geographic analysis is defined, the map tip chart simply uses lat long, like a speedy popup that requires no click. (Like NY Times maps’ maptips, but with a smart chart associated).

Example where speedy tips are sexy but don’t help create knowledge:
http://projects.nytimes.com/census/2010/map

Bonus: let the user “hang on” to a chart in session, so that they can easily build a comparison side by side visually, not just in their head. So, as I move around the map, if I want it to “stick” just click. That allows a user to see pie charts side by side for example.

Single-chart variation: as the user clicks, build a column chart or line chart one record per click, so that the user is building a story (and its chart) as they go.

Use case 4: sketch n chart
User is looking at a map of their city. They want to sketch an area on the map. They draw a polygon, and a chart appears, summarizing the data for the area they sketched. They draw another polygon. The chart expands to include a second element that summarizes the data for the second polygon.

Use case 5: chart what I want, not just what the data has
When comparing multiple features, I should be able to aggregate two columns into a new column. For example, I want to see the combined total of households making between $0-$15,000 and $15,000-$30,000 in an area in order to understand who may need support services in an emergency situation. Why: currently the database schema dictates what can be mapped or charted. Sometimes the user needs to see charts based on simple arithmetic of the raw data.

Use case 6: chart across layers
Example: I have total population in 1990, 2000, 2010 and 2014. Each is in its own layer or service. I want to show a simple chart that depicts total population from each of these services, brought together as a line or bar chart.

Use case 7: show me key values
From the chart I can see some useful information. If I touch the chart, the map highlights some key values (e.g. Features with the min or max value, features in a particular class, etc)

Use case 8: Sync map and chart
The map’s legend can inform the chart’s definition. If the map legend for income has 5 breaks, use those breaks and colors in the chart.

If the user touches the chart, chart author can choose to configure the chart to “light up” those features on the map, using a selection indicator or heat map. Easy undo.

Highlighter mode: let the user apply a digital highlighter pen to the chart, the corresponding features on the map are highlighted. Eg. Highlight the steepest part of a race course.

Chart author can configure the chart allow the user to instantly apply a filter to the map by touching the chart. Easy undo.

If I combine chart groups on the chart, combine them on the map and its legend also

Observations about charting contexts

What can I use to author a chart?
From a web map in AGOL (leverages legend info, popup info, AGOL item description/summary/credits)
From a map layer in AGOL (leverages legend info, popup info, AGOL item description/summary/credits)
From a REST endpoint (leverages legend info, item description/summary/credits)

Where can the chart appear?
Charts can appear on the map, or be directed to appear elsewhere on the map app, or be embedded independent of a map (like the Census example above)

Smart Charting
If customer wants to add a chart to a map, it should work like smart maps. Meaning, these are the 2 charts that best fit the data, please select one, or chose other for more options.

Where charts work
It would be great if the charts were available on desktop and online.

This is more of a discussion than actual issue. What should be our strategy if we want to do something very custom using d3 like chord diagram? We were thinking something about custom templates (specifications?) bypassing vega if needed. This will probably require some extensibility entry point in Cedar.

@ajturner, what are your thoughts on this?

cc @arthaddad @HeikoH

This is a general issue for thrashing out the Cedar API

There are 2 types of json documents involved with cedar.

Cedar Templates

These are the source, generic templates, based on the vega spec syntax, with an inputs hash appended.

An example of this would be json describing a generic bar chart - aka bar-chart.json

Stored in portals as type: chart-template

Cedar Chart

This is the combination of a vega spec, with the data source url / embedded data. This is the most commonly consumed form for presentation to the end-user.

An example of this would be dc-bar-chart-of-crimes-by-type-over-the-last-month.json.

Stored in portal as type: chart

API Philosophy

• templates/data can be fetched by cedar, but also may be passed in to avoid additional round-trips.

API Dependencies

• vega - for the visualization grammar and parsing into d3 based charts
• d3 because vega depends on it and also used to fetch data

Public API

moved to http://esridc.github.io/cedar/api-reference/

Would be nice to have the option of storing the query response to save an additional request.

Use case:
For story maps, we might like to drive the map interaction based on the chart. Present the user with a chart, zoom to feature with greatest value. Update chart with new query, zoom to new feature with greatest value.

when selecting features on the map, highlight the relevant chart features.

close, but needs more work

{
  "width": 700,
  "height": 400,
  "data": [
    {
      "name": "cars",
      "url": "http://opendata.arcgis.com/datasets/3979e158e3b941dbaac1227c4f0a200d_0/FeatureServer/0/query?where=1=1&outFields=*",
      "format": {
        "property": "features"
      }
    },
    {
      "name": "fields",
      "values": ["Q1 Actual", "Q2 Actual", "Q3 Actual", "Q4 Actual"]
    }
  ],
  "scales": [
    {
      "name": "ord",
      "type": "ordinal",
      "range": "width", "points": true,
      "domain": {"data": "fields", "field": "data"}
    },    
    {
      "name": "Q1 Actual",
      "range": "height", "zero": false, "nice": true,
      "domain": {"data": "cars", "field": "data.attributes['Q1 Actual']"}
    },
    {
      "name": "Q2 Actual",
      "range": "height", "zero": false, "nice": true,
      "domain": {"data": "cars", "field": "data.attributes['Q2 Actual']"}
    },
    {
      "name": "Q3 Actual",
      "range": "height", "zero": false, "nice": true,
      "domain": {"data": "cars", "field": "data.attributes['Q3 Actual']"}
    },
    {
      "name": "Q4 Actual",
      "range": "height", "zero": false, "nice": true,
      "domain": {"data": "cars", "field": "data.attributes['Q4 Actual']"}
    }
  ],
  "axes": [
    {"type":"y", "scale":"Q1 Actual",  "offset":{"scale":"ord", "value":"Q1 Actual"}},
    {"type":"y", "scale":"Q2 Actual",  "offset":{"scale":"ord", "value":"Q2 Actual"}},
    {"type":"y", "scale":"Q3 Actual",  "offset":{"scale":"ord", "value":"Q3 Actual"}},
    {"type":"y", "scale":"Q4 Actual",  "offset":{"scale":"ord", "value":"Q4 Actual"}}
  ],
  "marks": [
    {
      "type": "group",
      "from": {"data": "cars"},
      "marks": [
        {
          "type": "line",
          "from": {"data": "fields"},
          "properties": {
            "enter": {
              "x": {"scale": "ord", "field": "data"},
              "y": {"scale": {"field": "data"}, "group": "data.attributes", "field": "data"},
              "stroke": {"value": "steelblue"},
              "strokeWidth": {"value": 1},
              "strokeOpacity": {"value": 0.3}
            }
          }
        }
      ]
    },
    {
      "type": "text",
      "from": {"data": "fields"},
      "properties": {
        "enter": {
          "x": {"scale": "ord", "field": "data", "offset":-8},
          "y": {"group": "height", "offset": 6},
          "fontWeight": {"value": "bold"},
          "fill": {"value": "black"},
          "text": {"field": "data"},
          "align": {"value": "right"},
          "baseline": {"value": "top"}
        }
      }
    } 
  ]
}

There seems to be another open issue related to sizing/responsiveness but this one slightly different.

I took this default basic scatter.json and changed width/height to the values I needed (555px by 555px in this case). The chart got renderer in 809 x 615 box:

This issue is not specific to scatter plot, it is also reproducible for bar chart (the size discrepancy for bar chart is much less notable).

Please use this feature server to repro:
http://services.arcgis.com/pmcEyn9tLWCoX7Dm/arcgis/rest/services/CA_Hospitals/FeatureServer/0
... and use Mortality_Readm_Rate as X and Number_of_Patients as Y.

As a developer I want to react to events on the chart so I can orchestrate the rest of my application

Examples

dc school scatter plot - mouse over the graph, see the point on the map light up
dc school enrollment by type bar chart - move over graph, see schools of that type light up

Events

• mouse over, mouse out, click, double click
• objects to listen to: chart content, axes(?), legend(?)

Notes

vega views expose a nice eventing api via .on(, callbackfct), however, the event that is returned has a whole mess of vega/d3 noise in addition to the actual data. While that may be useful to a small subset of developers, for Cedar we should return a simpler payload that relates back to the data, and omits the noise.

In order to do this, we would need to wrap the vega events, and raise our own events. This would not be difficult, and could be implemented as an optional module. We would likely lean on an existing micro library for this so we have a well tested fluent (aka .on() ) style api.

requirements from @jpeterson

Currently, Cedar expects global vg variable but in presence of an AMD loader (e.g. Esri JS API), Vega defines itself as AMD module without creating global variables. Cedar fails to work in this scenario.

The Specification should include how the input mappings will be used in the query roll-up. This is not always easy to infer.

For example, when making a bar chart - we may say the height is the single value of an attribute, or it may be grouped by a category, or it may be a histogram which needs to make multiple calls to get bins.

The ascending and descending property is stripped from the query:
mergedQuery.orderByFields = dataset.mappings.count.field + "_SUM";

• Default looks good
• Ability to override with d3 commands

Generalized from #12

I zoomed the map to this area. While it is useful to use the popup to query parts of the map,
I really want the chart to instantly tell me something about the data using this map’s extent.

Example: in Urban Observatory, I see three population density maps side by side, but I really want to
know how many people are represented, in total, for each map’s extent

Pseudo code

//assumes we have a map object
var self = this;
Cedar.show(chartObj, function(chart){
  self.chart = chart;
});

map.on('zoom-end', function(evt){
  self.chart.query.bbox = Cedar.extentToBbox(map.extent);
  self.chart.update();
})

Cedar Impact

So, the first part of this would be to hold the mappings in the compiled chart vs actually replacing them in the template.

The second part would be to expose the query as part of the chart, and the ability to call .update(queryParams)

Chart Template Inputs

The inputs define the required

{
  "inputs": [
    {"name": "count", "type": ["numeric","string"], "required": true},
    {"name": "group", "type": ["string"], "required": false},
    {"name": "data", "type": ["url"], "required": true}
  ],
"datasources":[

 ],

...other properties...

}

Compiled Chart Inputs have values stored

{
  "inputs": [
    {"name": "count", "type": ["numeric","string"], "required": true, "value":"ZIP_CODE"},
    {"name": "group", "type": ["string"], "required": false, "value":"TOTAL_STUD_SUM"},
    {"name": "data", "type": ["url"], "required": true, "value":"http://someservice.com/featureserver/2"}
  ],

...other properties...

}

@ajturner

When I specify a dataset like this:

  var dataset = {
    "url":"http://sampleserver5.arcgisonline.com/ArcGIS/rest/services/LocalGovernment/CitizenRequests/FeatureServer/0",
    "mappings":{
      "group": {"field":"severity","label":"Severity"},
      "count": {"field":"OBJECTID","label":"Count"}
    }
  };

I'm expecting it to send a query w/ the following outStatistics to the service:

[{"statisticType":"count","onStatisticField":"OBJECTID","outStatisticFieldName":"OBJECTID_COUNT"}]

Instead it sends:

[{"statisticType":"sum","onStatisticField":"OBJECTID","outStatisticFieldName":"OBJECTID_SUM"}]

Which is what I would expect if I set mappings.sum instead of mappings.count.

Am I missing something? Is there another way to group by and get counts instead of sums for another field?

assign to @azhigimont

The responsive example is a good starting point, but it would be great if the Cedar API provided an option to indicate that the chart is responsive, and handled that so that the client code does not have to. I like the way that Chart.js does this with these two chart options:

    // Boolean - whether or not the chart should be responsive and resize when the browser does.
    responsive: false,

    // Boolean - whether to maintain the starting aspect ratio or not when responsive, if set to false, will take up entire container
    maintainAspectRatio: true,

If responsive is true, Chart.js will add a debounced handler for window resize that updates the chart height / width based on the new size of the chart's dom node the above maintainAspectRatio option.

I believe this can be done in Cedar b/c you get a reference to the chart's dom element in the .show() method. If the responsive option is truthy, you could add a debounced window resize event listener to call a new resize() method which would update the chart's width/height overrides based on the dom element's size and call update() just as in the example only instead of setting all the overrides, you'd only update the width and height overrides.

Thoughts?

In working on the Cedar code, I need to create the expected json structure, and try to cover the following scenarios:

• must contain all the information to render a chart
  • i.e. can be stored in AGO as an item or item data
• but still allows
  • modifications to the query
  • modifications to the chart template (axes titles etc)
  • simple re-rendering via cedar.update()
  • a chart editor app (cypress) to change the data source and field mappings

A Little History...

The original "template" was just a vega spec w/ tokens.

{ ... spec template ...}

We then extended that to have an "inputs" array so editor apps have the info needed to create the mappings...

{  
  "inputs": [
      {"name": "x","type": ["numeric"],"required": true},
      {"name": "y","type": ["numeric"],"required": true},
      {"name": "color","type": ["string"],"required": true}
    ],
   ... spec template ...
}

At this point, we could call cedar.create(template, mappings, dataUrl) and get a "compiled" chart json object (aka a vega spec) back out. That object would then be passed to cedar.show(elementId, chartJson) and the chart would be rendered. Woot!

The problem with that workflow is that in the compilation to the "vega spec" we have lost the parameterization, and thus it is a "one way" process. Put another way, if all cedar has in it's internal state is the compiled chart, we have limited options to manipulate the chart (i.e. change query etc).

In order to make cedar more flexible, I'm proposing that the "compiled vega spec" is ephemeral, and created as needed from a chart definition json.(totally open to changing terminology here)

The process chain would go like this:

chartTemplate + mappings + dataSource ==> chartDefinition

The api would be like this:

//constructor options
var cedar = new Cedar({chartDefinition: chartDefintionObjOrUrl})
var cedar = new Cedar({chartTemplate: chartTemplateObjOrUrl, mappings:{...}, dataSource:{...})
var cedar = new Cedar(); //must set template, mappings & dataSource properties before .show()

//once cedar has a full definition we can
//render the chart into an element
cedar.show(elementId);

//event handlers as before using .on
cedar.on('click', function(data){...do things..});

//Manipulate State: the query string... 
//if we using a real (ES5.1+) property it would look like this
cedar.query.where = "FOO > 12";
cedar.update();

//we could also make it chainable by using getter/setter functions
//thus allowing modifications and calls to update inline
cedar.query.where( "FOO > 12").update();

//could also pass in data - i.e. data is loaded in the map
//so get it from the feature layer in the map and pass in 
cedar.dataSource.data(...array of features...).update();

//finally we can get the full definition back out
//thus allowing apps to store this where ever they want
var def = cedar.definition();

Examples of the Json formats:

//chart template
{
  "inputs":[... array of inputs...],
  "specTemplate":{ ... template of a vega spec ...}
}

//mappings hash relating a datasource to the inputs
//main change here is the addition of label. If not present
//during compilation, it will be set to the value of the field
{
    "x": {
        "field": "POPULATION_ENROLLED_2008",
        "label": "Enrollment 2008"
      }
}

//dataSource
{
 "url":"...url to service...",
 "query": {... json hash of query params}
 "data":[...optional array of features...]
}

combined become...

//chart definition
{
  "config":{
    "dataSource":{ ... dataSource hash ...},
    "mappings": { ...mappings hash...}
  },
  "template": {
     "inputs":[... array of inputs...],
     "specTemplate":{ ... template of a vega spec ...}
  }
}

In this way, we always have all the info to re-create the vega spec, and also allow the query, or template to be modified in any way.

Here is an example of a whole definition object w real values and some inline comments https://gist.github.com/3d5e4784e2dd941507d3

/cc @ajturner

Using perhaps Keen to populate an interactive/configurable dashboard.

I’ve wired up mouseover on the line chart (events are logged to the console), but it seems that no matter where I hover in the chart I see the same data:

http://tomwayson.github.io/cedar/examples/time-responsive-events.html

@dbouwman points out that this is b/c w/ line charts there is only one mark (the line) and that wille the data that cedar gets back from vega is includes all the points that make up the line, cedar is only passing the first one back to the client event handler. He also suggested that we might be able to figure out the value from the array of points based on the mouse x position, but that we're probably better off waiting for vega v2's improved "interactions."

I did start looking at the v2 examples. The index_chart example here: http://idl.cs.washington.edu/projects/reactive-vega/examples/editor/ shows how to declaratively capture the changes to mousemove. Near as I can tell it’s getting the date along the x-scale at the mouse.x position and then using that as the input to a transform on the data (see below for .json).

Anyway, I wonder if that approach (getting data from the scales instead of the underlying data) makes more sense for line charts and whether or not there's a v1 way to do it.

"signals": [
    {
      "name": "xPos",
      "init": 265,
      "streams": [{"type": "mousemove", "expr": "p.x", "scale": "x", "invert": "true"}]
    },
    {
      "name": "indexDate",
      "init": 1104566400000,
      "streams": [{"type": "xPos", "expr": "date(xPos)"}]
    }
  ],
  "data": [
    {
      "name": "stocks",
      "url": "data/stocks.csv",
      "format": {"type": "csv", "parse": {"price":"number", "date":"date"}}
    },
    {
      "name": "index",
      "source": "stocks",
      "transform": [{
        "type": "filter",
        "test": "d.date + 1296000000 >= indexDate && d.date - 1296000000 <= indexDate"
      }]
    },
    {
      "name": "indexified_stocks",
      "source": "stocks",
      "transform": [{
        "type": "zip",
        "with": "index",
        "as": "index_term",
        "key": "symbol",
        "withKey": "symbol",
        "default": {"price": 0}
      }, {
        "type": "formula",
        "field": "indexed_price",
        "expr": "d.index_term.price > 0 ? (d.price - d.index_term.price)/d.index_term.price : 0"
      }]
    }
  ...

Time series data - linear chart that allows brushing and zoom.

See Vega v2 for examples with declarative interaction

ask @ffaubry

In some cases loading data takes several seconds. For anything above 1-2 sec we would like to show a progress bar. Please support events like update-start and update-end to facilitate this. This could be something similar to corresponding events in FeatureLayer.

Looks like the only way to load Cedar right now is by copying it locally to my HTTP server

ask @ffaubry & @mharrower

ask @ffaubry

This will allow for modification of a cedar object like...

//pseudo code
var cedar = new Cedar();
cedar.show('#chart", chartObject);
//modify the where clause
cedar.query.where = 'TOTAL_STUDENTS > 100';
//update the chart
cedar.update();

As a User, I want to visualize data from my service compared to data from another service.

For example, scatterplot of Sales by State from database compared with Median Income by State from demographics

Reproducible with this feature service:
http://services.arcgis.com/pmcEyn9tLWCoX7Dm/arcgis/rest/services/CA_Hospitals/FeatureServer/0

And these mappings:

{
  "group": {
    "field": "Hospital_Ownership"
  },
  "count": {
    "field": "Number_of_Patients"
  }
}

Right now the dataset hash is extended only when the query string is generated. However, this means that if the developer passes in a minimal dataset, then the query node will not be present.

Instead - set cedar._defiition.dataset to the default, and have the setter simply _.extend(current, new)