Another interesting insight from the dev process for the demo app (#15): If #32 gets added to Grout, then for data modelling purposes it would be useful to permit the user to define RecordTypes that are not fundamentally geospatial in nature. For instance, if we're collecting Records in a PosterRecordType, we might be interested in having a PosterSeries RecordType that each Poster can point to in a field like Poster.series that would allow us to group together posters that are identical but that are deployed in different locations. In this case, it wouldn't really make sense to have date/time or geospatial data associated with PosterSeries Records.

The schema editor currently uses an Azavea-maintained fork of Jeremy Dorn's json-editor library. Jeremy Dorn's version of the library has since been deprecated in favor of a community-maintained fork.

We should update the schema editor to use the actively-maintained version of JSON Editor. This work will involve auditing the Azavea fork to see what features we use that aren't in core, and making sure that those changes are included in whatever version we use (whether it means updating the Azavea fork, or trying to get those changes accepted into the core).

Once azavea/grout#9 comes in, the new types of Records (with new/optional geometries and optional temporal data) will have to be exposed in the schema editor.

As part of developing Ashlar, I want to build a demo app on top of it to demonstrate its capabilities. My two ideas are:

1. An app for archiving public posters for events stapled to street lamps in Philly
2. An app for cataloguing architectural features of Philly rowhouses

I prefer 1 to 2, but 1 was an idea brainstormed in tandem with some folks at William Way, so I need to talk with them and set some expectations first. If 2 seems more feasible, I need to figure out how to constrain the scope of it such that it's simple and doable with my current knowledge.

I should get this done this sprint so that development can start on the demo app as soon as possible.

The demo app needs to somehow be able to filter Events according to the Posters that reference them, and vice versa. There are a couple ways of doing this: #32 is the most complicated, but we could also implement a more naive solution (similar to what DRIVER does) and simply duplicate reference data inside the Records in question. Either way, let's figure out our approach this sprint cycle.

The frontend to the reference implementation should display a simple table of Records and demonstrate automatic filtering capabilities.

The reference implementation backend should allow a user to edit RecordTypes, RecordSchemas, and Records. While technically an "admin" interface, it won't deal with any kind of authentication stuff.

Right now, Records must be stored with point geometries. However, it seems reasonable to allow users to store polygon geometries, too (see e.g. my notes from the kickoff meeting for the demo app for an example use case).

Some more discussion of this issue with a proposed data model in #18 (comment).

There are a few style improvements I'd like to make to the demo app, including:

• Make the map/sidebar take up the full screen
• Flesh out the detail view modal
• Show short description on map popover
• Add "Read more" prompt to long description
• Make sidebar scrollable when filters overflow screen
• Display images
• Change title
• Icons
• Better typography
• Nicer colors
• Add new data

Looking at the code, it seems like the current implementation of the date filter doesn't factor the occurred_to field into its query. Here's the line that produces the ultimate queryset:

return queryset.filter(occurred_from__gte=min_date, occurred_from__lte=max_date)

This may be an artifact from a time when only occurred_from was supported. In order to consider occurred_to, it should probably be:

return queryset.filter(occurred_from__gte=min_date, occurred_to__lte=max_date)

Here are a few things we need to do to make that happen:

1. Make sure the tests cover date ranges (right now they only test Records where occurred_from = occurred_to, which is likely the reason we missed this)
2. Change the date filter to consider occurred_to, as above
3. Introduce field validation to make sure that if occurred_from or occurred_to is present in a create operation but its partner isn't, that the partner field gets assigned the value for the present field

This work is related to a refactor I'm already doing in #9, so I'll fix it there.

Right now the text-field searches only display exact matches. Search needs to be improved in the demo app.

@flibbertigibbet already put some effort into investigating JSONB searching in Django: azavea/fellowship-program-website#26 (comment)

Related to #14, the schema editor is currently written in a deprecated version of AngularJS. It should be updated to work with a modern framework.

The demo app should run tests and deploy via Travis.

The DRIVER frontend apps already sort of implement a JS API for making it easier to interact with Ashlar from the client side. Splitting this off into a separate module packaged via e.g. NPM would be a big boost to frontend development.

I have two talks to prepare on Grout for the week of 8/13:

• Project presentation, Wednesday 8/15 (~5 min lightning talk, project overview)
• TSM presentation, Friday 8/17 (~20 min talk, technical deep dive)

Both talks should be semi-finished by Monday, 8/13 for the practice run-through.

Rob prefers we use Google Slides for these talks

Before #20 comes in, the JS API needs to have tests.

The schema editor that was built into DRIVER is a nice component of the Ashlar toolkit, but it's currently tightly integrated with the reference implementation. It should live in its own library instead.

Some starting context: azavea/fellowship-program-website#26

There are several options for how to develop Ashlar as part of this fellowship. We "picked" one as part of the discussion above, but that doesn't have to be the final decision. So it would be good to have a brainstorming and planning session to revisit that discussion and make a final decision now that @jeancochrane has gotten started on the project.

The goals of this discussion will be:

1. Review possible development goals for this project and discuss their benefits and drawbacks
2. If we think there is an obviously best development path, choose it. If not, we should figure out what information we need in order to make a decision.
3. Either way we go with number 2, come up with a series of tasks to either begin implementing our preferred solution, or to begin gathering information necessary to make the decision.

There's no reason Grout shouldn't support Python 3.7, but currently we don't test against it. Let's update the tests to run against Python 3.7 so that we can add official support.

Let's have a meeting to talk about the future of Grout and put together a project roadmap.

That'll make it much easier for other people to install it.

The demo app needs to live online somewhere accessible.

Currently, Grout Server is configured to permit access to all HTTP methods from any user; it assumes that, like DRIVER, your API is only accessible to authenticated users.

It would be more useful to set the default permissions to IsAuthenticatedOrReadOnly, which would allow anyone to use read-only methods but would restrict write methods to logged-in users.

And make pull requests to the parent projects if / when appropriate.

Time permitting, it would be awesome to build a simple app using the Ashlar suite/toolkit to demonstrate its usefulness and put its different components to the test.

In putting together a reference implementation with Ashlar, I noticed that DRIVER implemented a nice authentication backend for Ashlar which includes protection for the API and some default user groups that allow you to define group-specific views out of the box. It'd be cool to abstract this out into an authentication module and pull it into Ashlar core.

Ashlar works well as a standalone DB server supporting a JS frontend, but it would also be nice to allow users to use it as a pluggable Django app. An important part of supporting this capability is porting the backend components (schema editor and data collector) to Django's admin system.

Alex gave some awesome feedback described in https://github.com/jeancochrane/philly-fliers/milestone/1/. I'd like to bring as many of these changes in as possible before the lightning talks on Wednesday.

One interesting insight gleaned from the demo app development process (#15) is that it could be useful for some use cases if we were to extend the relationship field type to permit a pointer to a Record from a different RecordType (e.g. if I have two RecordTypes, Poster and Event, there could exist a field Poster.event in the RecordSchema that points to a UUID for an Event Record).

This could get hairy for a few reasons:

1. Foreign key coherence is not guaranteed in NoSQL systems, so pointers could wind up empty
2. As far as we know, JSONSchema has no built-in foreign key validation
3. Introducing foreign keys could be a big performance hit to filtering if Postgres' jsonb data type does naive table scans (although the Django docs seem to indicate that jsonb permits indexing, which is perhaps a sign that filtering could have better performance)

Before jumping into this, we should do some research to see what the current state of the art is for foreign keys in JSONSchema.

Right now the default route for getting an authentication token from the Grout Server API is /api-token-auth/. All other API methods are served from /api/. This makes it annoying to run a Grout Server under a reverse proxy -- all calls to /api/ can be safely redirected to the running Grout Server, but /api-token-auth/ either A) requires its own location block or B) gets caught by the default / location block and passed to the wrong process/directory.

This whole situation would be simpler if the token request endpoint was underneath the /api/ location -- something like /api/token/.

Ashlar currently only supports Python 2.7. It would be nice to support Python 3.3 - 3.6.

Ashlar's models currently include tables and fields that are specific to the DRIVER implementation. For better abstraction, these tables should be removed from Ashlar and moved to DRIVER.

Time permitting, I'd love to write a blog post to wrap up the fellowship.

An open source project is no use without good docs. Ashlar in particular would benefit from a detailed description of the data model and API, as well as step-by-step instructions for incorporating it into different types of projects and customizing it.

Ashlar runs on Django 1.8, which is no longer supported as of April 1st. Ashlar should support the current 1.x LTS version (1.11).

Related to #17, Ashlar will be more useful the easier it is to plug into a project. We should improve packaging and design it with specific implementations in mind, including:

• Dockerized setup for running as a standalone DB server with a JS app
  x. Setups with and without backend UI
  x. Potentially, templates for different kinds of app
• PyPi distribution for Django projects

Postgres is Ashlar's heaviest dependency, and a barrier to simple deployment. It would be nice to allow alternative NoSQL database backends that wouldn't require so much configuration. This will involve some research to determine an appropriate library/service, since the database needs to be:

• Easy to configure and deploy
• Support simple filtering/querying
• Either free or cheap

In a twist of fate, it turns out that there's no built-in validation to make sure that a new Record matches its schema in the REST API. In DRIVER, the schema editor takes care of schema validation, which is why we hadn't noticed it before.

The RecordSchema model has a validate_json method that seems primed to perform this kind of validation:

def validate_json(self, json_dict):
    """Validates a JSON-like dictionary against this object's schema
    :param json_dict: Python dict representing json to be validated against self.schema
    :return: None if validation succeeds; jsonschema.exceptions.ValidationError if failure
             (or jsonschema.exceptions.SchemaError if the schema is invalid)
    """
    return jsonschema.validate(json_dict, self.schema)

We should build this validation into the Record viewset and make sure that it's tested.

I caught a couple of bugs in the schema editor this week, which you can see on the issues page. Time permitting, it'd be really nice to close these out, since a few of them prevent the schema editor from fully working.

djsonb is the only dependency currently preventing Ashlar from being compatible with Django 2.0+. Once that dependency is removed (#12), we should support these versions of Django.

At some point (most likely in the course of #12) we introduced a bug into Grout that is causing jsonb filters to never return Records, even when the condition should pass. Per @flibbertigibbet's description:

On entering any text in the search, I get no results, even if the text is an exact match. Clearing the filter still returns the one sample record.

For sprint 1, we're going to build a reference implementation for Ashlar as it exists now. This reference implementation will need a repo. The repo should live under the Azavea organization.

Unfortunately, someone has beaten us to the ashlar package name on PyPi. If distributing through PyPi is important to us, we should consider renaming the project.

Before getting started on the demo app, I'd like to cut a clear path forward in terms of the technologies that I'll use to implement it. In general terms, I can see possibilities:

• Pluggable Django app
  • Pro: Technology that I'm already very familiar with
  • Con: Not a great learning opportunity; puts most of our up-front investment in a narrow range of users (Django)
• Single-page JS app
  • Pro: Nice opportunity to learn a frontend framework
  • Con: More work to get me set up; deciding on what framework to use is riskier, sets up path dependence going forward

This might be a good opportunity to write up an ADR.

Related to #15 and #7.

Following #26, all references to "Ashlar" will need to change.

These steps include, but are not necessarily limited to:

• Rename the core library on GitHub
  • Either:
    1. Create a new repo for the project, fork Ashlar, and move the code over; or
    2. Rename the repo on GitHub (confirm that the old URL will continue to redirect)
• Update the core library code to refer to the new name instead of ashlar internally, in all imports and namespaces
• Rename repos that reference the name "Ashlar", including;
  - ashlar-2018-fellowship
  - ashlar-blueprint

The djsonb library was an awesome workaround for its time, but now that Django supports Postgres JSONB fields out of the box, we should remove the djsonb dependency and use the native JSONField type instead.

One of Ashlar's cool features is that you can query and filter arbitrarily-nested JSON over a REST API. There's been a lot of work in adjacent problem domains since 2015, however, and I think the syntax for the queries could be greatly improved by incorporating some lessons from those domains.

Relevant work includes:

• GraphQL: complex JSON-based web API query language
• glom: library/syntax for nested search of Python dicts