bkg / django-spillway Goto Github PK

Using generic filtering on the rest framework (with the common default filter 'django_filters.rest_framework.DjangoFilterBackend') does not work, because we cannot set the filterset_fields attribute when generating the views.

For example, the following fails when filtering by a 'layer' field on my model:

# urls.py
...
urlpatterns = [
    url(r'^locations/$',
        generics.GeoListView.as_view(
            queryset=Location.objects.all(),
            filterset_fields=('layer',)
        ),
        name='location-list'),
]

The error raised is: TypeError: GeoListView() received an invalid keyword 'filterset_fields'. as_view only accepts arguments that are already attributes of the class.

This isn't strictly an issue with spillway, since django-filter isn't in the rest framework core... the docs clearly show that a view should be inherited from generic, and the filterset_fileds attribute added within your own view class.

Thus, this issue can be worked around by adding the following views.py:

from spillway import generics

class GeoDetailView(generics.GeoDetailView):
    """Generic detail view providing vector geometry representations."""
    filterset_fields = ()

class GeoListView(generics.GeoListView):
    """Generic view for listing a geoqueryset."""
    filterset_fields = ()

class GeoListCreateAPIView(generics.GeoListCreateAPIView):
    """Generic view for listing or creating geomodel instances."""
    filterset_fields = ()

class RasterDetailView(generics.RasterDetailView):
    """View providing access to a Raster model instance."""
    filterset_fields = ()

class RasterListView(generics.RasterListView):
    """View providing access to a Raster model QuerySet."""
    filterset_fields = ()

which allows quick declaration of the API endpoints in urls.py as at the top of this issue.

That's the "right" way of doing it; so isn't a bug with spillway, but it means you have to have quite a bit of boilerplate.

Suggestion / Quick fix
I'd recommend adding:

    filterset_fields = ()

as an attribute in BaseGeoView and BaseRasterView from generics.py so that the users can really quickly and easily use your example without then having to refactor to get filtering working when they hit this problem.

Hope this helps!

Uploading files through the django admin raises IOError due to non-existent file. Need to handle temporary upload files in clean_fields().

This would make it possible to work with django and https://github.com/Leaflet/Leaflet.VectorGrid using Protobuf, and make it possible to use serveral stuff from MapBox.

I thing this work would be implement a new renderer. The best way should be user Mapinik to generate protobuf from serializer, but a renderer that just convert geojson in protobuf would be nice (with litle overhead, but I think it's ok).
Well, just an inprovement sugestion. I think this can promote this project, as there is no django app that make this happens...
UTFGrid would be great too.
I think this project is a good reference: https://github.com/TileStache/TileStache but its not django.
Thanks for this great work!

I believe you determine the name of the form field referring to the geometry from the model. See filters.py lines 34, 35.

modelfield = queryset.query._geo_field()
geom_field = view.form[modelfield.name]

In my case the first line yields geometry which causes a key error since the form field is defined as

class GeometryQueryForm(SpatialQueryForm):
    """Validates vector data options."""
    geom = forms.GeometryField(required=False)
    # Tolerance value for geometry simplification
    simplify = forms.FloatField(required=False)
    srs = SpatialReferenceField(required=False)

Is this a bug or am I missing something?

Falk

I am not sure whether the approach to reproject, simplify, and define precision in a FilterBackend really works across the board. Since FilterBackends are meant for filtering QuerySets and not so much for manipulating the objects within a QuerySet.

It doesn't seem to work for DetailViews because FilterBackends are not applied. The method filter_queryset simply does not exist.

Don't you think reprojection, simplification, and precision belong into the serialization since it influences how a very flexible geometry object will be displayed eventually. That is where we had it before.

Making it part of the QuerySet has the huge advantage that PostGIS can do the work and there is also the option to consider DetailViews as lists with a single element.

We could also add the application of FilterBackends to DetailView.

I am trying to pick the serializer dependant on whether the renderer is defined in spillway or somewhere else. I need this currently to avoid re-serialization or breaking changes to the API.

That is what I came up with (and it works):

if 'spillway' in inspect.getmodule(self.request.accepted_renderer).__name__:
    # override serialization settings here
    pass
else:
    pass

Thoughts?

I might remove this later when I have a breaking v2 version of the API.

Run into problem that with, height, and pixel sizes are not populated for existing data. How can I migrate without losing existing data.

Tried to determine values by uploading example in admin. Run into some file location issues.

Or can this problem be solved on the application (BEE) level? Let's chat tomorrow.

I probably need to have some more assistance or the coupling needs to be better documented.

Please test django-spillway against the newer packages. I think there are some adaptions necessary. Thank you

from greenwich import Raster fails in models.py

Might be a versioning problem though.

I have problems to JSON serialize dates before 1970, e.g.

https://dev-ecoengine.berkeley.edu/api/layers/vtmplots/tiles/11/327/792/

Causes a TypeError datetime.datetime(1928, 7, 28, 0, 0) is not JSON serializable

See my solution for bee here:

https://github.com/berkeley-gif/bee/blob/master/bee/base/renderers.py (class MyEncoder)

File "/home/falk/projects/bee/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  137.                 response = response.render()
File "/home/falk/projects/bee/env/local/lib/python2.7/site-packages/django/template/response.py" in render
  105.             self.content = self.rendered_content
File "/home/falk/projects/bee/env/local/lib/python2.7/site-packages/rest_framework/response.py" in rendered_content
  59.         ret = renderer.render(self.data, media_type, context)
File "/home/falk/projects/bee/env/local/lib/python2.7/site-packages/spillway/renderers/renderers.py" in render
  31.         return str(self._features(data))
File "/home/falk/projects/bee/env/local/lib/python2.7/site-packages/spillway/collections.py" in __str__
  78.         props = json.dumps({k: self[k] for k in keys}, cls=JSONEncoder)[1:-1]
File "/home/falk/projects/bee/env/local/lib/python2.7/site-packages/simplejson/__init__.py" in dumps
  296.         **kw).encode(obj)

Exception Type: TypeError at /api/layers/vtmquads/features/
Exception Value: __init__() got an unexpected keyword argument 'namedtuple_as_object'

It seems to be related to this issue:

simplejson/simplejson#37

?format=json returns the format as in the database

This might be intended behaviour but is currently not consistent with Ecoengine. Let's discuss issue to make it consistent and straight forward.

The removal of pagination is really useful and provides a huge performance advantage. However, it is not so useful on my tables with up to 5 millions entries. Can this be more configurable without too much of sub-classing. Maybe we could add a property default_pagination_behavior.

As I understand the specification you can do:

{"type": "FeatureCollection",
"crs": {},
"features": {}
}

Using

gdalinfo 2003_01.01.250m_16_days_NDVI.tif

I am getting

PROJCS["SINUSOIDAL Unspecified Datum. Semi-major axis: 6371007.181000, Semi-minor axis: 0.000000",
    GEOGCS["unknown",
        DATUM["unknown",
            SPHEROID["unnamed",6371007.181,0]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Sinusoidal"],
    PARAMETER["longitude_of_center",0],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]

... which has 395 characters with indention removed. The max_length attribute is hard to override because of the abstract model class. Do you have suggestions?

Get ImportError in my application as well as in python runtests.py after cloning repo in folder.

======================================================================
ERROR: tests.test_generics (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_generics
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/falk/projects/django-spillway/tests/test_generics.py", line 10, in <module>
    from spillway import generics
  File "/home/falk/projects/django-spillway/spillway/generics.py", line 4, in <module>
    from spillway import filters, forms, mixins, renderers, serializers
  File "/home/falk/projects/django-spillway/spillway/filters.py", line 4, in <module>
    from spillway import forms
  File "/home/falk/projects/django-spillway/spillway/forms/__init__.py", line 4, in <module>
    from spillway.forms.forms import (
  File "/home/falk/projects/django-spillway/spillway/forms/forms.py", line 4, in <module>
    from spillway.forms import fields
ImportError: cannot import name fields

Same error in 5 additional instances while running runtests.py

What am I missing?

NDArrayField should work with any geometry type. Right now, points are failing because Raster.clip() expects a polygon.

ERROR: test_query_params (bee.base.tests.test_serializers.TestFieldSelectionMixin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/falk/projects/bee/bee/bee/base/tests/test_serializers.py", line 70, in test_query_params
    item, context={'request': request})
  File "/home/falk/projects/bee/env/local/lib/python2.7/site-packages/spillway/serializers.py", line 33, in __init__
    renderer = self.context['request'].accepted_renderer
AttributeError: 'WSGIRequest' object has no attribute 'accepted_renderer'

I already switched to DRF's extension of the Django test framework.

Question: Should I populate this in my tests or could you use a default in spillway if this attribute is not present?

E.g. http://localhost:8000/api/tiles/tasmax_rcp60_r1i1p1_gfdl-esm2m-2032-02-16/2/0/100/

This might be intended behavior but it causes a 500 error (would 404 be appropriate?).

Tilestache in contrast starts over again once around the world so that any integer value is valid.

Allow for more flexible upload directory handling in AbstractRasterStore subclasses.

When pagination is requested, the serializer should still use the faster database functions provided by the spatial backend.

I saw that you changed the default serialization of spillway features to GeoJSON. Wouldn't it be a better option to leave the standard behavior and json rendering of the DRF in place and set the GeoJSON renderer as default in the settings if desired? That would help me enormously to:

1. Implement spillway for all resources without changing the API fields and vocabulary.
2. To implement other flavors of JSON (TopoJSON, Google, etc.).

Falk

Add crs key to geojson if srid different from ESPG:4326.

See also return from https://ecoengine.berkeley.edu/api/observations/?srs=3310&format=geojson

Be good to see this on the REST framework grid of Django Packages... https://www.djangopackages.com/grids/g/django-rest-framework/

We could also link to it from the main docs (perhaps here?) if you provided a pull request with a brief description in the third party packages section.

I have following setup

class Model1(model.Model):
    geometry = models.GeometryField()
    objects = GeoQuerySet.as_manager()

class Model2(models.Model):
   geometry = models.GeometryField()
   related = models.ForeignKey(Model1)
   objects = GeoQuerySet.as_manager()

which causes

Exception Type: ProgrammingError at /api/ranches/andre-ranch/pastures/
Exception Value: column reference "geometry" is ambiguous
LINE 1: SELECT (geometry) AS "geometry", "api_pastures"."id", "api_p...
                ^

I believe it becomes ambiguous on the join in GeoQuerySet where you determine the column of the database but not the table. View set is ReadOnlyGeoModelViewSet

PS: I implemented a solution which I could push to a fork. Basically:

def geo_db_column(queryset):
    column = queryset._geo_field().column
    table = queryset.query.tables[0]
    return '{}.{}'.format(table, column)

in place of self.geo_field.column

PyPi now provides GDAL 1.11.0 which breaks on the currently installed libgdal. Could we freeze it to 1.10.0 to make it work across installations for now? Or should we rework all the deployment scripts (rather not since I am pretty confused by the whole libgdal/libgdal1 packaging stuff, particular on 12.04).