See also: planetfederal#43

Currently layer import fails without GeoServer handlers listed in IMPORT_HANDLERS - my understanding is that django-osgeo-importer should also work with only GeoNode & not GeoServer available.

Not sure if this is a pending feature or just lack of documentation - either way, let's fix.

Low priority, and may involve angular-related complications, but this could be a quality of life issue for anyone who has to use importer heavily.

originally planetfederal#12

There is a known issue that makes OGR generate a "General Error" dealing with polygons near the equator:

http://lists.osgeo.org/pipermail/gdal-dev/2011-June/028939.html

Apparently it may be fixable by a 'tolerance' parameter.

This file exhibits the problem:

TM_WORLD_BORDERS_2005.zip

I looked to see where to manage test fixtures for the module but couldnt see it in the docs or file structure - maybe its more assumed django knowledge.

Also - there seems to be no good reason not to propagate the Exception thrown back to the client - maybe it is expected that the caller of the importer should handle this - and in this case MapStory is ignoring it - but a quick note explaining how to propagate diagnostics would make sense in the context of an "extensible" module.

Create an import handler which will configure data from vector layers to be provided by tegola server.
This handler should work without conflict with geoserver handlers for the same layeres.

test_wfs & test_arcgisjson are failing regularly in pull requests not due to changes in code but because the servers they contact aren't available. I propose mocking the web requests so test results are determined only by the state of the code.

Using /one-shot-demo/ to upload deleteme.zip worked previously but now fails. I think the problem was introduced with commit 9187ab0.

Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/jivan/.projects/django-osgeo-importer/osgeo_importer/utils.py", line 550, in import_all_layers
upload_layer.upload_file.id, configuration_options=configuration_options
File "/home/jivan/.virtualenvs/django-osgeo-importer/local/lib/python2.7/site-packages/celery/app/task.py", line 453, in delay
return self.apply_async(args, kwargs)
File "/home/jivan/.virtualenvs/django-osgeo-importer/local/lib/python2.7/site-packages/celery/app/task.py", line 551, in apply_async
link=link, link_error=link_error, **options)
File "/home/jivan/.virtualenvs/django-osgeo-importer/local/lib/python2.7/site-packages/celery/app/task.py", line 743, in apply
request=request, propagate=throw)
File "/home/jivan/.virtualenvs/django-osgeo-importer/local/lib/python2.7/site-packages/celery/app/trace.py", line 355, in eager_trace_task
uuid, args, kwargs, request)
File "/home/jivan/.virtualenvs/django-osgeo-importer/local/lib/python2.7/site-packages/celery/app/trace.py", line 253, in trace_task
I, R, state, retval = on_error(task_request, exc, uuid)
File "/home/jivan/.virtualenvs/django-osgeo-importer/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/jivan/.projects/django-osgeo-importer/osgeo_importer/tasks.py", line 87, in import_object
gi.handle(configuration_options=configuration_options, request_cookies=request_cookies, request_user=request_user)
File "/home/jivan/.projects/django-osgeo-importer/osgeo_importer/importers.py", line 116, in handle
config['handler_results'] = self.run_import_handlers(layer, config, **kwargs)
File "/home/jivan/.projects/django-osgeo-importer/osgeo_importer/importers.py", line 131, in run_import_handlers
self.handler_results.append({type(handler).name: handler.handle(layer, layer_config, *args, **kwargs)})
File "/home/jivan/.projects/django-osgeo-importer/osgeo_importer/handlers/init.py", line 20, in func_wrapper
return func(self, *args, **kwargs)
File "/home/jivan/.projects/django-osgeo-importer/osgeo_importer/handlers/geoserver/init.py", line 240, in handle
store = self.get_or_create_datastore(layer_config, request_user)
File "/home/jivan/.projects/django-osgeo-importer/osgeo_importer/handlers/geoserver/init.py", line 150, in get_or_create_datastore
if connection_string['type'] == 'geogig':
TypeError: 'NoneType' object has no attribute 'getitem'

error while trying to load library

please help.!!

ERROR: Command errored out with exit status 1:

 command: 'c:\users\hp\desktop\venv\scripts\python.exe' -c 'import sys, s

etuptools, tokenize; sys.argv[0] = '"'"'C:\Users\Hp\AppData\Local\Temp\
pip-install-_ogh68dm\gsconfig\setup.py'"'"'; file='"'"'C:\Users\Hp\A
ppData\Local\Temp\pip-install-_ogh68dm\gsconfig\setup.py'"'"';f=getattr(
tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"
', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg
_info --egg-base 'C:\Users\Hp\AppData\Local\Temp\pip-install-_ogh68dm\gsconfi
g\pip-egg-info'
cwd: C:\Users\Hp\AppData\Local\Temp\pip-install-_ogh68dm\gsconfig
Complete output (6 lines):
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Hp\AppData\Local\Temp\pip-install-_ogh68dm\gsconfig\setu
p.py", line 8
except IOError,e:
^
SyntaxError: invalid syntax
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check
the logs for full command output.

This is a result of the initial implementation being a part of a GeoNode application. This should be something more abstract, maybe just a Django setting.

Include functionality to translate string or numerical formatted fields for time to ISO 8601 standards.

For example a year field with the format of 1986.00 with data type as double, the user would enter the format key of YYYY.00 to produce the ISO 8601 value of 1986-01-01T00:00:00Z.

Example use cases:

• US vs European conventions on month and day order MM-DD-YYYY vs DD-MM-YYYY.
• Time fields that are typed as numerical or string data types to the data type expected by importer.
• Non-Standard date formats such as 9/20/23:30 to become 2020-09-01T23:30:00Z if defined by the key MM/YY/hh:mm

Since FGDB is/can be a directory, need to handle for zipping/unzipping.

In django-osgeo-importer, visit /uploads/new and click Choose File. Select the file linked below. It should continue to the next step of the upload process but halts with a UnicodeDecodeError.

https://eventkit.s3.amazonaws.com/0e473d86-6953-47d0-8dda-721b94039e1d/osm-vector/portauprince.gpkg

Handlers interact through the state of the persistence layer - for example configuring geoserver relies on the data uploaded by a previous handler.
Each handler should have its pre-conditions and post-conditions documented so future extensions can be integrated at different points in the chain as required. This will also provide a template to allow third party contributions for handlers to be assessed,

importer is extensible by design, but exactly how to add a new handler is not visible in documentation - does an extension need to be included in a fork of the project, or is it possible for another project to register handlers in some way? Currently its not very clear exactly what a handler should do - does it modify the uploaded content, or generate a new artefact that other handlers can use? How do they communicate? Some examples seem to generate metadata in the catalog - so what exactly is the architecture here? What does a handler which normalises content for example.

An example on adding a handler would be best.

Scenario - in the MapStory roadmap is a function to apply a set of attributes to an existing geometry - could one build a handler which takes a geometryless table and builds a view that joins it to an existing layer?

The file uploads successfully after making a few changes noted below but fails to import because its unable to open the file:

Request URL:http://mydomain.com/importer-api/data/152/

Response
{
   "complete":true,
   "date":"2016-04-03T06:39:34.878654",
   "file_size":"7.6KB",
   "file_type":null,
   "file_url":"https://static.s3.amazonaws.com/media/uploads/Sightings.zip",
   "id":152,
   "layers":[

   ],
   "metadata":"",
   "name":"Sightings.zip",
   "resource_uri":"/importer-api/data/152/",
   "size":7798,
   "state":"UPLOADED",
   "upload_dir":null,
   "user":"/api/profiles/1/"
}

This exception is raised and not handled gracefully by the caller.

https://github.com/ProminentEdge/django-osgeo-importer/blob/master/osgeo_importer/inspectors.py#L148

By handling the exception here:
https://github.com/ProminentEdge/django-osgeo-importer/blob/master/osgeo_importer/views.py#L65

and here:

https://github.com/ProminentEdge/django-osgeo-importer/blob/master/osgeo_importer/views.py#L87

and setting a default value for description in case of exception allowed the 200 Response referenced above. As noted in the JSON response, there are values missing but the url to the uploaded file is valid. A possible solution could be for the importer to leverage the storage backend to ensure it gets a handle on the file regardless of local or remote storage configuration.

from django.core.files.storage import default_storage as storage

layers = self.import_file(configuration_options=configuration_options)
  File "/home/mapstory/.virtualenvs/mapstory/src/osgeo-importer/osgeo_importer/importers.py", line 225, in import_file
    if feature.geometry().GetGeometryType() != target_layer.GetGeomType() and \
AttributeError: 'NoneType' object has no attribute 'GetGeometryType' 

test case: http://beta.mapstory.org/uploaded/uploads/Spring2015_VJYuMxp.csv

If users shoot themselves in the foot with a bad handler config (or if we shoot them in the foot with a bad default handler config, or documented handler config) then the UI seems to function as normal, the celery task reports success, etc. but nothing actually happens. For example, if there are no handlers or the handlers in the list are not actually doing anything, there is apparently nothing to detect and report that condition, so users simply see that importer did nothing while claiming to function as normal.

The 'layer' argument to import handlers comes from UploadLayer.name and can be a number of different values, including a name, a path, a geoserver store:name combo. Let's normalize this to an UploadLayer instance; from that, things like a name or a path can be easily grabbed & will make for clearer code.

Many packages & zip files contain extra files that shouldn't/can't be processed as layer data. These should be ignored instead of causing errors.

Examples:

• .shp.xml .sbn from shapefile .zips
• __MACOSX directory from zips made on Macs.
• .txt files with descriptions, licenses, or metadata

After import (and presumably some error - geoserver layer is created but not configured correctly - the importer is left repeatedly calling
/importer-api/data-layers//

manual inspection has this containing:
status: "PENDING",
task_id: "efc49592-62a7-4cfa-afad-92a229428595"

celery -A mapstory inspect active shows no active tasks.

whatever happens in import, the task status should not be left unresolved and the system polling forever.

https://pypi.python.org/pypi/django-osgeo-importer/

@bitner can you look into it?

Geonode 2.7.x uses Angular 1.4.0. ref

Whenever django-osgeo-importer is ready to bump from Geonode 2.6.x to 2.7.x we'll also need to bump Angular. Making note of this, and offering to handle that part of the bump when the time comes.

For our MapStory project using this importer we've created a branch to bump this importer from Angular 1.3 to 1.6 here and I've included all backwards compatible changes in the importer master branch via c17407c.

The bump to 1.4 will require bumping angular-resource, angular-cookies, and angular-mocks in the package.json and committing them in the /static/osgeo_importer/js/ directory because they are included in the {% _importer_scripts.html %} template assets.

I want to use django-osgeo-importer with geonode but this error appear when i try to upload any layer?

| Traceback (most recent call last):
|   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
|     response = wrapped_callback(request, *callback_args, **callback_kwargs)
|   File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
|     return view_func(request, *args, **kwargs)
|   File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
|     return self.dispatch(request, *args, **kwargs)
|   File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
|     return handler(request, *args, **kwargs)
|   File "/usr/local/lib/python2.7/site-packages/django/views/generic/edit.py", line 214, in post
|     if form.is_valid():
|   File "/usr/local/lib/python2.7/site-packages/django/forms/forms.py", line 184, in is_valid
|     return self.is_bound and not self.errors
|   File "/usr/local/lib/python2.7/site-packages/django/forms/forms.py", line 176, in errors
|     self.full_clean()
|   File "/usr/local/lib/python2.7/site-packages/django/forms/forms.py", line 393, in full_clean
|     self._clean_form()
|   File "/usr/local/lib/python2.7/site-packages/django/forms/forms.py", line 417, in _clean_form
|     cleaned_data = self.clean()
|   File "/usr/local/lib/python2.7/site-packages/osgeo_importer/forms.py", line 64, in clean
|     if not validate_inspector_can_read(cleaned_file_path):
|   File "/usr/local/lib/python2.7/site-packages/osgeo_importer/validators.py", line 51, in validate_inspector_can_read
|     data, inspector = importer.open_source_datastore(filename)
|   File "/usr/local/lib/python2.7/site-packages/osgeo_importer/importers.py", line 160, in open_source_datastore
|     return self.open_datastore(connection_string, self.source_inspectors, *args, **kwargs)
|   File "/usr/local/lib/python2.7/site-packages/osgeo_importer/importers.py", line 148, in open_datastore
|     data = insp.open()
|   File "/usr/local/lib/python2.7/site-packages/osgeo_importer/inspectors.py", line 145, in open
|     self.data = gdal.OpenEx(filename, open_options=open_options)
| AttributeError: 'module' object has no attribute 'OpenEx'

settings:

......
INSTALLED_APPS += ('osgeo_importer',)
#### osgeo_importer settings
OSGEO_DATASTORE = 'datastore'
OSGEO_IMPORTER_GEONODE_ENABLED = True
OSGEO_IMPORTER_VALID_EXTENSIONS = [
    'shp', 'shx', 'prj', 'dbf', 'kml', 'geojson', 'json', 'tif', 'tiff',
    'gpkg', 'csv', 'zip', 'xml', 'sld'
]
IMPORT_HANDLERS = [
    # If GeoServer handlers are enabled, you must have an instance of geoserver running.
    # Warning: the order of the handlers here matters.
    'osgeo_importer.handlers.FieldConverterHandler',
    'osgeo_importer.handlers.geoserver.GeoserverPublishHandler',
    'osgeo_importer.handlers.geoserver.GeoserverPublishCoverageHandler',
    'osgeo_importer.handlers.geoserver.GeoServerTimeHandler',
    'osgeo_importer.handlers.geoserver.GeoWebCacheHandler',
    'osgeo_importer.handlers.geoserver.GeoServerBoundsHandler',
    'osgeo_importer.handlers.geoserver.GenericSLDHandler',
    'osgeo_importer.handlers.geonode.GeoNodePublishHandler',
#     'osgeo_importer.handlers.mapproxy.publish_handler.MapProxyGPKGTilePublishHandler',
    'osgeo_importer.handlers.geoserver.GeoServerStyleHandler',
    'osgeo_importer.handlers.geonode.GeoNodeMetadataHandler'
]

PROJECTION_DIRECTORY = os.path.join(PROJECT_ROOT, "data")
DATABASES['datastore']= {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'test_datastore',
        'USER' : 'postgres',
        'PASSWORD' : 'root',
        'HOST' : 'localhost',
        'PORT' : '5432',
    }
.....

Unless someone uses the settings from _prj, they will not get any results, but the celery task will appear to succeed. This is really unfriendly behavior.

If we want the default config not to depend on geoserver per #58 that is fine, but the shipping config should do something.

Hi,

most of you probably know that I’m working on a new Importer with React.

We’re currently working on trying to make it easy configurable, and I came up with a few ideas how this could look like.

I would like your input on my current idea.
Thanks.

Idea

Have a config object, that is loaded at runtime and can be edited for each instance. There is a default which only handles name changes.

The default config would look like this:

config =  {
  steps: {
    1: {
      title: 'Layer Name'
      fields: [
        { title: '', api_name: 'layer_name', type: 'text'}
      ]
    }
  }
};

The basic idea is that each step in the create layer window is represented by it’s position number.
Each Step has a title and fields. The Layername step only has one item. But for example the Configure time item would have multiple.

Here is a more detailed example

var config = {
  steps: {
    1: {
      title: 'Layer Name',
      fields: [
        { title: 'Layer', api_name: 'layer_name', type: 'text'}
      ]
    },
    2: { 
      title: 'Dates', 
      fields: [
        {title: 'Start Date', api_name: 'start_date', type: 'fields'},
        {title: 'End Date', api_name: 'end_date', type: 'fields'},
        {title: '', api_name: 'configureTime', type: 'hidden', value: true, parent: 'start_date‘}
        {title: '', api_name: 'convert_to_date', type: 'hidden', value: (d) => { return [d.start_date]}, parent: 'start_date'}
    ]},
    3: {
      title: 'Version Control',
      fields: [
        {title: '', api_name: 'editable', type: 'switch', values: [true, false]},
        {title: '', api_name: 'geoserver_store', type: 'hidden', value: {'type': 'geogig'}, parent: 'editable'}
    ]}
  }
}

As you can see there is even a idea about adding functions to it. They would be called with the results.
Hidden fields provide a way to add values to the configuration without displaying them.
The parent item ensures that this field only get’s added when the parent is filled.

What do you think? Feedback very much welcome and appreciated.

trying to find what formats are handled - and in the context of an extensible module - hers and improvement idea:

provide a means for the handlers to report what formats they support and some way to provide user documentation. If the handlers do this and the module provides the API to expose it to user apps then reusable documentation can be bundled with any third party extensions.

Docs might be scoped to multiple levels:

1. packaging (i.e. requirements for a zip file - such as internal structure assumptions
2. file format - extension, encodings etc
3. fields
4. process
5. outputs (user visible)

intermediate handlers the user doesnt directly interact with might only populate process documentation to record what is being done.

Originally filed at planetfederal#24, the following was written by @sarasafavi

Given the following files:

foo.zip: contains a copy of bar.zip
bar.zip: contains baz.tif, etc

The following traceback occurs:

IOError at /uploads/new
[Errno 2] No such file or directory: u'/tmp/tmpeh7O5t/bar.zip'
Request Method: POST
Request URL:    http://exchange/uploads/new
Django Version: 1.8.15
Exception Type: IOError
Exception Value:    
[Errno 2] No such file or directory: u'/tmp/tmpeh7O5t/bar.zip'
Exception Location: /usr/local/lib/python2.7/shutil.py in copyfile, line 82
Python Executable:  /env/bin/python
Python Version: 2.7.11
Python Path:    
['/mnt/exchange',
 '/env/lib/python27.zip',
 '/env/lib/python2.7',
 '/env/lib/python2.7/plat-linux2',
 '/env/lib/python2.7/lib-tk',
 '/env/lib/python2.7/lib-old',
 '/env/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7',
 '/usr/local/lib/python2.7/plat-linux2',
 '/usr/local/lib/python2.7/lib-tk',
 '/env/lib/python2.7/site-packages',
 '/mnt/geonode',
 '/mnt/importer',
 '/mnt/exchange',
 '/mnt/exchange',
 '/mnt/exchange']
Server time:    Mon, 26 Sep 2016 17:18:35 -0500

Environment:


Request Method: POST
Request URL: http://exchange/uploads/new

Django Version: 1.8.15
Python Version: 2.7.11
Installed Applications:
('djmp',
 'hypermap.aggregator',
 'hypermap.dynasty',
 'hypermap.search',
 'hypermap',
 'appearance',
 'flat',
 'exchange.core',
 'geonode',
 'geonode.contrib.geogig',
 'geonode.contrib.slack',
 'django_classification_banner',
 'maploom',
 'solo',
 'haystack',
 'corsheaders',
 'osgeo_importer',
 'modeltranslation',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.admin',
 'django.contrib.sitemaps',
 'django.contrib.staticfiles',
 'django.contrib.messages',
 'django.contrib.humanize',
 'django.contrib.gis',
 'pagination',
 'taggit',
 'treebeard',
 'friendlytagloader',
 'geoexplorer',
 'leaflet',
 'django_extensions',
 'autocomplete_light',
 'mptt',
 'djcelery',
 'storages',
 'pinax_theme_bootstrap_account',
 'pinax_theme_bootstrap',
 'django_forms_bootstrap',
 'account',
 'avatar',
 'dialogos',
 'agon_ratings',
 'announcements',
 'actstream',
 'user_messages',
 'tastypie',
 'polymorphic',
 'guardian',
 'geonode.people',
 'geonode.base',
 'geonode.layers',
 'geonode.maps',
 'geonode.proxy',
 'geonode.security',
 'geonode.social',
 'geonode.catalogue',
 'geonode.documents',
 'geonode.api',
 'geonode.groups',
 'geonode.services',
 'geonode.geoserver',
 'geonode.upload',
 'geonode.tasks')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'pagination.middleware.PaginationMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'geonode.security.middleware.LoginRequiredMiddleware',
 'geonode.security.middleware.LoginRequiredMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware')


Traceback:
File "/env/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  132.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/env/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  22.                 return view_func(request, *args, **kwargs)
File "/env/lib/python2.7/site-packages/django/views/generic/base.py" in view
  71.             return self.dispatch(request, *args, **kwargs)
File "/env/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
  89.         return handler(request, *args, **kwargs)
File "/env/lib/python2.7/site-packages/django/views/generic/edit.py" in post
  215.             return self.form_valid(form)
File "/mnt/importer/osgeo_importer/views.py" in form_valid
  177.             shutil.move(each.name, tofile)
File "/usr/local/lib/python2.7/shutil.py" in move
  302.             copy2(src, real_dst)
File "/usr/local/lib/python2.7/shutil.py" in copy2
  130.     copyfile(src, dst)
File "/usr/local/lib/python2.7/shutil.py" in copyfile
  82.     with open(src, 'rb') as fsrc:

Exception Type: IOError at /uploads/new
Exception Value: [Errno 2] No such file or directory: u'/tmp/tmpeh7O5t/bar.zip'

UploadLayer should only relate to UploadedData through UploadFile.

file names and other user input are reflected in resource names and used in SQL statements. These need to be slug-ified or other approaches to make sure malicious SQL cannot be injected, or simple syntax errors generated.
Check for clashes with SQL reserved keywords as well.

Hi,

I'm getting the following error on pip install:

error: can't copy 'osgeo_importer/static/osgeo_importer/img/one_shot_demo': doesn't exist or not a regular file

The error appears both when installing from this repo and from pypi.

Cheers

/uploads/new endpoint does now work for uploading

https://github.com/GeoNode/django-osgeo-importer/blob/master/osgeo_importer/handlers/geoserver/__init__.py#L177

The above line continues the loop as long as the status is not 'FINISHED', however, this allows for unnecessary compute cycles when a terminal status such as 'FAILED' is returned which should be handled so that client is updated immediately that the import failed rather than waiting for Geoserver to abort the task after an unspecified length of time. The sequence of events I've been able to extract from the logs are as follows:

1. Several dozen request, regardless of the size of the dataset, are made to the Geoserver /tasks/.json endpoint each returning a 200 with the payload seen in the example below.
2. Eventually Geoserver clears the SecurityContextHolder
3. 404 is returned with a Task Not Found message
4. Importer fails

Below is an example response from Geoserver with the FAILED status.

{
"task": {
"id": 11,
"status": "FAILED",
"transactionId": "91a0f517-1fcf-465e-861c-aa0993c8c4b7",
"description": "postgis import table yoyoma_events into repository: postgresql://localhost:5432/geogig_data/public/cdavis-layers?user=<user>&password=<pwd>",
"href": "http://example.io/geoserver/geogig/tasks/11.json",
"error": {
"message": "Unable to obtain connection: ERROR: function postgis_lib_version() does not exist\n  Hint: No function matches the given name and argument types. You might need to add explicit type casts.\n  Position: 8",
"stackTrace": "java.lang.RuntimeException: Unable to obtain connection: ERROR: function postgis_lib_version() does not exist\n  Hint: No function matches the given name and argument types. You might need to add explicit type casts.\n  Position: 8\n\tat org.geotools.jdbc.JDBCDataStore.createConnection(JDBCDataStore.java:2078)\n\tat org.geotools.jdbc.JDBCDataStore.createTypeNames(JDBCDataStore.java:978)\n\tat org.geotools.data.store.ContentDataStore.entry(ContentDataStore.java:582)\n\tat org.geotools.data.store.ContentDataStore.ensureEntry(ContentDataStore.java:618)\n\tat org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:393)\n\tat org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:360)\n\tat org.geotools.data.store.ContentDataStore.getSchema(ContentDataStore.java:344)\n\tat org.locationtech.geogig.geotools.plumbing.ImportOp.checkPreconditions(ImportOp.java:360)\n\tat org.locationtech.geogig.geotools.plumbing.ImportOp._call(ImportOp.java:146)\n\tat org.locationtech.geogig.geotools.plumbing.ImportOp._call(ImportOp.java:79)\n\tat org.locationtech.geogig.repository.AbstractGeoGigOp.call(AbstractGeoGigOp.java:153)\n\tat org.locationtech.geogig.rest.AsyncContext$CommandCall.call(AsyncContext.java:240)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.postgresql.util.PSQLException: ERROR: function postgis_lib_version() does not exist\n  Hint: No function matches the given name and argument types. You might need to add explicit type casts.\n  Position: 8\n\tat org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2270)\n\tat org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1998)\n\tat org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:570)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:406)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:286)\n\tat org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)\n\tat org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)\n\tat org.geotools.data.postgis.PostGISDialect.getVersion(PostGISDialect.java:1198)\n\tat org.geotools.data.postgis.PostGISDialect.initializeConnection(PostGISDialect.java:237)\n\tat org.geotools.jdbc.JDBCDataStore.createConnection(JDBCDataStore.java:2069)\n\t... 17 more\n"
}
}
}

When importing layers from some gpkg files the map which is shown in the UI is a global map of the world instead of a map matching the bounding box of the layer.

The layer created in geoserver at: https://github.com/GeoNode/django-osgeo-importer/blob/master/osgeo_importer/handlers/geoserver/__init__.py#L167 has bad bounding limits for layers from the file which shows the problem; this is probably why the map is global.

san_diego_downtown_generic-osm-data-20161202.gpkg: Shows problem
tiger_roads.gpkg: No Problem
portauprince.fixedcolnames.gpkg: No problem; has multiple layers & rtrees like problem gpkg.

example_gpkgs.zip

This may be N/A or infeasible, but it was an idea we had to improve the information user receives on what is happening "in the background" and maybe mitigate any perceptions that importer tends to fail mysteriously

This issue was previously planetfederal#14

import_by_path is deprecated.

RemovedInDjango19Warning: import_by_path() has been deprecated. Use import_string() instead.
  OSGEO_IMPORTER = import_by_path(OSGEO_IMPORTER)

Is this feasible? What would be required? The idea here is that for apps which support storing documents, users may just want to sling documents in along with other kinds of files. Exchange will need to supplement importer with a document upload capability in the near future (e.g. via a separate tab on its uploads page), but it might be nicer for users to include it all in the same importer interface.

The importer should support the ability to import tilesets from geopackages and configure them to be served in the GeoNode instance using django-mapproxy

No reason to reference the layer name in more than one place. As long as no handlers modify the layer name, this should be a quick fix.

During import:

• Grab projection type, resolution via gdal.
• Include that data along with file location in ImportLayer or as part of config passed to handlers.

Add new django-mapproxy handler which:

• Copy the uploaded gpkg to [yet-to-be-chosen-location]
• Create djmp models.Tileset from details collected during import
• Do something to let django-mapproxy know to serve the new layer (Tileset.seed()?)

If an underlying database issue arises during import there should be some way to propagate this to the administrator at least - via some logs perhaps - and configuration errors should be propagated to the end user.
A verbosity switch in the configuration may be the best way to deal with this.

This then becomes a key part of documentation for building a plug-in handler - what and how to propagate diagnostics for different verbosity levels.

When importing a layer with global bounds, the estimated space needed is more than a petabyte.
The same data with tight bounds & empty levels removed imports without problem.

Per conversation with Joseph Svrcek, what's likely causing the issue:
So the specifications don't necessarily require tight bounds on the bounding box. So I made the bounds global. There is also no requirement on the number of levels in the geopackage so I prepopulated all of them. If the estimator is saying, Hmm this geopackage has 20 layers and global bounds then it would guess that there was 366,503,875,925 tiles in the geopackage....https://wiki.openstreetmap.org/wiki/Tile_disk_usage

estimation_gpkgs.zip

Right now in the Exchange integration it is taking a fair amount of work to ensure that this importer substitutes in for the default import routes. It would be nice if we could reduce that to just a typical urlpatterns += thing.

originally planetfederal#10

All of the test data is now public-read, however the setup steps will fail if AWS keys are not present.

When uploading a file to process, long filenames (specifically those > 64 characters long) yield value too long for type character varying(64)

https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits excluding ReiserFS/Reiser4 general consensus is 255 chars