petrjasek / eve-elastic Goto Github PK

View Code? Open in Web Editor NEW

69.0 11.0 21.0 398 KB

Elasticsearch data layer for eve framework

License: GNU General Public License v3.0

Python 99.79% Shell 0.21%

eve-elastic's Introduction

Eve-Elastic

Eve-Elastic is elastic data layer for eve REST framework.

Features

fulltext search
filtering via elasticsearch filter dsl
facets support
aggregations support
elasticsearch mapping generator for schema

License

Eve-Elastic is GPLv3 licensed.

Supported elastic versions

It supports elastic 7 versions.

Install

$ pip install Eve-Elastic

Usage

Set elastic as your eve data layer.

import eve
from eve_elastic import Elastic

app = eve.Eve(data=Elastic)
app.run()

Config

There are 2 options for Eve-Elastic taken from app.config:

ELASTICSEARCH_URL (default: 'http://localhost:9200/') - this can be either single url or list of urls
ELASTICSEARCH_INDEX - (default: 'eve')
ELASTICSEARCH_INDEXES - (default: {}) - resource to index mapping
ELASTICSEARCH_FORCE_REFRESH - (default: True) - force index refresh after every modification
ELASTICSEARCH_AUTO_AGGREGATIONS - (default: True) - return aggregates on every search if configured for resource

Query params

Eve-Elastic supports eve like queries via where param which work as term filter.

On top of this, there is a predefined query_string query which does fulltext search.

$ curl http://localhost:5000/items?q=foo&df=name

q - query (default: *)
df - default field (default: _all)

Filtering

For more sophisticated filtering, you can use filter query param which will be used as filter for the query, using elastic filter dsl.

Facets

To add a facets support for specific resource, add facets into its datasource:

DOMAIN = {
    'contacts': {
        'datasource':
            'facets': {
                'urgency': {'terms': {'field': 'urgency'}},
                'versioncreated': {'date_histogram': {'field': 'versioncreated', 'interval': 'hour'}}
            }
        }
    }

You will find more info about facets in elasticsearch docs.

eve-elastic's People

Contributors

Stargazers

Watchers

eve-elastic's Issues

getting connection error

my code is shown below.

import eve
from eve_elastic import Elastic
app = eve.Eve(data=Elastic)

error

(venv)weber@weber-desktop:/var/www/lunar-cloud-web-ui/kukunako$ python test_server.py
Traceback (most recent call last):
File "test_server.py", line 4, in
app = eve.Eve(data=Elastic)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve/flaskapp.py", line 136, in init
self.data = data(self)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve/io/base.py", line 100, in init
self.init_app(self.app)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve_elastic/elastic.py", line 149, in init_app
self.put_mapping(app)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve_elastic/elastic.py", line 195, in put_mapping
indices.put_mapping(index=index or self.index, doc_type=resource, body=mapping, ignore_conflicts=True)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(_args, params=params, *_kwargs)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 276, in put_mapping
params=params, body=body)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))

Does eve elastic work together with mongodb?

What I mean is:
Is this extension meant to REPLACE mongodb when using eve or does it work TOGETHER? Something like when data is inserted in mongo, it is sent to be indexed by elastic...

Thanks!

How to run eve-elastic search?

I am running python eve successfully. then I am imported eve-elastic and then just added data = Elastic,
and then run my server.
Then getting connection refused error.
My eve localhost machine running on 127.0.0.1:8000 port.
Why I am getting connection refused error

How does one go about integrating eve with sqlalchemy with flask?

Correct spelling

In features list, please correct word "aggragations" to "aggregations"

404 & 405 Errors

I've encountered two big issues when attempting to make relatively simple use of this Eve backend. First, it seems items are not available at their endpoint: "/{resource}/{_id}". And second, it seems that only GET requests work.

With issues this obvious, I'm inclined to say that I'm using this wrong. So, below is a simple setup that replicates the issues:

# settings.py

RESOURCE_METHODS = ['GET','POST']
ITEM_METHODS = ['GET','PUT','PATCH','DELETE']
PUBLIC_RESOURCE_METHODS = ['GET','POST']
PUBLIC_ITEM_METHODS = ['GET','PUT','PATCH','DELETE']

DOMAIN = {
    'test': {
        'schema': {
            'foo': {
                'type': 'string'
            } 
        },
        'datasource': {
            'backend': 'elastic'
        }
    }
}

# app.py
from eve_elastic import Elastic
from eve import Eve

Eve(data=Elastic).run()

Also, here is a quick demostration with curl:

$ # POST a resource
$ curl -XPOST -H 'Content-Type: application/json' localhost:5000/test -d '{foo: bar}'
{
  "_status": "OK",
  "_updated": "Sat, 16 May 2015 20:25:53 GMT",
  "_etag": "3014835b5fe61de1d5995d3b5487d6f85dba6c68",
  "_created": "Sat, 16 May 2015 20:25:53 GMT",
  "_id": "AU1eZ4v3w-8a2P9WOYVU",
  "_links": {
    "self": {
      "title": "Test",
      "href": "test/AU1eZ4v3w-8a2P9WOYVU"
    }
  }
}
$ # Now, check that it exists
$ curl localhost:5000/test
{
  "_links": {
    "parent": {
      "title": "home",
      "href": "/"
    },
    "self": {
      "title": "test",
      "href": "test"
    }
  },
  "_items": [
    {
      "_type": "test",
      "foo": "bar",
      "_id": "AU1eZ4v3w-8a2P9WOYVU",
      "_links": {
        "self": {
          "title": "Test",
          "href": "test/AU1eZ4v3w-8a2P9WOYVU"
        }
      },
      "_etag": "3014835b5fe61de1d5995d3b5487d6f85dba6c68",
      "_created": "Sat, 16 May 2015 20:25:53 GMT",
      "_updated": "Sat, 16 May 2015 20:25:53 GMT"
    }
  ],
  "_meta": {
    "page": 1,
    "max_results": 25,
    "total": 1
  }
}
$ # Ok, now try to GET the item endpoint
$ curl localhost:5000/test/AU1eZ4v3w-8a2P9WOYVU
{
  "_status": "ERR",
  "_error": {
    "code": 404,
    "message": "The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again."
  }
}
$ # Ok, that doesn't work??

So, I tried adding an additional lookup:

# New settings.py
RESOURCE_METHODS = ['GET','POST']
ITEM_METHODS = ['GET','PUT','PATCH','DELETE']
PUBLIC_RESOURCE_METHODS = ['GET','POST']
PUBLIC_ITEM_METHODS = ['GET','PUT','PATCH','DELETE']

DOMAIN = {
    'test': {
        'schema': {
            'foo': {
                'type': 'string'
            } 
        },
        'additional_lookup': {
            'url': 'regex("[\w]+")',
            'field': 'foo'
        },
        'datasource': {
            'backend': 'elastic'
        }
    }
}

Now, the item endpoint works. But, attempting to PUT, PATCH, DELETE doesn't.

$ # Now the item endpoint works
$ curl localhost:5000/test/bar
{
  "_status": "OK",
  "_updated": "Sat, 16 May 2015 20:25:53 GMT",
  "_etag": "3014835b5fe61de1d5995d3b5487d6f85dba6c68",
  "_created": "Sat, 16 May 2015 20:25:53 GMT",
  "_id": "AU1eZ4v3w-8a2P9WOYVU",
  "_links": {
    "self": {
      "title": "Test",
      "href": "test/AU1eZ4v3w-8a2P9WOYVU"
    }
  }
}
$ # But, attempting to PUT, PATCH, DELETE doesn't
$ curl -XPUT -H 'Content-Type: application/json' -H 'If-Match: 3014835b5fe61de1d5995d3b5487d6f85dba6c68' localhost:5000/test/bar -d '{"foo: rab"}'
{
  "_status": "ERR",
  "_error": {
    "message": "The method is not allowed for the requested URL.",
    "code": 405
  }
}

Schema: object hierarchy

Hi,

I have defined a schema in Eve to be used with a collection of products.
In this schema i added an 'images' attribute of type list of objects. It is expressed as follows :

    'name': {
        'type': 'string',
        'minlength': 3,
        'maxlength': 150,
        'required': True,
    },
    'images': {
        'type': 'list',
        'schema': {
            'type': 'dict',
            'schema': {
                'name': {
                    'type': 'string',
                    'required': True
                },
                'url': {
                    'type': 'string',
                    'required': True
                }
            }
        }
    },

This is a valid Cerberus format in Eve.
Inserting an object with a POST request works fine with a mongodb backend.

Request example:

POST on http://localhost:5000/products
with body containing :

{
  "name": "bird",
  "images": [{
    "name": "bird1",
    "url": "http://localhost/images/bird1.jpg"
  },
  {
    "name": "bird2",
    "url": "http://localhost/images/bird2.jpg"
  }]
}

With mongodb i receive a 201 response status whereas with elasticsearch i get a 500 response error and the data is not inserted nor the index created.

Am i missing something ?
Is any additional configuration required to make this work ?

Thanks

There is no tag for release 2.4

Mention PyMongo dependency

Apparently, this package needs PyMongo (https://pypi.python.org/pypi/pymongo), but doesn't say so. And from looking at the code (which imports bson.objectid.ObjectID) it's not evident that another package named bson is not the one needed (https://pypi.python.org/pypi/bson), which causes confusion. The README then should also point out, why PyMongo is really needed, since some people (including me) would not expect this if they want to use Eve with Elasticsearch instead of MongoDB.

Invite to join the @pyeve org

Hi, FYI I created the @pyeve organization and moved all eve repositories over there. If you feel like doing that, you can join the org and move eve-elastic there too (you'd still be maintainer of course)

Is there any support for nested documents?

And if yes, is there any tests/examples.

thanks!

Error on dependency ciso8601

When I install, its throw following error on ciso8601 package.

Collecting ciso8601>=1.0.2 (from Eve-Elastic==2.1->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/61/d8/e17159b6037a26e3facdfeec8750dd17312284a915c5e0da79562b982035/ciso8601-2.0.1.tar.gz
    Complete output from command python setup.py egg_info:
    error in ciso8601 setup command: 'tests_require' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in unittest2 ; python_version < '3' at  ; python_version < '3'

How to connect in a elasticsearch cluster?

Error when update or delete an item

Update: Problem with DELETE resolved after dowgrading to Eve 0.7.10 where common.get_document() still passed only 3 arguments to app.data.find_one() but still persist with PUT/PATCH.

In my simple app, I can create and get new items just fine but when I tried a PATCH/PUT/DELETE request then I got the error:

TypeError: find_one() takes 3 positional arguments but 5 were given

The error doesn't appear when Mongodb is used instead of Elasticsearch.

My following setup and code:

Python version: 3.6
Elasticsearch version: 2.4.6
Eve version: 0.8.1
Eve-Elastic version: 2.5.0

Source code

# settings.py
import os

ELASTICSEARCH_URL = os.environ.get("ELASTICSEARCH_URL", "http://localhost:9200")
ELASTICSEARCH_INDEX = os.environ.get("ELASTICSEARCH_INDEX", "eve")

RESOURCE_METHODS = ["GET", "POST"]
ITEM_METHODS = ["GET", "PATCH", "PUT", "DELETE"]

DOMAIN = {
    "test": {
        "schema": {"foo": {"type": "string"}},
        "item_url": 'regex("[\w,.:_-]+")',
        "datasource": {"backend": "elastic"},
    },
}

# run.py
from eve import Eve
from eve_elastic import Elastic

if __name__ == "__main__":
    Eve(data=Elastic).run()

API Call

curl -X DELETE \
  http://localhost:5000/test/AWjlAAzY-Wkn3SL9OgCa \
  -H 'If-Match: 16e79eb4b035c2d25b08f5bb46eeadd8002e4230'

Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/eve/flaskapp.py", line 1105, in __call__
    return super(Eve, self).__call__(environ, start_response)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/eve/endpoints.py", line 100, in item_endpoint
    response = deleteitem(resource, **lookup)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/common.py", line 317, in rate_limited
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/eve/auth.py", line 80, in decorated
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/common.py", line 1251, in decorated
    r = f(resource, **combined_args)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/delete.py", line 46, in deleteitem
    return deleteitem_internal(resource, concurrency_check=True, **lookup)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/delete.py", line 97, in deleteitem_internal
    original = get_document(resource, concurrency_check, original, **lookup)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/common.py", line 88, in get_document
    resource, req, check_auth_value, force_auth_field_projection, **lookup
TypeError: find_one() takes 3 positional arguments but 5 were given

It was tested with new versions of Elasticsearch like 6 and 7?

can't pass query parameters to elastic search

I installed the latest eve via pip install, when I try the example curl http://localhost:5000/items?q=foo&df=name i noticed the query parameter foo was never send to elastic search. By tracing the code, I found this line 'args = getattr(req, 'args', {})' in method find() returns null, actually there is no 'args' attribute of ParsedRequest req. It seems eve only accept parameter 'where' or the alias defined by QUERY_WHERE. Anything necessary configuration that I missed or this is a defect?

Error with simple first app

I am trying to run simple basic app to get eve_elastic running.
I have following settings.py and app.py files.
When I do a request:
curl localhost:5000/test

I get following error, here I am assuming eve_elastic would create test index in ES

# settings.py

RESOURCE_METHODS = ['GET','POST']
ITEM_METHODS = ['GET','PUT','PATCH','DELETE']
PUBLIC_RESOURCE_METHODS = ['GET','POST']
PUBLIC_ITEM_METHODS = ['GET','PUT','PATCH','DELETE']

DOMAIN = {
    'test': {
        'schema': {
            'foo': {
                'type': 'string'
            } 
        },
        'datasource': {
            'backend': 'elastic'
        }
    }
}

# app.py
from eve_elastic import Elastic
from eve import Eve

Eve(data=Elastic).run()

Call

a@a-virtual-machine ~/api/eve/app1 $ curl localhost:5000
{"_links": {"child": [{"href": "test", "title": "test"}]}}
a@a-virtual-machine ~/api/eve/app1 $ 

a@a-virtual-machine ~/api/eve/app1 $ curl localhost:5000/test
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

Error Serverside

(env) a@a-virtual-machine ~/api/eve/app1 $ python app.py 
WARNING:elasticsearch:GET /eve/test/_search [status:400 request:0.054s]
[2018-01-29 12:03:34,248] ERROR in app: Exception on /test [GET]
Traceback (most recent call last):
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/endpoints.py", line 54, in collections_endpoint
    response = get(resource, lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/common.py", line 284, in rate_limited
    return f(*args, **kwargs)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/auth.py", line 79, in decorated
    return f(*args, **kwargs)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/common.py", line 1040, in decorated
    r = f(resource, **combined_args)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/get.py", line 39, in get
    return get_internal(resource, **lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/get.py", line 112, in get_internal
    return _perform_find(resource, lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/get.py", line 184, in _perform_find
    cursor = app.data.find(resource, req, lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve_elastic/elastic.py", line 481, in find
    hits = self.elastic(resource).search(body=query, **args)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 539, in search
    doc_type, '_search'), params=params, body=body)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 110, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 114, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'parsing_exception', u'no [query] registered for [filtered]')

"where" searches

When using the
/endpoint?where=field=="value"
or
/endpoint?where={"field":"value"}
The data returned does not appear filtered. However, using the q=value syntax, the search appears to work normally.

I can do the same filters using mongoDB as the backend with the same data.