Result can be made more intuitive and user-friendly by using something like https://github.com/cdgriffith/Box to 'wrap' the result data dict.

(Perhaps this could be useful for Response as well?)

The idea is to pass in a version argument in order to alter the results of get_document, get_es, & possibly get_type_name. The best way I see this working out is to add the argument with default set to None.

Adding such an enhancement would allow anyone to facilitate an ES transition/migration as they needed it.

See the docs.

Need to add non-model based examples, in-code arbitrary individual index/delete and bulk indexing calls, search form usage.

ElasticsearchIndexMixin is designed to support a stand-alone class or a Model-based class - so either of the following is intended:

class MyFancyIndex(ElasticsearchIndexMixin):
    ...

and

class BlogPost(models.Model, ElasticsearchIndexMixin):
    ...

ElasticsearchIndexMixin could detect whether it's applied as a mixin on a Model based class and sanely default many of its @classmethods to be driven from the Model's meta info, etc.

Alternatively, a ElasticsearchIndexModelMixin could be derived from ElasticsearchIndexMixin providing this functionality. This is probably the cleaner solution.

Hello James,

Your project is very interesting and quite unique as I have not found similar ressources elsewhere.

I know quite well Elasticsearch and I am familiar with the use of the official elasticsearch-py Python API, but I am currently struggling to set up a Django website with elasticsearch as the database.

I have modified the settings.py file as you did, but I don't understand how I can modified the models.py file. I am a beginner in Django.

Do you the ideas clear on what to do to this models.py in order to link to elasticsearch?

Regards,
SD

Gleaned from the discussion with @avelis in #20, during the bulk reindexing scenario only the following line:

data = {'delete' if delete else 'index': data}

would likely be better as:

data = {'delete' if delete else 'create': data}

Even better, for flexibility, probably having a safe default parameter into bulk_index() makes sense, like:

def bulk_index(cls, es=None, index_name='', queryset=None, create_only=False):
  ...
  action = 'delete' if delete else ('create' if create_only else 'index')
  data = {action: data}

This would prevent any possible overwriting of data in the new index during the reindex process. That said, this would never happen given the current structure and usage of this library. It becomes more relevant in a parallel index writing strategy discussed in #20.

See ES bulk docs for more information.

The current default return value of get_type_mapping is {}.

Verify that this simply skips the setting of that mapping and allows future indexing to auto-generate a mapping. This would be the default Elasticsearch behaviour.

Not sure if this feature keeps this library simple as the title suggests. However, I think for large indexes this could be helpful in speeding up commands.

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/vagrant/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/vagrant/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/vagrant/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/vagrant/venv/local/lib/python2.7/site-packages/raven/contrib/django/management/__init__.py", line 41, in new_execute
    return original_func(self, *args, **kwargs)
  File "/vagrant/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/vagrant/venv/local/lib/python2.7/site-packages/simple_elasticsearch/management/commands/es_manage.py", line 76, in handle
    self.subcommand_rebuild(requested_indexes, no_input)
  File "/vagrant/venv/local/lib/python2.7/site-packages/simple_elasticsearch/management/commands/es_manage.py", line 112, in subcommand_rebuild
    results, aliases = rebuild_indices(indices=indexes)
  File "/vagrant/venv/local/lib/python2.7/site-packages/simple_elasticsearch/utils.py", line 174, in rebuild_indices
    }}, index=index_name)
  File "/vagrant/venv/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/vagrant/venv/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 618, in put_settings
    '_settings'), params=params, body=body)
  File "/vagrant/venv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/vagrant/venv/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
    self._raise_error(response.status, raw_data)
  File "/vagrant/venv/local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'illegal_argument_exception')

I put an exception handler around es.indeces.put_settings and grabbed the info section of the ES error:

Can't update non dynamic settings[[index.merge.policy.merge_factor]] for open indices [[...]]

Hi,

simple_elasticsearch.utils module needs signals, but it does not commited.

Environment

OS: MacOSX 10.10
Python: 3.4.2

Repro steps

• install library from HEAD pip install -e git+https://github.com/jaddison/django-simple-elasticsearch.git#egg=django-simple-elasticsearch
• type python manage.py es_manage --list

Expected result

• No import error.

Stacktrace

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/key/Documents/virtualenv/photoshare/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/key/Documents/virtualenv/photoshare/lib/python3.4/site-packages/django/core/management/__init__.py", line 354, in execute
    django.setup()
  File "/Users/key/Documents/virtualenv/photoshare/lib/python3.4/site-packages/django/__init__.py", line 21, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/key/Documents/virtualenv/photoshare/lib/python3.4/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/Users/key/Documents/virtualenv/photoshare/lib/python3.4/site-packages/django/apps/config.py", line 202, in import_models
    self.models_module = import_module(models_module_name)
  File "/Users/key/Documents/virtualenv/photoshare/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/Users/key/Documents/virtualenv/photoshare/photoshare/apps/album/models.py", line 11, in <module>
    from apps.event.models import Event
  File "/Users/key/Documents/virtualenv/photoshare/photoshare/apps/event/models.py", line 7, in <module>
    from simple_elasticsearch.mixins import ElasticsearchIndexMixin
  File "/Users/key/Documents/virtualenv/photoshare/src/django-simple-elasticsearch/simple_elasticsearch/mixins.py", line 5, in <module>
    from .utils import queryset_iterator
  File "/Users/key/Documents/virtualenv/photoshare/src/django-simple-elasticsearch/simple_elasticsearch/utils.py", line 11, in <module>
    from .signals import post_indices_create, post_indices_rebuild
ImportError: No module named 'simple_elasticsearch.signals'

Note

simple_elasticsearch/utils.py

from .signals import post_indices_create, post_indices_rebuild  # <- misssing file!

usage examples/guidelines for the es_manage command are needed in the docs

queryset_iterator in utils.py relies on an int primary key. Having a uuid primary key breaks it, which prevents rebuilding indices.

For now I have changed line 74 mixins.py commit fb13b30 from:
for i, obj in enumerate(queryset_iterator(queryset, cls.get_query_limit())):
to:
for i, obj in enumerate(queryset):

And that works. FYI I am new to this whole elasticsearch thing.

Observing this bulk client api call Is there a way to pass any exceptions up or a connections setting I could set so that any potential exceptions with ES bulk API are easier to catch? The ES client does return a response but the mixin class does not return the body for inspection. There is the helpers file bulk function call but it doesn't take a queryset argument and the actions must be built procedurally.

Thanks again for making this library.

Hi,

Is there any wait to deal with parent/child relationship using this app.
I need to do something like this :
https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-parent-child.html

If it's not possible to do it yet could you please give me some description of the files inside the project and I will try to implement it in a pull request.

M0dM

So I've set up the Elasticsearchmixin, and it works well. However, I notice that when I update a field within a model. That field is updated in Elasticsearch due to the post_save but all the other models that have foreign key relationships to that model field don't update within Elasticsearch. Is there a way that when you change the username of a specific user, it not only updates that user, but the username within every model that user is tied to?

Default to '1' if unable to cast to int.

Is there current support for connecting to an AWS elasticsearch node, instead of localhost/regular IP driven? This has the requirement of needing to send over access & token information. Amazon has a boto library that may help.

If not, how hard/long do you think it would be to allow that support? I imagine you wouldn't be doing it yourself in a short time span, if ever.