Git Product home page Git Product logo

yiielasticsearch's Introduction

YiiElasticSearch

Elastic Search client for Yii.

Installation

Install via composer, requires php >= 5.3

Configuration

Add the following to your application config:

'components' => array(
    'elasticSearch' => array(
        'class' => 'YiiElasticSearch\Connection',
        'baseUrl' => 'http://localhost:9200/',
    ),
    ...
)

Also make sure, that you include the autoloader of composer. We recommend to add this line to your index.php and maybe also your yiic.php:

// Include composer autoloader
require_once(__DIR__.'/protected/vendor/autoload.php');

Make sure to modify the path so that it matches the location of your vendor/ directory.

Usage

Index your ActiveRecords

Attach the YiiElasticSearch\SearchableBehavior to any of your ActiveRecords to make it easy to index and search your normal models with elasticsearch.

class MyModel extends CActiveRecord
{
    public function behaviors()
    {
        return array(
            'searchable' => array(
                'class' => 'YiiElasticSearch\SearchableBehavior',
            ),
        );
    }
}

Now when MyModel instances are saved or deleted they will be automatically indexed or deleted in elasticsearch as appropriate.

Define an index for a record

By default your records will be stored in an index that uses your sanitized application name (Yii::app()->name). To change it you can define

class MyModel extends CActiveRecord
{
    public $elasticIndex = 'myindex';

or, if you need more control, create a method

class MyModel extends CActiveRecord
{
    public function getElasticIndex()
    {
        return 'myindex';
    }

Define a type for a record

By default the lower case class name will be used as type name in elasticsearch. If you want to change that you can define

class MyModel extends CActiveRecord
{
    public $elasticType = 'mymodel';

or, again, if you need more control, create a method

class MyModel extends CActiveRecord
{
    public function getElasticType()
    {
        return 'mymodel';
    }

Customize indexed data

By default all attributes are stored in the index. If you need to customize the data that should be indexed in elasticsearch, you can override these two methods.

class MyModel extends CActiveRecord
{
    /**
     * @param DocumentInterface $document the document where the indexable data must be applied to.
     */
    public function populateElasticDocument(DocumentInterface $document)
    {
        $document->setId($this->id);
        $document->name     = $this->name;
        $document->street   = $this->street;
    }

    /**
     * @param DocumentInterface $document the document that is providing the data for this record.
     */
    public function parseElasticDocument(DocumentInterface $document)
    {
        // You should always set the match score from the result document
        if ($document instanceof SearchResult)
            $this->setElasticScore($document->getScore());

        $this->id       = $document->getId();
        $this->name     = $document->name;
        $this->street   = $document->stree;
    }

Query records

You can specify queries using the YiiElasticSearch\Search object. This object provides a simple OO wrapper for the vanilla elasticsearch search API.

For example:

$search = new \YiiElasticSearch\Search("myindex", "mymodel");
$search->query = array(
    "match_all" => array()
);

// start returning results from the 20th onwards
$search->offset = 20;

With a search you can either perform a 'raw' query, e.g.

$resultSet = Yii::app()->elasticSearch->search($search);

This will return a result set that is a very simple wrapper around the raw elastic search response.

Alternatively, when combined with a SearchableBehavior you can use data providers, e.g.

$dataProvider = new \YiiElasticSearch\DataProvider(MyModel::model(), array(
        'search' => $search
));

The data from $dataProvider->data is a list of ActiveRecords, just like from an ordinary CActiveDataProvider. So you can use it in any list or grid view.

Raw requests

You can also use the connection component to send raw requests to elasticsearch.

// Will be an instance of a Guzzle\Http\Client
$client = Yii::app()->elasticSearch->client;

$mapping = array(
   'country' => array(
        'properties' => array(
            'name' => array(
                'type' => 'string',
            ),
        ),
    ),

// Create a mapping
$request = $client->put('myindex', array("Content-type" => "application/json"));
$request->setBody(array('mapping' => $mapping));

$response = $request->send();

$result = $response->getBody();

Console Maintenance

The extension comes with two simple maintenance commands that can be helpful to find out what's going on in your index. To configure them, add this to your console.php configuration:

'commandMap' => array(
    'elastic' => array(
        'class' => 'YiiElasticSearch\ConsoleCommand',
    ),
    'zerodowntimeelastic' => array(
        'class' => 'YiiElasticSearch\ZeroDowntimeConsoleCommand',
    ),
),

This will allow you to use yiic elastic and yiic zerodowntimeelastic on the console. Here are the commands help:

Console Command

This is the maintenance command for the elasticSearch component.

ACTIONS

  index --model=<model> [--skipExisting]

    Add all models <model> to the index. This will replace any previous
    entries for this model in the index. Index and type will be auto-detected
    from the model class unless --index or --type is set explicitely.
    If --skipExisting is used, no action is performed if there are already
    documents indexed under this type.


  map --model=<model> --map=<filename> [--skipExisting]
  map --index=<index> --map=<filename> [--skipExisting]

    Create a mapping in the index specified with the <index> or implicitly
    through the <model> parameter. The mapping must be available from a JSON
    file in <filename> where the JSON must have this form:

        {
            "tweet" : {
                "properties": {
                    "name" : {"type" : "string"},
                    ...
            },
            ...
        }

    If --skipExisting is used, no action is performed if there's are already
    a mapping for this index.


  list [--limit=10] [--offset=0]
  list [--model=<name>] [--limit=10] [--offset=0]
  list [--index=<name>] [--type=<type>] [--limit=10] [--offset=0]

    List all entries in elasticsearch. If a model or an index (optionally with
    a type) is specified only entries matching index and type of the model will be listed.


  delete --model=<name> [--id=<id>]

    Delete a document from an index. If no <id> is specified the whole
    index will be deleted.

  help

    Show this help

ZeroDowntime Command

This is a zero downtime maintenance command for the elasticSearch component. More details: https://www.elastic.co/blog/changing-mapping-with-zero-downtime

ACTIONS

  * index --models=<model1>,...
    Add all models to the index.

  * status --models=<model1>,...
    Displays actual indexes and aliases

  * schema --models=<model1>,...
        [--version=201512121212] [--forceMigrate=false] [--bulkCopy=true] [--updateAlias=true] [--deleteIndexes=true]

    Creates schema for the given models. Steps:
     1, compare mapping
     2, if migration is needed, create the new schema version, always create new if <forceMigrate> is true
     3, bulk copy the previous data if <bulkCopy> is true
     4, update aliases if <updateAlias> is true
     5, delete indexes if <deleteIndexes> is true


  * bulkCopy --from=index/type --to=index2/type [--properties=]
    Bulk copy all data

  * changeAlias --from=value --to=value [--old=]
    Change alias

  * deleteIndex --indexesToDelete=value
    Delete index with all types
    
  help

    Show this help

yiielasticsearch's People

Contributors

bartaakos avatar burci avatar maxsh avatar mikehaertl avatar phpnode avatar qymaen avatar schmunk42 avatar urmaul avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yiielasticsearch's Issues

MapperParsingException for any active record

Hi,

I am having an Yii 1.x application and i am working on adding an elastic search server to it. Using the yiielasticsearch extension has been very helpful as the data moves seamlessly between the two.

However i have this error whenever i try and create any data:

MapperParsingException[failed to parse [total_paid]]; nested: NumberFormatException[For input string: "0.00"];

if i manually cast (double) to total_paid (in populateElasticDocument) it moves to the next error about created_at - which is a date. Now, i believe this should be automattic and we'd require casting only in special cases.

Any ideas?

Aggregations

I have made changes to allow for the support of aggregations. We can discuss any points around this here.

I will submit a pull request soon, if you wish to integrate this feature.

Search with dataprovider

Hi, could you help me and teach me using dataProvider?
how exactly parameter $search.
i have script below:

$search = array();
$dataProvider = new \YiiElasticSearch\DataProvider(Channel::model(), array(
'search' => $search
));

in the view :

widget('zii.widgets.grid.CGridView', array( 'dataProvider' => $dataProvider, 'columns' => array( 'id', 'title' ), )); ?>

but only error show like this :
exception 'CException' with message
'IndexMissingException[[exploreattractionsfindtraveltours] missing]' in
D:\xampp\htdocs\tourexplora\protected\vendors\YiiElasticSearch\Connection.php:177
Stack trace:
#0

D:\xampp\htdocs\tourexplora\protected\vendors\YiiElasticSearch\Connection.php(144):
YiiElasticSearch\Connection->perform(Object(Guzzle\Http\Message\EntityEnclosingRequest))
#1

D:\xampp\htdocs\tourexplora\protected\vendors\YiiElasticSearch\DataProvider.php(127):
YiiElasticSearch\Connection->search(Object(YiiElasticSearch\Search))
#2 D:\xampp\htdocs\tourexplora\yii\framework114\web\CDataProvider.php(168):

YiiElasticSearch\DataProvider->fetchData()
#3

D:\xampp\htdocs\tourexplora\yii\framework114\zii\widgets\CBaseListView.php(111):
CDataProvider->getData()
#4

D:\xampp\htdocs\tourexplora\yii\framework114\zii\widgets\grid\CGridView.php(339):
CBaseListView->init()
#5

D:\xampp\htdocs\tourexplora\yii\framework114\web\CBaseController.php(147):
CGridView->init()
#6

D:\xampp\htdocs\tourexplora\yii\framework114\web\CBaseController.php(172):
CBaseController->createWidget('zii.widgets.gri...', Array)
#7 D:\xampp\htdocs\tourexplora\protected\views\test\elastic2.php(12):

CBaseController->widget('zii.widgets.gri...', Array)
#8

D:\xampp\htdocs\tourexplora\yii\framework114\web\CBaseController.php(126):
require('D:\xampp\htdocs...')
#9

D:\xampp\htdocs\tourexplora\yii\framework114\web\CBaseController.php(95):
CBaseController->renderInternal('D:\xampp\htdocs...', Array, true)
#10 D:\xampp\htdocs\tourexplora\yii\framework114\web\CController.php(869):

CBaseController->renderFile('D:\xampp\htdocs...', Array, true)
#11 D:\xampp\htdocs\tourexplora\yii\framework114\web\CController.php(782):

CController->renderPartial('elastic2', Array, true)
#12

D:\xampp\htdocs\tourexplora\protected\controllers\TestController.php(995):
CController->render('elastic2', Array)
#13

D:\xampp\htdocs\tourexplora\yii\framework114\web\actions\CInlineAction.php(49):
TestController->actionElastic2()
#14 D:\xampp\htdocs\tourexplora\yii\framework114\web\CController.php(308):

CInlineAction->runWithParams(Array)
#15 D:\xampp\htdocs\tourexplora\yii\framework114\web\CController.php(286):

CController->runAction(Object(CInlineAction))
#16 D:\xampp\htdocs\tourexplora\yii\framework114\web\CController.php(265):

CController->runActionWithFilters(Object(CInlineAction), Array)
#17

D:\xampp\htdocs\tourexplora\yii\framework114\web\CWebApplication.php(282):
CController->run('elastic2')
#18

D:\xampp\htdocs\tourexplora\yii\framework114\web\CWebApplication.php(141):
CWebApplication->runController('test/elastic2')
#19

D:\xampp\htdocs\tourexplora\yii\framework114\base\CApplication.php(180):
CWebApplication->processRequest()
#20 D:\xampp\htdocs\tourexplora\index.php(11): CApplication->run()
#21 {main}

REQUEST_URI=/tourexplora/test/elastic2

Behaviors doesn't work

I have fetched data succesfully from elastic search and add behavior funciton in my model like this:(DbBaseModel has extended CActiveRecord)
image
but after I save data,it hasn't been inserted into elastic search.
image
I debug and find the below function hasn't been executed,so what should I do?
image

Timestamp Behavior conflict

I have an issue with zii timestamp behavior when creating\updating items.

Here is how behavior config:

'AutoTimestampBehavior' => array(
    'class' => 'zii.behaviors.CTimestampBehavior',
    'createAttribute' => 'created_at',
    'updateAttribute' => 'updated_at',
    'setUpdateOnCreate' => true,
),

When you are creating item - you are getting an error:

MapperParsingException[failed to parse [created_at]]; nested: ElasticsearchIllegalArgumentException[unknown property [expression]];

When you are updating item - you are getting an error:

MapperParsingException[failed to parse [updated_at]]; nested: ElasticsearchIllegalArgumentException[unknown property [expression]];

I've fixed it by adding ignore of attributes with type of CDbExpression in SearchableBehavior.php

public function populateElasticDocument(DocumentInterface $document)
{
    $document->setId($this->owner->getPrimaryKey());
    foreach($this->owner->attributeNames() as $name) {
        if($this->owner->{$name} instanceof \CDbExpression) {
            continue;
        }
        $document->{$name} = $this->owner->{$name};
    }
}

It's pretty bad solution because created_at value will be passed as string only after the first update of the document, and updated_at attribute will be ignored everytime, because it's passed as CDbExpression on each update.

Console index action ignores default scope

In ConsoleCommand::actionIndex() for getting models used simple sql-query to the database. But for count used CActiveRecord::count(), so when you expect that in index would be added some part of rows actually would be added everything.

Add mapping feature to models

Alternatively to the mapping file, we could keep the mapping definition inside the model.

I'd suggest a syntax like this:

public function elasticMapping()
{
    return array(
        'id' => array('type' => 'integer'),
        'name' => array('type' => 'string', 'index'=>'not_analyzed'),
        ...
    );
}

Then ./yiic elastic map --model=<model> would use the data above to find the right mapping.

Not sure, if we should also automate this for the regular auto-indexing on save(). This would require a check on every save operation, whether the mapping exists. Probably a bit too extreme.

Bulk index

I wish to have the ability to bulk upload documents from the console application.

I will try and make changes soon.

I will submit a pull request soon, if you wish to integrate this feature.

Wrong gate error:

Wrong gate error: 
Argument 2 passed to YiiElasticSearch\ResultSet::__construct() must be of the type array, null given, called in /var/www/www-root/data/www/florida.croc.one/protected/extensions/YiiElasticSearch/src/YiiElasticSearch/Connection.php on line 156 and defined

what could be the problem?

 $ curl http://localhost:9200/
{
  "name" : "Kamuu",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.1",
    "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp" : "2016-04-04T12:25:05Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

Yiinitializr support

I'm using Yiinitializr on couple of projects and found out that console index command wont work as expected.
yiic elastic index --model=Post

Here is a log:

C:\xampp\htdocs\my_app\src\webapp>yiic elastic index --model=Post
PHP Error[2]: include(Post.php): failed to open stream: No such file or directory
in file C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\yiisoft\yii\framework\YiiBase.php at line 427
#0 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\yiisoft\yii\framework\YiiBase.php(427): autoload()
#1 unknown(0): autoload()
#2 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\yiisoft\yii\framework\db\ar\CActiveRecord.php(395): spl_autoload_call()
#3 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\elasticsearch\YiiElasticSearch\ConsoleCommand.php(384): model()
#4 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\elasticsearch\YiiElasticSearch\ConsoleCommand.php(130): YiiElasticSearch\ConsoleCommand->getModel()
#5 unknown(0): YiiElasticSearch\ConsoleCommand->actionIndex()
#6 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\yiisoft\yii\framework\console\CConsoleCommand.php(172): ReflectionMethod->invokeArgs()
#7 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\yiisoft\yii\framework\console\CConsoleCommandRunner.php(71): YiiElasticSearch\ConsoleCommand->run()
#8 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\yiisoft\yii\framework\console\CConsoleApplication.php(92): CConsoleCommandRunner->run()
#9 C:\xampp\htdocs\my_app\src\webapp\common\lib\vendor\yiisoft\yii\framework\base\CApplication.php(180): CConsoleApplication->processRequest()
#10 C:\xampp\htdocs\my_app\src\webapp\yiic.php(18): CConsoleApplication->run()

Convert SearchableActiveRecord to a behavior

Enforcing a base class is always bad. What if I need my own base class in a project? I could extend from SearchableActiveRecord, but what if most of my models are not searchable?

From a quick look i think, the class can easily be converted to a behavior instead.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.