Craft Scout provides a simple solution for adding full-text search to your entries. Scout will automatically keep your search indexes in sync with your entries.
License: MIT License
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
I am curious about the solution for multisite. We recently ran into this issue as well where entry of the second language would overwrite the entry of the first language in our algolia index.
As far as I can see this was solve by adding site id to objectID as well as adding some checks in canIndexElement.
Why does it work when running the index commands in CLI, but not when editing an entry in Craft prior to 1.0.0? I would have thought the getElementQuery function would handle this because $this->critera either had a site handle or siteId present and thus would not return an element if the entry was of a different site handle or siteId? If this worked as expected (my expectations may be wrong) that would eliminate the need for checking site handle and id in the canIndexElement function in v1.0.0?
Why was site id added to objectID? What purpose does it serve? The recommended solution for multilingual search in Algolia is to either have separate indexes or language specific keys in the same index. https://www.algolia.com/doc/tutorials/full-text-search/multi-language-search/multilingual-search/
Adding siteId to objectID breaks splitElementIndex as it is not handled there, it relies on the pre v1.0.0 where it was just element->id.
The comparison check for siteId added in v1.0.0 expects a comparison of same types, however, I specify siteId as an int in criteria whereas the siteId from Craft is of type string, making this comparison fail. I recommend adding changing it to if (isset($this->criteria['siteId']) && intval($element->site->id) !== intval($this->criteria['siteId'])) { if its necessary and can't be solved by the getElementQuery.
Because of a recent move to docker for development environments we experience a timeout issue connecting to Algolia because the network can be slow on some machines. This is easily solved by adding an option to set connection timeout on the algolia client instance. $this->client->setConnectTimeout(10); The value should be possible to specify in config/scout.php and not hardcoded of course.
I can make a pull request to fix some of these issues, but as question 1 and 2 was not clear to me I wanted to find out why it was solved this way before fixing 3 and 4.
I have sync enabled in config, but whenever I update an entry it is removed from Algolia.
From the queue status in the side bar I see it tries to do the following:
See the attached image.
Seems that the event hook for Category::EVENT_BEFORE_DELETE calls the old deIndexElements rather than the new indexElements method.
I intend to submit a PR which replaces the deIndexElements call with indexElements.
Stack trace:
yii\base\UnknownMethodException: Calling unknown method: rias\scout\Scout::deIndexElements() in /my-project-dir/vendor/yiisoft/yii2/base/Component.php:300
Stack trace:
#0 /my-project-dir/vendor/rias/craft-scout/src/Scout.php(126): yii\base\Component->__call('deIndexElements', Array)
#1 [internal function]: rias\scout\Scout->rias\scout\{closure}(Object(craft\events\ModelEvent))
#2 /my-project-dir/vendor/yiisoft/yii2/base/Event.php(310): call_user_func(Object(Closure), Object(craft\events\ModelEvent))
#3 /my-project-dir/vendor/yiisoft/yii2/base/Component.php(636): yii\base\Event::trigger('craft\\elements\\...', 'beforeDelete', Object(craft\events\ModelEvent))
#4 /my-project-dir/vendor/craftcms/cms/src/base/Element.php(1865): yii\base\Component->trigger('beforeDelete', Object(craft\events\ModelEvent))
#5 /my-project-dir/vendor/craftcms/cms/src/services/Elements.php(922): craft\base\Element->beforeDelete()
#6 /my-project-dir/vendor/craftcms/cms/src/controllers/CategoriesController.php(541): craft\services\Elements->deleteElement(Object(craft\elements\Category))
#7 [internal function]: craft\controllers\CategoriesController->actionDeleteCategory()
#8 /my-project-dir/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#9 /my-project-dir/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#10 /my-project-dir/vendor/craftcms/cms/src/web/Controller.php(104): yii\base\Controller->runAction('delete-category', Array)
#11 /my-project-dir/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('delete-category', Array)
#12 /my-project-dir/vendor/craftcms/cms/src/web/Application.php(282): yii\base\Module->runAction('categories/dele...', Array)
#13 /my-project-dir/vendor/craftcms/cms/src/web/Application.php(542): craft\web\Application->runAction('categories/dele...', Array)
#14 /my-project-dir/vendor/craftcms/cms/src/web/Application.php(266): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#15 /my-project-dir/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#16 /my-project-dir/public/index.php(21): yii\base\Application->run()
#17 {main}
Odd question, but we have all of our sections in one large index, but our Solspace Calendar events are a different element type. Since I can't add to an index more than once apparently, what are my options, if any for merging craft\elements\Entry and Solspace\Calendar\Elements\Event into one index?
Hi,
i use your great plugin but i'm not familiar with php. I want to add a data field to my mapping so that i can get the img url. i think i have to combine two criteria models or? Can you please give me a hint?
"mappings" => [ [ 'indexName' => 'xyz', 'elementType' => \craft\elements\Entry::class, 'criteria' => [ 'section' => 'xyz' ], 'transformer' => function(craft\elements\Entry $entry) { return [ 'title' => $entry->title, 'id' => $entry->id, 'uri' => $entry->uri, 'url' => $entry->url, 'slug' => $entry->slug, 'enabled' => $entry->enabled, 'archived' => $entry->archived, 'postDate' => $entry->postDate, 'dateCreated' => $entry->dateCreated, 'dateUpdated' => $entry->dateUpdated, 'expiryDate' => $entry->expiryDate, 'hero' => $entry->heroImage ]; } ], ],
It seems deleting an entry from the entries index page, like below, does not de-index the entry:
Often saving an entry with many relations can cause A LOT of operations to Algolia, which is more expensive, and takes longer.
Ideally, operations would be batched, using: https://www.algolia.com/doc/api-reference/api-methods/batch/
Really great work on this plugin!
I can't seem to get eager loading to work when I have my scout.php file setup like so:
<?php
use craft\elements\Entry;
return [
"sync" => true,
"connect_timeout" => 1,
"application_id" => getenv("ALGOLIA_APP_ID"),
"admin_api_key" => getenv("ALGOLIA_API_KEY"),
"mappings" => [
[
"indexName" => "my_index",
"elementType" => Entry::class,
"criteria" => [
"section" => "test",
"with" => [
"photo"
]
],
"transformer" => function (Entry $entry) {
Craft::dd($entry->photo); // Outputs an craft\elements\db\AssetQuery instead of array
}
]
],
];Not sure if I'm missing something on my end!
Hi
Can you tell me if the plugin will work with Commerce products and variants?
I've been unable to get Scout to send any POST requests for entries. Any time I create or update an entry, I see a DELETE request made. I'm also seeing requests made for entries that are not in the article section. Based on the following config, I expected only article entries to be synced with Algolia.
This is my current, simplified config:
return [
"sync" => true,
"connect_timeout" => 1,
"application_id" => $app_id,
"admin_api_key" => $api_key,
'mappings' => [
[
'indexName' => "test_index",
'elementType' => \craft\elements\Entry::class,
'criteria' => [
'section' => 'article'
],
'transformer' => function (craft\base\Element $element) {
return $element->toArray();
},
],
],
];
I initially set up a more complex config, and it seemed to work, i.e., records were being added to the index. But at some point, Scout started sending DELETE requests on every save/update. This persisted even when I simplified the config. And in fact, if I delete the config entirely so that my scout.php is totally empty, I'm still seeing DELETE posts being sent. I assume config settings are saved in the database. Perhaps those settings got corrupted somehow?
I've tried running the flush/refresh/import commands, still no luck.
Last note: I did update Craft around the same time this issue arose. 3.1.21.1 -> 3.1.22. I tried downgrading, but the issue was still present.
I've simplified my scout.php as much as possible:
<?php
return [
"sync" => true,
"application_id" => "removed",
"admin_api_key" => "removed",
"mappings" => [
[
'indexName' => 'pages',
'elementType' => \craft\elements\Entry::class,
'criteria' => [
['section' => 'contact']
]
]
],
];
But when running ./craft scout/index/import I get an error:
Exception 'yii\base\UnknownPropertyException' with message 'Setting unknown property: craft\elements\db\EntryQuery::0'
in /path/vendor/yiisoft/yii2/base/Component.php:209
Stack trace:
#0 /path/vendor/craftcms/cms/src/elements/db/ElementQuery.php(425): yii\base\Component->__set('0', Array)
#1 /path/vendor/craftcms/cms/src/elements/db/EntryQuery.php(227): craft\elements\db\ElementQuery->__set('0', Array)
#2 /path/vendor/yiisoft/yii2/BaseYii.php(546): craft\elements\db\EntryQuery->__set('0', Array)
#3 /path/vendor/rias/craft-scout/src/models/AlgoliaIndex.php(247): yii\BaseYii::configure(Object(craft\elements\db\EntryQuery), Array)
#4 /path/vendor/rias/craft-scout/src/console/controllers/IndexController.php(76): rias\scout\models\AlgoliaIndex->getElementQuery()
#5 [internal function]: rias\scout\console\controllers\IndexController->actionImport('')
#6 /path/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#7 /path/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#8 /path/vendor/yiisoft/yii2/console/Controller.php(148): yii\base\Controller->runAction('import', Array)
#9 /path/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('import', Array)
#10 /path/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('scout/index/imp...', Array)
#11 /path/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction('scout/index/imp...', Array)
#12 /path/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(craft\console\Request))
#13 /path/craft(22): yii\base\Application->run()
#14 {main}
Hi @Rias500
I would appriciate a functionality to disable the auto save functionality of the package. Lots of change is going on, so we are updating the index daily in a bulk mode. So it would be nice to just load the env and disable the event listener for save by flag.
Like this it would be supereasy to manage when to save the index
Thank you for your help and great work!
Is it possible to define searchableAttributes via the plugin on save?
I have tried defining them in the indexSettings and settings arrays - but they don't seem to end up defined on my index
'mappings' => [
[
'indexName' => getenv('ENVIRONMENT') . '_services',
'indexSettings' => [
'settings' => [
'searchableAttributes' => [
'title',
'intro',
'body'
]
],
'forwardToReplicas' => 'true',
],
'elementType' => \craft\elements\Entry::class,
'criteria' => [
'section' => 'services'
],
'transformer' => function (craft\base\Element $element) {
return [
'title' => $element->title,
'id' => $element->id,
'url' => $element->url,
'intro' => $element->introTaglineLede,
'body' => $element->bodyText
];
}
],
...
]
I currently have a single Scout mapping that dumps a large amount of content for populating the Algolia index. I also have 'sync' => true is set in the Scount config, but nothing seems to be syncing for either 1) adding new entries content, nor 2) content updates.
Running ./craft scout/index/refresh articles_index works as expected, but in order for the search index to be updated, I have to run that command every time, which isn't ideal. I can set it up as a cron job, but it seems implied that the plugin will do this itself with some kind of syncing.
Part of the problem here might be that it's unclear to me what events this plugin listens for in regards to when to update the Algolia index. Perhaps the docs could use some expansion there, as well.
How can I get new/updated content syncing automatically? Thanks.
return [
"sync" => true,
"connect_timeout" => 2,
"application_id" => getenv('ALGOLIA_APPLICATION_ID'),
"admin_api_key" => getenv('ALGOLIA_ADMIN_API_KEY'),
"mappings" => [
[
'indexName' => getenv('ALGOLIA_ARTICLES_INDEX'),
'elementType' => \craft\elements\Entry::class,
'criteria' => [
'section' => ManualsTransformer::SECTION_MANUAL,
'type' => ManualsTransformer::ENTRY_TYPE_MANUAL,
'with' => [
// Eager load manual image
ManualsTransformer::IDENTIFIER_IMAGE,
// Eager load manual category
ManualsTransformer::IDENTIFIER_CATEGORY,
// Eager load:
// * all descendants of this entry (chapters)
// * that are level = 2 (ensures we don't get stuff that doesn't follow structure)
// * that are 'chapter' type only
[
'descendants',
[
'level' => 2,
'type' => ManualTransformer::ENTRY_TYPE_CHAPTER,
'limit' => 500
]
],
// Eager load topic articles for each chapter
['descendants.topicArticles', ['limit' => 1000]],
// Eager load meta keywords tag
[
'descendants.topicArticles.metaSearchKeywords',
['limit' => 500]
],
// Eager load subtopic group tag
[
'descendants.topicArticles.subtopcicGroupTag',
['limit' => 100]
],
// Eager load article steps
[
'descendants.topicArticles.topicArticleStepBuilder',
['limit' => 1000]
],
// Eager load article content blocks
[
'descendants.topicArticles.topicArticleContentBuilder',
['limit' => 1000]
],
// Eager load article content step blocks
[
'descendants.topicArticles.topicArticleStepBuilder.step',
['limit' => 1000]
]
],
'limit' => 25
],
'transformer' => 'ManualsCompleteTransformer',
'splitElementIndex' => ['article']
]
]
];I initially was confused by scout/index/import, as it wasn't clear if I was importing into Craft (which I realize you wouldn't ever do…), or into Algolia.
At any rate, if not renaming, perhaps just clarifying in the task command desc (adding to Algolia or from Craft or something)
The queue should first be checked if the elementId you are indexing already exists in the queue
During development I have had Scout installed alongside another plugin (trendyminds/algolia) which was handling the actual search on my site.
But now I'm trying to install the same plugins elsewhere and finding that I can't have both Scout and Alogolia installed together. There seems to be a conflict between the versions of algolia/algoliasearch-client-php
Scout requires 1.28.0 but Algoila requires ^2.2
With Scout installed I get the following message when trying to install Algolia
Problem 1
- Installation request for trendyminds/algolia ^2.0 -> satisfiable by trendyminds/algolia[2.0.0].
- Conclusion: remove algolia/algoliasearch-client-php 1.28.0
- Conclusion: don't install algolia/algoliasearch-client-php 1.28.0
- trendyminds/algolia 2.0.0 requires algolia/algoliasearch-client-php ^2.2 -> satisfiable by algolia/algoliasearch-client-php[2.2.0, 2.2.1, 2.2.2, 2.2.3].
- Can only install one of: algolia/algoliasearch-client-php[2.2.0, 1.28.0].
- Can only install one of: algolia/algoliasearch-client-php[2.2.1, 1.28.0].
- Can only install one of: algolia/algoliasearch-client-php[2.2.2, 1.28.0].
- Can only install one of: algolia/algoliasearch-client-php[2.2.3, 1.28.0].
- Installation request for algolia/algoliasearch-client-php (locked at 1.28.0) -> satisfiable by algolia/algoliasearch-client-php[1.28.0].
I can't find a way around this. Is there some way to change the requirement for Scout?
We currently have a website up and running with 2 sites (for 2 different countries), When i create a new entry, for an example a page, it will create 2 items in Algolia (both with a different siteId) But when i remove the item, only the item with siteId 1 will be removed (in Algolia), not the other one. (If i change the storeView to the second domain, it will remove only the item for the selected store, but craft will remove the item from both sites, not just one)
The data (Algolia hits array)
hits.txt
Kind Regards,
Import is failing if a mapped field does not have a getIsPost method. In this instance, we're using Easy Address Field, which does not have that method.
Config file: scout.php.zip
Error and stack trace below:
Exception 'yii\base\UnknownMethodException' with message 'Calling unknown method: craft\console\Request::getIsPost()'
in /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/base/Component.php:300
Stack trace:
#0 /var/www/calendar-ochsner.test/vendor/studioespresso/craft-easyaddressfield/src/services/FieldService.php(79): yii\base\Component->__call('getIsPost', Array)
#1 /var/www/calendar-ochsner.test/vendor/studioespresso/craft-easyaddressfield/src/fields/EasyAddressFieldField.php(122): studioespresso\easyaddressfield\services\FieldService->getField(Object(studioespresso\easyaddressfield\fields\EasyAddressFieldField), Object(Solspace\Calendar\Elements\Event), NULL)
#2 /var/www/calendar-ochsner.test/vendor/craftcms/cms/src/base/Element.php(1937): studioespresso\easyaddressfield\fields\EasyAddressFieldField->normalizeValue(NULL, Object(Solspace\Calendar\Elements\Event))
#3 /var/www/calendar-ochsner.test/vendor/craftcms/cms/src/base/Element.php(793): craft\base\Element->normalizeFieldValue('locationDetails')
#4 /var/www/calendar-ochsner.test/config/scout.php(35): craft\base\Element->__get('locationDetails')
#5 [internal function]: craft\services\Config->{closure}(Object(Solspace\Calendar\Elements\Event))
#6 /var/www/calendar-ochsner.test/vendor/league/fractal/src/Scope.php(373): call_user_func(Object(Closure), Object(Solspace\Calendar\Elements\Event))
#7 /var/www/calendar-ochsner.test/vendor/league/fractal/src/Scope.php(315): League\Fractal\Scope->fireTransformer(Object(Closure), Object(Solspace\Calendar\Elements\Event))
#8 /var/www/calendar-ochsner.test/vendor/league/fractal/src/Scope.php(234): League\Fractal\Scope->executeResourceTransformers()
#9 /var/www/calendar-ochsner.test/vendor/rias/craft-scout/src/models/AlgoliaIndex.php(108): League\Fractal\Scope->toArray()
#10 /var/www/calendar-ochsner.test/vendor/rias/craft-scout/src/models/AlgoliaIndex.php(170): rias\scout\models\AlgoliaIndex->transformElement(Object(Solspace\Calendar\Elements\Event))
#11 /var/www/calendar-ochsner.test/vendor/rias/craft-scout/src/models/AlgoliaIndex.php(133): rias\scout\models\AlgoliaIndex->indexElement(Object(Solspace\Calendar\Elements\Event))
#12 /var/www/calendar-ochsner.test/vendor/rias/craft-scout/src/console/controllers/scout/IndexController.php(98): rias\scout\models\AlgoliaIndex->indexElements(Array)
#13 [internal function]: rias\scout\console\controllers\scout\IndexController->actionImport('')
#14 /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#15 /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#16 /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/console/Controller.php(148): yii\base\Controller->runAction('import', Array)
#17 /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('import', Array)
#18 /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('scout/index/imp...', Array)
#19 /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction('scout/index/imp...', Array)
#20 /var/www/calendar-ochsner.test/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(craft\console\Request))
#21 /var/www/calendar-ochsner.test/craft(22): yii\base\Application->run()
#22 {main}
Hi Rias, I'm getting "Unknown command: scout/index/import" when trying to run the import for the first time.
Edit**
This wasn't working locally with MAMP but I did get it to work using a local virtual box container.
An entry already in the index does not seem to be removed if you disable the entry.
Running Scout 0.4.2 on Craft 3.0.24
Now that I've gotten the index imported I have a few questions:
I can see that when I perform an ...entries(search...) in craft, algolia gets a request, but the query is always empty. Is there any specific config I've missed?
The contents of matrix elements aren't indexed, just the matrix element itself, is there any way to index this as well?
When saving a custom element type added with a different plugin, an exception is thrown. This only happens while Scout is installed.
Serialization of 'Closure' is not allowed
Does not seem to matter whether the element is new or not. This issue was not happening prior to upgrading to Craft 3-RC9.
Craft version: 3-RC9 (Pro)
PHP version: 7.1
In my logs:
2018-02-06 19:07:47 [127.0.0.1][1][v9ve9k2fd9d1cm6l7mpmvdu5il][error][Exception] Exception: Serialization of 'Closure' is not allowed in /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2-queue/src/serializers/PhpSerializer.php:24
Stack trace:
#0 [internal function]: ArrayObject->serialize()
#1 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2-queue/src/serializers/PhpSerializer.php(24): serialize(Object(rias\scout\jobs\IndexElement))
#2 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2-queue/src/Queue.php(159): yii\queue\serializers\PhpSerializer->serialize(Object(rias\scout\jobs\IndexElement))
#3 /Users/emma/Sites/bouwjobs/vendor/craftcms/cms/src/queue/Queue.php(148): yii\queue\Queue->push(Object(rias\scout\jobs\IndexElement))
#4 /Users/emma/Sites/bouwjobs/vendor/rias/craft-scout/src/Scout.php(70): craft\queue\Queue->push(Object(rias\scout\jobs\IndexElement))
#5 [internal function]: rias\scout\Scout->rias\scout\{closure}(Object(craft\events\ModelEvent))
#6 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2/base/Event.php(221): call_user_func(Object(Closure), Object(craft\events\ModelEvent))
#7 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2/base/Component.php(565): yii\base\Event::trigger('craft\\base\\Elem...', 'afterSave', Object(craft\events\ModelEvent))
#8 /Users/emma/Sites/bouwjobs/vendor/craftcms/cms/src/base/Element.php(1718): yii\base\Component->trigger('afterSave', Object(craft\events\ModelEvent))
#9 /Users/emma/Sites/bouwjobs/plugins/bouwjobsstripe/src/elements/Plan.php(553): craft\base\Element->afterSave(true)
#10 /Users/emma/Sites/bouwjobs/vendor/craftcms/cms/src/services/Elements.php(478): dadamotion\bouwjobsstripe\elements\Plan->afterSave(true)
#11 /Users/emma/Sites/bouwjobs/plugins/bouwjobsstripe/src/controllers/PlansController.php(127): craft\services\Elements->saveElement(Object(dadamotion\bouwjobsstripe\elements\Plan), false)
#12 [internal function]: dadamotion\bouwjobsstripe\controllers\PlansController->actionSave()
#13 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#14 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#15 /Users/emma/Sites/bouwjobs/vendor/craftcms/cms/src/web/Controller.php(80): yii\base\Controller->runAction('save', Array)
#16 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('save', Array)
#17 /Users/emma/Sites/bouwjobs/vendor/craftcms/cms/src/web/Application.php(241): yii\base\Module->runAction('bouwjobs-stripe...', Array)
#18 /Users/emma/Sites/bouwjobs/vendor/craftcms/cms/src/web/Application.php(451): craft\web\Application->runAction('bouwjobs-stripe...', Array)
#19 /Users/emma/Sites/bouwjobs/vendor/craftcms/cms/src/web/Application.php(212): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#20 /Users/emma/Sites/bouwjobs/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#21 /Users/emma/Sites/bouwjobs/web/index.php(42): yii\base\Application->run()
#22 /Users/emma/.composer/vendor/laravel/valet/server.php(133): require('/Users/emma/Sit...')
#23 {main}
Great work on v2 @riasvdv
I've been playing around with muti-sites and even though you refer to them in your code comments, I'm wondering of it might be an idea to include an example in the readme!?
Here's my solution that means you don't have to create a specific index each time a new site is added:
config/scout.php
return [
'indices' => [] + siteIndices(),
];
/**
* Iterate over sites and create index for each
*
* @return array
* @throws Exception
*/
function siteIndices()
{
$indices = [];
foreach (Craft::$app->getSites()->getAllSites() as $site) {
$indices[] = \rias\scout\ScoutIndex::create($site->handle . '_Search')
->elementType(\craft\elements\Entry::class)
->criteria(function (\craft\elements\db\EntryQuery $query) use ($site) {
return $query->siteId($site->id)->section('pages');
})
->transformer(function (\craft\elements\Entry $entry) {
return [
'title' => $entry->title
];
});
}
return $indices;
}
Hi @Rias500,
Thanks for this plugin. I am currently looking at migrating a previous Algolia search mapping setup from Search Plus using your plugin. One thing that would be nice is to have Algolia admin API key and application ID exposed as twig variables so they output in Twig templates when building the front end search i.e.
{{ craft.scout.algoliaApplicationId }}
{{ craft.scout.algoliaAdminApiKey }}
Would this be something you be happy to have added to your plugin?
This may of happened when I updated to the latest version.
It looks like when i am creating a new entry it adds it to Agolia, but then removes it. E.g, I created the entry in the websitePages section and its ID is 1895, see below logs where it appears to add, remove, add and then remove it.
Here is my queue.log:
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::beforeExec] [16862] Updating element slugs and URIs (attempt: 1) - Started
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::afterExec] [16862] Updating element slugs and URIs (attempt: 1) - Done (time: 0.023s)
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::beforeExec] [16863] Adding element 1_1895 to index (attempt: 1) - Started
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::afterExec] [16863] Adding element 1_1895 to index (attempt: 1) - Done (time: 0.070s)
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::beforeExec] [16864] Removing element 1_1895 from index (attempt: 1) - Started
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::afterExec] [16864] Removing element 1_1895 from index (attempt: 1) - Done (time: 0.047s)
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::beforeExec] [16865] Adding element 1_1895 to index (attempt: 1) - Started
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::afterExec] [16865] Adding element 1_1895 to index (attempt: 1) - Done (time: 0.061s)
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::beforeExec] [16866] Removing element 1_1895 from index (attempt: 1) - Started
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::afterExec] [16866] Removing element 1_1895 from index (attempt: 1) - Done (time: 0.048s)
2018-10-29 09:27:11 [-][1][-][info][craft\queue\QueueLogBehavior::beforeExec] [16867] Deleting stale template caches (attempt: 1) - Started
2018-10-29 09:27:13 [-][1][-][info][craft\queue\QueueLogBehavior::afterExec] [16867] Deleting stale template caches (attempt: 1) - Done (time: 1.649s)
Here is my scout config:
<?php
return [
"sync" => true,
"application_id" => "xxx",
"admin_api_key" => "xxxxxxxx",
"mappings" => [
[
'indexName' => 'removedfordemo',
'elementType' => \craft\elements\Entry::class,
'criteria' => [
'section' => 'websitePages',
'type' => 'websitePages'
],
'transformer' => function(craft\elements\Entry $entry) {
Craft::$app->getConfig()->getGeneral()->generateTransformsBeforePageLoad = true;
// Begin defining the response data for this entry
$data = [
'title' => $entry->title,
'slug' => $entry->url,
//'textArea' => $entry->textArea,
'textArea' => ! empty($entry->textArea) ? (string) $entry->textArea : 'No preview text available',
//'heroBanner' => ! empty($entry->heroBanner->one()->) ? (string) $entry->heroBanner->one()->url : '/images/site/mc-placeholder-480x324.jpg',
'status' => $entry->status
];
// Get the first asset in a "heroImage" Assets field (if there is one)
$heroBanner = $entry->heroBanner->one();
$data['heroBanner'] = $heroBanner ? $heroBanner->getUrl('mailchimpThumbnails') : '/images/site/mc-placeholder-480x324.jpg';
return $data;
},
],
[
'indexName' => 'removedfordemo',
'elementType' => \craft\elements\Entry::class,
'criteria' => [
'section' => 'pupilsWork',
],
'transformer' => function(craft\elements\Entry $entry) {
Craft::$app->getConfig()->getGeneral()->generateTransformsBeforePageLoad = true;
$data = [
'title' => "Pupils Work: $entry->title",
'slug' => $entry->url,
'textArea' => ! empty($entry->textArea) ? (string) $entry->textArea : 'No preview text available',
'status' => $entry->status
];
$heroBanner = $entry->overviewImage->one();
$data['heroBanner'] = $heroBanner ? $heroBanner->getUrl('mailchimpThumbnails') : '/images/site/mc-placeholder-480x324.jpg';
return $data;
},
],
],
];
As you might remember I'm using the plugin for a multi-languages setup, add first I made a PR to change the id and prepend the site ID to the objectId. But that's not enough to have a localised search. The docs recommend to have an index per languages.
I'm really not saying this commit in my fork is the solution, it works for me, but might be a hint or an idea for you?
Please check this commit.
Then in the config I do this:
[
'indexName' => 'news__en_us',
'elementType' => Entry::class,
'criteria' => [
'section' => 'news',
'site' => 'en_us'
],
'transformer' => new SearchIndexEntryTransformer(),
],
[
'indexName' => 'news__nl_nl',
'elementType' => Entry::class,
'criteria' => [
'section' => 'news',
'site' => 'nl_nl'
],
'transformer' => new SearchIndexEntryTransformer(),
],
I've installed via the plugin section in Craft, set up the config file, but can't run the import command. I get:
Unknown command: scout/index/import
/**
* Refresh one or all indexes.
*
* @param string $index
*
* @throws Exception
* @throws \yii\base\InvalidConfigException
*
* @return int
* @throws \Exception
*/
public function actionRefresh($index = '')
{
$this->actionFlush($index);
$this->actionImport($index);
}
I am curious about the solution for multisite. We recently ran into this issue as well where entry of the second language would overwrite the entry of the first language in our algolia index.
As far as I can see this was solve by adding site id to objectID as well as adding some checks in canIndexElement.
Why does it work when running the scount/index/* commands in CLI, but not when editing an entry in Craft prior to 1.0.0? I would have thought the getElementQuery function would handle this because $this->critera either had a site handle or siteId present and thus would not return an element if the entry was of a different site handle or siteId? If this worked as expected (my expectations may be wrong) that would eliminate the need for checking site handle and id in the canIndexElement function in v1.0.0?
Why was site id added to objectID? What purpose does it serve? The recommended solution for multilingual search in Algolia is to either have separate indexes or language specific keys in the same index. https://www.algolia.com/doc/tutorials/full-text-search/multi-language-search/multilingual-search/
Adding siteId to objectID breaks splitElementIndex as it is not handled there, it relies on the pre v1.0.0 where it was just element->id.
The comparison check for siteId added in v1.0.0 expects a comparison of same types, however, I specify siteId as an int in criteria whereas the siteId from Craft is of type string, making this comparison fail. I recommend adding changing it to if (isset($this->criteria['siteId']) && intval($element->site->id) !== intval($this->criteria['siteId'])) { if its necessary and can't be solved by the getElementQuery.
Because of a recent move to docker for development environments we experience a timeout issue connecting to Algolia because the network can be slow on some machines. This is easily solved by adding an option to set connection timeout on the algolia client instance in ScoutService.php. $this->client->setConnectTimeout(10); The value should be possible to specify in config/scout.php and not hardcoded of course.
I can make a pull request to fix some of these issues, but as question 1 and 2 was not clear to me I wanted to find out why it was solved this way before fixing 3 and 4.
This bug seems to be back, I believe this is due to the checks in canDeindexElement() specifically, it returns the same final response as canIndexElement(), a call to getElementQuery() which returns false for a disabled entry. I think it should just return true?
I have been trying to index to algolia and I ran across the issue below.
./craft scout/index/import
Adding elements from index allProductData.[> ] 0% (0/1108) ETA: n/aException 'yii\base\UnknownPropertyException' with message 'Getting unknown property: craft\elements\db\CategoryQuery::0'
in /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/base/Component.php:154
Stack trace:
#0 /vol/site/build.20190820-172247-344c9d6a/vendor/craftcms/cms/src/elements/db/ElementQuery.php(491): yii\base\Component->__get(0)
#1 /vol/site/build.20190820-172247-344c9d6a/vendor/craftcms/cms/src/elements/db/ElementQuery.php(597): craft\elements\db\ElementQuery->__get(0)
#2 /vol/site/build.20190820-172247-344c9d6a/config/scout.php(59): craft\elements\db\ElementQuery->offsetGet(0)
#3 [internal function]: craft\services\Config->{closure}(Object(craft\commerce\elements\Product))
#4 /vol/site/build.20190820-172247-344c9d6a/vendor/league/fractal/src/Scope.php(373): call_user_func(Object(Closure), Object(craft\commerce\elements\Product))
#5 /vol/site/build.20190820-172247-344c9d6a/vendor/league/fractal/src/Scope.php(315): League\Fractal\Scope->fireTransformer(Object(Closure), Object(craft\commerce\elements\Product))
#6 /vol/site/build.20190820-172247-344c9d6a/vendor/league/fractal/src/Scope.php(234): League\Fractal\Scope->executeResourceTransformers()
#7 /vol/site/build.20190820-172247-344c9d6a/vendor/rias/craft-scout/src/models/AlgoliaIndex.php(108): League\Fractal\Scope->toArray()
#8 /vol/site/build.20190820-172247-344c9d6a/vendor/rias/craft-scout/src/models/AlgoliaIndex.php(159): rias\scout\models\AlgoliaIndex->transformElement(Object(craft\commerce\elements\Product))
#9 /vol/site/build.20190820-172247-344c9d6a/vendor/rias/craft-scout/src/models/AlgoliaIndex.php(127): rias\scout\models\AlgoliaIndex->indexElement(Object(craft\commerce\elements\Product))
#10 /vol/site/build.20190820-172247-344c9d6a/vendor/rias/craft-scout/src/console/controllers/Scout/IndexController.php(91): rias\scout\models\AlgoliaIndex->indexElements(Array)
#11 [internal function]: rias\scout\console\controllers\scout\IndexController->actionImport('')
#12 /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#13 /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#14 /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/console/Controller.php(148): yii\base\Controller->runAction('import', Array)
#15 /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('import', Array)
#16 /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('scout/index/imp...', Array)
#17 /vol/site/build.20190820-172247-344c9d6a/vendor/craftcms/cms/src/console/Application.php(93): yii\console\Application->runAction('scout/index/imp...', Array)
#18 /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application->runAction('scout/index/imp...', Array)
#19 /vol/site/build.20190820-172247-344c9d6a/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(craft\console\Request))
#20 /vol/site/build.20190820-172247-344c9d6a/craft(22): yii\base\Application->run()
#21 {main}
I am running stable craft 3 and commerce 2. Below is my scout config file:
<?php
use Craft;
use craft\i18n\Locale;
use craft\elements\Entry;
use craft\commerce\base\Model;
use craft\commerce\models\LineItem;
use craft\commerce\base\Purchasable;
use craft\commerce\base\PurchasableInterface;
use craft\commerce\elements\Order;
use craft\commerce\events\LineItemEvent;
use craft\commerce\elements\Product;
use craft\commerce\models\ProductType as Type;
use craft\commerce\elements\Variant;
use craft\commerce\models\PaymentCurrency;
use craft\commerce\models\TaxRate;
use craft\commerce\controllers\TaxRatesController as TaxRates;
use craft\commerce\Plugin as Commerce;
use craft\helpers\UrlHelper;
use superbig\imgix\Imgix;
use craft\elements\Asset;
use craft\elements\Category;
use craft\elements\GlobalSet;
use craft\elements\MatrixBlock;
use craft\elements\Tag;
use craft\elements\User;
return [
"sync" => true,
'connect_timeout' => 4,
"application_id" => "NHF40KMICV",
"admin_api_key" => "97545c7b55bc08bf3600a5bb9f4c6ce5",
"mappings" => [
[
'indexName' => 'allProductData',
'elementType' => Product::class,
'criteria' => [
'type' => craft\commerce\Plugin::getInstance()->getProductTypes()
],
'transformer' => function (Product $product) {
$paymentCurrency = PaymentCurrency::class;
$title = $product->title;
$url = $product->url;
$brand = (string)$product->productBrand[0];
$designer = (string)$product->productDesigner[0];
$num_decimals = (intval($product->defaultVariant->price) == $product->defaultVariant->price) ? 0 :2;
$price = number_format($product->defaultVariant->price,$num_decimals);
$primaryCurrency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrency();
$currency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrencyIso();
$currencySymbol = Craft::$app->getLocale()->getCurrencySymbol($currency);
$variants = $product->variants;
if(!empty($product->productImages->first())){
$frontImage = $product->productImages->first();
} else {
$frontImage = $product->defaultVariant->productImages->first();
}
$fm = Imgix::$plugin->imgixService->transformImage( $frontImage, [ 'width' => 500 ], ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$body = strip_tags((string) $product->body, '<strong><em><br>');
$totalPrice = $currencySymbol.$price;
$related = array(
'targetElement' => $product,
'field' => 'productBlock.product'
);
$typesCriteria = Entry::find()
->section('products')
->relatedTo($related);
$types = array_map(function($el) {
return [
'title' => $el->title,
'id' => $el->id,
'level' => $el->level,
];
}, $typesCriteria->all());
$typesByLevel = [];
foreach ($types as $type) {
$typesByLevel['types.level' . $type['level']][] = $type;
}
$variantAmount = count($variants);
return array_merge([
'title' => $title,
'id' => $product->id,
'frontImage' => $fm,
'type' => $product->type->name,
'types' => $types,
'designer' => $designer,
'brand' => $brand,
'price' => $totalPrice,
'productPrice' => (int)$product->defaultVariant->price,
'material' => $product->variantMaterials[0]->title,
'url' => $url,
'variants' => $variantAmount
], $typesByLevel);
},
],
[
'indexName' => 'allSiteData',
'elementType' => Product::class,
'criteria' => [
'type' => craft\commerce\Plugin::getInstance()->getProductTypes()
],
'transformer' => function (Product $product) {
$paymentCurrency = PaymentCurrency::class;
$title = $product->title;
$url = $product->url;
$brand = (string)$product->productBrand[0];
$designer = (string)$product->productDesigner[0];
$num_decimals = (intval($product->defaultVariant->price) == $product->defaultVariant->price) ? 0 :2;
$price = number_format($product->defaultVariant->price,$num_decimals);
$primaryCurrency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrency();
$currency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrencyIso();
$currencySymbol = Craft::$app->getLocale()->getCurrencySymbol($currency);
$variants = $product->variants;
if(!empty($product->productImages->first())){
$frontImage = $product->productImages->first();
} else {
$frontImage = $product->defaultVariant->productImages->first();
}
$fm = Imgix::$plugin->imgixService->transformImage( $frontImage, [ 'width' => 500 ], ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$body = strip_tags((string) $product->body, '<strong><em><br>');
$totalPrice = $currencySymbol.$price;
$related = array(
'targetElement' => $product,
'field' => 'productBlock.product'
);
$typesCriteria = Entry::find()
->section('products')
->relatedTo($related);
$types = array_map(function($el) {
return [
'title' => $el->title,
'id' => $el->id,
'level' => $el->level,
];
}, $typesCriteria->all());
$typesByLevel = [];
foreach ($types as $type) {
$typesByLevel['types.level' . $type['level']][] = $type;
}
$variantAmount = count($variants);
return array_merge([
'title' => $title,
'id' => $product->id,
'frontImage' => $fm,
'type' => $product->type->name,
'types' => $types,
'designer' => $designer,
'brand' => $brand,
'price' => $totalPrice,
'productPrice' => (int)$product->defaultVariant->price,
'material' => $product->variantMaterials[0]->title,
'url' => $url,
'variants' => $variantAmount
], $typesByLevel);
},
],
[
'indexName' => 'allSiteData',
'elementType' => Entry::class,
'criteria' => [
'section' => ['journal']
],
'transformer' => function(craft\base\Element $element) {
$frontImage = $element->catalogueGridImage[0];
$fm = '';
if(!empty($frontImage)){
$image = Imgix::$plugin->imgixService->transformImage( $frontImage, ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$fm = $image->url;
} else {
$fm = '/images/filler.jpg';
}
return [
'title' => $element->title,
'url' => $element->url,
'frontImage' => $fm,
];
},
],
[
'indexName' => 'allSiteData',
'elementType' => Entry::class,
'criteria' => [
'section' => ['products']
],
'transformer' => function(craft\base\Element $element) {
$frontImage = $element->catalogueGridImage[0];
$fm = '';
if(!empty($frontImage)){
$image = Imgix::$plugin->imgixService->transformImage( $frontImage, ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$fm = $image->url;
} else {
$fm = '/images/filler.jpg';
}
return [
'title' => $element->title,
'url' => $element->url,
'frontImage' => $fm,
];
},
],
[
'indexName' => 'allSiteData',
'elementType' => Entry::class,
'criteria' => [
'section' => ['products']
],
'transformer' => function(craft\base\Element $element) {
$frontImage = $element->catalogueGridImage[0];
$fm = '';
if(!empty($frontImage)){
$image = Imgix::$plugin->imgixService->transformImage( $frontImage, ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$fm = $image->url;
} else {
$fm = '/images/filler.jpg';
}
return [
'title' => $element->title,
'url' => $element->url,
'frontImage' => $fm,
];
},
],
[
'indexName' => 'allSiteData',
'elementType' => Category::class,
'criteria' => [
'group' => ['brands']
],
'transformer' => function(craft\base\Element $element) {
if($element->group->name == 'Brands'){
$frontImage = $element->photo[0];
$fm = '';
if(!empty($frontImage)){
$image = Imgix::$plugin->imgixService->transformImage( $frontImage, ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$fm = $image->url;
} else {
$fm = '/images/filler.jpg';
}
return [
'title' => $element->title,
'url' => $element->url,
'frontImage' => $fm,
];
}
},
],
[
'indexName' => 'allSiteData',
'elementType' => Category::class,
'criteria' => [
'group' => ['designers']
],
'transformer' => function(craft\base\Element $element) {
if($element->group->name == 'Designers'){
return [
'title' => $element->title,
'url' => $element->url,
];
}
},
],
[
'indexName' => 'allDesigners',
'elementType' => Category::class,
'criteria' => [
'group' => ['designers']
],
'transformer' => function(craft\base\Element $element) {
if($element->group->name == 'Designers'){
return [
'title' => $element->title,
'url' => $element->url,
];
}
},
],
[
'indexName' => 'allBrands',
'elementType' => Category::class,
'criteria' => [
'group' => ['brands']
],
'transformer' => function(craft\base\Element $element) {
if($element->group->name == 'Brands'){
$frontImage = $element->photo[0];
$fm = '';
if(!empty($frontImage)){
$image = Imgix::$plugin->imgixService->transformImage( $frontImage, ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$fm = $image->url;
} else {
$fm = '/images/filler.jpg';
}
return [
'title' => $element->title,
'url' => $element->url,
'frontImage' => $fm ?? null,
];
}
},
],
[
'indexName' => 'allProducts',
'elementType' => Product::class,
'criteria' => [
'type' => craft\commerce\Plugin::getInstance()->getProductTypes()
],
'transformer' => function (Product $product) {
$paymentCurrency = PaymentCurrency::class;
$title = $product->title;
$url = $product->url;
$brand = (string)$product->productBrand[0];
$designer = (string)$product->productDesigner[0];
$num_decimals = (intval($product->defaultVariant->price) == $product->defaultVariant->price) ? 0 :2;
$price = number_format($product->defaultVariant->price,$num_decimals);
$primaryCurrency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrency();
$currency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrencyIso();
$currencySymbol = Craft::$app->getLocale()->getCurrencySymbol($currency);
$variants = $product->variants;
if(!empty($product->productImages->one())){
$frontImage = $product->productImages->one();
} else {
$frontImage = $product->defaultVariant->productImages->one();
}
$fm = Imgix::$plugin->imgixService->transformImage( $frontImage, [ 'width' => 500 ], ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$body = strip_tags((string) $product->body, '<strong><em><br>');
$totalPrice = $currencySymbol.$price;
$related = array(
'targetElement' => $product,
'field' => 'productBlock.product'
);
$typesCriteria = Entry::find()
->section('products')
->relatedTo($related);
$types = array_map(function($el) {
return [
'title' => $el->title,
'id' => $el->id,
'level' => $el->level,
];
}, $typesCriteria->all());
$typesByLevel = [];
foreach ($types as $type) {
$typesByLevel['types.level' . $type['level']][] = $type;
}
$variantAmount = count($variants);
$postDate = date('Ymd', strtotime($product->postDate->getTimestamp() * 1000));
$rates = craft\commerce\Plugin::getInstance()->getTaxRates();
$allRates = $rates->getAllTaxRates();
$rateAmount = '';
foreach ($allRates as $rate) {
$rateAmount = $rate->rate;
$taxAmount = number_format($rateAmount,2);
$rateTrimmed = ltrim($taxAmount,"0.");
$rateAmount = $rateTrimmed;
}
$percentRounded = round($product->defaultVariant->price,3,PHP_ROUND_HALF_UP);
$percent = ($percentRounded/100)*$rateAmount;
$incVat = $product->defaultVariant->price + $percent;
$total = number_format($incVat);
// not sale price
$totalPrice = $currencySymbol.$total;
if(!empty($product->productBrand->one()->discountPercent) && $product->productBrand->one()->hasDiscount == true){
$discountPercent = $product->productBrand->one()->discountPercent;
$salePercent = (($incVat)/100)*$discountPercent;
$salePrice = $incVat - $salePercent;
$salePriceRound = number_format($salePrice);
$salePriceFinal = $currencySymbol.$salePriceRound;
}
return array_merge([
'title' => $title,
'id' => $product->id,
'dateAdded' => (int)date($postDate),
'frontImage' => $fm ?? null,
'type' => $product->type->name,
'types' => $types,
'designer' => $designer,
'brand' => $brand,
'price' => $totalPrice,
'salePrice' => $salePriceFinal ?? null,
'productPrice' => $incVat,
'material' => $product->variantMaterials[0]->title,
'colour' => $product->variantColor[0]->title ?? null,
'productionTime' => $product->dispatchTimes[0]->title ?? null,
'url' => $url,
'variants' => $variantAmount
], $typesByLevel);
},
],
]
];
Is it possible to skip entry somehow, and not from the criteria query, but from the transformer view?
I have been tried to return null, but it failed, also tried to return empty array, which is not working either.
In some cases I need more complex logic, what I really would like to put algolia, and what not...
Thank you for the great, and free product!
I'd like to remove an entry from the Algolia index if it is later disabled, how would this be possible with the plugin? Thanks
I have an index where I am attempting to set several of my attributes as facets per the README.md. My index posts just fine, but the facets aren't set. No errors are kicked back, either.
[
'indexName' => getenv('ENVIRONMENT') . '_news',
'indexSettings' => [
'settings' => [
'attributesForFaceting' => [
'boardsCommissions',
'departments',
'projects',
'electedOfficials',
'topics'
],
],
'forwardToReplicas' => 'true',
],
'elementType' => \craft\elements\Entry::class,
'criteria' => [
'section' => 'news'
],
'transformer' => function(craft\elements\Entry $entry) {
$boardsCommissions = [];
foreach($entry->boardsCommissions->all() as $value)
$boardsCommissions[] = $value->title;
$departments = [];
foreach($entry->departments->all() as $value)
$departments[] = $value->title;
$electedOfficials = [];
foreach($entry->electedOfficials->all() as $value)
$electedOfficials[] = $value->title;
$projects = [];
foreach($entry->projects->all() as $value)
$projects[] = $value->title;
$topics = [];
foreach($entry->topics->all() as $value)
$topics[] = $value->title;
return [
'title' => $entry->title,
'url' => $entry->url,
'newsImage' => ! empty($entry->newsImage->one()) ? (string) $entry->newsImage->one()->url : null,
'summary' => (string) $entry->summary,
'body' => (string) $entry->body,
'mediaContact' => (string) $entry->mediaContact,
'boardsCommissions' => $boardsCommissions,
'departments' => $departments,
'projects' => $projects,
'electedOfficials' => $electedOfficials,
'topics' => $topics,
];
},
],
// END NEWS INDEX
We're returning an empty array as noted in the docs. In cases where we do this and we also have splitElementIndex the following error occurs.
PHP Notice 'yii\base\ErrorException' with message 'Undefined index: body' in vendor/rias/craft-scout/src/models/AlgoliaIndex.php:219
Is there any plans to support long documents? We currently exceed the limits of algolia row size (20kb) and Algolia refers to https://www.algolia.com/doc/guides/ranking/distinct/#distinct-to-index-large-records
Our problem comes from trying to index text fields added to a matrix field, we dont know how many text fields are added, and as such, they may exceed the limit.
My initial plan was to add support for splitting up an article in accordance to the link above, but thought I'd check if this was something that was already thought of in this plugin or a planned feature?
./craft scout/index/import
Adding elements from index .[===============] 100% (0/0) ETA: n/a
scout.php
return [
'application_id' => '...',
'admin_api_key' => '...',
'mappings' => [
[
'indexName' => 'poc_magasin',
'elementType' => \craft\elements\Entry::class,
'criteria' => [
'section' => 'cityPlanningArticles'
],
'transformer' => function (craft\base\Element $element) {
return $element->toArray();
},
],
],
];
Nothing in the logs that says much about what is happening, or not happening.
Hi,
I am running the latest version of Craft 3 and Commerce 2. I am trying to import via CLI products from the backend. I get too a point where I can index 260 but that is it. I have ran this command:
php -d memory_limit=512M ./craft scout/index/import
But this still only reaches 260 and I know there are more like 890 products in total. Please can you check what that might be? Here is my code below:
"mappings" => [
[
'indexName' => 'allProductData',
'elementType' => Product::class,
'criteria' => [
'type' => ['Chairs, Benches and Stools','Side chairs','Armchairs','Stacking and Folding Chairs','Bar Stools','Low Stools','Benches','Lounge Seating','Sofas','Lounge chairs ','Daybeds & chaises ','Sofa beds ','Tables and Desks ','Dining ','Cafe and Bar ','Coffee and Side ','Console ','Home office ','Extending Tables ','Storage ','Shelving ','Cupboards and Drawers ','Sideboards & Credenzas ','Small Storage ','Systems ','Bedroom Furniture ','Beds ','Bedside Tables ','Bedroom Storage ','Wardrobes ','Home Accessories ','Mirrors & clocks ','Kitchenware ','Decorative objects ','Small storage ','Lighting ','Pendant lighting ','Ceiling Lights ','Wall Lights ','Floor lamps ','Table Lamps ','Office ','Task chairs ','Conference chairs ','Conference tables ','Home office ','Outdoor ','Outdoor Lounge ','Outdoor Seating ','Outdoor Tables ','Outdoor Lighting ','Stacking and Folding Chairs ','Bar and Counter Stools ','Chaises, Benches and Daybeds ','Ottomans and Poufs ','Consoles & Dressers, Vanity Units ','Rugs, Cushions and Textiles ','Desk Lamps ','Hooks and coatstands ','Bathroom lights ','Dressing Tables ','Desk lamps ','Bespoke ','Ex Display']
],
'transformer' => function (Product $product) {
$paymentCurrency = PaymentCurrency::class;
$title = $product->title;
$url = $product->url;
$brand = (string)$product->productBrand[0];
$designer = (string)$product->productDesigner[0];
$num_decimals = (intval($product->defaultVariant->price) == $product->defaultVariant->price) ? 0 :2;
$price = number_format($product->defaultVariant->price,$num_decimals);
$primaryCurrency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrency();
$currency = Commerce::getInstance()->getPaymentCurrencies()->getPrimaryPaymentCurrencyIso();
$currencySymbol = Craft::$app->getLocale()->getCurrencySymbol($currency);
$variants = $product->variants;
if(!empty($product->productImages->first())){
$frontImage = $product->productImages->first();
} else {
$frontImage = $product->defaultVariant->productImages->first();
}
$fm = Imgix::$plugin->imgixService->transformImage( $frontImage, [ 'width' => 500 ], ['lossless' => 0,'auto' => 'compress','fm' => 'jpg']);
$body = strip_tags((string) $product->body, '<strong><em><br>');
$totalPrice = $currencySymbol.$price;
// $typesCriteria = Craft::$app->elements->getCriteria(craft\elements\Entry::class, [
// 'section' => 'products',
// 'relatedTo' => [
// 'targetElement' => $product,
// 'field' => 'productBlock.product',
// ]
// ]);
$related = array(
'targetElement' => $product,
'field' => 'productBlock.product'
);
$typesCriteria = Entry::find()
->section('products')
->relatedTo($related);
$types = array_map(function($el) {
return [
'title' => $el->title,
'id' => $el->id,
'level' => $el->level,
];
}, $typesCriteria->all());
$typesByLevel = [];
foreach ($types as $type) {
$typesByLevel['types.level' . $type['level']][] = $type;
}
$variantAmount = count($variants);
return array_merge([
'title' => $title,
'id' => $product->id,
'frontImage' => $fm,
'type' => $product->type->name,
'types' => $types,
'designer' => $designer,
'brand' => $brand,
'price' => $totalPrice,
'productPrice' => (int)$product->defaultVariant->price,
'material' => $product->variantMaterials[0]->title,
'url' => $url,
'variants' => $variantAmount
], $typesByLevel);
},
]
]
Hi @Rias500. Just to let you know that the recent 1.2 release isn't available to install through composer. Looks like packagist has not been updated yet. The update is detected through Craft plugin store, but composer can't see it.
Thanks!
I've disabled sync and set up cron jobs to run every morning to prevent too many operations running and using up all the quota. It would be useful for permitted CMS users to have a button to refresh the index manually, e.g. if they push a new event that they want to be immediately searchable on the day.
Is this something that's in the pipeline?
I have 5000 entry, currently it hits the memory limit and doesn't import all of them. Would be nice to have a feature to chuck them or another way around hitting the memory limit when having more posts!
From what I gather, here's what currently happens:
When an element is saved, Scout loops through every mapping, and indexes where it can, and de-indexes where it can't.
That means if I have 5 mappings, I end up with jobs like this:
That seems problematic in terms of Algolia operations AND just in feedback to the user/dev. It seems like if saved element doesn't match a criteria, we should be able to bail earlier and not even run a job at all, and thus not incur pointless algolia operations.
With multiple sites, when you run scout/index/import, it will only index as the default site.
On the other hand – when you save an entry, it will index the element for every site that element is enabled in.
Seems like the behavior should be consistent.
NOTE: this only applies if your criteria does not specify site/siteId explicitly. Unfortunately, those params cannot (yet) accept arrays or null. The way I've worked around it is to have an explicit scout mapping for each site, with the only thing being different is the siteId param of the criteria.
Steps to reproduce:
No phperrors.log created, but this has appeared in queue.log:
2018-03-11 15:54:29 [127.0.0.1][1][-][error][UnexpectedValueException] UnexpectedValueException: Error at offset 0 of 2 bytes in /Users/emma/Sites/c3test/vendor/yiisoft/yii2-queue/src/serializers/PhpSerializer.php:32
Stack trace:
#0 [internal function]: ArrayObject->unserialize('[]')
#1 /Users/emma/Sites/c3test/vendor/yiisoft/yii2-queue/src/serializers/PhpSerializer.php(32): unserialize('O:28:"rias\\scou...')
#2 /Users/emma/Sites/c3test/vendor/yiisoft/yii2-queue/src/Queue.php(195): yii\queue\serializers\PhpSerializer->unserialize('O:28:"rias\\scou...')
#3 /Users/emma/Sites/c3test/vendor/yiisoft/yii2-queue/src/cli/Queue.php(139): yii\queue\Queue->handleMessage('4', 'O:28:"rias\\scou...', '300', 1)
#4 /Users/emma/Sites/c3test/vendor/craftcms/cms/src/queue/Queue.php(95): yii\queue\cli\Queue->handleMessage('4', 'O:28:"rias\\scou...', '300', 1)
#5 /Users/emma/Sites/c3test/vendor/craftcms/cms/src/controllers/QueueController.php(84): craft\queue\Queue->run()
#6 [internal function]: craft\controllers\QueueController->actionRun()
#7 /Users/emma/Sites/c3test/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#8 /Users/emma/Sites/c3test/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#9 /Users/emma/Sites/c3test/vendor/craftcms/cms/src/web/Controller.php(74): yii\base\Controller->runAction('run', Array)
#10 /Users/emma/Sites/c3test/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('run', Array)
#11 /Users/emma/Sites/c3test/vendor/craftcms/cms/src/web/Application.php(237): yii\base\Module->runAction('queue/run', Array)
#12 /Users/emma/Sites/c3test/vendor/craftcms/cms/src/web/Application.php(445): craft\web\Application->runAction('queue/run', Array)
#13 /Users/emma/Sites/c3test/vendor/craftcms/cms/src/web/Application.php(221): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#14 /Users/emma/Sites/c3test/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#15 /Users/emma/Sites/c3test/web/index.php(21): yii\base\Application->run()
#16 /Users/emma/.composer/vendor/laravel/valet/server.php(147): require('/Users/emma/Sit...')
#17 {main}
Task runs and clears self as expected if there is no Checkboxes field added to the entry.
I'm trying to update Scout from 1.2.1 to 1.3.0 and I get the following error when I run craft update scout in the console:
<warning>Package "craftcms/vue-asset" listed for update is not installed. Ignoring.</warning>
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Conclusion: remove craftcms/element-api 2.5.4
- Conclusion: don't install rias/craft-scout 1.3.0
- Installation request for rias/craft-scout 1.3.0 -> satisfiable by rias/craft-scout[1.3.0].
- craftcms/element-api 2.5.4 requires league/fractal ^0.16.0 -> satisfiable by league/fractal[0.16.0].
- craftcms/element-api 2.5.4 requires league/fractal ^0.16.0 -> satisfiable by league/fractal[0.16.0].
- craftcms/element-api 2.5.4 requires league/fractal ^0.16.0 -> satisfiable by league/fractal[0.16.0].
- Can only install one of: league/fractal[0.18.0, 0.16.0].
- Can only install one of: league/fractal[0.18.0, 0.16.0].
- Can only install one of: league/fractal[0.18.0, 0.16.0].
- Can only install one of: league/fractal[0.18.0, 0.16.0].
- rias/craft-scout 1.3.0 requires league/fractal ^0.18.0 -> satisfiable by league/fractal[0.18.0].
- Conclusion: don't install league/fractal 0.18.0
- Installation request for craftcms/element-api 2.5.4 -> satisfiable by craftcms/element-api[2.5.4].
<warning>Running update with --no-dev does not mean require-dev is ignored, it just means the packages will not be installed. If dev requirements are blocking the update you have to resolve those problems.</warning>
I've tried updating from 1.2.1 -> 1.2.2, 1.2.1 -> 1.2.3 and 1.2.1 -> 1.3.0 and every time I get this error. I'm not sure how to proceed, so any help is gratefully received!
Hey @Rias500 - we updated your plugin (thanks!) the other day and we've noticed that deleted entries are not being removed from Algolia so our entry count and Algolia count is off. Any chance you can help sort? We use FeedMe to import stock and has been working perfectly in hand with Scout.
Not sure if this is an issue with Craft or Scout however...
When pulling content from a matrix in the transform, querying the matrix will return the matrix in its original state.
for example:
Matrix (handle: contentBlocks) with one block (handle: text) containing plain text field
Initial saved entry with two (contentBlocks) blocks of type text
When adding a new block of type text, saving and sending to Scout though the following transformer:
'transformer' => function (craft\base\Element $element) {
$preQueryCount = count($element->contentBlocks);
$allTextBlocks = $element->contentBlocks->type('text')->all();
$postQueryCount = count($element->contentBlocks);
return [
...
];
}
$preQueryCount will be 3 however $postQueryCount will be 2 and $element->contentBlocks will now be the element in it's initial state.
I believe this has only started happening since Craft 3.2.
Has the import command changed or am i missing something? Seems not be able to find the command
Hey! I have two Algolia APIs that I'd like to sync content to. Is this something that you've done before? Reason being is that it's two accounts of products into one website. The search functions are completely separate but need to forward two different channels to two different Algolia APIs.
Any help would be awesome!
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.