Current version : 1.3.3 (changelog)

Nolan Lawson

Health On the Net Foundation

Apache 2.0.

Extension of the ExtendedDisMaxQueryParserPlugin that splits queries into a "normal" query and a "synonym" query. This enables proper query-time synonym expansion, with no reindexing required.

This also fixes lots of bugs with how Solr typically handles synonyms using the SynonymFilterFactory.

For more details, read my blog post on the subject.

The following tutorial will set up a working synonym-enabled Solr app using the example/ directory from Solr itself, running in Jetty.

Protip: The unit tests will do these steps automatically.

Download the latest JAR file depending on your Solr version:

Download Solr from the Solr home page. For this tutorial, we'll use Solr 3.6.2. You do not need the sources; the tgz or zip file will work fine.

Extract the compressed file and cd to the example/ directory.

Now, you need to bundle the hon-lucene-synonyms-*.jar file into webapps/solr.war. Below is a script that will work quite nicely on UNIX systems. Be sure to change the /path/to/my/hon-lucene-synonyms-*.jar part before running this script.

mkdir myjar
cd myjar
jar -xf ../webapps/solr.war 
cp /path/to/my/hon-lucene-synonyms-*.jar WEB-INF/lib/
jar -cf ../webapps/solr.war *
cd ..

Note that this plugin will not work in any location other than the WEB-INF/lib/ directory of the solr.war itself, because of issues with the ClassLoader.

Update: We have tested to run with the jar in $SOLR_HOME/lib as well, and it works (Jetty).

Download example_synonym_file.txt and copy it to the solr/conf/ directory (solr/collection1/conf/ in Solr 4.x).

Download example_config.xml and copy the contents into solr/conf/solrconfig.xml (solr/collection1/conf/solrconfig.xml in 4.x), just before the </config> tag at the end.

This defines the analyzer that will be used to generate synonyms.

Protip: You can customize this analyzer based on your synonym set. E.g. if your synonyms are all two words or less, you can safely set maxShingleSize to 2.

Solr 4.3+ Protip: For Solr 4.3 and up, we support loading Tokenizers and Token Filters by service name through the new SPI method. That means you may put synonym instead of solr.SynonymFilterFactory or shingle instead of solr.ShingleFilterFactory, if you'd like to make your configuration more succinct.

Start up the app by running java -jar start.jar. Jetty may print a ClassNotFoundException, but it shouldn't matter.

In your browser, navigate to

http://localhost:8983/solr/select/?q=dog&debugQuery=on&qf=text&defType=synonym_edismax&synonyms=true

You should see a response like this:

<response>
  ...
  <result name="response" numFound="0" start="0"/>
  <lst name="debug">
    <str name="rawquerystring">dog</str>
    <str name="querystring">dog</str>
    <str name="parsedquery">
        +(DisjunctionMaxQuery((text:dog)) (((DisjunctionMaxQuery((text:canis)) 
        DisjunctionMaxQuery((text:familiaris)))~2) DisjunctionMaxQuery((text:hound)) 
        ((DisjunctionMaxQuery((text:man's)) DisjunctionMaxQuery((text:best)) 
        DisjunctionMaxQuery((text:friend)))~3) DisjunctionMaxQuery((text:pooch))))
    </str>
    <str name="parsedquery_toString">
        +((text:dog) ((((text:canis) (text:familiaris))~2) (text:hound) 
        (((text:man's) (text:best) (text:friend))~3) (text:pooch)))
    </str>
    <lst name="explain"/>
    <str name="QParser">SynonymExpandingExtendedDismaxQParser</str>
    ...
  </lst>
</response>

Note that the input query dog has been expanded into dog, hound, pooch, canis familiaris, and man's best friend.

Boost the non-synonym part to 1.2 and the synonym part to 1.1 by adding synonyms.originalBoost=1.1&synonyms.synonymBoost=1.2:

+((text:dog)^1.1 (((((text:canis) (text:familiaris))~2) (text:hound) 
(((text:man's) (text:best) (text:friend))~3) (text:pooch))^1.2))

Apply a minimum "should" match of 75% by adding mm=75%25:

+((text:dog) ((((text:canis) (text:familiaris))~1) (text:hound) 
(((text:man's) (text:best) (text:friend))~2) (text:pooch)))

Observe how phrase queries are properly handled by using q="dog" instead of q=dog:

+((text:dog) ((text:"canis familiaris") (text:hound) (text:"man's best friend") (text:pooch)))

Keep in mind that you must add defType=synonym_edismax and synonyms=true to enable the parser in the first place.

Also, you must either define qf in the query parameters or defaultSearchField in solr/conf/schema.xml, so that the parser knows which fields to use during synonym expansion.

If you enable debugging (with debugQuery=on), the plugin will output helpful information about how synonyms are being expanded.

The following are parameters that you can use to tweak the synonym expansion.

Download the code and run:

mvn install

Since there are several branches depending on the Solr version, there's also a build script that will git checkout each branch, build it, and put the compiled jar files into the target/s3 directory:

./build_all_versions.sh

Basically, my strategy is to maintain a main master/solr-4.3.x branch, with offshoot branches solr-3.x, solr-4.0.0, and solr-4.1.0. When I need to backport changes to older versions of Solr, I just git rebase each of the offshoots.

Python-based unit tests are in the test/ directory. You can run them using these commands:

# install the solrpy and nose packages
sudo easy_install nose
sudo easy_install solrpy

# downloads, builds, and launches Solr on localhost:8983
./run_solr_for_unit_tests.py

# run some Python unit tests against the local Solr
nosetests test/

• v1.3.3
  • Fixed #33: synonyms are now weighted equally, regardless of how many there are per word.
  • Fixed #31: synonyms are no longer given extra weight when using the params bq, bf, and boost.
  • debugQuery=on now gives more helpful debug output.
  • Fixed #9, #26, #32, and #34. Note that this is a documentation change; not a code change, so to get the benefits of this "fix," you'll need to manually perform Step 6 again.
• v1.3.2
  • Added synonyms.ignoreQueryOperators option (#28)
  • Added synonyms.bag option (#30)
  • The run_solr_for_unit_tests.py script now downloads the proper version of Solr.
• v1.3.1
  • Avoid luceneMatchVersion in config (#20)
• v1.3.0
  • Added support for Solr 4.3.0 (#19)
  • New way of loading Tokenizers and TokenFilters
  • New XML syntax for config in solrconfig.xml
• v1.2.3
  • Fixed #16
  • Verified support for Solr 4.2.0 with the 4.1.0 branch (unit tests passed)
  • Improved automation of unit tests
• v1.2.2
  • Added synonyms.constructPhrases option to fix #5
  • Added proper handling for phrase slop settings
• v1.2.1
  • Added support for Solr 4.1.0 (#4)
• v1.2
  • Added support for Solr 4.0.0 (#3)
• v1.1
  • Added support for Solr 3.6.1 and 3.6.2 (#1)
  • Added "Getting Started" instructions to clarify plugin usage (#2)
• v1.0
  • Initial release