Entity Extractor

Entity Extractor built for Rasa using Microsoft Recognizers Text for extractig common entities like dates, time, currency, age, email id and others.

I have built the Entity Extactor as an alternative for Duckling Entity Extractor, since there some problems using Duckling as it requires one server running for entity extraction as well as it's not available directly on Windows system.

Please note that this entity extractor doesn't call's any cloud api, it works offline.

How to setup?

Note: Make sure you have already installed and setup the Rasa on your system

Step 1 : Install the Recognizer's suite with the following command: pip install recognizers-text-suite
Step 2 : Copy the MSRTEntityExtractor.py file in your project directory:
Step 3 : Add the reference of our Entity Extractor inside the Rasa NLU pipeline configuration file, as shown below:

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/

language: "en"

pipeline:
- name: "WhitespaceTokenizer"
- name: "RegexFeaturizer"
- name: "CRFEntityExtractor"
- name: "EntitySynonymMapper"
- name: "CountVectorsFeaturizer"
- name: "EmbeddingIntentClassifier"
- name: "MSRTEntityExtractor.MSRTExtractor"

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: KerasPolicy
  - name: MappingPolicy

Step 4 : Now you can train your model & test for the Entity Extractor

Check this sample Bot for reference: MoodBot

Samples

Type	Text	Output
Email Id	my email id is [email protected]	`"entities": [ { "start": 15, "end": 28, "text": "[email protected]", "entity": "email", "value": "[email protected]", "extractor": "MSRTExtractor" } ]`
DateTime	next Thursday at 8pm	`"entities": [ { "start": 0, "end": 3, "text": "next", "entity": "ordinal", "value": "0", "extractor": "MSRTExtractor" }, { "start": 0, "end": 19, "text": "next thursday at 8pm", "entity": "datetime", "value": "2020-01-02 20:00:00", "extractor": "MSRTExtractor" } ]`
Date	31st dec.	`"entities" : [ { "start" : 0 , "end" : 3 , "text" : "31st" , "entity" : "ordinal" , "value" : "31" , "extractor" : "MSRTExtractor" }, { "start" : 0 , "end" : 7 , "text" : "31st dec" , "entity" : "date" , "value" : "2019-12-31" , "extractor" : "MSRTExtractor" } ]`
Currency	Show me products below ₹5000	`"entities" : [ { "start" : 24 , "end" : 27 , "text" : "5000" , "entity" : "number" , "value" : "5000" , "extractor" : "MSRTExtractor" }, { "start" : 23 , "end" : 27 , "text" : "₹5000" , "entity" : "currency" , "value" : "5000" , "extractor" : "MSRTExtractor" } ],`
Phone Number	my contact number is 1234567890	`"entities" : [ { "start" : 21 , "end" : 30 , "text" : "1234567890" , "entity" : "number" , "value" : "1234567890" , "extractor" : "MSRTExtractor" }, { "start" : 21 , "end" : 30 , "text" : "1234567890" , "entity" : "phonenumber" , "value" : "1234567890" , "extractor" : "MSRTExtractor" } ]`
Age	I am 23 years old	`"entities" : [ { "start" : 5 , "end" : 6 , "text" : "23" , "entity" : "number" , "value" : "23" , "extractor" : "MSRTExtractor" }, { "start" : 5 , "end" : 16 , "text" : "23 years old" , "entity" : "age" , "value" : "23" , "extractor" : "MSRTExtractor" }, { "start" : 5 , "end" : 12 , "text" : "23 years" , "entity" : "datetimeV2.duration" , "value" : [ { "timex" : "P23Y" , "type" : "duration" , "value" : "725328000" } ], "extractor" : "MSRTExtractor" } ]`
Number	I have two apples	`"entities" : [ { "start" : 7 , "end" : 9 , "text" : "two" , "entity" : "number" , "value" : "2" , "extractor" : "MSRTExtractor" } ]`
Ordinal	7th	`"entities" : [ { "start" : 0 , "end" : 2 , "text" : "7th" , "entity" : "ordinal" , "value" : "7" , "extractor" : "MSRTExtractor" } ]`
Percentage	one hundred percents	`"entities" : [ { "start" : 0 , "end" : 10 , "text" : "one hundred" , "entity" : "number" , "value" : "100" , "extractor" : "MSRTExtractor" }, { "start" : 0 , "end" : 19 , "text" : "one hundred percents" , "entity" : "percentage" , "value" : "100%" , "extractor" : "MSRTExtractor" } ]`
Temperature	Set the temperature to 30 degrees celsius	`"entities" : [ { "start" : 23 , "end" : 24 , "text" : "30" , "entity" : "number" , "value" : "30" , "extractor" : "MSRTExtractor" }, { "start" : 23 , "end" : 40 , "text" : "30 degrees celsius" , "entity" : "temperature" , "value" : "30" , "extractor" : "MSRTExtractor" } ]`
Quarters	1st Quarter of 2020	`"entities" : [ { "start" : 15 , "end" : 18 , "text" : "2020" , "entity" : "number" , "value" : "2020" , "extractor" : "MSRTExtractor" }, { "start" : 0 , "end" : 2 , "text" : "1st" , "entity" : "ordinal" , "value" : "1" , "extractor" : "MSRTExtractor" }, { "start" : 0 , "end" : 18 , "text" : "1st quarter of 2020" , "entity" : "daterange" , "value" : { "start_date" : "2020-01-01" , "end_date" : "2020-04-01" }, "extractor" : "MSRTExtractor" } ]`

Language Support

Currently I have added suppport for English language, if you want to add other language, just edit the line here:

language = Culture.English

Chinese: Culture.Chinese
Dutch: Culture.Dutch
English: Culture.English
French: Culture.French
Italian: Culture.Italian
Japanese: Culture.Japanese
Korean: Culture.Korean
Portuguese: Culture.Portuguese
Spanish: Culture.Spanish
Turkish: Culture.Turkish

For more info., check the reference

References:
- Microsoft Recognizers Text

amedat04 / entityextractor Goto Github PK