Git Product home page Git Product logo

nli_project's Introduction

TOURIST GUIDE ASSISTANT

This project is based in 4 main modules:

  • SPR: Speach Recognition Module --> Two tasks 1.Convert client's voice in text, 2.Convert NLU text in the agent's voice
  • NLU: Natural Language Understanding Module --> Analyze and make predictions on the previous text using Machine Learning techniques
  • DM: Dialogue Manager Module --> Flow process control. The main module
  • NLG: Natural Language Generator --> Prediction text formatter

Additionally the DM has two Conversational trackers:

  • ct_interest: Conversational tracker in charge of flow conversations related to places of interest
  • ct_restaurant: Conversational tracker in charge of flow conversations related to restaurants and food in general

The complete flow process can be represented with the following graph:

Requirements

  • Docker
  • Pulseaudio

Usage

Install pulseaudio. Then execute the run.sh script (needs Docker).

After installation you can also do a quick test, just launch test.py

Authors

Natalia Pattarone

Sayan Goswami

Iñaki Lorente

nli_project's People

Contributors

iggyrrieta avatar say4n avatar npattarone avatar

Stargazers

Roger G. March avatar

Watchers

 avatar  avatar  avatar

nli_project's Issues

[DM] - Dialogue Act Restaurant Search

A) We should create submodules for separate dialogue acts (specific Classes):

  • restaurant search
  • interesting places (we can use 2nd Notebook I sent to search with Wikipedia API)
  • google maps integration (for directions)
  • weather??? (later)

B) Types of interactions: utterances, intentions, entities (slots)

  • restaurant search for particular cuisine
  • places of interest (by direct question or by open question)
  • weather??? (later)

[DM] - Dialogue Act Google Maps Integration

A) We should create submodules for separate dialogue acts (specific Classes):

  • restaurant search
  • interesting places (we can use 2nd Notebook I sent to search with Wikipedia API)
  • google maps integration (for directions)
  • weather??? (later)

B) Types of interactions: utterances, intentions, entities (slots)

  • restaurant search for particular cuisine
  • places of interest (by direct question or by open question)
  • weather??? (later)

Create flow architecture

Check Gdrive doc

Now this projects contains 3 initial proposed scripting that work following the modules idea found on the previous link.

Inside folder imutils:

  • dmanager.py --> Dialogue manager. Contains a class with dialogue functions. From here we move to all other scripting

  • processing_layer.py --> Audio Input to text Output processing scripting

  • semantic_layer.py --> Context, History and connection to DB (maybe we can split this)

Inside folder media:

  • Here, all audios are saved

Inside folder templates:

  • Just the html

Inside folder doc:

  • Requirement doc and all other coding-related doc

Inside folder notebooks:

  • Probably you like to test using notebooks and also it is good to explain how things work. Now there is only a notebook about pre-processing and classification tasks.

Now the project is made using Flask so there is also a app.py script. We will only keep this if we decide to use Flask.

[DM] - Dialogue Act Interesting Places

A) We should create submodules for separate dialogue acts (specific Classes):

  • restaurant search
  • interesting places (we can use 2nd Notebook I sent to search with Wikipedia API)
  • google maps integration (for directions)
  • weather??? (later)

B) Types of interactions: utterances, intentions, entities (slots)

  • restaurant search for particular cuisine
  • places of interest (by direct question or by open question)
  • weather??? (later)

GTT (The speech recognition system) is working a bit weird

As the title says, it is not as good as we would like it to be. The main problem at least in my computer is that it sometimes fails to reproduce the audio. This is solved using a try-catch but it makes the system slow, waiting it gets up again.

  • Look for other SR systems
  • Try to improve this one (some quick research)
  • Maybe increase to 10sec for the user to speak

Syntactic/Semantic Analysis

  • PoS Tagging
  • Dependency Parser

We can use the example Natu sent (first Notebook), which uses spaCy.

TODO:

  • Create the model (improve training) for the dialog examples we have on the Doc
  • From the inputs, by using spacy, you select main words depending on the utterance detected

[DM] - Dialogue Management Model Training for Interactions

A) We should create submodules for separate dialogue acts (specific Classes):

  • restaurant search
  • interesting places (we can use 2nd Notebook I sent to search with Wikipedia API)
  • google maps integration (for directions)
  • weather??? (later)

B) Types of interactions: utterances, intentions, entities (slots)

  • restaurant search for particular cuisine
  • places of interest (by direct question or by open question)
  • weather??? (later)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.