tmhoward / h2sl Goto Github PK
View Code? Open in Web Editor NEWH2SL Natural Language Symbol Grounding Project
License: GNU General Public License v2.0
H2SL Natural Language Symbol Grounding Project
License: GNU General Public License v2.0
H2SL (Human to Structured Language) provides a framework for realtime natural langugage symbol grounding in the context of understanding robot instructions. Copyright (C) 2014 by the Massachusetts Institute of Technology Developed by Thomas M. Howard and Matthew R. Walter at the Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, Massachusetts USA, with partial support from the U.S. Army Research Laboratory under the Robotics Collaborative Technology Alliance, Cooperative Agreement W911NF-10-2-0016 and National Science Foundation National Robotics Initiative Award #1427547. Contributors to this project include: Derya Aksaray Jacob Arkin Rohan Paul Thomas Howard Matthew Walter This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, see <http://www.gnu.org/licenses/gpl-2.0.html> or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. OVERVIEW H2SL provides a set of base classes for developing a natural language symbol grounding inferface. Example symbols, features, and data is provided, but is not intended to serve as a drop-in natural language interface for robotic system. To adapt to new applications, one can replace the symbols, features, and examples with those that are relevant to your system and domain. The probablistic graphical models included in this project consist of the Distributed Correspondence Graph (DCG), Hierarhical Distributed Correspondence Graph (HDCG), Adaptive Distributed Correspondence Graph (ADCG), and Hierarhical Adaptive Distributed Correspondence Graph (HADCG). For more details about the mathematical basis of each model, please refer to the following papers and articles: T.M Howard, S. Tellex, and N. Roy. A natural language planner interface for mobile manipulators. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 6652โ6659. IEEE, May 2014. O. Propp, I. Chung, M.R. Walter, and T.M. Howard. On the performance of hierarchical distributed correspondence graphs for efficient symbol grounding of robot instructions. In Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 2015. R. Paul, J. Arkin, N. Roy, and T.M. Howard. Effcient grounding of abstract spatial concepts for natural language interaction with robot manipulators. In Proceedings of the 2016 Robotics: Science and Systems Conference, June 2016. Examples of applications of this framework to various problems in natural language symbol grounding of robot instructions include inferring LTL from language, assistive robotics, inferring maps and behaviors, expressing homotopy constraints and multi-modal interactions with human-robot teams. For more details of these applications, please refer to the following papers and articles: F. Duvallet, M.R. Walter, T.M. Howard, S. Hemachandra, J. Oh, S. Teller, N. Roy, and A. Stentz. A probabilistic framework for inferring maps and behaviors from natural language. In Proceedings of the 14th International Symposium on Experimental Robotics, July 2014. S. Hemachandra, F. Duvallet, T.M. Howard, N. Roy, A. Stentz, and M.R. Walter. Learning models for following natural language directions in unknown environments. In Proceedings of the IEEE International Conference on Robotics and Automation. IEEE, May 2015. D. Yi, T.M. Howard, K. Seppi, and M. Goodrich. Expressing homotopic requirements for mobile robot navigation through natural language instructions. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 2016. J. Oh, T.M. Howard, M. Walter, D. Barber, M. Zhu, S. Park, A. Suppe, L. Navarro-Serment, F. Duvallet, A. Boularias, O. Romero, J. Vinokrov, T. Keegan, R. Dean, C. Lennon, B. Bodt, M. Childers, J. Shi, K. Daniilidis, N. Roy, C. Lebiere, M. Hebert, and A. Stentz. Integrated intelligence for human-robot teams. In Proceedings of the 2016 International Symposium on Experimental Robotics, October 2016. A. Boteanu, J. Arkin, T.M. Howard, and H. Kress-Gazit. A model for verifable grounding and execution of complex language instructions. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 2016. A. Broad, J. Arkin, N. Ratliff, T. M. Howard, and B. Argall. Real-time natural language corrections for assistive robotic manipulators. International Journal of Robotics Research, 36(5-7):684โ698, May 2017. J. Arkin, M. Walter, A. Boteanu, M. Napoli, H. Biggie, H. Kress-Gazit, and T.M. Howard. Contextual awareness: Understanding monologic natural language instructions for autonomous robots. In Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication, August 2017. TUTORIAL (DCG) To train the model from a corpus of labeled examples, run ... h2sl-llm-train (examples files) --feature_set=(feature_set.xml) --output=(llm.xml) To run the Distributed Correspondence Graph (DCG) demo, run ... h2sl-dcg-demo --world=(world.xml) --llm=(llm.xml) --grammar=(grammar.xml) --command=(command string) -output=(output.xml) To run the Distributed Correspondence Graph (DCG) test, run ... h2sl-dcg-test (example files) --llm=(llm.xml) --grammar=(grammar.xml) To run the Graphical User Interface (GUI), run ... h2sl-gui-demo --world=(world.xml) --llm=(llm.xml) --grammar=(grammar.xml)
-add num_phrases() and num_words() functions.
-add to_tikz function
Want to have a protected member that is a pointer to the parent phrase class.
feature_object_matches_child
feature_object_matches_child_container_object
feature_object_merge_abstract_container_index_region_container
feature_container_merge_abstract_container_region_container
feature_container_merge_container_pair
feature_object_merge_container_pair
feature_object_merge_abstract_container_region_container
feature_object_merge_object_property_spatial_relation
feature_object_merge_abstract_container_spatial_relation
feature_object_merge_object_container
feature_object_merge_single_object_different_container
feature_object_merge_object_different_container
feature_object_merge_object_property_container
feature_object_merge_object_property_region_container
feature_object_merge_object_region_container
feature_object_type_matches_child_abstract_container
Transfer the LLM training process and the arguments file.
To be done after doing DCG.
Add the spatial features left.
Transfer code to re-create the results for the RSS/IJRR paper. Account for the new models and navigation domain.
Target files:
move the examples in the corpus from h2sl_nsf_nri_mvli
create another folder containing examples with no actions
Abstract out the search space class to encapsulate vector of groundings and CVs.
Forgot to add newly transferred features to xml file, I will add them
feature_region_container_type
feature_region_container_container_type
Hi There!
Can the 'adcg' branch reproduce results from your RSS paper (http://www.roboticsproceedings.org/rss12/p37.html)?
Regards
Add value functions to each feature that allows for compatibility with the monologue context function argument.
-Create Demo program for each class
-Add dummy data
-Add class_name() function
-fix any issues with class functions
Introduce hand-crafted example files for testing in the navigation domain (doors, buildings).
Update the data structures for pre-computed data structures for the spatial characteristics. Change from arrays to maps.
This will lead to changes in the spatial features
For example, "the rightmost block" will now have an index symbol expressed as well as an object_property symbol.
Create a demo program and dummy data to test the container.cc functions; add a script to run the program.
Integrate ADCG model into h2sl project.
Branch to cleanly add an initial push of the navigation dataset.
Extending the symbol space of h2sl based on h2sl_nsf_nri_mvli
The h2sl feature_set.xml has the following:
<feature_matches_child_object invert="0"/>
<feature_matches_child_object invert="1"/>
<feature_matches_child_region invert="0"/>
<feature_matches_child_region invert="1"/>
<feature_matches_child_constraint invert="0"/>
<feature_matches_child_constraint invert="1"/>
These are now templated:
feature_object_matches_child.h and .ipp
The individual feature_matches_child_region etc. have now gone away but still live in the feature_set.xml file.
How do we load in templated features from the feature_set.xml file. Do we have an example?
Jake could you discuss this with Tom and let us know.
Thanks
feature_abstract_container_type
feature_abstract_container_color
feature_abstract_container_number
feature_abstract_container_index
Transfer the NL interface. Not super important right now.
feature_object_property_type
feature_object_property_relation_type
Add template-based features to simplify derived models
Update the demo program to match the abstract_container demo; add a script to run the demo.
Add demo program to test the class functions for object.cc
Anticipation is that the to/from xml functions are not capturing the position/orientation information.
New branch created for ongoing work on adding spatial regions, exploration to guide grounding of unknown symbols. This supports work by Nicole Glabinski (in Nick Roy's group).
Generate a grammar file for the new door_dataset
Make sure the feature_set.xml file has all instances of all features we need. Also, ensure we are not duplicating features by including specialized an templated features that have the same value function.
Add the symbol dictionary in inference functions.
ToDo:
-update abstract_container to have a "class_name" function
-generate a demo program to test the class functions
-fix any issues uncovered during testing
Copy constructor has small error
Cleanup and restructure the data folder. Will also add more navigation example data.
Add a demo program to test the class functions for number.cc.
want choice of whether project builds static and/or dynamic libraries
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.