PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World [ACL 2021] This repo contains code and data for PIGLeT. If you like this paper, please cite us:
@inproceedings{zellers2021piglet,
title={PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World},
author={Zellers, Rowan and Holtzman, Ari and Peters, Matthew and Mottaghi, Roozbeh and Kembhavi, Aniruddha and Farhadi, Ali and Choi, Yejin},
booktitle ={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics},
year={2021}
}
See more at https://rowanzellers.com/piglet
- You can get data yourself by sampling trajectories in sampler/ and then converting them to
tfrecord
(which is the format I used) in tfrecord/. I also have the exact tfrecords I used atgs://piglet-data/physical-interaction-tfrecords/
-- they're big files so I turned on 'requester pays' for them. - You can pretrain the model and evaluate it in model/interact/train.py and model/interact/intrinsic_eval.py
- Alteratively feel free to use my checkpoint:
gs://piglet/checkpoints/physical_dynamics_model/model.ckpt-5420
- You can process data (also in tfrecord format) using data/zeroshot_lm_setup/prepare_zslm_tfrecord.py, or download at
gs://piglet-data/text-data/
. I have both 'zero-shot' tfrecord data, basically a version of BookCorpus and Wikipedia where certain concepts are filtered out, as well as non-zero shot (regularly processed). This was used to evaluate generalization to new concepts. - Train the model using model/lm/train.py
- Alternatively, feel free to just use my checkpoint:
gs://piglet/checkpoints/language_model/model.ckpt-20000
- Everything you need for this is in model/predict_statechange/ building on both the physical dynamics model and language model pretrained.
- I have annotations in data/annotations.jsonl for training and evaluating both tasks -- PIGPeN-NLU and PIGPeN-NLG.
- Alternatively you can download my checkpoints at
gs://piglet/checkpoints/pigpen-nlu-model/
for NLU (predicting state change given english text) orgs://piglet/checkpoints/pigpen-nlg-model/
for NLG.
That's it!
I used TPUs for this project so those are the only things I support right now, sorry!
I used tensorflow 1.15.5 and TPUs for this project. My recommendation is to use ctpu
to start up a VM with access to a v3-8
TPU. Then, use the following command to install dependencies:
curl -o ~/miniconda.sh -O https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh && \
chmod +x ~/miniconda.sh && \
~/miniconda.sh -b -p ~/conda && \
rm ~/miniconda.sh && \
~/conda/bin/conda install -y python=3.7 tqdm numpy pyyaml scipy ipython mkl mkl-include cython typing h5py pandas && ~/conda/bin/conda clean -ya
echo 'export PATH=~/conda/bin:$PATH' >>~/.bashrc
source ~/.bashrc
pip install "tensorflow==1.15.5"
pip install --upgrade google-api-python-client oauth2client
pip install -r requirements.txt