facebookarchive / babi-tasks Goto Github PK
View Code? Open in Web Editor NEWTask generation for testing text understanding and reasoning
License: Other
Task generation for testing text understanding and reasoning
License: Other
Example: qa16_basic-induction_train.txt
1 Julius is a rhino.
2 Brian is a swan.
3 Brian is white.
4 Greg is a frog.
5 Julius is yellow.
6 Greg is gray.
7 Bernhard is a lion.
8 Lily is a rhino.
9 Lily is green.
10 ???
1 Lily is a frog.
2 Bernhard is a frog.
3 Bernhard is green.
4 Brian is a lion.
5 Brian is white.
6 Julius is a swan.
7 Julius is green.
8 Lily is green.
9 Greg is a swan.
10 What color is Greg? green 9 6 7
Similar to Issue #1 , data generated from task 17 also seems problematic. With the following command:
babi-tasks 17 10
Here is one generated problematic example:
1 The rectangle is below of the red square.
2 The red square is below of the rectangle.
3 The rectangle is above of the red square.
4 The pink rectangle is to the left of of the red square.
5 Is the rectangle below the red square? no 2 1
6 Is the pink rectangle to the left of the rectangle? yes 2 1
7 Is the pink rectangle above the rectangle? no 2 1
8 Is the rectangle to the right of the pink rectangle? yes 2 1
9 Is the rectangle above the red square? yes 2 1
10 Is the rectangle below the red square? no 2 1
11 Is the red square below the rectangle? yes 2 1
12 Is the pink rectangle to the right of the red square? no 2 1
if I want debug it, for example, I edit the source and run it immediately without reinstall it by luarocks, how can I do that
The dataset (The SimpleQuestions dataset) provided on the project page seems to be unavailable
https://research.fb.com/downloads/babi/
That is the broken link: https://www.dropbox.com/s/tohrsllcfy7rch4/SimpleQuestions_v2.tgz
Can you please make it available again?
The dataset (The SimpleQuestions dataset) provided on the project page seems to be unavailable
https://research.fb.com/downloads/babi/
That is the broken link: https://www.dropbox.com/s/tohrsllcfy7rch4/SimpleQuestions_v2.tgz
I can get simplequestion.txt from https://s3.amazonaws.com/fairdata/parlai/simplequestions/simplequestions.tar.gz,
but I can't get subset of freebase,fb2m, fb5M.
Maybe it's my problem, but can you provide a link like https://s3.amazonaws.com
I tried to generate some task data for question 19 with the following default command:
babi_tasks 19 1000 > test_19.txt
In the resulting file I got, I observed some conflicting facts, for example
1 The office is south of the garden.
2 The office is west of the hallway.
3 The kitchen is west of the garden.
4 The garden is south of the office.
5 What is the path from hallway to garden? w,n 1 2
1 and 4 cannot hold at the same time. Another example is
1 The kitchen is west of the garden.
2 The hallway is north of the kitchen.
3 The office is south of the garden.
4 The office is east of the garden.
5 What is the path from kitchen to office? e,s 3 1
Here 3 and 4 are confusing. And one more:
1 The bedroom is west of the bathroom.
2 The garden is north of the bedroom.
3 The hallway is west of the bedroom.
4 The hallway is north of the garden.
5 What is the path from bedroom to garden? w,s 3 4
In this one there are two paths from bedroom to garden, (1) using fact 2; (2) using 3 and 4. The generated ground truth answer ignored the more direct solution, and the two solutions are actually conflicting, as fact 2 says garden is north of the bedroom but fact 3 and 4 together imply that garden is south of the bedroom.
Is this a common problem?
Hello , I was doing a project which requires to generate a conversations artificially, like the The Dialog-based Language Learning dataset of Facebook AI. Can I do that. Please Note I want the domain of my conversations to be realted to a student library.
Hi I generated a simple example for task 4:
asim@XX:~/bAbI-tasks/: babi-tasks 4 1
1 The office is south of the garden.
2 The hallway is south of the office.
3 What is south of the hallway? office 2
I consistently see such examples being generated with incorrect answers.
I am getting incorrect/inconsistent answers generated from tasks 5 and 17.
For task 5:
...
11 John journeyed to the kitchen.
12 Daniel gave John the milk.
13 Daniel moved to the bathroom.
14 Mary journeyed to the kitchen.
15 John gave Daniel the milk.
16 Who received the milk? John 12
17 Sandra moved to the bedroom.
18 Sandra moved to the garden.
19 John moved to the hallway.
20 Sandra went to the office.
21 John gave Daniel the apple.
22 John journeyed to the bathroom.
23 Daniel gave John the apple.
24 Who received the apple? John 23
Note that lines 12 and 15 both involve trading milk between Daniel and John. In this case the answer to "Who received the milk?" (16) is John. On lines 21 and 23, there is a trade of the apple, and in this case the answer to "Who received the apple?" (24) is also John. But the order of the trade is reversed between the two cases, so it seems like at least one of these answers is incorrect. (In particular, if the question is supposed to correspond to the most recent transfer, then the first answer, on line 16, is incorrect.)
For task 17:
1 The pink triangle is above of the square.
2 The red square is to the right of of the pink triangle.
...
10 Is the red square to the left of the pink triangle? yes 1 2
This is wrong according to the facts given. (Also, I think there is an extra "of" in the generated facts.)
Since these are randomly generated, these exact examples might not be directly reproducible, but I was able to generate similar inconsistencies by repeatedly running the generation code for these two tasks.
Based on the Task name, this shouldn't be the case or am I wrong.
I am getting stories such as -:
1 Sandra travelled to the kitchen.
2 Sandra grabbed the milk.
3 Daniel journeyed to the bedroom.
4 Sandra dropped the milk.
5 Daniel journeyed to the hallway.
6 Sandra took the milk.
7 Sandra went to the bedroom.
8 Where was the milk before the bedroom? kitchen 1 2 6 7
function Grid:to_coordinates(i)
return (i - 1) % self.width + 1, math.floor((i - 1) / 10) + 1
end
Should the number 10 here actually be self.width?
What is the syntax for using world:load(filename) to add new worlds?
Let's say the file I wanted to load was world.txt.
I've tried world:load(world.txt)
, world:load world.txt
, and other similar formats with varying spacing, but I only get syntax error near unexpected token '('
or syntax error near unexpected token 'world.txt'
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.