Comments (3)
Hi,
Thanks for your finding! We are working on to fix this bug. However, this would not affect your performance significantly because of the relatively small amount of examples.
Best,
Tao
from spider.
We fixed the bug. However, considering this doesn't affect the performance significantly (also some results from different groups were already reported), we are going to release the new evaluation script in our future release.
from spider.
I am still experiencing this. This is running evaluation after training (model rat-sql):
{
"index": 38,
"beams": [
{
"orig_question": "What are the names of the singers who performed in a concert in 2014?",
"model_output": {
"_type": "sql",
"select": {
"_type": "select",
"is_distinct": false,
"aggs": [
{
"_type": "agg",
"agg_id": {
"_type": "NoneAggOp"
},
"val_unit": {
"_type": "Column",
"col_unit1": {
"_type": "col_unit",
"agg_id": {
"_type": "NoneAggOp"
},
"col_id": 9,
"is_distinct": false
}
}
}
]
},
"sql_where": {
"_type": "sql_where",
"where": {
"_type": "Eq",
"val_unit": {
"_type": "Column",
"col_unit1": {
"_type": "col_unit",
"agg_id": {
"_type": "NoneAggOp"
},
"col_id": 13,
"is_distinct": false
}
},
"val1": {
"_type": "Terminal"
}
}
},
"sql_groupby": {
"_type": "sql_groupby"
},
"sql_orderby": {
"_type": "sql_orderby",
"limit": false
},
"sql_ieu": {
"_type": "sql_ieu"
},
"from": {
"_type": "from",
"table_units": [
{
"_type": "Table",
"table_id": 1
},
{
"_type": "Table",
"table_id": 3
}
],
"conds": {
"_type": "Eq",
"val_unit": {
"_type": "Column",
"col_unit1": {
"_type": "col_unit",
"agg_id": {
"_type": "NoneAggOp"
},
"col_id": 8,
"is_distinct": false
}
},
"val1": {
"_type": "ColUnit",
"c": {
"_type": "col_unit",
"agg_id": {
"_type": "NoneAggOp"
},
"col_id": 21,
"is_distinct": false
}
}
}
}
},
"inferred_code": "SELECT singer.Name FROM singer JOIN singer_in_concert ON singer.Singer_ID = singer_in_concert.Singer_ID WHERE singer.Age = 'terminal'",
"score": -2.1817409903378575
}
]
}
from spider.
Related Issues (20)
- The hardness in SQL evaluation
- Not supporting any MySQL functions! HOT 5
- Instance 1222 in train json
- [Bug] Training Instance 3153, gold query refers to a table that doesn't exist in the db. HOT 5
- The results are inconsistent with the notes
- Tokenize new data HOT 3
- Exact Match for JOINs without the ON clause
- I couldn't find any measure of query difficulty in the Spider dataset HOT 3
- how is the test data generate?
- WHERE IN Clause for Multiple Values NOT Supported (where country in ('A','B','C','D') while Parsing HOT 1
- get_tables.py HOT 1
- Noob question about dev.json HOT 1
- Possible erroneous gold SQL ?
- Personal and Sensitive Information in the dataset HOT 1
- question about test HOT 2
- get schema method
- Some answers in evaluation_examples/dev.sql are incorrect HOT 2
- Invalid gold query for document_management HOT 3
- Expectation for data format in seq2seq_attention_copy
- Support with Large Language Models
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spider.