aixcoder-plugin / nl2code-dataset Goto Github PK
View Code? Open in Web Editor NEWAix-bench, the Java benchmark for code synthesis problem.
License: MIT License
nl2code-dataset's People
Contributors
Stargazers
Watchers
Forkers
amaal04nl2code-dataset's Issues
pass@k的计算方式
前一段时间关注到了skcoder，在aixbench上效果很好，所以想尝试复现一下在其实验结果。
但是有两个问题，希望能够解答🙏🏻
1 想问一下skcoder这篇论文中的aixbench用的测试集就是这个仓库里的文件吗？
2 如何计算pass@k呢，在humaneval中是通过重复生成200个答案，然后通过采样的方式获得稳定的pass@k的指标，这个仓库中实现代码我看是直接把生成的代码封装成类跑一下evaluation查看是否通过，这样的结果应该算作是pass@1。但是选取这样生成一次的结果来计算指标的偏差可能存在过大的问题，所以想问一下skcoder中pass@k的计算方式和这个仓库一样吗？还是说有什么其他等效的方法。
希望能得到您的回复，感谢感谢！
生成好代码后，怎么测试？
需要将项目构建成maven吗，因为发现有一些包不存在
关于测试集和testcases的两个问题
首先感谢公开本测试集！
其次，有两个问题没有在README中看到，烦请回答一下：
- 测试集是否与AiXCoder模型的训练集隔离？
- testcases是从原项目收集的吗？还是人工补充的呢？
Copilot的结果是如何测试的
Copilot的结果是如何测试的？通过Codex API接口吗？
人类评价分数
请问能公开对各个代码样本的人类评价分数吗
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.