aixcoder-plugin / nl2code-dataset Goto Github PK
View Code? Open in Web Editor NEWAix-bench, the Java benchmark for code synthesis problem.
License: MIT License
Aix-bench, the Java benchmark for code synthesis problem.
License: MIT License
前一段时间关注到了skcoder,在aixbench上效果很好,所以想尝试复现一下在其实验结果。
但是有两个问题,希望能够解答🙏🏻
1 想问一下skcoder这篇论文中的aixbench用的测试集就是这个仓库里的文件吗?
2 如何计算pass@k呢,在humaneval中是通过重复生成200个答案,然后通过采样的方式获得稳定的pass@k的指标,这个仓库中实现代码我看是直接把生成的代码封装成类跑一下evaluation查看是否通过,这样的结果应该算作是pass@1。但是选取这样生成一次的结果来计算指标的偏差可能存在过大的问题,所以想问一下skcoder中pass@k的计算方式和这个仓库一样吗?还是说有什么其他等效的方法。
希望能得到您的回复,感谢感谢!
需要将项目构建成maven吗,因为发现有一些包不存在
首先感谢公开本测试集!
其次,有两个问题没有在README中看到,烦请回答一下:
Copilot的结果是如何测试的?通过Codex API接口吗?
请问能公开对各个代码样本的人类评价分数吗
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.