sc-forks / solidity-coverage Goto Github PK
View Code? Open in Web Editor NEWCode coverage for Solidity smart-contracts
License: MIT License
Code coverage for Solidity smart-contracts
License: MIT License
solidity-coverage instrument <files>
require
a cli.js file for our unit testsGnosis has a test that takes like 10 minutes to cover and the file i/o at the LOG opcode could be part of that. It would be nice to find where the vm exits so we could write the logs in bulk.
And this, while visually balanced, is also insane.
Speeding things up is critical if the coverage tool is going to get plugged into a fuzzer
What do you think about allowing the user to specify the solc
version being used?
The use-case would be a compiler regression that affects contracts, such as this one.
Of course, one can always downgrade/pin the solidity-coverage
version, but features developed since then will be unavailable without getting the offending solc
version as well.
One way to achieve this without dealing with different NPM packages may be using legacy solc versions, and specifying it in the .solcover.js
file (though I have not tried this).
Anyway, let me know if this makes any sense to you :)
Two cases:
From zeppelin CI build 433 at the start of the Claimable test. i.e. testrpc-sc launches ok, runs for a while and crashes.
/home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/ethereumjs-testrpc-sc/node_modules/web3-provider-engine/node_modules/solc/soljson.js:1
(function (exports, require, module, __filename, __dirname) { var Module;if(!Module)Module=(typeof Module!=="undefined"?Module:null)||{};var moduleOverrides={};for(var key in Module){if(Module.hasOwnProperty(key)){moduleOverrides[key]=Module[key]}}var ENVIRONMENT_IS_WEB=typeof window==="object";var ENVIRONMENT_IS_WORKER=typeof importScripts==="function";var ENVIRONMENT_IS_NODE=typeof process==="object"&&typeof require==="function"&&!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_WORKER;var ENVIRONMENT_IS_SHELL=!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_NODE&&!ENVIRONMENT_IS_WORKER;if(ENVIRONMENT_IS_NODE){if(!Module["print"])Module["print"]=function print(x){process["stdout"].write(x+"\n")};if(!Module["printErr"])Module["printErr"]=function printErr(x){process["stderr"].write(x+"\n")};var nodeFS=require("fs");var nodePath=require("path");Module["read"]=function read(filename,bi
NotFoundError: Key not found in database
at /home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/level-sublevel/shell.js:101:18
at /home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/level-sublevel/nut.js:121:19
From Gnosis build 100.1 at the start of the run, before truffle compile.
/home/travis/build/gnosis/gnosis-contracts/node_modules/solc/soljson.js:1
(function (exports, require, module, __filename, __dirname) { var Module;if(!Module)Module=(typeof Module!=="undefined"?Module:null)||{};var moduleOverrides={};for(var key in Module){if(Module.hasOwnProperty(key)){moduleOverrides[key]=Module[key]}}var ENVIRONMENT_IS_WEB=typeof window==="object";var ENVIRONMENT_IS_WORKER=typeof importScripts==="function";var ENVIRONMENT_IS_NODE=typeof process==="object"&&typeof require==="function"&&!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_WORKER;var ENVIRONMENT_IS_SHELL=!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_NODE&&!ENVIRONMENT_IS_WORKER;if(ENVIRONMENT_IS_NODE){if(!Module["print"])Module["print"]=function print(x){process["stdout"].write(x+"\n")};if(!Module["printErr"])Module["printErr"]=function printErr(x){process["stderr"].write(x+"\n")};var nodeFS=require("fs");var nodePath=require("path");Module["read"]=function read(filename,binary){filename=nodePath["normalize"](filename);var ret=nodeFS["readFileSync"
Error: Cannot find module 'levelup/lib/errors'
at Function.Module._resolveFilename (module.js:485:15)
at Function.Module._load (module.js:437:25)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/home/travis/build/gnosis/gnosis-contracts/node_modules/level-sublevel/shell.js:4:14)
at Module._compile (module.js:569:30)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
Hi,
using solidity-coverage 0.2.11 I get the following error on my contract:
/home/jerome/src/eth/contracts/coverageEnv/contracts/HeroCoin.sol:449:24: ParserError: Expected primary expression.
returns (uint256) {/__FunctionCoverageHeroCoin('/home/jerome/src/eth/contracts/contracts/HeroCoin.sol',23);
^
Compiliation failed. See above.
Cleaning up...
There was an error generating coverage. Possible reasons include:
1. Another application is using port 8555
2. Truffle crashed because your tests errored
Error: ENOENT: no such file or directory, open './allFiredEvents'
Exiting without generating coverage...
My truffle unit tests compile and pass.
Environment:
solcover 0.2.1
solc 0.4.11
ethereumjs-testrpc 4.0.1
truffle 3.4.5
When I run ./node_modules/.bin/solidity-coverage
after installing in my project, I get the following error.
Launching testrpc on port 8555
Launching test command (this can take a few seconds)...
Using network 'development'.
SyntaxError: Error parsing /home/travis/prg/protocol/coverageEnv/contracts/governance/GovernanceProtocol.sol: Expected "contract", "import", "library", "pragma", "using", comment, end of input, end of line, or whitespace but "e" found. Line: 6, Column: 31
at peg$buildStructuredError (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/build/imports_parser.js:543:12)
at Object.peg$parse [as parse] (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/build/imports_parser.js:4142:11)
at Object.parse (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/index.js:34:23)
at /usr/local/lib/node_modules/truffle/node_modules/truffle-compile/profiler.js:200:36
at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/index.js:79:5
at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/internal/onlyOnce.js:12:16
at next (/usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/whilst.js:68:18)
at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/index.js:64:7
at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/fs.js:65:5
at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/internal/once.js:12:16
Cleaning up...
There was an error generating coverage. Possible reasons include:
1. Another application is using port 8555
2. Truffle crashed because your tests errored
Error: ENOENT: no such file or directory, open './allFiredEvents'
Exiting without generating coverage...
May be worth nothing that there is no configuration stub generated in my truffle.json
after this error, which should be there if I understand the readme correctly.
This is almost definitely something wrong on my end though, since it looks like you guys got it to work on the melonport protocol just fine, which is the code that I'm currently trying the coverage tool on.
A new version of ganache-core
is being published w/ various bugfixes and improvements. It's also moving from [email protected]
to [email protected]
.
Todo:
Related to #63 by virtue of changing the vm.
ethereumjs-vm-sc
so that it doesn't add our events to the logs.Strategy: in coverage.js
, write topics to a file called scTopicList
. In opFns.js
load contents of that file into an array and check the topics of each Log OP against it. If it's ours, don't add it to the logs array?
Unlike npm
, yarn
does not install binaries of dependencies to ./node_modules/.bin
(see here). As a result when we try and run testrpc-sc
, it fails. This could be fixed by executing ./node_modules/ethereumjs-testrpc-sc/bin/testrpc
rather than ./node_modules/.bin/testrpc-sc
. Alternatively, the onus could be on the use a tool like yarn-bin-fix.
A PR discussion at Open-Zeppelin caught this. And here is Istanbul on the same issue.
I wonder if we could (reliably) pre-process / instrument these into conditionals somehow. . .
A list of steps from #63:
Other notes:
ganache-core
installs with [email protected]
and [email protected]
blockGasLimit = options.gasLimit || "0xfffffffffff"
defaultTransactionGasLimit = '0xfffffff'
solidity-coverage assumes the Truffle config is at truffle.js
. There is an alternative filename truffle-config.js
as seen in the documentation.
Generalising this issue to contain all cases of valid solidity that we know we can't instrument...
These are either going to need edge cases coded, or use of debug_traceTransaction
to avoid instrumentation all together.
Hopefully, nothing here would be something realistically used in 'real' solidity contracts, and are just odds and ends for us to look at.
Instead of having a huge list that requires updates, we should just check for the existence of the function in the table before calling it
e.g:
parse[modifier.type] &&
parse[modifier.type](contract, modifier);
var (, , , , , sharePrice) = performCalculations();
is not being parsed correctly - statement instrumentation is getting injected in front of the function call.
Found while testing against the Aragon contracts. They have a line like this:
vote.voted[voter] = isYay ? 1 : 2;
The left hand side parses as a member expression and there's a summary exit in the instrumented.js
code for AssignmentExpressions
to notify us if we encounter a node-type we didn't expect. Only handling DeclarativeExpressions and Identifiers at the moment.
Opening a PR for this shortly.
exec.js
should be in /bin
/lib
And possibly PR these in? If useful?
NPM package name should be changed back solidity-coverage
after fixing paths to testrpc in exec.js
. Squash commits in Master, and re-run zeppelin-solidity tests to verify this all works.
We changed our path to execute testrpc-sc from the customary /.bin
path to ./node_modules/ethereumjs-testrpc-sc/bin/testrpc
in order to accommodate yarn in #48 and now package-lock.json
has broken pattern with regular npm install
and nests the dependency inside the solidity-coverage
folder.
Just head over heels in love with this multi-package manager locking mania that has swept code recently.
Likelyhood of avoiding package-lock is zero since its constantly hanging out in the untracked files, driving everyone insane. Likelyhood of avoiding yarn also remote. . . .
Options are pretty much incoherent ATM - esp. if someone uses the testrpc options one. Gaslimit should be hardcoded into testrpc-sc if it isn't already.
There's also a problem with the port . . . for example someone who doesn't specify a port but DOES use testrpcOptions
will automatically run on 8545.
Also replace all instances of ${coverageDir}
which are not part of a larger string. Should be coverageDir
.
For robustness, recommend:
Problem:
$ ./node_modules/solidity-coverage/bin/exec.js
Generating coverage environment
mkdir: path already exists: ./coverageEnv
cp: copyFileSync: could not write to dest file (code=ENAMETOOLONG):coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/build/contracts/MultiSigWalletWithTimeLockExceptRemoveAuthorizedAddress.json
cp: cannot create directory './coverageEnv': No such file or direct
Solution: death
If we do this, we can set it to 'solidity-coverage' or something similar, and people can use web3.version.network
in their tests to decide if they're running against coverage if they need to execute slightly differently (most likely, varying gas costs / responses to gas used).
EDIT: I know this can already be achieved with a custom testrpc
options, but having it by default would be nice.
This is in question at issue #56.
melonproject/protocol for example has a utilities folder located at project root which the tests require from e.g. require('../utilities/<file>')
truffle.js
/ show zep fix..solcover.js
and that user verifies no background instance of testrpc is running.Also add link to github repo in package.
I think there is a race condition causing the test command to fail when TestRPC is not ready to accept requests by the time the tests begin making them. I was able to possibly address this via a workaround.
For more context, see the following Travis builds:
https://travis-ci.org/gnosis/gnosis-contracts/builds/253195228
https://travis-ci.org/gnosis/gnosis-contracts/builds/253307021
And this commit:
Try to run this on /0xProject/contracts. The tests run as is without issue (using npm run test
, but throw an error when called via solidty-coverage. It seems to have something to do with the npm scripts:
"transpile": "rm -rf ./transpiled; copyfiles ./build/**/* ./transpiled; tsc",
"test": "npm run transpile; truffle test",
Error message:
Launching test command (this can take a few seconds)...
> [email protected] test /Users/primary/Projects/0x_contracts/coverageEnv
> npm run transpile; truffle test
> [email protected] transpile /Users/primary/Projects/0x_contracts/coverageEnv
> rm -rf ./transpiled; copyfiles ./build/**/* ./transpiled; tsc
sh: copyfiles: command not found
sh: tsc: command not found
npm ERR! Darwin 16.6.0
npm ERR! argv "/Users/primary/.nvm/versions/node/v7.10.0/bin/node" "/Users/primary/.nvm/versions/node/v7.10.0/bin/npm" "run" "transpile"
npm ERR! node v7.10.0
npm ERR! npm v4.2.0
I also tried changing the script to
rm -rf ./transpiled; ./node_modules/.bin/copyfiles ./build/**/* ./transpiled; ./node_modules/.bin/tsc
This gets me a new error, which seems closer to success:
error TS5023: Unknown compiler option 'baseUrl'.
error TS5023: Unknown compiler option 'allowJs'.
This bug was introduced in 0.1.9 during by the exec.js
refactor. Changed a hardcoded path to use a variable in instrumentation sequence and it's off by a character. This was caught trying to debug a failure to get Coveralls reports at Aragon. Opening a PR and publishing a fix for this shortly.
Zeppelin, for example. High memory footprint issues have been solved in the latest testrpc They've also moved to a webpack build so we'll have to replicate that. Going to be a little tricky.
When I ran solidity-coverage
for the first time, it gave me this error:
Invalid input source specified.
Compiliation failed. See above.
After some debugging, it turns out that the problem was the paths on my imports. I'd been importing "Contract.sol"
instead of "./Contract.sol"
in my contracts. Since solidity-coverage is more particular about the current working directory than truffle is, someone else might run into this issue. It's not a solidity-coverage bug, but I'm creating an issue in the hopes that the next person to have that problem comes across it.
/* istanbul ignore . . . */
strategy${coverageDir}/contracts/Migrations.sol
is always ignored.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.