Multiple customers have run into transient issues loading symbols from a remote symbols server (for files which definitely have symbols published to the server)--a run will fail to find symbols once, but work later on.

Can we investigate if the underlying library contains automatic retries when it fails to load a .PDB? If it does not, it would likely be helpful to offer a command line option for retrying the lookup when the .PDB isn't found. (If the underlying library already retries, more retries is likely overkill.)

Malformed command line tend to result in args that are improperly evaluated as file args. We should detect IO exceptions and add a useful message asking users to ensure args are properly quoted, that all switches have an expected arg value following it, etc.

We can only use some heuristics based on breaks we typically observe, due to dependency on CommandLineLibrary. Otherwise, we'd need to update the internals of that code to cover some new cases.

Hi

I am unable to implement BinSkim of my projects build output, because among the files it produces on our drop server, ildasm.exe is found. BinSkim issues an ExceptionLoadingPdb error:

binskim analyze ildasm.exe --sympath SRV*http://msdl.microsoft.com/download/symbols -v
Analyzing...
Analyzing 'ildasm.exe'...
[...]
error ERR997.ExceptionLoadingPdb : BA2006 : 'ildasm.exe' was not evaluated for check 'BuildWithSecureTools' because its PDB could not be loaded. (E_PDB_NOT_FOUND (File not found))
[...]

One or more rules was disabled for an analysis target, as it was determined not to be applicable to it (this is a common condition). Pass --verbose on the command-line for more information.

Analysis did not complete due to one or more unrecoverable execution conditions.
Unexpected fatal runtime condition(s) observed: ExceptionLoadingPdb

I am able toto verify the binaries can be dfoudn on teh server
e:\rainfnd>\symbols\tools\x86\symchk.exe Source\Servicing\AXCreateDeployablePackageTool\AXCreateDeployablePackageBase\ildasm.exe /s SRV*http://msdl.microsoft.com/download/symbols /od
SYMCHK: ildasm.exe PASSED - PDB: ildasm.pdb DBG: <N/A>

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

What can I do to get BinSkim to find the symbols?

Thanks
Bo

Microsoft.CodeAnalysis collides directly with Roslyn root namespace. As it turns out, taking a dependency in this tool on Roslyn will create a name collision (with the Roslyn Location object).

pdb files are not loading when we use --sympath. if binary file and its respective pdb in the same path then pdb files getting loaded (with/without the attribute "--sympathy"). otherwise, binskim throwing below exception
error ERR997.ExceptionLoadingPdb : BA2006 : 'MyFile.dll' was not evaluated for check 'BuildWithSecureTools' because its PDB could not be loaded. (E_PDB_NOT_FOUND (File not found))
error ERR997.ExceptionLoadingPdb : BA2014 : 'MyFile.dll' was not evaluated for check 'DoNotDisableStackProtectionForFunctions' because its PDB could not be loaded. (E_PDB_NOT_FOUND (File not found))
error ERR997.ExceptionLoadingPdb : BA2002 : 'MyFile.dll' was not evaluated for check 'DoNotIncorporateVulnerableDependencies' because its PDB could not be loaded. (E_PDB_NOT_FOUND (File not found))
error ERR997.ExceptionLoadingPdb : BA2007 : 'MyFile.dll' was not evaluated for check 'EnableCriticalCompilerWarnings' because its PDB could not be loaded. (E_PDB_NOT_FOUND (File not found))
error ERR997.ExceptionLoadingPdb : BA2011 : 'MyFile.dll' was not evaluated for check 'EnableStackProtection' because its PDB could not be loaded. (E_PDB_NOT_FOUND (File not found))
error ERR997.ExceptionLoadingPdb : BA2013 : 'MyFile.dll' was not evaluated for check 'InitializeStackProtection' because its PDB could not be loaded. (E_PDB_NOT_FOUND (File not found))

Current behavior: any error when running BinSkim, regardless of severity/etc., will give an exit code of '1'. This includes issues that often happen if you run the tool across products with rely on third party libraries, like not finding PDB files.

This makes it difficult to integrate the tool into other orchestration using the 'recurse' option--where we may expect to be missing private symbols for some libraries.

Expected behavior: Missing PDB files should be logged as an error, users integrating the tool should be able to differentiate that case from other errors via the exit code.

spectre mitigations currently are not enabled if optimizations are disabled

According to the Sarif spec, the runInfo.commandLineArguments field is required. Current BinSkim output has this value as null.

Sarif Spec:

General { #runInfo-General }

A runInfo object describes the invocation of the static analysis tool that produced the issues specified
in the containing runLog object (§[#runLog]).

~ IsoNote
The information in the runInfo object makes it possible to precisely repeat a run of a static analysis tool,
and to verify that the issues reported in the log file were generated by an appropriate invocation of the tool.
~

commandLineArguments property { #runInfo-commandLineArguments }

A runInfo object shall contain a property named commandLineArguments whose value is a string
containing the command line arguments with which the tool was invoked.
This string shall not include the file name or path to the executable itself.

~ IsoExample
Suppose a tool is invoked with the command line

    C:\Tools\CodeScanner\CodeScanner.exe /input C:\Code\*.cc

Then the value of the commandLineArguments property should be "/input C:\Code\*.cc"."

I'd like to setup binskim on our gated check-in builds.
In this builds we do not sign binaries as they are discarded.
Is it possible run binskim with policy that skips this check?
How to do it?
Command-Line Documentation say

-p, --policy Path to policy file that will be used to configure analysis. Pass value of 'default' to use built-in settings.

But how should policy file look like?

Thank you.
Palo Misik

The msdia140.dll file is missing the flag for Control Flow Guard (CFG). Could this DLL be re-built and updated?

The Microsoft/visualfsharp repo uses this tool in our official build to ensure assemblies were signed appropriately. It would be beneficial if this tool could also peek inside VSIX (zip archive) and MSI packages (COM structured storage?) to ensure our installers were created with the signed versions of binaries.

I'm running the following (dlls I want to check are directly placed under bin directory):

BinSkim.exe analyze bin -c default -r --output BinSkim.report

And I get this error:

error ERR997.NoValidAnalysisTargets : No valid analysis targets were specified.

Analysis did not complete due to one or more unrecoverable execution conditions.
Unexpected fatal runtime condition(s) observed: NoValidAnalysisTargets

These are rebased automatically as part of JIT, address location determined by whether loading exe is marked /highentropyva or not. The specified base address for these assemblies, therefore, is not relevant to runtime security.

Update signing layout to include a folder level that specifies codesign cert ID
e.g. bld\bin\LayoutForSigning<cert id>\net46\BinSkim.exe
This simplifies signing since we will be signing third-party assemblies with a different cert

This command line throws E_PDB_NOT_FOUND:

binskim.exe analyze Skype.exe --sympath SkypeWithLib.pdb
    --recurse --config default --output MyRun7.sarif

The problem is that --sympath is supposed to specify a directory where symbol files can be found. The symbol file name must match the assembly name.

To avoid this unclear error message, check that the argument to --sympath is a directory, not a file.

Currently the check returns the most recent /d2GuardSpecLoad supporting toolchains

The readme.md file has some sections that should be shown monospaced, e.g.:

command line options here

The text also directs the user to run test.cmd, which I believe is now called BuildAndTest.cmd.

The command line example also omits the required analyze argument.

@BillyONeal @tikiwanMS @davidknise @bobschumaker

is there a good reason for leaving these checks distinct? it seems as though we should just tack on the check to initialize the stack protector after verifying that it is enabled. you might want a distinct check in a world where that specific aspect of analysis should be configurable on its own. that doesn't seem the case here.

The policy BA2006 incorrectly assumes that .obj files built with NASM were built with MASM:

error: BA2006: 'x.dll' was compiled with one or more modules which were not built using minimum required tool versions (compiler version 17.0.65501.17016, linker version 11.0.65501.17016). More recent tool chains contain mitigations that make it more difficult for an attacker to exploit vulnerabilities in programs they produce. To resolve this issue, compile and/or link your binary with more recent tools. If you are servicing a product where the tool chain cannot be modified (e.g. producing a hotfix for an already shipped version) ignore this warning.
Modules built outside of policy:
aes-x86_64.obj built with MASM compiler version 8.0.26708.0 (Front end version 0.0.0.0)

These object files have been built with NASM 2.12.02 (http://www.nasm.us/)

When you link your CFG-protected executable or DLL with a static library which was not compiled with /guard:cf then BinSkim will not report any issue. Flags are set, table exists. Still, if there is an indirect call in this linked library then it won't be protected. Calls to __guard_check_icall_fptr were not inserted during compilation time.

Is there any reasonable way how to extend this rule to cover also these false negatives? Maybe additional support for scanning COFF libs assuming this information could be checked there?

We are getting HighEntropyVACheck on a dotnet core generated assembly defined by this csproj

https://github.com/PowerShell/PowerShell/blob/master/src/powershell-win-core/powershell-win-core.csproj
I don't think this is valid, for a dotnet assembly.

Build instructions are here: https://github.com/PowerShell/PowerShell/blob/master/docs/building/windows-core.md

The command line documentation in the repository is out of date--we're missing details on the different verb(s), such as 'analyze'/similar, which are required.

If you run BinSkim's 'exportConfig' command with a folder as the output location, you get an unhelpful exception.

For example, if you run it with the desktop folder:
.\BinSkim.exe exportConfig "C:\Users{username}\Desktop"
System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users{username}\Desktop'.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
at System.IO.File.InternalWriteAllText(String path, String contents, Encoding encoding, Boolean checkHost)
at Microsoft.CodeAnalysis.Sarif.Driver.ExportConfigurationCommandBase.Run(ExportConfigurationOptions exportOptions)

The workaround is to explicitly specify a full filename (e.x. "binskim.exe exportconfig C:\users{username}\Desktop\binskimconfig.json"). However, the current behavior isn't particularly user friendly.

Expected behavior: BinSkim should either chose a default filename to output to if given a folder (e.x. 'binskimconfig.json' or 'binskimconfig.xml'), or give a more helpful error message (e.x. "Error: Please specify a full file name, not a folder"), rather than throwing an exception.

In order to identify modules that contribute no executable code. Generally, these should be ignored by the tool.

the url should be updated to https://github.com/Microsoft/binskim/releases/download/v1.2.4-beta/binskim1.2.4-beta.x86.zip or some other working url.

fix should be on line 24

Users are encountering an issue with long sympaths, which ends up giving them an error. There is no support ticket currently associated with this, but the thread title is "Information about BinSkim error"

Per: https://blogs.msdn.microsoft.com/vcblog/2018/01/15/spectre-mitigations-in-msvc/

New flags are being added to VC++ that add fence instructions to prevent spectre-style exploits (necessary if your application deals with data which crosses a trust boundary). It looks like they're only available in 2017.5 and 2017.6 for now, but ports are planned for 2017 RTM and 2015.3.

It'd be useful to detect their usage (we'll need to determine if symbols are/aren't necessary) and implement a check (possibly an experimental one, as these are very new?).

In the BinSkim rule descriptor output (binskim.rules.sarif.txt), I suggest:

Line 14: “Address Space Layour Randomization” (sp.)

Line 17: “increaseing the effectiveness” (sp.)

Line 17: “address space layout randomization” was capitalized on Line 14 but not here.

Line 29: “was built with a dependency on version of {1}” – did you mean “on a version of {1}”? or “on version {1} of {2}” (e.g., “on version 2.0 of Foo.exe”)?

Line 48: “in order to take advantage of” -- consider “to take advantage of” ;-) (and wherever that construction occurs)

Line 52: “built with a tool chain that satisfies…” – is “tool chain” our own jargon? How about “was built with tools that satisfy…”

Line 63: “… there is a greater likelihood that X, Y, and Z do not exist…” – consider “it is less likely that X, Y, and Z exist…”

I had made some changes in the IOperation dev branch to check the exit code at every step of BuildAndTest.cmd and upon integration from master, found nuget packaging is failing with:

File not found: '..\..\..\src\ReleaseHistory.md'.

@michaelcfanning

I ran BinSkim and Binscope against the same binary and found an important difference.

When BinScope doesn't find private symbols (and is therefore unable to perform various analyses), it fails that check with a message like:

<FILE> cannot be processed by ATLVersionCheck, because ATLVersionCheck requires private symbols which BinScope was not able to find next to the binary or on your symbol path (%_NT_SYMBOL_PATH% or /SymPath). The specific error generated by MSDIA while resolving symbols was: E_PDB_NOT_FOUND (File not found) Obtain matching private symbols for the binaries being tested and run BinScope again.

When BinSkim is run against the same binary, I see a more cryptic:

<FILE>: error BA2007: '<FILE>' contains code from an unknown language, preventing a comprehensive analysis of the compiler warning settings. The language could not be identified for the following modules: <file> ... 

<FILE>: error BA2011: '<FILE>' contains code from an unknown language, preventing a comprehensive analysis of the stack protector buffer security features. The language could not be identified for the following modules: <file> ...

Analysis finished but was not complete due to non-fatal runtime errors.
Unexpected runtime condition(s) observed: RuleNotApplicableToTarget

Are these error messages "the same" -- if so, can the user be given more information (and direction to include symbol files)? Without this, the user will be more likely to think they "passed" -- under the presumption that no output is "good".

Repro:

BinSkim.exe analyze bin -p default -r -o binskim.report

Result:

Unhandled Exception: System.InvalidOperationException: Sequence contains more than one matching element
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
   at CommandLine.Core.TypeLookup.FindTypeDescriptorAndSibling(String name, IEnumerable`1 specifications, StringComparer comparer)
   at CommandLine.Core.Switch.<>c__DisplayClass0_0.<Partition>b__0(Token t)
   at System.Linq.Enumerable.WhereArrayIterator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at CSharpx.EnumerableExtensions.Memorize[T](IEnumerable`1 source)
   at CommandLine.Core.TokenPartitioner.Partition(IEnumerable`1 tokens, Func`2 typeLookup)
   at CommandLine.Core.InstanceBuilder.<>c__DisplayClass0_0`1.<Build>b__7()
   at CommandLine.Core.InstanceChooser.MatchVerb(Func`3 tokenizer, IEnumerable`1 verbs, IEnumerable`1 arguments, StringComparer nameComparer, CultureInfo parsingCulture, IEnumerable`1 nonFatalErrors)
   at CommandLine.Core.InstanceChooser.<>c__DisplayClass0_0.<Choose>b__0()
   at CommandLine.Parser.ParseArguments(IEnumerable`1 args, Type[] types)
   at Microsoft.CodeAnalysis.IL.BinSkim.Main(String[] args)

(expanded version of them works)

BinSkim isn’t able to properly evaluate its checks against a WiX-toolset generated executable because it does not build valid private symbols. Without these private symbols, developers cannot find out the C++ compiler version used, whether a WiX link contains vulnerable ATL libraries or if stack protection is enabled.

Most rules have a leading sentence that is an effective shortDescription. For several, we could trim some details around security justification in order to simply capture the prescriptive guidance.

The first sentence of BA2016, for example, is this:

Binaries should be marked as NX compatible in order to help prevent execution of untrusted data as code.

This could be reduced to the following rule abstract:

Binaries should be marked as NX compatible.

Perhaps add an option to 'flatten' all newlines in emitted output?

For notifications that users have disabled specific mitigations at the function/etc level (via #pragma or declspec). Ideally, these notifications would be visible in logs/output by default but not break the build/return an error code. We can use the rich error result mechanism for this. We should propose a new 'audit' or 'review' designation to the SARIF design team.