Consider using a commit message structure like: https://www.conventionalcommits.org/en/v1.0.0/

Add a GH action that automatically updates packages with are downloaded from GH URL (such as GitHub releases) and sends a PRs. If the test suite passes, we can just merge it. A relatively easy way to get the last version of many of our tool.

Add option for helper functions to allow user's to specify if tool should be ran as admin

Package Name

bytecodeviewer

Tool Name

Bytecode Viewer

Tool's version number

2.11.2

Category

Java

Tool's authors

Konloch

Tool's description

A lightweight user-friendly Java/Android Bytecode Viewer, Decompiler and more.

Package type

ZIP_EXE

Download URL

https://github.com/Konloch/bytecode-viewer/releases/download/v2.11.2/Bytecode-Viewer-2.11.2.jar

Download SHA256 Hash

536ad387424106083f76cd0cb7c051a22aff21f08663ba2539c11f1ddef9147f

Why is this tool a good addition?

To analyze Java/Android files.

What's the problem?

Attempting to get AV information similar to https://www.powershellgallery.com/packages/CimSweep/0.6.0.0/Content/Auditing%5CAntiVirusInfo.ps1 sometimes works. Other times it doesn't.

Steps to Reproduce

1. .\installl.ps1

Environment

Windows 10 x64 1809

Detected by test suite

I don't know

Update documentation at: https://github.com/mandiant/VM-Packages/tree/main/packages/common.vm

In the past we had bugs related to usernames with spaces. If possible use a username with spaces in GH actions.

It needs everything else in the milestone v1.

What's the problem?

Chocolatey installed 1/1 packages. 
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Chocolatey v1.2.0
Installing the following packages:
python2.vm
By installing, you accept licenses for the packages.

python2.vm v2.7.18 (forced)
python2.vm package files install completed. Performing other installation steps.
Downloading python2.vm 
  from 'https://www.python.org/ftp/python/2.7.18/python-2.7.18.msi'

Download of python-2.7.18.msi (18.72 MB) completed.
Hashes match.
Installing python2.vm...
python2.vm has been installed.
2022/11/14 04:14:17 [python2.vm] vm.common.psm1 [+] ERROR : Invalid path: C:\Python27
2022/11/14 04:14:17 [python2.vm] vm.common.psm1 [+] ERROR : [ERR] Invalid path: C:\Python27
At C:\ProgramData\_VM\vm.common\vm.common.psm1:271 char:5
+     throw $err_msg
+     ~~~~~~~~~~~~~~
ERROR: Invalid path: C:\Python27
The install of python2.vm was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\python2.vm\tools\chocolateyinstall.ps1'.
 See log for details.

Chocolatey installed 0/1 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - python2.vm (exited -1) - Error while running 'C:\ProgramData\chocolatey\lib\python2.vm\tools\chocolateyinstall.ps1'.
 See log for details.

Steps to Reproduce

See e.g. https://github.com/mandiant/VM-Packages/actions/runs/3458340832/jobs/5777311898

Environment

CI

Detected by test suite

Yes

I like how capa has multiple badges, one of which is the number of rules. I think vm-packages should also have a badge that counts the number of packages!

Seems that happens via a GitHub Action: https://github.com/mandiant/capa-rules/blob/master/.github/workflows/sync.yml

We should have a GitHub Action that compares packages in the GitHub repository to those hosted on MyGet and let us know if they differ. For example, we if remove a local package or rename one, this action should point out a conflict so we can take the appropriate actions to remedy it.

Package Name

regshot

Tool Name

Regshot-x64-Unicode

Tool's version number

1.9.1

Category

Utilities

Tool's authors

maddes, regshot, xhmikosr

Tool's description

Regshot is a small, free and open-source registry compare utility that allows you to quickly take a snapshot of your registry and then compare it with a second one - done after doing system changes or installing a new software product.

Package type

ZIP_EXE

Download URL

https://sourceforge.net/projects/regshot/files/regshot/1.9.1-beta/Regshot-1.9.1-beta_r321.7z

Download SHA256 Hash

5933d59f591e1e68ce7819904f8cb1118fc935bdfe89581599d0560ec9b97cd6

Why is this tool a good addition?

Aids finding differences in the registry after running your malware sample.

We are currently opening console applications using cmd.

$executableCmd  = Join-Path ${Env:WinDir} "system32\cmd.exe"

I think it would be nice to use a environment variable to allow to change the default console.

The local tests and the daily.yml and ci.yml GH actions actions are quite similar, so that we often need to modify the same thing in all of them. Avoid code duplication by exporting the actions to a script to use it in the GH actions and for local testing.

This should also make easier for new contributors to test their changes locally.

• Intro about chocolatey (packages structure, important details for us)
• Document create_package_template.py in the Quick Start
• Document requirements for local testing (same as to install FLARE-VM)

Update FLARE-VM installer at https://github.com/mandiant/flare-vm to use packages built/pushed from this repository

What's the problem?

Issue introduced in #103. $built_pkgs is now a list of True and False and not a list of packages names.

PS VM-Packages > $built_pkgs = Get-ChildItem $built_pkgs_dir | Foreach-Object { ([regex]::match($_.BaseName, '(.*?[.](?:vm)).*').Groups[1].Value) -and $_ -notin $exclude_tests}
FLARE 11/21/2022 1:57:05 PM
PS VM-Packages > $built_pkgs
True
True
True

Steps to Reproduce

See https://github.com/mandiant/VM-Packages/actions/runs/3511009362/jobs/5881380101 or run scripts/test/test_install.ps1 locally.

Detected by test suite

As there was no package update in #103, the issue was not detected by the CI. But now it fails for every package update. See #106.

Some packages use the same url for different versions which breaks the package every release. We could try to update them automatically. We could use something similar to the daily run that updates the hash if choco fails because the hash doen't match.

PRs remain open after automatic generated package has been merged. See for example: #52

Ensure documentation is ready and clear to new contributors by adding a My first package walkthrough section.

Related to #28 and #27.

Currently in PR we only test modified packages to reduce the testing time. However, there are cases were a package can break if another package is modified. For example if we modify a function in common, we could break all packages that use that function, that are not tested if they are not modified.

Different approaches to handle this better:

• Test in all PR all packages. This take some time, but ensure everything works.
• Test all packages only in PRs that modify common. For most PR the testing time is still small, but we ensure that modifying common doesn't break any other package. This only solve the problem for common, but not other dependencies
• Test only modified packages (current). Prioritize small testing time. If something breaks we will realize in the daily run.

@vm-packages opinions?

What's the problem?

$toolDir = Join-Path ${Env:SystemDrive} "tools\Cygwin" -Resolve

Steps to Reproduce

change default env var from Tools to

Environment

see recent comments in mandiant/flare-vm#423

Detected by test suite

No

Use the information provided in an "New package proposal" issue to create a PR. We would need some improvements in the package creator script and a GH action that takes care of the automation.

Needs #25

When users update the default ${Env:RAW_TOOLS_DIR} location it also modifies the ${Env:ChocolateyToolsLocation} location and this is the location where Chocolatey defaults to install tools. For tools / dependencies that install to the default location, we need to use ${Env:ChocolateyToolsLocation}.

Some packages (such as notepadplusplus.vm renamed in #98) haven't been pushed to myget because our action only checks modified nuspec files and not renamed ones if they are not modified. Note that in a package change of name, the version is not modified. We need to:

• Fix action.
• Push packages manually (idea: we could have an action to repush packages so that we don't need to do this manually, but not sure how much we will use it)
• [optional] removed unused/old packages

Add "basic URL file" to create_package_template.py and to the issue template and GH action that create PRs from issues.

What's the problem?

See https://github.com/mandiant/VM-Packages/runs/8303111845?check_suite_focus=true#step:5:20

May be an issue with the line breaks?

Steps to Reproduce

add send PR label to METAPACKAGE issue

Environment

CI

Detected by test suite

No

Per #50:

The code in create_package_template_from_json.py could be integrated in create_package_template.py

Similar to #26

@MalwareMechanic what is the easiest way to get the latest version of a dependency? Is there an API from the community feed? Do we need to use some command of choco and parse the output?

Add the following logs to the artifacts: %PROGRAMDATA%\chocolatey\lib-bad\*\install_log.txt

Packages may create a per-install log to use for whatever they wish, see:

The packages libraries.python*.vm use this to redirect output from pip when installing Python modules. It'd be good to grab these per-install log files as well. You'll likely need to use a regex to find them, but they should be in a subdirectory within lib-bad somewhere.

Some dependencies are not fixed and they should to avoid an update breaks the package. We have CI now in place to update them automatically if the package still works.

should we install the same packages in python 3 as in python 2? For example pefile and vivisect could be installed in Python 3 (maybe they didn't supported py3 when this file was created).

Package Name

dnspyex

Tool Name

dnSpyEx

Tool's version number

6.2.0

Category

dotNet

Tool's authors

0xd4d, ElektroKill

Tool's description

dnSpyEx is a unofficial continuation of the dnSpy project which is a debugger and .NET assembly editor. You can use it to edit and debug assemblies even if you don't have any source code available.

Package type

METAPACKAGE

Dependency

dnSpyEx

Shim path

bin\dnSpy.Console.exe

Why is this tool a good addition?

dnSpy rules!

Currently, only 7zip 15.05 (with NSIS support) gets installed.

• add the current version via metapackage
• change existing package to indicate it's 15.05

The Python script should check the following:

• Ensure file paths use lowercase
• Verify correct versioning: 3 first segments of the tool's version, fourth segment reserved for the date of hotfix
• Verify if nuspec version is not increased (by geting both revisions)
• Verify .nuspec has only the required fields
• Verify the package depends on “common.vm”
• Verify .ps1 imports the “common.vm”
• Check if first line is “error action stop”

Previously documented in #6 (splitting that issue up now)

I think having two badges that show the number of "passing" and "failing" packages would be more informative versus a daily badge that simply says "success/fail".

Add issue templates to help tracking new packages proposals.

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository#creating-issue-forms

For common and more involved install mechanisms there should exist a helper function like VM-Install-Raw-GitHub-Repo etc.

See https://github.com/mandiant/VM-Packages/wiki/Contributing#example-4 for a good candidate of tools run via cmd.exe, for example.

Ideas for linting:

• Powershell style linter (@Ana06 - #6)
• Ensure file paths use lowercase
• Verify .nuspec has only the required fields
• Verify nuspec version has the correct format
• Verify correct versioning update
  • fail if nuspec is not modified (@Ana06 - #8)
  • fail if nuspec version is not increased (by geting both revisions)
• Verify the package depends on “common.vm”
• Verify .ps1 imports the “common.vm”
• Verify first line is “error action stop” in install
• send a comment into PR with the problem

We are very close to start using these packages in FLARE-VM. We have mentioned a few times that we want to give a thought to the current categories, trying to create more concrete categories that make it easier to find tools.

I think these are the current categories:

• Android
• Debuggers
• Delphi
• Developer Tools
• Disassemblers
• dotNet
• Flash
• Forensic
• Hex Editors
• Java
• Javascript
• Net
• Office
• PDF
• Pentest
• PowerShell
• Python
• Text Editors
• Utilities
• VB
• Web Application

This is also needed for #25

@mandiant/flare-vm ideas?

What's the problem?

Chocolatey v1.2.0
Installing the following packages:
7zip.vm
By installing, you accept licenses for the packages.

7zip.vm v15.05 (forced)
7zip.vm package files install completed. Performing other installation steps.
Downloading 7zip.vm 64 bit
  from 'https://sourceforge.net/projects/sevenzip/files/7-Zip/15.05/7z1505-x64.exe/download'
2022/11/13 03:43:49 [7zip.vm] vm.common.psm1 [+] ERROR : [ERR] The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://sourceforge.net/projects/sevenzip/files/7-Zip/15.05/7z1505-x64.exe/download'. Exception calling "GetResponse" with "0" argument(s): "The request was aborted: Could not create SSL/TLS secure channel."
At C:\ProgramData\chocolatey\helpers\functions\Get-WebFile.ps1:330 char:8
+        throw "The remote file either doesn't exist, is unauthorized,  ...
+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://sourceforge.net/projects/sevenzip/files/7-Zip/15.05/7z1505-x64.exe/download'. Exception calling "GetResponse" with "0" argument(s): "The request was aborted: Could not create SSL/TLS secure channel."
The install of 7zip.vm was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\7zip.vm\tools\chocolateyinstall.ps1'.
 See log for details.

Chocolatey installed 0/1 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - 7zip.vm (exited 404) - Error while running 'C:\ProgramData\chocolatey\lib\7zip.vm\tools\chocolateyinstall.ps1'.
 See log for details.

Is the download potentially blocked via Sourceforge/Cloudflare?

Steps to Reproduce

See e.g. https://github.com/mandiant/VM-Packages/actions/runs/3453675077/jobs/5764461768

Environment

CI

Detected by test suite

Yes

Send out survey for getting more insight about the packages our users want.

MAP is distributed using the following link for all releases: http://sandsprite.com/idef/map

This makes that an update breaks the package as the hash changes. 😭 We can't ignore the checksum because the URL is using HTTP and Chocolatey complains (with good reasons). 😭😭

Create a Get-VM-Info command that users can run and copy-paste in the issue when reporting a bug. @mr-tz's idea, see #40 (comment)

Update bug issue template with this command.

Document different between VM-Install-From-Zip and VM-Install-Raw-GitHub-Repo (also in the ISSUE template, or link to the added documentation).

I think the syntax of "$variable.exe" is more readable. I checked and we use ($variable + ".exe") some throughout the code base. Let's make this more uniform

./packages/processdump.vm/tools/chocolateyinstall.ps1:14:  $executablePath = Join-Path $toolDir ($toolName + "32.exe")
./packages/processdump.vm/tools/chocolateyinstall.ps1:29:  $shortcut = Join-Path $shortcutDir ($toolName + "32.lnk")
./packages/processdump.vm/tools/chocolateyinstall.ps1:32:  Install-BinFile -Name ($toolName + "32") -Path $executablePath
./packages/processdump.vm/tools/chocolateyinstall.ps1:38:    $executablePath = Join-Path $toolDir ($toolName + "64.exe")
./packages/processdump.vm/tools/chocolateyinstall.ps1:51:    $shortcut = Join-Path $shortcutDir ($toolName + "64.lnk")
./packages/processdump.vm/tools/chocolateyinstall.ps1:54:    Install-BinFile -Name ($toolName + "64") -Path $executablePath
./packages/processdump.vm/tools/chocolateyuninstall.ps1:7:VM-Remove-Tool-Shortcut ($toolName + "32") $category
./packages/processdump.vm/tools/chocolateyuninstall.ps1:8:VM-Remove-Tool-Shortcut ($toolName + "64") $category
./packages/processdump.vm/tools/chocolateyuninstall.ps1:10:Uninstall-BinFile -Name ($toolName + "32")
./packages/processdump.vm/tools/chocolateyuninstall.ps1:11:Uninstall-BinFile -Name ($toolName + "64")

• config.chrome.defaultbrowser.vm (to make default browser)
• config.chrome.removeshortcut.vm (delete desktop shortcut)
• config.chrome.disableupdates (to disable auto-update for chrome to allow quieter PCAPs)

Add the following priority packages:

• cyberchef (cyberchef.vm - #15)
• notepadplusplus (notepadplusplus.vm - #4)
• pd64.exe (processdump.vm - #19)
• wireshark (wireshark.vm - #12)

It seems that installing notepadplusplus.vm fails sometimes with access is denied: https://github.com/mandiant/VM-Packages/runs/6088818786?check_suite_focus=true It successes in some daily runs and fails in others.

I forgot to add the upload of logs to the daily in #22, I'll send a PR in a moment that may help us figuring out what the problem is.

From @mr-tz comment in #6 (comment):

Make the linter output easy to see, currently lines like Progress: Downloading PSScriptAnalyzer 1.20.0... 0% add a lot of noise.

This refers to the Powershell linter and it is not related to the new Python linter.