rsm-hcd / andculturecode.cli Goto Github PK

Backfill tests for webpack-publish module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.

Create a command that allows you to add migrations, remove migrations and update the database given a project and a startup project for Entity Framework Core.

i.e
and-cli --migrations -[add|remove] [MigrationName] -p [Project] -s [StartupProject]

... or something

Problem
Recently upgraded a project to the latest dotnet SDK version. In doing so I no longer had a valid version of dotnet-ef. When running the and-cli migration command, while it does what is expected and outputs possible issues, the and-cli could reconcile it for the user.

Possible reasons for this include:

• You misspelled a built-in dotnet command.
• You intended to execute a .NET Core program, but dotnet-ef does not exist.
• You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.

Solution
When the migration command is run, do an initial check to see if the global dotnet-ef tool is installed. If not, automatically run the install...

dotnet tool install -g dotnet-ef

Now that we have a few modules broken out with tests and TravisCI to automatically run tests, it would be a good time to go back and fill in tests for existing modules.

As part of another initiative, I've already started filling in some tests - just wanted to add a task here to document the progress. These modules will be tested as part of this task:

dir.js
dotnet-build.js
dotnet-clean.js
dotnet-path.js
dotnet-restore.js

This may include some minor refactors to clean up existing code.

Problem
The dotnet-test command runs all of the test assemblies by the solution file itself. Normally, this should be fine, however - the underlying dotnet test runner fires off multiple assemblies to run at the same time, even if those assemblies are set to run tests within the suite in sequence. There's an issue where integration tests that are accessing the same database can and will collide, and the easiest workaround we have for this for the time being is to force the test assemblies to run in serial.

Solution

• Update the dotnest-test command to have a flag for running the solution "by project", which will look for any *.Test*.csproj files in the dotnet solution directory, and call dotnet test with those projects.

• Add a few tests if you have time. Right now, the project isn't hooked up to TravisCI so we don't have the benefit of the existing tests running automatically, so it's not a dealbreaker, but it will be nice to have for when we do.

Problem
When running the nuget --publish <version> command I experienced an I/O error in creating the assets. Even though this occurred, the process kept going. It created the nupkg just fine so in my case, it ended up sending up the previously built assembly version with the new nupkg file.

Solution
While the nuget pack command does attempt to do a build, perhaps we consider doing an explicit dotnet build ourselves and error check that.

Problem
ShellJS exec, no matter the setting, is leveraging node exec. The primary issue is that exec does not pipe output to the parent process correctly via streaming. Thus we have had to make work arounds to reformat the output.

Solution
Switch from using shelljs.exec to nodejs spawn (see webpack-test) for an example. This will allow for the identical experience to webpack dev server from formatting to overtaking the shell.

const child = spawn(this.cmds.webpack, { stdio: "inherit", shell: true });
child.on("exit", (code, signal) => {
    if (code !== 0) {
        echo.error(`Exited with error ${signal}`);
        shell.exit(code);
    }

    dir.popd();
    echo.newLine();
    echo.message("Exited webpack");
});

I want to be able to run a command that updates a nuget package that is referenced in multiple projects without having to navigate to each project and run dotnet add package <packagename> -v <version>

The dotnet --kill command is not cross platform, the kill call fails on Mac OS with the following output:

and-cli dotnet --kill 
[and-cli] Stopping dotnet build servers via (dotnet build-server shutdown)...
Shutting down MSBuild server...
Shutting down VB/C# compiler server...
VB/C# compiler server shut down successfully.
MSBuild server shut down successfully.
[and-cli] Finished shutting down build servers.
[and-cli] Force killing dotnet PIDs... via (kill --force)
/bin/sh: line 0: kill: -force: invalid signal specification
[and-cli] [ERROR] Could not kill dotnet PID Brandon
/bin/sh: line 0: kill: -force: invalid signal specification
[and-cli] [ERROR] Could not kill dotnet PID Brandon
[and-cli] Finished force killing lingering dotnet processes.

While you're in there, some of this code could be more defensively error checking as well, so it doesn't continue on if, for example, the dotnet build-server shutdown call fails.

A quick Google search led me to fkill, which looks promising.

Backfill tests for echo module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.

A really cool and helpful command would be an OSS contribution setup related one. Where it would both perform initial setup of a "workspace" for open source contributions as well as potentially sync an existing one with any "new" repos. Of course there will likely need to be discussion about what all is entailed in this setup. My initial thoughts for features are...

• Project look up: Check github AndcultureCode organization for existing AndcultureCode repos following our established naming pattern AndcultureCode.{language/technology}.{projectName}
• Clone Project setup: Clone 'main' branch of repo ie. AndcultureCode.CSharp.Core
• Fork Project setup: Fork repo and clone fork of 'main' branch of repo in parallel to core project ie. MyAndcultureCode.CSharp.Core
• Sync of workspace: If a given 'clone' or 'fork' project exists (given established naming convention) ignore! This would allow simple setup of new projects ongoing
• Documentation: Update AndcultureCode contributing.md with this command

https://github.com/AndcultureCode/AndcultureCode.Cli/network/alert/package-lock.json/bl/open

Remediation
Upgrade bl to version 0.9.5 or later. For example:

"dependencies": {
"bl": ">=0.9.5"
}
or…
"devDependencies": {
"bl": ">=0.9.5"
}

Details
WS-2018-0074
moderate severity
Vulnerable versions: < 0.9.5
Patched version: 0.9.5
Versions of bl before 0.9.5 and 1.0.1 are vulnerable to memory exposure.

bl.append(number) in the affected bl versions passes a number to Buffer constructor, appending a chunk of uninitialized memory

I stumbled upon the dotnet-cli module and it could use some cleanup based on newer patterns we have. Here's a quick list of what I suggest, ordered by priority:

• Check exit code from shell.exec and echo an error message if the code is not successful (0). This should kill the process chain with shell.exit(result.code) so that any commands leveraging it will not try to keep working on a failed CLI command.
• Consider refactoring from shell.exec to child_process.spawnSync, which streams output and retains original colors.
  • If doing this refactor, make sure to update the cmd() function with the existing pattern in dotnet-test, dotnet-build, dotnet-clean, etc. It is a simple way to separate out the 'base' command from the arguments, as well as a toString() override for output purposes
• Add test coverage
• Reference the absolute path of the Cli.dll executable in description() to allow for easier debugging & visibility on where the executable is actually being executed from.
• Remove reference to LMS in the description

Problem
Nuget.org can be very slow to propagate packages across its network. Causing 30 minute to 1 hour delays before those packages can be consumed. This is a productivity killer.

Solution
Use a secondary package management resource that is more performant and leverage nuget.org as a baseline/fallback. Normal users will just leverage nuget.org, but those doing heavy contributions can additionally configure github packages as a source that should offer better performance.

https://help.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-dotnet-cli-for-use-with-github-packages

https://help.github.com/en/packages/publishing-and-managing-packages/about-github-packages

Backfill tests for migration module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.

A common deployment picture of frontend application is on remote storage, such as; Amazon S3 or Azure Blob. The first use case is to take conventionally compiled webpack frontend build artifacts and deploy them to AWS S3.

At this time, it will be an explicit command, but in not so distant future we will allow configuration of this and other settings on an environment basis via .env (dotenv in this case) files.

Command: and-cli deploy --aws-s3 --webpack --public-url={s3-bucket-url} --access-key={s3-access-key} --secret-key={s3-secret-key}

Currently the and-cli is written to be abstract and work for our projects. Inevitably we will want to write some additional commands that we want to remain outside of the and-cli and reside in the project repository.

This issue is to create a plugin model for the and-cli to allow a project to extend the commands. While there could be many ways to do this, it would be ideal from a usage perspective not to have to add any additional configuration for to work.

There was a period of time on 6/24/20 where the dotnet image for TravisCI was down, and the and-cli project was failing to build (due to new dependency on that image for integration testing our and-cli dotnet command).

Looks like it has since been restored (link), so we should revert the changes to skip those integration tests and the changes in .travis.yml

Not super high priority, but it would help me sleep better at night 😂

Description
Investigate how much of a lift porting the project over to Typescript would be. Document any caveats or potential roadblocks, changes in development workflow, npm package publishing, etc.

It looks like the dotnet-build module was not written to properly handle the exit status of the dotnet command itself, in that it returns a zero exit status (which is generally considered a successful return). While developers may not see this bug in day to day use, it will cause an issue if this cli command is ever hooked up to a CI pipeline.

Brandon@local ~/TestProject/ (development) -> and-cli-dev dotnet --build
[and-cli] Building solution (via dotnet build dotnet/TestProject/TestProject.sln --no-restore)...
Microsoft (R) Build Engine version 16.2.32702+c4012a063 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Core -> /Users/Brandon/TestProject/dotnet/TestProject/Business/Core/bin/Debug/netstandard2.0/Core.dll
  Conductors -> /Users/Brandon/TestProject/dotnet/TestProject/Business/Conductors/bin/Debug/netstandard2.0/Conductors.dll
  Data.SqlServer -> /Users/Brandon/TestProject/dotnet/TestProject/Infrastructure/Data.SqlServer/bin/Debug/netcoreapp2.2/Data.SqlServer.dll
Controllers/BaseController.cs(7,9): error CS1519: Invalid token ';' in class, struct, or interface member declaration [/Users/Brandon/TestProject/dotnet/TestProject/Presentation/Web/Web.csproj]

Build FAILED.

Controllers/BaseController.cs(7,9): error CS1519: Invalid token ';' in class, struct, or interface member declaration [/Users/Brandon/TestProject/dotnet/TestProject/Presentation/Web/Web.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:02.39
Brandon@local ~/TestProject/ (development) -> echo $?
0

Should be a pretty simple fix to handle the return from shell.exec(...) in the dotnet-build.js module, similar to how the dotnet-publish.js module is handling it.

Maybe we could support user-defined aliases (think git aliases) for the and-cli. On unimplemented command, before throwing an error, maybe we could check for user-defined aliases at ~/.and-cli-config. This would be convenient for defining aliases like

migrate-test-db=dotnet -cRb -- --cli test db migrate

so then you could just run

and-cli migrate-test-db

Backfill tests for node-clean module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.

Recently some of the engineers have desired an easier way to set environment variables when running commands through the cli. While this could be baked in directly to our current use-case (and-cli dotnet), it might make more sense for there to be a 'global' option where you can set environment variables for any command, such as:

and-cli dotnet --env ASPNETCORE_ENVIRONMENT=Working

I assume we would want the ability to set multiple variables in one command:
and-cli dotnet --env ASPNETCORE_ENVIRONMENT="Working" --env AllowedHosts="*"

Nested environment variables should use the Object__NestedVariable syntax, such as:
and-cli dotnet --env ConnectionStrings__ProjectDb="..."

If we build it as a 'global' option instead of just into the dotnet command, we would be able to set variables when running webpack, webpack-test etc., without having to update each individual command.

Some of the engineers expressed preference in these variables only persisting for the command run they were run with, not for the entire shell session. I could see value in both. For clearing them out after the command exits, you'd need to do something like this (example bash commands):

bscot@BSCOTT-PC ~ -> export SOMEVAR=SOMEVALUE; and-cli dotnet; unset SOMEVAR

I'm not set on this specific naming convention, but maybe --shell-env SOMEVAR=SOMEVALUE could be the standard 'sticky' environment variable which persists for the shell session, where --env or -e could be the 'process' environment variable which is immediately unset upon exit.

ShellJS provides a wrapper around Node's process.env, which you can index into with just a string: see http://documentup.com/shelljs/shelljs#envvar_name

Problem
The default for the Elastic beanstalk CLI is to timeout if a deploy takes longer than 10 minutes. Depending up on your rolling strategy, health monitoring, etc... that could very well likely happen. This results in the CLI returning an error even though beanstalk will likely continue on a deploy successfully.

ERROR: TimeoutError - The EB CLI timed out after 10 minute(s). The operation might still be running. To keep viewing events, run 'eb events -f'. To set timeout duration, use '--timeout MINUTES'.
[and-cli] [ERROR]  - Failed to deploy to AWS beanstalk

Solution
Lets increase the default timeout (--timeout) flag sent to awsebcli from 10 minutes to 20 minutes. Additionally, lets provide an optional argument to override it. That way we don't need to keep moving the "needle" and we don't hold up edge use cases that happen to exceed the limit legitimately.

It seems that the commands are not getting read right when used in WSL. I'm using Ubuntu 18.04. When I type just and-cli the help menu is displayed but when I give it a command I get an error that the command does not exist.

There is an increasing likelihood for regressions as the code base grows from an actual integration perspective. Investigate options for adding automated integration tests for our parent commands.

Example:

• https://medium.com/@zorrodg/integration-tests-on-node-js-cli-part-1-why-and-how-fa5b1ba552fe

Problem
The dotnet-test command will always rebuild the project in between runs, which can be add up over time. If you're intending on running the command multiple times in a row with the same assemblies, there is no need to rebuild every time. One use-case of this is to force the dotnet test runner to run the solution's test assemblies sequentially, and not in parallel.

Solution

• Update the dotnest-test command to have a flag for skipping the clean, build, and restore steps

• The default behavior should still be to clean, build, and restore unless explicitly opting out

• The build and restore process can be done with the existing dotnet-build module

• Add appropriate test coverage

Need a way to conventionally execute npm run test command from CRA as well as other frontend projects.

ie. and-cli webpack-test

Additionally, provide support to switch it to continuous integration mode (ie. --ci flag)

Description
Similar to our CSharp packages we need to have the automated testing suite configured to run during development as well as with Travis CI. First things first, configure jest automated test suite

The deploy command and its subcommands have grown to a point that they would benefit from some supplementary documentation. While there are many ways this could go a few ideas are...

• Diagram showing high-level flow of sub/command structure and the steps it handles for deployments
• Written play by play of how the deploys work (both aws and azure branches of it)
• Provide actual examples of how it might be composed in a CI environment (like jenkins) to provide a complete deployment process on projects

• Extract 'dotnet' module from and-cli-dotnet command in the root of the project, similar to how dotnet-build, dotnet-clean, etc.
  • Perform any necessary refactors to allow arguments to be passed in for configuration of how the module runs (for example, the migration module has several setter methods for configuration)
• Add any applicable unit tests

Wanted to raise an issue so we can document discussion around revving the project to a 1.0.0 version, since it has matured quite a bit, and the testing suite has begun to take form.

Speaking of versioning, do we want to start tagging releases/versions in Github itself? That might make it easier to track down code changes between versions, and write up release notes.

Problem
Currently the dotnet-build.js module will perform a shell.exit(0) (on success). The issue is that these modules are by nature to be shared. Right now this isn't being shared as a sub-step, but if it were and the parent command was hoping to do additional work after a successful build, it wouldn't occur.

Solution
Remove any shell.exit(0) in modules and leave that to the parent. At the time of writing this, the only known is the dotnet-build.js module.

Devise strategy for documentation related tooling

Problem
I'm finding a common issue where folks run the nuget --publish command and forget to commit and push the changes that were made by the command (typically .csproj) files. This causes confusion and if there happened to be other changes while publishing, while rare, those could be lost if another happens to publish.

Solution
Extend the nuget --publish command to offer the user to perform the git work when it is successful.

When successful, offer prompt along the lines of "Don't forget to commit and push your git changes! Would you like me to do this for you? (Y/n)?"

When 'Y', runs the following git commands under the hood...
git add -A
git commit -m "Rev'd to version {x.y.z}"
git push

It would be nice to have a general git module to consolidate common git commands that will be used in the project. As other commands are built that utilize git, this would help keep code DRY.

When this is implemented, and-cli-deploy-azure-web-app.js should be refactored to utilize the shared module, if it makes sense.

During active development of webpack based frontend build pipelines, we leverage the webpack dev server and have our background proxy those routes to that server. The ideal is to have a very easy way to opt-in to running your local development environment in "release mode" without much additional configuration.

Goals

• Compile release artifacts minified, gzipped, and versioned
• Deploy developer-machine version to configured storage (s3, azure blob)
• Read .env and conventional dotenv files to configure storage settings
  • DEPLOY_PUBLIC_URL
  • DEPLOY_ACCESS_KEY
  • DEPLOY_SECRET_KEY

Command: and-cli webpack --debug-release

The current COMMANDS.md is fine, but at this current size has become a bit hard to read and manage. Investigate options and port existing documentation.

As with our other commands, it is great to run the various pieces of a project seemlessly without needing to change directory a whole lot and remember nuances between different tooling. For starters, we need a frontend related command that will allow us to extend and conventionally run projects utilizing webpack (no knowledge of framework).

Ultimately it will defer to the conventional npm run start command that is used by a variety of different frameworks and tools (ie. create-react-app / react-scripts).

Command: and-cli webpack

As some time has passed since this project was originally started, the team has decided upon some looser formatting standards in favor of tooling to do the work for us. (yay! 🎉 ) See this issue for further discussion.

Unless there are any objections, I am suggesting we add Prettier to the project and do a full format on the project as-is. We'll want to set up either a git hook or an npm script to run the formatter after a build to ensure code coming in or changing meets our standard formatting style. This should lessen the burden on developers new to the project.

A pretty standard configuration can be found in one of our JavaScript repositories, such as AndcultureCode.JavaScript.Core

After configuration of jest, we need to go further and configure the code coverage

Backfill tests for node-restore module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.

• Pipe code coverage to codecov.io
• Add travis badge to readme
• Add codecoverage badge to readme
• Additional PR to AndcultureCode: add documentation to AndcultureCode readme around setup of JavaScript/Node repositories

Add support for optional --watch flag to dotnet-test command. Ultimately prefixes dotnet test with dotnet watch test.

This would only work when run on an entire solution (.sln) or a single project (.csproj).

The current readme is already quite large. Devise proposal and refactor existing README.

Backfill tests for dotnet-publish module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.

Backfill tests for user-prompt module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.

While you're in there, consider renaming the file and any commands that import it to just prompt. Not sure why I named it user-prompt. Who else would we be prompting?

Description
The 'install' command should more gracefully modify the developer's .bashrc file. In my case, it did not add a new line before adding the two aliases, so one of the aliases was dropped onto the ending bracket of my .bashrc and and unexpected EOF was thrown when resourcing the file:
bash: /home/brandon/.bashrc: line 625: syntax error: unexpected end of file

}alias and-cli='npx and-cli'
alias and-cli-dev='/home/brandon/AndcultureCode.Cli/cli.js'

Additionally, a simple comment about where/why these aliases were added might be nice for someone referencing their bash file, something linking back to the repo or npm package

Problem
[and-cli] Deploying to AWS beanstalk. Check AWS console for realtime output. This could take a few minutes...
Traceback (most recent call last):
File "C:\Python27\Scripts\eb-script.py", line 6, in
from pkg_resources import load_entry_point
File "c:\python27\lib\site-packages\pkg_resources_init_.py", line 3017, in
@call_aside
File "c:\python27\lib\site-packages\pkg_resources_init.py", line 3003, in call_aside
f(*args, **kwargs)
File "c:\python27\lib\site-packages\pkg_resources_init.py", line 3030, in _initialize_master_working_set
working_set = WorkingSet.build_master()
File "c:\python27\lib\site-packages\pkg_resources_init.py", line 661, in _build_master
return cls.build_from_requirements(requires)
File "c:\python27\lib\site-packages\pkg_resources_init.py", line 674, in build_from_requirements
dists = ws.resolve(reqs, Environment())
File "c:\python27\lib\site-packages\pkg_resources_init.py", line 853, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'botocore<1.13,>=1.12.29' distribution was not found and is required by awsebcli
[and-cli] [ERROR] - Failed to deploy to AWS beanstalk

Solution
Turns out we must specify explicit awscli and awsebcli versions to ensure their subdependencies align.

Backfill tests for zip module

As a general rule of thumb, I like to test exit paths (error or otherwise) to ensure the module is killing the process when expected. If we are returning from an erroneous exec or spawnSync call , ensure we are forwarding the exit code properly. (See dotnet-build.test.js for an example of this)

Add coverage where you see the most value - Jest will usually mark description() functions as having missed coverage, but seeing as we only use these for printing out text to the UI help menu, there's little value in testing that.