paypal / butterfly Goto Github PK

It probably will need:

• REST API
• Messaging queue for upgrades processing
• Pluggable integration (via JPA) with a DBMS as external system to persist upgrade requests, metrics, and any other necessary meta-data

1. Add the "-l" option right after the usage section
2. Mention where log files are placed

There should be a pre and a post transformation validation, as part of the extensions API.

In case of upgrades, the pre validation will happen only once, right before the first upgrade step, and the post validation will happen only once as well, right after the last upgrade step.

Requirements:

1. Pre-validation: checks if the application is eligible to be transformed, and fails the transformation before it even starts if the application is not valid
2. Post-validation: checks if the application is in a valid and acceptable state after the transformation. Depending on the validation criteria (which is defined by the extension) the whole transformation can be declared as success or fail, even after all transformation operations have been already performed.
3. Transformation validations API is part of Extensions API
4. Pre and post transformation validations can be turned ON or OFF
5. Pre and post transformation validations should be able to run TUs
6. Pre and post transformation validations must not be able to run TOs, or modify the app in any ways (except post-transformation manual instructions)
7. Post transformation validations can be used to gather meta-data about the transformed application and produce one ore more post-transformation manual instruction, depending on the state of the transformed application.

Add many real extensions for many common uses cases, building a catalog of extensions that many people can use, helping Butterfly adoption.

Add a UtilityCondition that checks if a regular expression matches any line in the specified text file. It should return true only if the specified text file has one or more lines that match the given regular expression.

Metrics JSON object currently does not include fields that are null. It would be easier for report tools that consume these metrics to handle them if those null fields were still present, but set to null.

Print the date of transformation Butterfly execution, so it is possible to tell when it was run from execution logs.

Print that from a layer beyond the facade, so that this information is not tied to the CLI and will be available even if Butterfly was run in other scenarios, like from a Java application.

PomAddDependency implements AddElement, which introduces a problem in the case of WarnButAdd, since pom files cannot have duplicated dependencies added to it. Other AddElement implementations might have the same problem.

Search for all implementations of AddElement that have this issue and solve them as well.

Upgrade the most important third-party jars, a few examples below, to their latest versions.

• Spring Boot
• maven-model
• maven-invoker
• commons-io
• commons-lang
• lightcouch
• gson
• xmlunit
• yamlbeans
• reflections
• javaparser
• guava
• zip4j
• testng
• mockito
• powermock

Improve sample-extension by adding the upgrade templates mentioned in the "Running" document, then mention them in that document.

1. Rename attribute to compareAttribute
2. Add a new instance variable called compareRelative, to be used as an alternative to compareAttribute, pointing directly to the comparison file, but relatively to the transformed application folder, and without the need for a transformation context attribute.

They could be as simple as the POM TOs (having one TO per use case), or there could be just one single TO, which would perform the transformation based in XSTL.

They could be implemented using one of the two technologies below:

1. DOM (read once, model, and write brand new file). Elements to be modified can be found using XPath.
2. StAX (streaming based, modification on top of original file).

Print extension name and version during Butterfly execution, so it is possible to tell the extension version from execution logs.

Print that from a layer beyond the facade, so that this information is not tied to the CLI and will be available even if Butterfly was run in other scenarios, like from a Java application.

Butterfly has one limitation when performing certain modifications to Maven pom files. Depending on the modification, it ends up modifying reformatting the whole file, changing indentation and the order of the tags. That happens because Butterfly reads the file, parses it, builds a model in memory, applies the transformation, then writes it back to the same file.

When it writes the file with the transformation, it uses the model in memory as source, which means it looses entirely the original formatting and comments the file used to have. Some users don't like it because many times the transformation is very trivial, and would change just a few lines, but still the result is a file that looks entirely different.

The goal of this issue is to create a new set of Maven POM file Transformation Operations that are able to perform isolated modifications, preserving formatting, order and comments by using StAX.

In addition to that, also undo the changes in this commit, uncommenting all unit tests that assure formatting and comments are preserved.

A few TOs have already been implemented using StAX, such as #84 and #92.

StAX references

• https://docs.oracle.com/javase/tutorial/jaxp/stax/index.html
• https://en.wikipedia.org/wiki/StAX
• http://tutorialspointexamples.com/stax-xml-parser-java-tutorial-examples/
• http://www.xml.com/pub/a/2003/09/17/stax.html

1. Add the following How-To documents:
   1. How to create an extension from scratch
   2. How to create a custom TU or TO
   3. How to integrate Butterfly into a Java application
2. Add an introduction section explaining:
   1. The definition and differences between a TU and a TO
   2. The basics of setting the file a TU or TO is supposed to perform against, explaining all versions of methods relative and absolute
   3. The transformation context attribute and how to use it
3. Add sections about usage of:
   1. Every available TU and TO
   2. Conditional execution
   3. Dependency management
   4. Resilience, including perform and execution result types
   5. Error handling, including abortion triggers
   6. Pre and post validation (after #9 is added)
   7. Advanced TU execution modes
      1. Configurable utilities
      2. Anonymous utilities
      3. Multiple execution
      4. Group execution
      5. In-loop execution
   8. Post-transformation features
      1. Manual steps Reporting
      2. Metrics

In TransformationUtility class, rename:

• checkForBlankString to validateMandatoryString
• checkForEmptyString to validateOptionalString
• checkForNull to validateMandatoryObject

Also, validateOptionalString method should return a String, which must be:

• null, if the property value is an empty String or null
• the actual value, if the property value is not null neither an empty String

Add option -n to CLI to set the name of the transformation folder, instead of using the original application folder name followed by a timestamp suffix.

Implement continuous delivery through automated deployment via Travis CI.
To do so, the generated jars need to be signed. The necessary key to sign the jars need to be encrypted, checked in to the repository, and then decrypted by Travis CI when executing the CI job.

More information at:

• https://docs.travis-ci.com/user/encrypting-files/
• http://www.debonair.io/post/maven-cd/

A few TransformationOperation classes already implement AddElement or ChangeOrRemoveElement. A few of them actually also extend AddElementTO or ChangeOrRemoveElementTO abstract classes.

However, there are many other Transformation Operations that could also rely on these interfaces, or abstract classes, like RemoveProperty for example.

This issue is about identifying them all and applying one of those interfaces or abstract class.

Even though Butterfly has a metrics feature, based on JSON metric objects, it does NOT provide at this point any schema nor data base to persist these objects.

Still, if at any point that metrics data base is created as a Butterfly project (for example under issue #11), then add the following calculated fields (non persistent) in the metrics DB:

Totals

performTotal = utilitiesCount + operationsCount;
skippedTotal = performResultSkippedConditionCount + performResultSkippedDependencyCount;
toExecutionTotal = toExecutionResultErrorCount + toExecutionResultNoOpCount + toExecutionResultSuccessCount + toExecutionResultWarningCount;
tuExecutionTotal = tuExecutionResultErrorCount + tuExecutionResultNullCount + tuExecutionResultValueCount + tuExecutionResultWarningCount;
executionTotal = tuExecutionTotal + toExecutionTotal;
executionErrorTotal = toExecutionResultErrorCount + tuExecutionResultErrorCount;

Rates

if (performTotal > 0) performErrorRate = 100 * (float) performResultErrorCount / performTotal;
if (executionTotal > 0) executionErrorRate = 100 * (float) executionErrorTotal / executionTotal;
if (toExecutionTotal > 0) toExecutionErrorRate = 100 * (float) toExecutionResultErrorCount / toExecutionTotal;
if (toExecutionTotal > 0) toExecutionWarningRate = 100 * (float) toExecutionResultWarningCount / toExecutionTotal;
if (tuExecutionTotal > 0) tuExecutionErrorRate = 100 * (float) tuExecutionResultErrorCount / tuExecutionTotal;
if (tuExecutionTotal > 0) tuExecutionWarningRate =  100 * (float) tuExecutionResultWarningCount / tuExecutionTotal;

TransformationUtilityLoop should allow setting a maximum number of iterations, to avoid infinite loops.

Add a "print" method to TransformationTemplate to print their instructions in an output stream, and another method to return it as a string.

The transformation template human-readable manual instructions could then be could automatically generated and displayed to users curious about what changes the transformation is supposed to do.

Problem definition

Every Transformation Utility and Operation input parameters need to be validated before execution. These parameters are the fields that are set during transformation definition time or during transformation execution time (usually based on the transformation context).

If an utility pre-validation fails, an exception should be thrown, the execution should be interrupted, and a perform result of type PRE_VALIDATION should be returned (containing the validation exception).

Potential solutions

There are two potential solutions for this problem:

1. Using BeanValidations 2.0
2. Using an abstract method called preValidation. It should be added to TransformationUtility class and be called in the perform method right before execution.

Which exception to throw, TransformationDefinitionException or PreValidationException?

Some TUs and TOs currently are currently throwing TransformationDefinitionException.

Other option would be a new type, called PreValidationException, could be created. It should extend TransformationException, which should extend ButterflyRuntimeException. TransformationUtilityException should also extend TransformationException.

Acceptance criteria

1. Define a transformation time TU and TO pre-validation .
2. Implement it for EVERY transformation utility and operation. Move any pre-validation done currently in any utility execution methods to the new mechanism. Notice that in many cases there will be a TODO comment referring to it.
3. Update EVERY transformation utility and operation unit test adding pre-validation test cases.
4. If any new exception type is introduced, the exceptions class diagram in Design Documents should be updated accordingly.
5. Update README and end user documentation accordingly.

Address Codacy static code analysis issues and make sure "error prone" category is at least 90%.

Find all lines of code that creates temporary files, and change its logic to rely on File.createTempFile (letting the OS decide the temporary directory), instead of doing so manually via regular file creation API.

Add a helper and protected final method for it in TransformationOperation, to be called getOrCreateReadFile. Behind the scenes that, should call File.createTempFile passing butterfly as prefix.

Sometimes logging messages show color code characters around the log level, instead of coloring it. See the example below.

This behavior has been seen when running Butterfly in Windows machines or Jenkins CI jobs. It has not been seen when running Butterfly in Mac OS. Having said that, it might be OS specific.

Example of logging statement with the problem (notice the characters around the log level INFO):

[09:36:53.323] [[34mINFO[0;39m] 	1	 - Packaging for POM file /pom.xml has been changed to jar
[09:36:53.438] [[34mINFO[0;39m] 	2	 - Parent for POM file /pom.xml has been set to org.springframework.boot:spring-boot-starter-parent:pom:1.5.6.RELEASE

Example of logging statement without the problem (notice the log level INFO is print correctly):

[09:36:53.323] [INFO] 	1	 - Packaging for POM file /pom.xml has been changed to jar
[09:36:53.438] [INFO] 	2	 - Parent for POM file /pom.xml has been set to org.springframework.boot:spring-boot-starter-parent:pom:1.5.6.RELEASE

The log level coloring configuration can be seen here.

Add equals and hashCode methods to all TUs. Use PomAddPlugin as reference.

Add option -f to CLI to to modify the original application folder directly, instead of creating a separated transformation folder. Whenever using this option, options -o, -z and -n should be ignored.

Develop a metrics implementation (artifact named butterfly-metrics-log) that persist transformation metrics as debug logging statements.

Add a Transformation Operation offering modifications similar to what the Linux command sed offers by leveraging a Java library that provides sed functionality.

Further information about sed can be found here.

There are a few Java libraries that offer sed capability, like https://github.com/tools4j/unix4j. It is part of the scope of this issue to identify the best library to be used.

Add a Transformation Utility to build Maven model out of a pom.xml file not present in the application folder. It should download the pom file from the Maven repo (local or remote) and build a Maven model out of it.

The idea is to support finding out about Maven artifacts (used as parent or dependencies) that are not present in the application folder.

A few interesting and potentially useful references:

• http://aether.jcabi.com/
• https://stackoverflow.com/questions/11835076/api-to-download-dependencies-from-pom-xml
• https://stackoverflow.com/questions/15450383/using-maven-to-download-dependencies-to-a-directory-on-the-command-line

1. Make the sample-app more realistic and more complex
2. Change its name also to something more realistic
3. Change the sample extension accordingly, including its name too
4. Change functional tests and end user documentation accordingly as well

Reading and writing files multiple times are definitely expensive operations, since they are IO related.

Often Butterfly would open, read, write and close the same file multiple times during the same transformation. There could be a way to avoid that, or at least reduce the amount of IO, when manipulating specific type of files that can be entirely modeled and kept in memory, like properties files or Maven pom files, as long as they are not too large.

There could be a way to delay writing the file if multiple changes on it will be performed, performing all of them on one single object in memory, and then writing only once in the end.

Add XmlXPathElement, to be used to retrieve data from XML files using XPath expressions.
In addition, deprecate XmlElement.

Add option to remove the transformed app folder in case the transformation failed. Even if present, this option should be ignored if option -f is present, which is to be developed in #14.

The letter for this option hasn't been chosen at the time this issue was written.

An intermediate upgrade step is an upgrade step that can't be the last one to be executed in an upgrade path.

They though are not usually created as so initially. They are set to be intermediate later, after additional following upgrade steps are added and, for any reason, it is decided that an application would not be considered in a valid state if it was upgraded to a particular version. The upgrade step then that takes the application to that version gets set as intermediate.

This feature is helpful when, for example, it is discovered that a particular framework version has an unacceptable bug, and no application is supposed to stay in that version.

InsertLine TU currently only allows insertion after the matched line (in case of regex based insertions). It should allow placing the new line before as well. This decision, before or after, should be configurable.

Problem defition

There are inconsistent results whenever the file a TU is supposed to execute on does not exist.

Some TUs return NULL (or NO_OP in case of TOs), others ERROR with a FileNotFoundException, others ERROR with simple TransformationUtilityException or TransformationOperationException, and others don't do nothing, and then undesired consequences might happen later, like a NullPointerException.

Note

Not every TU has to have a file to perform on. For example, ManualInstruction doesn't.

Solution to be implemented

The file should be checked automatically by a logic in TransformationUtility class in the perform method before calling execution. If the file is null or not found, a PreValidationException should be thrown (but issue #29 has to be completed first then).

In case of TUs that allow that file to be null or non-existent (like ManualInstruction or FileExists for example), TransformationUtility class should have a protected method, called skipFileCheck, to so this check is not performed.

Removing redundant code

After this solution is implemented, check every TU and TO to find and remove any occurrence of validation to the file it is supposed to execute on.

As an alternative to manually placing the extension jar in the extensions folder, users could also install the extension by using a butterfly install <extension> command.

The <extension> term in the command above refers to the extension Maven artifact coordinates, and should be set in the format groupid:artifactid:version.

With this command, butterfly would automatically download the jar from the Maven repo (local or remote), and pace it in theextensions folder.

Requirements

1. Optionally, the user might specify only groupid:artifactid. In this case Butterfly should install the latest version of the extension available in Maven repository.
2. Before trying to download and install the extension, Butterfly should first check if that extension, and specified version, is already installed or not.
   1. If it is, a message Extension <groupid:artifactid:version> already installed should be displayed.
   2. If the extension is already installed, but in an older version, then the old version should be replaced by the specified newer version, and message Extension <groupid:artifactid> updated from version <old version> to version <new version> should be displayed.
   3. If the extension is already installed, but in a newer version, then Butterfly should not replace it, but display A newer version of <groupid:artifactid> is already installed. Uninstall it first if you want to replace it with an older version. instead.
3. Introduce also command butterfly uninstall <groupid:artifactid> to uninstall an extension. After conclusion, message Extension <groupid:artifactid> has been uninstalled should be displayed.

Design and implementation requirements

1. Rely on Maven utilities, libraries and API as much as possible to obtain the extension jar, as opposed to manually locate the jar in the file system, or download it from the internet using a HTTP client.
2. Even though the CLI project will be modified, to include install and uninstall options, place the code for this feature under a new project called butterfly-extension-manager. The idea is to not couple it with the CLI because it might be useful in the future in non-CLI scenarios too, like under a Butterfly server exposed to end users via a web front-end.

Butterfly removes execution flag from modified files.
Steps to reproduce this issue:

1. Turn on execution flag from any file that is supposed to be transformed (use chmod command in Linux or Mac OS)
2. Run the transformation
3. Check the permissions flag from that file in the transformed app folder, and notice the execution flag is off

Explore easier ways to define transformation templates. One alternative solution would be using a DSL, instead of Java.

These are the motivations for this story:

1. Make transformation template definition less verbose, easier and closer to natural language
2. Simplify and make it easy to develop and maintain extensions
3. Remove the Java requirement for extension development
4. Keep backward compatibility (if possible)
5. Support every feature present in the current transformation template definition mechanism (Java based)
6. Provide end user documentation

Kotlin and Spek

Probably the best option to develop this DSL would be Kotlin. An alternative would be Groovy. Kotlin seems to be better because it has better Java interoperability and is a more modern language.
Also, if Kotlin is the chosen one, Spek could be used for tests.

MavenGoal will run its output handlers regardless of exit code (0 or not). However, very often these output handlers are only useful if exit code is non-zero (because they analyze failure results). The problem is that even in 0 exit code case, those output handlers still run, and that is a big problem, because they are all extremely expensive.

This task is about investigating if there is a way to make output handlers configurable, so that they would run only in one of these cases:

1. Only if exit code is zero
2. Only if exit code is non-zero
3. Always

1. It should be placed under /tests/integration-tests folder, which already exists
2. The sample app in /tests/sample-app should be used as test app, and be transformed using sample-extension, which also already exists
3. Execute butterfly against sample-app, transform it to be Spring Boot based using sample-extension, and then compare the transformed application (created from Butterfly) with sample-app-transformed-baseline. If they differ, test should fail, if they are identical, test should pass
4. If this test execution is NOT too long, this test project should be a module of Butterfly root pom, which means it should run every time Butterfly is built
5. If this test execution is too long, it should be executed from a regression CI job, triggered by any change in Butterfly repo

There should be a pre and a post upgrade step validation, as part of the extensions API.

The pre validation will happen right before each upgrade step, and the post validation will happen right after each upgrade step.

Requirements:

1. Pre-validation: checks if the application is eligible to go through the upgrade step, and fails the transformation if the application is not in a valid state
2. Post-validation: checks if the application is in a valid and acceptable state after the upgrade step transformation. Depending on the validation criteria (which is defined by the extension) the upgrade can proceed to the next upgrade step or fail the whole upgrade, even after all transformation operations have been already performed.
3. Upgrade step validations API is part of Extensions API
4. Upgrade step pre and post validations can be turned ON or OFF
5. Upgrade step pre and post validations should be able to run TUs
6. Upgrade step pre and post validations must not be able to run TOs, or modify the app in any ways (except post-transformation manual instructions)
7. Upgrade step post validations can be used to gather meta-data about the transformed application and produce one ore more post-transformation manual instruction, depending on the state of the transformed application.

Some times when an TUs/TOs fails execution a TransformationUtilityException/TransformationOperationException is thrown. Some other times, a non-Butterfly exception, more specific to the problem, is thrown. instead.

To make this behavior consistent, make sure always a TransformationUtilityException or TransformationOperationException is thrown, encapsulating the original non-Butterfly problem specific exception.

Butterfly should be configurable via an external configuration file. Requirements:

1. This file format should be either properties or yaml (research pros and cons of both and decide the best option)
2. This file should be named .butterfly and placed under $HOME/, similar to .gitconfig for example.
3. The properties to be configurable should be classified as:
   1. Global properties: Butterfly properties independent from the extension to be run
   2. Extension specific properties: properties defined and used by installed extensions
4. A few examples of global properties:
   1. Toggle to turn metrics ON (true) or OFF (false). If ON, at least one metrics jar would have to be present to have metrics generated. Default to false.
   2. Toggle to stop upgrades (true) if an upgrade step has produced manual instructions. That would give a chance for the user to complete the manual instructions, and then, run Butterfly again, to continue the upgrade from that point. If off, the upgrade continues all to the end and all generated post-transformation manual instructions are collected and reported in the end. Default to false.
5. Global and extension properties can be overridden by setting the property as a CLI option parameter.

Update end user documentation accordingly.