Git Product home page Git Product logo

scripts-1's Introduction

Scripts

Welcome to FlossWare Scripts! This represents a collection of reusable scripts we hope you will find useful.

History

This project got its start as a way to provide scripts for [Jenkins] (http://jenkins-ci.org) as part of our [Continuous Delivery] (http://en.wikipedia.org/wiki/Continuous_delivery) initiative. We currently use an [OpenShift] (https://www.openshift.com) instance of [Jenkins] (https://jenkins-camponotus.rhcloud.com) and faced challenges publishing artifacts from our [Github Project] (https://github.com/FlossWare/java) to [Bintray] (https://bintray.com/flossware/maven/java/view). We realized that we needed a repository of reusable scripts (presently bash functions) as well as provide [Jenkins] (http://jenkins-ci.org) and [OpenShift] (https://www.openshift.com) scripts that others can we reuse - ideally side stepping issues we discovered thereby helping folks to get on with "the business at hand."

Conventions

Directories

The following represents our directory structure:

  • bash - bash utility scripts and functions.
  • bash/bintray - Bintray related scripts for creating, listing, publishing and deleting artifacts.
  • bash/continuous-delivery - continuous delivery related scripts. Presently only "short cuts" for preparing [Maven] (http://maven.apache.org) and RPM artifacts for release.
  • bash/openshift - OpenShift related scripts that can help with rev'ing artifact versions, pushing out artifacts to Bintray, Github, Bitbucket, etc.
  • test/bash - contains our unit tests (and test suite) for the bash utility scripts.

Naming

Scripts

Scripts are named using lower case words separated by dashes (like git-utils.sh). All utility scripts end in -utils.sh (like common-utils.sh).

Functions

We are somewhat inconsistent in our naming of functions. Initially we started using all lower case dash separated words (like generate-unique). However, we'd like to standardize upon lower camel (like getBackground) per [Issue #48] (FlossWare-Archives#48).

Functionality

Continuous Delivery

For us, continuous delivery relates to utilizing:

In reality we are providing the "delivery team," "version control" and "build & unit test" portion of the [Continuous Delivery] (http://en.wikipedia.org/wiki/Continuous_delivery) process as follows:

  • A developer works on a feature (likely in a branch) and when the feature is completed, merges the feature in and makes it available. In [Git] (http://git-scm.com), it'd be equivalent to pushing master out to remote as in: git push origin master
  • [Jenkins] (http://jenkins-ci.org) monitors the source control system and performs the following when a change is observed:
    • Retrieves the latest version of the code base.
    • Revs the appropriate versioning file. For [Maven] (http://maven.apache.org) that's the pom.xml and for RPM's, that's a spec file. More about respective versioning can be found below.
    • Build the source and "do the needful." This is the portion that you will be responsible to provide - this may include invoking [Maven] (http://maven.apache.org) and instructing it to run all tests or how to build using an RPM spec file. More about building will be found below.
    • Commit the versioning file (now containing a rev'd version). For spec files, automatic logs of changes since the last rev are included in the %changelog section.`
    • Publish an artifact to a consumable repository such as [Bintray] (https://bintray.com).

If you are using [Jenkins] (http://jenkins-ci.org) at [OpenShift] (https://www.openshift.com), we provide the necessary plumbing so your work can be rev'd and pushed back out to [GitHub] (https://github.com/) or [Bintray] (https://bintray.com). Please note, we are assuming [Jenkins] (http://jenkins-ci.org) at [OpenShift] (https://www.openshift.com) and [Git] (http://git-scm.com) below. One can easily adapt our work to other [CI] (http://en.wikipedia.org/wiki/Continuous_integration) environments or non [OpenShift] (https://www.openshift.com).

Jenkins

Configuration

As mentioned above, we use [Jenkins] (http://jenkins-ci.org). There is a "gotcha" you need to recognize: [Jenkins] (http://jenkins-ci.org) not only builds and manages versioned files (like pom.xml or a spec file), but it also publishes those changes back to a remote [Git] (http://git-scm.com) repository. Therefore, when building, you need to instruct [Jenkins] (http://jenkins-ci.org) to exclude changes from itself. If you do not, it will rev, build, commit, checkin...repeat - over and over again. To avoid this, configure [Jenkins] (http://jenkins-ci.org) as follows:

  • Install the [Git Plugin] (https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin) if not already installed.
  • Click on the Jenkins link (upper left), then "Manage Jenkins" and "Configure System."
  • Navigate to the "Git plugin" section and provide values for "Global Config user.name Value" and "Global Config user.email Value." We opted to name our user name "jenkins."
  • Click the "Save" button.
  • For your [Jenkins] (http://jenkins-ci.org) job, choose it and click "configure."
    • Navigate down to the "Source Code Management" section for [Git] (http://git-scm.com).
    • Add two "Additional Behaviour" options:
      • "Checkout to a specific local branch." For most cases use "master" as the branch name.
      • "Polling ignores commits from certain users." For the "Excluded Users" enter the name of the [Jenkins] (http://jenkins-ci.org) user you configured above for "Global Config user.name Value."
    • Click the "Save" button.
FlossWare Scripts Job

We presently do not yet have the capability to RPM-ify our scripts (this will be resolved in [Issue #30] (FlossWare-Archives#30)). However, in [OpenShift] (https://www.openshift.com), we won't be able to install any RPM we generate and have to, instead, have a [Jenkins] (http://jenkins-ci.org) job that monitors our [Scripts Git Repo] (https://github.com/FlossWare/scripts.git) for changes to the master branch. Our job is entitled Flossware-scripts and it can be utilized by other jobs refering to the scripts in the form of $WORKSPACE/../FlossWare-scripts/bash. Below you will see how we execute the [FlossWare Scripts] (https://github.com/FlossWare/scripts).

Additionally, you may want to consider installing the [Build Blocker Plugin] (https://wiki.jenkins-ci.org/display/JENKINS/Build+Blocker+Plugin) and block any of your jobs from building if a build of [FlossWare Scripts] (https://github.com/FlossWare/scripts) is executing. Please note, a build of [FlossWare Scripts] (https://github.com/FlossWare/scripts) may be misleading - we simply need the job to update (aka checkout the latest) when there are changes to the [scripts] (https://github.com/FlossWare/scripts).

OpenShift

If you use a [Git] (http://git-scm.com) hosting service such as [GitHub] (https://github.com/), the [Jenkins] (http://jenkins-ci.org) ~/.ssh directory is inaccessible as it's owned by root. If you want [Jenkins] (http://jenkins-ci.org) to affect change to your versioning files (pom.xml or spec file), you will need a way to provide an SSH key so commits can be automatically propogated back. Presently we are defaulting to an SSH directory at ${HOME}/app-root/runtime/.ssh. Additionally, we've provided the script bash/openshift/openshift-keygen.sh that can create the aforementioned directory and setup a passwordless SSH key. You can then add the public key ${HOME}/app-root/runtime/.ssh/id_rsa.pub to your [Git] (http://git-scm.com) hosting service.

The [OpenShift] (https://www.openshift.com) script bash/openshift/openshift-push-to-gitrepo.sh will ensure that the ${HOME}/app-root/runtime/.ssh/id_rsa.pub is used.

Bintray

Below we denote examples that indirectly call out to [Bintray] (https://bintray.com) to publish artifacts. Two notable examples are:

These two scripts reside with the other [Bintray] (https://bintray.com) scripts and honor the same command line parameters. More information can be found below in the Bintray section.

Maven

Pre Build

For a pom.xml, versioning increments the third digit of an assumed three digit versioning scheme as denoted in the version element (namely incrementing Release): Major.Minor.Release. As an exmaple:

  <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<groupId>org.flossware</groupId>
	<artifactId>scripts-test-maven</artifactId>
	<version>1.0.3</version>

The next version will be 1.0.4 as seen below:

  <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<groupId>org.flossware</groupId>
	<artifactId>scripts-test-maven</artifactId>
	<version>1.0.4</version>

Add an "Execute Shell" command before any [Maven] (http://maven.apache.org) invocation line as follows:

cd $WORKSPACE &&
$WORKSPACE/../FlossWare-scripts/bash/continuous-delivery/prepare-maven-git-release.sh
Post Build

After your [Maven] (http://maven.apache.org) invocation line (something akin to "Invoke top-level Maven targets" within the "Build" section of your job), you will want to have an "Execute Shell" command like:

cd $WORKSPACE &&
$WORKSPACE/../FlossWare-scripts/bash/openshift/openshift-push-to-gitrepo.sh

Should you use an artifact hosting service such as [Bintray] (https://bintray.com), you will want to do something similar to the following:

cd $WORKSPACE &&
$WORKSPACE/../FlossWare-scripts/bash/openshift/openshift-push-to-gitrepo.sh &&
$WORKSPACE/../FlossWare-scripts/bash/bintray/maven-publish.sh --bintrayUser someUser --bintrayKey abcdefghijklmnopqrstuvwxyz0123456789ABCD --bintrayAccount someAccount --bintrayPackage somePackage

The line $WORKSPACE/../FlossWare-scripts/bash/openshift/openshift-push-to-gitrepo.sh will commit and tag your pom.xml and push it out to either [GitHub] (https://github.com/) or [Bitbucket] (https://bitbucket.org/).

Please note, supply the appropriate values to the maven-publish.sh script based upon your account - see the Bintray section below for more information.

RPM

Pre Build

For a spec file, versioning increments the Release section of the file (as in Release: xyz). As an example:

Summary: A set of scripts to help aid in Salesforce.com development and deployment
Name: flosswareScriptsTest
Version: 1.0
Release: 1

The next release will be 2, as can be seen below:

Summary: A set of scripts to help aid in Salesforce.com development and deployment
Name: flosswareScriptsTest
Version: 1.0
Release: 2

Add an "Execute Shell" command before any statements you use to build your RPM as follows:

cd $WORKSPACE &&
$WORKSPACE/../FlossWare-scripts/bash/continuous-delivery/prepare-rpm-git-release.sh /path/to/your/file.spec
Post Build

After your RPM building invocation line, you will want to have an "Execute Shell" command like:

cd $WORKSPACE &&
$WORKSPACE/../FlossWare-scripts/bash/openshift/openshift-push-to-gitrepo.sh

Should you use an artifact hosting service such as [Bintray] (https://bintray.com), you will want to do something similar to the following:

cd $WORKSPACE &&
$WORKSPACE/../FlossWare-scripts/bash/openshift/openshift-push-to-gitrepo.sh &&
$WORKSPACE/../FlossWare-scripts/bash/bintray/rpm-publish.sh --bintrayUser someUser --bintrayKey abcdefghijklmnopqrstuvwxyz0123456789ABCD --bintrayAccount someAccount --bintrayRepo someRepo --bintrayPackage somePackage --bintrayFile /path/to/the.rpm --bintrayContext /path/to/the.specFile

The line $WORKSPACE/../FlossWare-scripts/bash/openshift/openshift-push-to-gitrepo.sh will commit and tag your spec file and push it out to either [GitHub] (https://github.com/) or [Bitbucket] (https://bitbucket.org/). The tag is generated from the spec file (using the Version dash Release) and a comment is added to [Git] (http://git-scm.com). Additionally, git commit messages since the last tag will be included in the %changelog section.

As an example:

%changelog
* Mon Dec 29 2014 OpenShift <jenkins@jenkins-camponotus.rhcloud.com> 1.0-2
- Fixed the null pointer.
* Sun Dec 28 2014 OpenShift <jenkins@jenkins-camponotus.rhcloud.com> 1.0-1
- Broke the build.

Please note, supply the appropriate values to the rpm-publish.sh script based upon your [Bintray] (https://bintray.com) account - see the Bintray section below for more information.

All in One Build

You can just lump everything together to pre build, create the rpm and publish like so (using the above as an example):

cd $WORKSPACE &&
$WORKSPACE/../FlossWare-scripts/bash/continuous-delivery/prepare-rpm-git-release.sh /path/to/your/file.spec &&
$WORKSPACE/someScriptToCreateYourRpm.sh &&
$WORKSPACE/../FlossWare-scripts/bash/openshift/openshift-push-to-gitrepo.sh &&
$WORKSPACE/../FlossWare-scripts/bash/bintray/rpm-publish.sh --bintrayUser someUser --bintrayKey abcdefghijklmnopqrstuvwxyz0123456789ABCD --bintrayAccount someAccount --bintrayRepo someRepo --bintrayPackage somePackage --bintrayFile /path/to/the.rpm --bintrayContext /path/to/the.specFile

Bintray

We've included a number of scripts to create, list and delete artifacts using the [Bintray API] (https://bintray.com/docs/api). While not all command line parameters are used by each script, the following bulleted list denotes the recognized options (some familiarity with [Bintray] (https://bintray.com) may provide better clarity):

  • --bintrayUser - your user name.
  • --bintrayKey - your API key.
  • --bintrayAccount - your account name (really your organization name).
  • --bintrayRepo - the repo type like maven, rpm, etc.
  • --bintrayLicenses - for now, a single entry license name you use when creating a package. For example GPL-3.0.
  • --bintrayPackage - the package name.
  • --bintrayDescription - a description like a package description.
  • --bintrayVersion - a version of an artifact.
  • --bintrayFile - the actual local artifact you will publish.
  • --bintrayContext - this is a [FlossWare Scripts] (https://github.com/FlossWare/scripts) specific item and it denotes the context in which you are working. For example, if you are publishing RPMs, its the spec file used to build the RPM or a pom.xml.

All of the scripts support the --help command line argument which will emit the aforementioned names and descriptions to the console.

Scripts

Our scripts are divided into utilities and commands. The commands perform an action but those actions may require related functionality which is encapsulated in utilties. For example, the version commands [version-create.sh] (https://github.com/FlossWare/scripts/blob/master/bash/bintray/version-create.sh) and [version-delete.sh] (https://github.com/FlossWare/scripts/blob/master/bash/bintray/version-delete.sh) both require a repo, package and version to work correctly. This functionality is encapsulated in the [version-utils.sh] (https://github.com/FlossWare/scripts/blob/master/bash/bintray/bintray-version-util.sh) script.

All commands require the following parameters: --bintrayUser, --bintrayKey and --bintrayAccount.

Package

The package scripts allow you to create, delete and list your [Bintray] (https://bintray.com) packages. The required package parameters are: --bintrayRepo and --bintrayPackage.

Version

The version scripts allow you to create and delete your [Bintray] (https://bintray.com) versions. The required version parameters are: --bintrayRepo, --bintrayPackage and --bintrayVersion.

Content

The content scripts allow you to create and publish your [Bintray] (https://bintray.com) content. The required content parameters are: --bintrayRepo, --bintrayPackage and --bintrayVersion.

Publishing

The publishing scripts allow you to publish your [Maven] (http://maven.apache.org) and RPM [Bintray] (https://bintray.com) content.

Unit Testing

Without a doubt, we discovered writing bash functions was simple - testing them for "the long haul" much more challenging. Our "knee-jerk" reaction was to write "one off" tests - but that doesn't ensure our quality should we need to make changes. With that said, we've provide a test harness vary similar in nature to [JUnit] (http://junit.org) in what we call the [test-utils.sh] (https://github.com/FlossWare/scripts/blob/master/bash/test-utils.sh) framework. If you are familiar with [JUnit] (http://junit.org) this should "hopefully" be obvious. We do unit test (or try to) all our functions as can be found [here] (https://github.com/FlossWare/scripts/tree/master/test/bash).

How To

This section is a work in progress. Please stay tuned.

Covered Code

The following unit tests have high test coverage:

Miscellaneous

Plugins

While not yet written, we have a longer term goal of providing some [Jenkins plugins] (https://wiki.jenkins-ci.org/display/JENKINS/Plugins) for things like publishing to [Bintray] (https://bintray.com). Stay tuned for things to come!

scripts-1's People

Contributors

sfloess avatar

Watchers

avatar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.