vlead / continuous-integration Goto Github PK

Support for the websocket application layer protocol needs to be implemented in the Centos container.

@sivashankerN

As one of the requirement of ci-service is to make the lab/service auto deployable using ADS service after build, tests suite ran successfully. For this is to happen ci-service has to make a call to ADS's deployment REST API which runs on 5000 port.

As of now there is no way to reach 5000 port on ADS machine from ci-service.
could please make necessary changes to make this happen as earliest

Currently,
All the vlead services/labs project's make file has provision to run the test cases. By doing this we don't know the exact test report details using the status of make command to proceed further action. Hence after make runs we should list all the test.py files and run them individually to find out the test reports

• Dashboard
  VLEAD should provide a dashboard for statistics or analytics on a current
  basis:

  1. Number of total number of registered users
  2. Number of current users using the labs
  3. Performance metrics of all the servers:
     cpu usage, memory, number of requests etc.
  4. Lab level statistics etc.
  5. Get the collge cloud statistics.

  This allows for making decisions about the architecture regarding scaling
  etc.

• Projection of the system
  Any system that is built should have throbs that emit it's state.

Continuous Integration service has to be deployed using ADS service on testing, staging environments.

This issue will be updated against on functional requirements & design captured at #1 and #2

CI service' REST API should create an issue when build runs and also create/assign labels to the issue based on the result.

1. if build failed to run create an issue and assign 'build label' label
2. if build success create an issue and assign 'build successful' and close the issue

Unable to access the base4 server and AWS.

https://stackoverflow.com/questions/30669887/jenkins-integrating-jira-with-jenkins-for-creating-ticket-automatically-when

Configure jenkins server on three different vlead cluster environments testing, staging and production

The primary concern here is that how to provide a JRE to run the Jenkins CI tool on our servers since they aren't Java Application servers by default. In case this issue isn't resolved containerizing the application might have to be considered.

This issue captures design of CI

http://www.tornadoweb.org/en/stable/websocket.html

Jenkins: Open Sourced, can be hosted locally, can be configured to suit our needs, only issue is it needs a Java application server(ex: IBM WebSphere)
Travis CI: Very easy to setup with Github. Travis provides host environment, Travis installs required softwares for every build which is a time consuming process.
Buildbot: you do not need to change buildbot code to "extend" it, it is actually quite easy to import your own packages in its configuration in which you can sub-class most of the features with your own additions, buildbot is the most "general purpose" automatic build tool. Running tests however isn't very efficient.
Go CI: Go lets you distribute your builds across different systems and monitor them all in one place. Setting this up isn't very easy however and it has to be hosted externally.
Drone.io: Built on docker, easy setup,some cases of self hosted Drone not synchronizing commits with Github.
Tox: Tox is an automation tool providing packaging, testing and deployment of Python software right from the console or CI server. It is a generic virtualenv management and test command line tool

As of now the current CI tool we have built will allow any developer's repository to be integrated.We need to restrict this to repositories only in the vlead organisation.

Test case to be written for REST API : https://github.com/vlead/continuous-integration/blob/develop/src/runtime/rest/api.org#api-accepts-the-json-sent-by-the-git-hub-webhook-via-a-post-request

Implemented REST API /ci_hook and registered the same API on github repository to invoke whenever a git commit is made

REST API : /api/sample_build_lab is to be implemented for developer to check their github code for given a branch and repository url.

Currently ADS is using ubuntu12.04 64 bit openvz template to deploy service/lab since 14.04 is also LTS version we should move all labs into 14.04 64 bit OS.
To do that we should create a OpenvZ template which has the environment for running make and emacs setup. ADS will use this template to install the service/lab

@sivashankerN

As one of the requirement of ci-service is to copy/rsync the build documentation of lab/service to doc-server (where all project docs are available to users) after build operation is successful.

Hence ci-service programatically has to rsync build files to doc-server for which port 22 on doc-server is to be in function.

Please look into this as earliest

Performance Improvement
1. Build scalable flask applications using
gunicorn, nginx on ubuntu

Configure ci-service to use the OVPL(Auto Deployment Service) after the post build operation.

In order to continuous integration happen between github and jenkins, a web hook should be invoked from github repository to jenkins API (which is to be accessible over publicly)

Run Jenkins through ADS

By default jenkin server runs on 8080 port. Since vlead-cluster doesn't allow requests from 8080 port we should make it running on 80 port

Documentation can be found https://github.com/vlead/continuous-integration/blob/develop/src/runtime/rest/api.org#the-rest-api

The output console prototype for ADS is more or less ready.We are not able to move ahead due to issues with hosting it on the base4(staging) and AWS(production).The simplest form of a websocket based application is also facing hosting issues in the above environments.We tried using both Centos and Ubuntu instances.We tried using an Apache server on top also.
We need to get through this bottle neck.Research needs to be done from the scratch on how to host a websocket based server/application.

Body text

Research on alternatives to serve the purpose of websocket is required.A few of my suggestions are
1.Server Sent Events https://medium.com/code-zen/python-generator-and-html-server-sent-events-3cdf14140e56
2.HTTP Streaming
2.HTTP Long Polling

Please suggest a few other alternatives or give opinions about these if possible.

Build ran successfully and docs are moved to doc server

Build ran successfully and docs are moved to doc server

Build ran successfully and docs are moved to doc server

@psrikar97

Please create a doc server (use ubuntu14.04 template) using ADS on base4

Using ssh-plugin copy build artifacts from jenkins machine to doc server over ssh

Feedback from audience

1. All pull requests also to be tested using Jenkins
2. Same jenkin server is used to test & deploy service/lab in testing, staging and production environment
   This issue addressed on #12

Build ran successfully and docs are moved to doc server

Setup jenkins server on development environment (local machine) and try to implement the workflow for deploying docs on docs server