Git Product home page Git Product logo

infosys / openidp Goto Github PK

View Code? Open in Web Editor NEW
99.0 31.0 55.0 73.04 MB

An open source platform to accelerate and scale DevOps adoption across the enterprise. Contact [email protected] for any queries or for latest updates.

Home Page: https://www.infosys.com/services/agile-devops/offerings/infosys-devops-platform.html

License: MIT License

Shell 0.55% Dockerfile 0.03% Java 27.33% HTML 10.88% Groovy 5.18% Batchfile 0.02% PowerShell 0.01% Python 0.02% TypeScript 46.70% JavaScript 7.08% CSS 2.07% FreeMarker 0.07% SCSS 0.04%
devops automation platform centralized simplified

openidp's Introduction

Infosys DevOps Platform



About the repository

Codacy Badge

This repository is intended to hold the code base along with the required dependencies for the Infosys DevOps Platform (IDP) OSS version.

Getting Started

Below instructions will help the users to get to know the structure of this repository. See WIKI for notes on how to deploy the platform on a live system.

Details on project modules

Module/File Description Tech-stack details
Dashboard This microservice is designed for collecting relevant data during pipeline execution in Jenkins with the help of IDP's custom utilities. The data is then stored in Postgres and fed to Grafana dashboard, which is being used as the visualisation tool for providing trends and insights about applications.
  1. Programming Language : Java, for custom utilities
  2. Dependency Management: Maven
  3. Database : Postgres
  4. Metrics Dashboard : Grafana
DSL This module allows the developer to describe Jenkins jobs using Groovy-based language. The module uses Jenkins Job DSL Plugin which creates, updates and controls the Jenkins jobs. This allows to embrace the concept of "Configuration as a Code", thereby reducing MTTR of CICD Server (Jenkins)
  1. Programming Language : Groovy
  2. Dependency Management: Gradle
UI This module holds all the IDP user interface code and its related dependencies
  1. Framework : Angular 5
  2. Dependency Management: NPM
Services This module is divided into number of sub-parts.
  1. Spring RESTful service module for creating stateless services for implementing business logic. These services can be independently deployed and scaled up using a load balancer
  2. OAuth2 authorization module for enhanced security
  3. Module implementing Zuul Proxy to be used as an edge gateway which would route the request through Authorization Server (if Access token is not available/expired)
  4. Database layer for performing CRUD operations on backend database with respect to the operation being performed
  5. Swagger for automatic documentation of RESTful Services
  1. Framework : Spring Boot, Spring MVC, OAUth2.0
  2. Dependency Management: Maven
  3. Database: Postgres
Scheduler This Java based component is for timed execution of pipelines.
  1. Framework : Spring Boot
  2. Dependency Management: Maven
Jenkins This folder contains the necessary configuration files and plugins for Jenkins to make it work smoothly with IDP. It also holds the custom tools being used by IDP for performing various operations. These custom tools are placed in Jenkins custom tools directory, from where they can be copied to any of the build agents for running some specific errands, for instance, pushing the analysis results to dashboard while the build is still running. -
docs This folder contains the assets used in the Wiki pages. -
Data Files This folder contains the database initialization scripts for Postgres. It also consists of the Grafana configuration file (grafana.ini) with all the necessary configurations for integrating Grafana dashboard with IDP -

Please read WIKI for detailed documentation.

openidp's People

Contributors

dependabot[bot] avatar idp-oss avatar jaskirat8 avatar krishnakanthbn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openidp's Issues

Change emojis in Capabilities wiki page for better understanding

Describe the bug
Thumbs up-down emoji is creating confusion

To Reproduce
Steps to reproduce the behavior:

  1. Go to https://github.com/Infosys/openIDP/wiki/Capabilities
  2. Navigate role based access section. The current emojis used are not visible properly.

Expected behavior
Change them to cross and mark. You can review the changes in my wiki fork.
Repo link: https://github.com/manishbansal8843/OpenIDP-Wiki.git
Branch: manishbansal8843-patch-1

Screenshots
NA

Desktop (please complete the following information):
NA

Additional context
NA

Angular CI-CD support

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

build.sh: Syntax error line 137b

Describe the bug
A clear and concise description of what the bug is.
after Running sh build.sh
getting this error
build.sh: 137: build.sh: Syntax error: "do" unexpected (expecting "fi")
i am using master branch

openIDP : New Release

Hello All,

The new version of openIDP has been recently released. There have been major changes to enhance UI experience, installation automation experience and overall stability of the platform.

We strongly recommend you to update your forked repository with the latest code from the "master" branch. We look forward to your feedback and collaboration.

Reference links:

Thank you all.
@Chids-gs
@agrawallalit
@ahilmathew
@ajshastri
@ameyaagashe
@AnandSagar92
@ank-kumar
@arjunsreepad
@avsprashant
@AzmathR1
@bala-27
@basiljames
@beejhuff
@bhavith
@cloudarch77
@cswaroop
@cvsaraswathi
@daxgames
@deivendranj
@DevduttMalakar
@devopsomatic
@dipjyotighosh
@epasham
@erickcp
@esimionato
@himanshumps
@hiteshprajapati
@hoangpm
@invertednoob
@Jay-Wani
@jayanmai
@jaynoel
@letmelearncode
@lizarragarigoberto
@mahendrasoma
@manishbansal8843
@milanchawla
@nunna1989
@praveennadipi
@rajneeshsharma76
@ravulasureshgithub
@rghorpade80
@saaga
@saikiran232
@Sanyam101
@SaravanaKumarssp
@satya-deswal
@sauravsingh51
@SChandan18
@shaheershukur
@shaileshre
@smartysid
@snj07
@sorabh2008
@subir0071
@swachalit
@udayinfy
@vabhigit
@vairamuthu-shanmugaraj
@venkateshmadala
@virajput
@viveks03
@VM-HL-2017
@InfyMegha

Error while deploying IDP stack

Describe the bug
Below error is coming while deploying IDP stack.

Deploying IDP Stack
Creating network IDP_default
Creating service IDP_grafana
Creating service IDP_scheduler
Creating service IDP_config
Creating service IDP_jenkins
Creating service IDP_eureka
Creating service IDP_kafka
Creating service IDP_app
Creating service IDP_keycloak
Creating service IDP_oauth
Creating service IDP_services
Creating service IDP_dashboard
Creating service IDP_postgres
failed to create service IDP_postgres: Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded
root@MyOpenIDPHackMachine:~/openIDP# docker service list
ID                  NAME                MODE                REPLICAS            IMAGE                         PORTS
wbjg7pq1m1ds        IDP_app             replicated          1/1                 idp/idpapp:latest             *:8080->8080/tcp
m4wemh9lhr33        IDP_config          replicated          1/1                 idp/idpconfig:latest          *:8888->8888/tcp
2vwh1t7j7rdl        IDP_dashboard       replicated          1/1                 idp/idpdashboard:latest       *:8184->8184/tcp
lcz9jpzrg1fk        IDP_eureka          replicated          1/1                 idp/idpeureka:latest          *:8761->8761/tcp
thubqn4aznd0        IDP_grafana         replicated          1/1                 grafana/grafana:5.2.1         *:3000->3000/tcp
amgwu7vtcvrm        IDP_jenkins         replicated          1/1                 idp/jenkins:latest            *:8085->8080/tcp, *:50000->50000/tcp
shadjk5qrzzy        IDP_kafka           replicated          0/1                 wurstmeister/kafka:0.11.0.1
lci0qbs3ad64        IDP_keycloak        replicated          0/1                 jboss/keycloak:4.0.0.Beta1    *:8989->8080/tcp
hy1q7vfcgfn9        IDP_oauth           replicated          1/1                 idp/idpoauth:latest           *:8181->8181/tcp
kf2rncrossbm        IDP_postgres        replicated          0/1                 postgres:9.6.1                *:5432->5432/tcp
9ok80rgc7585        IDP_scheduler       replicated          1/1                 idp/idpscheduler:latest       *:8222->8222/tcp
pocv4utlds3o        IDP_services        replicated          0/1                 idp/idpservices:latest        *:8889->8889/tcp

Use openjdk:8-jre-alpine image instead of openjdk:8u151-jdk

Is your feature request related to a problem? Please describe.
Using openjdk:8u151-jdk as a base image for runtime is unnecessarily bloating up the size of docker containers.
Describe the solution you'd like
Use openjdk:8-jre-alpine as a base image.
Describe alternatives you've considered
NA
Additional context
NA

Unable to resolve class : EmailSender, IDPJobInfo, SCMStage, BuildStage, etc

I created a pipeline for Maven based Java Project. After triggering the pipeline it automatically created several jobs. But SCM job due to some issue resulted in unstable condition. While Pipeline Job results in failure. The build job was not build at all. When I opened the configure section of the pipeline job, It showed me several errors in the code. Most of them are unable to resolve the class error. I'm attaching the screenshot of my jenkins page.

jenkins-page

Kindly help me. Thanks!!

SKIP_TESTS parameter should be false by default

Describe the bug
In build.sh, SKIP_TESTS parameter should be false by default. Also, it's description should be corrected in wiki page here.
To Reproduce
NA

Expected behavior
SKIP_TESTS parameter should be false by default in build.sh.
Also, the wiki page should read like this.

SKIP_TESTS
Default Value: false
Allowed Values: true | false
Implications: If set true, will skip the test during build. If set false, then test cases will execute with build section.

Screenshots
NA
Desktop (please complete the following information):
NA

Additional context
NA

Getting 403 forbidden during login

Application was working fine yesterday. Today, I updated some java code and fired the build.sh. Now, during login, i am getting 403 forbidden error in developer's console during get request to below url:

http://ip.address:8889/idprest/userService/jobid1/getUserRolesPermissions
Further, i inspected the logs of oauth application, below are the logs.

2018-12-21 12:16:09.949  INFO 48 --- [nio-8181-exec-8] o.i.i.c.a.LoginBL                        : Auth Status : 0
2018-12-21 12:16:09.949  INFO 48 --- [nio-8181-exec-8] o.i.i.c.a.LoginBL                        : authenticated!!!!

Keycloak logs are as follows:

11:28:42,309 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-4) SQL Error: 0, SQLState: 23505
11:28:42,310 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-4) ERROR: duplicate key value violates unique constraint "uk_b71cjlbenv945rb6gcon438at"
  Detail: Key (realm_id, client_id)=(master, idp) already exists.
11:28:42,310 INFO  [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (default task-4) HHH000010: On release of batch it still contained JDBC statements
11:43:17,382 WARN  [org.keycloak.events] (default task-10) type=LOGIN_ERROR, realmId=master, clientId=account, userId=null, ipAddress=myipaddress, error=invalid_redirect_uri, redirect_uri=http://ip.address:8989/auth/realms/master/account/login-redirect?path=password
11:43:18,258 WARN  [org.keycloak.events] (default task-13) type=LOGIN_ERROR, realmId=master, clientId=account, userId=null, ipAddress=myipaddress, error=invalid_redirect_uri, redirect_uri=http://ip.address:8989/auth/realms/master/account/login-redirect?path=password

One other observations:

During full build, the posgres server data was not re-initialised. I feel like, this is the main reason behind keycloak exceptions.

Any idea how can i resolve this issue.

Log aggregation using ELK stack

Is your feature request related to a problem? Please describe.
As of now, I need to poke each container using below command to get the logs out.
docker logs -f <Container_id>
Describe the solution you'd like
It would have been awesome if some log aggregation was used like ELK stack.
Describe alternatives you've considered
NA
Additional context
NA

Unable to deploy war file

I'm trying to build and deploy maven project. Build Job is running properly after triggering the pipeline and it also creates .war file. The deployment Job also seems normal and executing successful but after opening the url in tomcat server 404 Page Not Found error use to come. I've tried to deploy the war file on the same tomcat server in which jenkins is running. Also I tried to deploy it onto another machine as well. But still no luck. When I went inside the tomcat manager folder(inside the container) where jenkins.war file is located, I'm unable to find my deployed war file. I'm attaching the log files of Deploy Job as well as Pipeline Job. Kindly help me!

Deploy_Job_Logs.txt
Piepline_logs.txt

jenkins_page

Thanks

Review State of Devops 2019 report & Prepare Roadmap

State of Devops 2019 Report:
https://services.google.com/fh/files/misc/state-of-devops-2019.pdf

Is your feature request related to a problem? Please describe.
This feature request is towards making openidp more relevant and usable.

Describe the solution you'd like
The link to the state of devops report for 2019 is attached. It would make sense to go through it and brainstorm on how the onward trajectory of openidp should look like.

Describe alternatives you've considered
Jenkins, Jenkins X, Circle CI, prow etc.

Additional context
Add any other context or screenshots about the feature request here.

Default login as per Installation steps not working

Describe the bug
We did all installation steps on Linux box but on step 7 below, not able to login on OpenIDP UI using following default user/password and blocked for our pilot.

Could someone pl. let me know user/password or any additional configuration needed, if support is still available?

Steps to reproduce the behavior:
Step 7(Installation): Once all the services are up and running, you can open a web browser (Chrome/IE) and check the URL by providing your system's host name followed by service name idpapp, if the installation and configuration has been completed without any error, IDP Login page appears. Eg: http://hostname/idpapp/

Error message: Username/password incorrect.
Username: idpadmin
Password: idpadmin@123

Expected behavior
Able to log in using default user/password.

Screenshots
Snapshot attached.

Deployment in Cloud

OpenIDP is deployed on docker, How we can directly deploy on GCP GKE or AWS ECS / EKS?

I have to go through the documentation but did not find a way to do it.

New technology support

As of now, OpenIDP does not support PHP, If we want to add that support then do you have any documentation for how we can extend OpenIDP with new technology support and how we will do further development?

How to configure my own Jenkins?

Sorry to bother you.

This seems too complicated to atleast install and try. The underlying arch uses Jenkins and other open surce tools. This seems to be an wrapper written over existing tools.

Just outa couriosity, wanted to check how this so called "product" from Infosys is better than existing CI-CD tools.

Clarify me few points:

  1. How to configure my Jenkins?
  2. How to use SCM without username or password? Shouldn't there be a way to connect with Access Token provided by Git?
  3. How to hook into a repo and automatically trigger a build when a change is committed?
  4. Why there were many dummy constants like "asadad", "adadada", "dahhfjfhs" in few JSON?
  5. Is there any "easy" way to atleast test this "product"?

What or How does openIDP different than existing tools?

If openIDP is better, then why TravisCI is used for showing build status? Why cant Infosys host its openIDP on some public server and build via same tool?

Too many open issues or questions unanswered!!!

Image 'paasmule / java-maven-git-alpine: 3.3-jdk-8-alpine' does not exist in the docker hub

Describe the bug
I clone the project and I run bash build.sh then:

`[root@localhost openIDP]# bash build.sh
Evaluating Hardware
Clearing Memory for Setup ...
CPU Load: 0.40
Memory Usage: 1/31GB (3.23%)
Disk Usage(Datafiles Directory): 601/1113760GB (1%)
Disk Usage (Docker Data Directory): 7433/51175GB (15%)
Your Setup Failed to meet recommended Memory Requirements (32GB). Continue ?

  1. Yes
  2. No
    #? 1
    Recommended Disk Space for Docker Images not available (15360MB). Continue ?
  3. Yes
  4. No
    #? 1
    Checking Pre-requisites
    Checking for Docker and Swarm Mode
    Docker Check: Passed
    Checking if interactive session
    Running in Interactive Session
    Checking for selected ports open status
    Starting port check .......
    Checking HTTP Server Port (80): Passed
    Checking CI Engine Client Port (50000): Passed
    Port Check: Passed
    Compiling openidp parent pom .......
    Unable to find image 'paasmule/java-maven-git-alpine:3.3-jdk-8-alpine' locally
    docker: Error response from daemon: pull access denied for paasmule/java-maven-git-alpine, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
    See 'docker run --help'.`

Image 'paasmule / java-maven-git-alpine: 3.3-jdk-8-alpine' does not exist in the docker hub

Expected behavior
image 'paasmule / java-maven-git-alpine: 3.3-jdk-8-alpine' exists in docker hub.

Screenshots
Screen Shot 2020-09-25 at 15 15 03

Desktop (please complete the following information):

  • OS: Centos7
  • Docker version 19.03.13, build 4484c46d9d

Running in VMWARE

Ubuntu 16.04 image running in VMWARE
12GB allocated
150 GB HD space

Checking Config Server failed at end of build.sh

I have tried all the bug issues already covered here as of 1-6-2019, including setting the build.sh to only do the healthcheck. Really need help getting this going in a VM for a POC.

Deployment Error- Port Check: Failed

Hi,
Getting Port Check Failed error while deployment. If anyone knows the resolution steps, plese help.

Checking Pre-requisites
Checking for Docker and Swarm Mode
Docker Check: Passed
Checking for selected ports open status
docker: invalid reference format: repository name must be lowercase.
See 'docker run --help'.
Port Check: Failed
Specified ports are either occupied or not open. Please refer above error for checking which one actually is not open

Unit tests are failing after making SKIP_TESTS=false

Describe the bug
Unit tests are failing after making SKIP_TESTS=false
To Reproduce
Steps to reproduce the behavior:

  1. Edit build.sh and update SKIP_TESTS=false.
  2. Execute sh build.sh

Getting below error.

Failed tests:
BuildIntervalTest.testBuildInterval0:75 expected:<buildInt[rvl12]> but was:<buildInt[erval]>
BuildInfoTest.testBuildInfo0:80 expected:<sonar_url[]> but was:<sonar_url[21]>
TransitionStatusTest.testupdateQueryValues:168 expected:<...artdate = '1970-01-2[6', enddate='1970-01-0]1', Remarks='remarks...> but was:<...artdate = '1970-01-2[5',enddate='1969-12-3]1', Remarks='remarks...>
ServiceIdentityTest.testOrgInfoNull:31 expected null, but was:<0>

Tests run: 357, Failures: 4, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] idpservices 1.0.0 .................................. SUCCESS [ 51.059 s]
[INFO] entities ........................................... FAILURE [ 43.355 s]
[INFO] utils .............................................. SKIPPED
[INFO] dataapi ............................................ SKIPPED
[INFO] businessapi ........................................ SKIPPED
[INFO] restapi ............................................ SKIPPED
[INFO] authservice 1.0.0 .................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Expected behavior
Unit tests should not fail. Build should be success.
Screenshots
NA
Desktop (please complete the following information):

  • OS: Ubuntu 16.x LTS
  • Browser NA

Additional context
NA

Getting permission errors while executing build.sh

Describe the bug
Getting permission denied error while executing build.sh. This is happening while docker container is getting executed while "Compiling Cloud config". Error thrown is as below:

Can not write to /root/.m2/copy_reference_file.log. Wrong Permissions?

To Reproduce
Steps to reproduce the behavior:

  1. Spin up a t2.micro instance on AWS with RHEL image.
  2. su root
  3. Install git and docker.
  4. Initialize docker swarm.
  5. Execute sh build.sh

Expected behavior
build.sh should get executed without any issues.

Screenshots
NA
Desktop (please complete the following information):

  • OS: RHEL
  • Browser: NA
  • Version 7

Additional context
NA

Health check fail while starting

Describe the bug
When I am running the buil.sh script, I am getting below error

Performing Health Checks .......... Checking Config Server: Failed

System configaration
I am using AWS Ubuntu 16.X version

Dashboard - Environment and Unified Applications views to be added

Is your feature request related to a problem? Please describe.
This is an enhancement request to the dashboard

Describe the solution you'd like
User would like to have visibility on deployment statuses on each environment in the application. And for each environment, user would want to see the versions of components deployed and their history.

User would like to have a cross applications view to compare applications against parameters such as build status, deployment status, code coverage, code quality and also identify applications that are in RED, AMBER, GREEN zones based on these status.

Describe alternatives you've considered
NA

Additional context
NA

Syntax error in build.sh ?

When I try to run build.sh I am having syntax error line 138. Can that be fixed ? Please update install instructions if necessary. Thank you.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.