lf-engineering / easycla-api Goto Github PK

Summary

Create a new templates table and support service for the EasyCLA system.

Background

The EasyCLA system requires a standard template for Project Managers to select from in order to establish legal documents for their project CLA. This issue will provide a database table and API services to support creating, searching, updating and deleting the templates.

User Story

• As a Project Manager, I want to select the Apache Style CLA template for my project.
• As an LF Admin, I want to create a new CLA template for projects to use.
• As an LF Admin, I want to update an existing CLA template for projects to use.
• As an LF Admin, I want to delete an existing CLA template.

Tasks

• Develop a database schema to hold CLA templates. Columns should include a unique id/pk, date created, date modified, template name, template description, template version, a text blob to hold the template text encoded as an HTML fragment (see: https://github.com/communitybridge/easycla/blob/master/cla-backend-go/template/repository.go#L546), and a JSON blob to hold the template meta-data (see: https://github.com/communitybridge/easycla/blob/master/cla-backend-go/template/repository.go#L412)
• Develop and implement the Swagger specification to allow CRUD operations
• Develop and implement the repository/DB API to allow CRUD operations
• Develop and implement the service layer to support the API
• Develop and implement the request/response handler - connect the logic through the API
• Add the security layer - only users with the LF Admin role can create/update/delete. All users with the project manager role assigned can read for now.

Acceptance Criteria

• Demonstrate the working API in the DEV environment.

References

• EasyCLA v1 templates

Summary

This task is to create an API for Repository Management.

Background

A CLA Group is a logical organization under a single Foundation (also often called a Project) that groups a collection of related source code organizations (e.g. GitHub Organization or Gerrit Group) and/or repositories under a single CLA legal grouping/policy. This task will define, implement, validate and test the API services supporting creating, updating, viewing, searching and deleting of repositories under a CLA Group.

User Story

As a project manager, I want to add a set (one or more) of GitHub repositories to my project.
As a project manager, I want to delete a set (one or more) of GitHub repositories to my project.
As a project manager, I want to search the GitHub repositories by the repository name under my project.

Tasks

1. Define a swagger specification for creating one or more repositories under a given project.

   1. The swagger specification will accept the foundation/project id (SFID) and the CLA Group id, the repository type (GitHub, Gerrit, GitLab, etc.), and one or more repositories to add.

2. Define a swagger specification for deleting a set (one or more) repositories under a CLA Group

3. Define a swagger specification for searching or listing a list of repositories under a Project/CLA Group. Include pagination params as well: page_size/limit, offset, order_by as the list of parameters.

   1. Should be able to search all repositories under the Project/CLA Group
   2. Should be able to search/filter by repository type
   3. Should be able to search/filter by repository name
   4. Should be able to search/filter by repository organization

4. Define the repository database schema using dbmate. Fields might include:

   1. repository_id: pk GUID type v4 or something similar
   2. repository_type: a string representing the repository type (GitHub, Gerrit, GitLab, etc.)
   3. repository_name - the name of individual repository on GitHub/Gerrit/GitLab, etc.
   4. repository_organization_name - the name of the repository parent organization
   5. repository_url - the repository URL, such as github.com/communitybridge/easycla
   6. repository_enabled - a flag to indicate if this repository is enabled for this Project/CLA Group (may not need this) - default TRUE
   7. project_id - the SFDC project id
   8. cla_group_id - the CLA Group ID
   9. date_created - the date the repository was added

5. Create a repo implementation for the CRUD operations defined above

6. Create a service layer to support the REST operations defined above

7. Create an API handler to bin the Swagger specification to the implementation

8. Add the handler/service/repo to the setup to stitch it together

9. Create an audit event in the audit table when a repository is added, updated or deleted

10. test/validate CRUD operations

Acceptance Criteria

The "done" criteria when this feature or problem is resolved. Such as:

1. Unit Tests added and running in CI
2. Functional Tests updated to cover feature, if applicable
3. Demonstrate the set of capabilities to the product team while the code is
   running in the DEV environment.

References

1. EasyCLA v1 repository schema: https://confluence.linuxfoundation.org/display/PROD/EasyCLA+Version+1+Database+Schema+Walkthrough#EasyCLAVersion1DatabaseSchemaWalkthrough-Repositories
2. UX Mock Ups that leverage this service: https://wireframepro.mockflow.com/view/app-easycla#/page/D2f18f9c3dbacbc216821b379b96a0347

Summary

This task is to create an API for CLA Group Management

Background

A CLA Group is a logical organization under a single Foundation (also often called a Project) that groups a collection of related source code organizations (e.g. GitHub Organization or Gerrit Group) and/or repositories under a single CLA legal grouping/policy.

The CLA Group consists of:

1. CLA Group Name
2. A reference to the parent Foundation/Project (this will be a SFID)
3. A pair of CLA templates (CCLA and ICLA)
4. Flags to indicate which CLA templates are active/enabled for this CLA Group
5. A collection of one or more Project Managers (references to user objects)
6. References to the source code organizations/repositories

This task will define, implement, validate and test the API services supporting creating, updating, viewing, searching and deleting CLA Groups.

Tasks

1. Define a swagger specification for creating a CLA group
   1. The swagger specification will accept the foundation id (SFID), CLA Group Name, one or more Project managers (SFID's of users)
2. Define a swagger specification for updating a CLA group
3. Define a swagger specification for deleting a CLA group
4. Define a swagger specification for searching or listing a list of CLA groups by foundation id. Include pagination params as well: page_size/limit, offset, order_by as the list of CLA groups should be managed.
5. Define a swagger specification for searching or listing a list of CLA groups by a project manager id. Include pagination params as well: page_size/limit, offset, order_by as the list of CLA groups should be managed.
6. Define the CLA group database schema using dbmate. Fields might include:
   1. id: pk GUID type v4 or something similar
   2. CLA Group Name
   3. Parent foundation id
   4. date created
   5. date updates/last updated
   6. ccla enabled flag (boolean)
   7. icla enabled flag (boolean)
   8. CCLA and ICLA template definitions will be defined in another table
7. Create a repo implementation for the CRUD operations defined above
8. Create a service layer to support the REST operations defined above
9. Create an API handler to bin the Swagger specification to the implementation
10. Add the handler/service/repo to the setup to stitch it together
11. Create an audit event in the audit table when a CLA Group is added, updated or deleted (we will backfill the API user info later)
12. test/validate CRUD operations

This is the first piece of several to establish the Project/CLA Group/Template setup required for on-boarding a project.

User Story

1. As an LF admin, I want to create, update and delete CLA groups under a Foundation to support project setup and maintenance.
2. As a Project Manager, I want to view my CLA groups under a given Foundation.

Acceptance Criteria

• Unit tests for create/delete audit events
• Create function tests for audit events (see @dealako for CI/CD automation)
• Demonstration of audit service to product owners.

References

See existing cla-project-{STAGE} table concepts/schema already defined in DynamoDB for an example.

Summary

Develop the backend service to support the LFX EasyCLA Organization Dashboard (Member Manager View).

Background

The LFX EasyCLA Organization Dashboard front-end, request/response models are documented in the [EasyCLA v2 PRD](https://confluence.linuxfoundation.org/display/PROD/EasyCLA+v2+PRD#EasyCLAv2PRD-OrganizationDashboard(MemberManagerView). Enhance the existing API to incorporate the referenced path/endpoint.

User Story

As an LFX Community Bridge EasyCLA Organization Manager, I can view the list of Foundations affiliated with my Organization.

Tasks

• Create a orgs package in the main folder
• Define the Swagger API specification for the GET easycla-api.lfcla.com/v4/orgs/{salesForceOrganizationId} endpoint (see swagger/cla.yml)
• Provide a handler in the new orgs package to map the API spec to a specific handler function. For examples, see EasyCLA v1 go backend or Organization Service
• Add the handler to the server.go setup/configuration
• Create a service implementation - define the interface and implementation
• Create a mock repository implementation - define the interface and return sample data outlined in the PRD write-up. We will create an in-memory database and real RDS repository implementation later.
• Connect the server to the handler to the service to the mock repository.
• Test the service implementation locally via cURL, PostMan or similar. Results should closely match PRD examples.
• Follow the Fork/Clone/Branch/Commit/PR model to submit these changes

Acceptance Criteria

The "done" criteria when this feature or problem is resolved. Such as:

1. Unit Tests added and running in CI
2. Functional Tests updated to cover feature, if applicable
3. Demonstrate the set of capabilities to the product team while the code is
   running locally.

References

• EasyCLA v2 PRD: https://confluence.linuxfoundation.org/display/PROD/EasyCLA+v2+PRD

Summary

Implement the framework for the GitHub Activity Callback.

Background

In order to support EasyCLA management of checks on GitHub, a callback handler must be implemented. We will register a GitHub app within the LFX Community Bridge organization which will include a callback URL. This URL will be implemented in this API and support a number of events including open PR, close PR, reopen PR, sync and a few others. This handler will initially support parsing the messages to determine the event type.

User Story

As a developer, I want the EasyCLA bot to check the CLA database when a PR is opened.

Tasks

• Work with @dealako to register a new community bridge EasyCLA bot for the easycla-api v4 system.
• Define a swagger specification to support the API callback/webhook from GitHub
• Ensure the callback/webhook does not require LF API Gateway/ACS authentication (pass-through, similar to api-docs and health and status)
• Payload model should be a generic payload since the same endpoint will receive multiple message types (I think)
• Implement the API handler
• Application should fetch the webhook secret from SSM
• Application should validate the webhook secret in the handler
• Handler should invoke the service layer
• Service layer should parse/inspect the payload and branch based on the message type
• The initial implementation phase will stop at the parsing of the payload. Should be able to extract the PR:
  - commits
  - author/user info
  - SHAs from the commit
• Validate/demonstrate using a test repo

Acceptance Criteria

1. Demonstrate system receiving a GH webhook for open, close and reopen PR.
2. Demonstrate parsing of the message type
3. Demonstrate parsing of the payload to extract commit details, author info and SHAs.

References

1. Python implementation
2. GitHub API Documentation
3. Pull Request Documentation

Summary

Create an Audit Service for recording events in the EasyCLA system.

Background

Since the EasyCLA system processes permissions relating to legal CLA documents, a comprehensive audit log needs to record all access and permission events. Examples include add permission, delete permission, create a signature, delete a signature, add/update/delete whitelist entry.

Task

• Review the dbmate documentation
• Use dbmate new create_audit_table to construct a SQL script to provision and migrate the audit table definition. Include this in the PR.

Suggested columns (will refine):

• in the -- migrate:up define a database table schema.
• id: pk GUID type v4 or something similar
• event: string (we will have a set of enum values)
• user: string (id of the user/person who created the event - e.g. logged in user)
• project id: string (id of the SFID project)
• company id: string (id of the organization/company)
• event date time (date-time of the event)
• event_data: blob, contain event details, such as: {"github_whitelist": ["prasannamahajan", "ddeal"]}

Also:

• create a Swagger specification for SearchEvent , should include search by event type, by the user, project id, company/org, and by a time window. Support any combination of these parameters. Also, include pagination params as well: page_size/limit, offset, order_by as the list of events may/will be large.
• create a repo implementation for SearchEvent
• create a service layer for SearchEvent
• add the handler/service/repo to the setup/glue it together for SearchEvent.
• create a repo implementation for CreateEvent
• create a service layer for CreateEvent - note: this service will NOT have a public endpoint or definition in swagger.
• test/validate creating and retrieving audit values

We’ll use this for auditing everything in the backend/events, etc.

Acceptance Criteria

• Unit tests for create/delete audit events
• Create function tests for audit events (see @dealako for CI/CD automation)
• Demonstration of audit service to product owners

References

We have similar examples of a Search Endpoint used in the Platform Core Services, such as the Organization Service: https://github.com/LF-Engineering/organization-service/blob/master/swagger/organization-service.yaml#L61 (private repo).

Summary

Data migration from the existing DynamoDB database to the v2 RDS instance.

Background

The EasyCLA v2 system will need to migrate existing data from the old v1 system into the new database model. This will require a migration script that can read from the existing DynamoDB database from {DEV, STAGING, and PROD} and write to the Aurora RDS PostgreSQL database tables in {DEV, STAGING, and PROD}. We will initially test by exporting data from DynamoDB DEV to a local PostgreSQL instance on the developer's machine. We will work to migrate the CLA specific data including signatures, permissions, and other metadata. We will not transfer data that is currently duplicated in our 'system of record' database: Salesforce.

User Stories

1. As a developer, I want to leverage a script to export data from DynamoDB and import the data into AWS RDS PostgreSQL.
2. As a developer, I want to re-run the export multiple times without duplicating records in the RDS database.
3. As a developer, I want to specify the STAGE to export/import - e.g. LOCAL, DEV, STAGING, PROD environments. This will allow us to run a migration in each environment.
4. As a developer, I want to specify the RDS host, port, user, password and database details. Deployment of the RDS system will require separate connection information for each environment
5. As a developer, I want to see a report of what was exported/imported. A summary should be provided describing how many records were processed and any errors that occurred. The report should also include how long the process took.
6. As a developer, I want to run the migration in --dry-run mode which exercises the code but does not import the data.

Tasks

• Extend the existing tools folder scripts to include a python migration script
• Include any additional libraries in the requirements.txt file (e.g. dynamodb and postgresql drivers)
• provide the main routine with command-line options via the click library
• allow the user to specify the AWS region/credentials
• allow the user to specify the PostgreSQL database connection details
• provide a README with documentation on how to set up and run the tool with working examples (don't show credentials)
• Extract the data from the signatures table
• Extract the data from the companies table
• Extract the data from the projects table
• Extract the data from the users table
• Extract the data from the user-permissions table
• Extract the data from the repositories table
• Extract the data from the company-invitations table
• Extract the data from the github-orgs table
• Extract the data from the gerrit-instances table
• Import data into the relevant RDS tables (schema is TODO for some of the tables)
• Provide a migration report of what was exported/imported
• Private a report indicating how long the process took (this will give us a gauge on how long it will take for other environments).

Acceptance Criteria

The "done" criteria:

1. DEV data is migrated from v1 to v2 migrated.
2. Demonstrate the set of capabilities to the product team while the code is
   running in the DEV environment.

References

See @dealako for script setup examples and usage of existing v1 python models.

Summary

Build out the EasyCLA v2 API service to support the LFX CLA Member Project page.

Background

This task will build out the EasyCLA API v2 swagger, handler, service, and repository/db backend logic to support the LFX CLA Member Project page.

Tasks

[ ] Provide an API endpoint in the v2 swagger specification to support the LFX CLA Member Project page.
[ ] Develop the request/response/error models within the swagger specification - leverage the UI mock data and existing data models to outline the fields
[ ] Generate the code using go-swagger, review, iterate
[ ] Write a handler to process the incoming request - add to the server startup logic
[ ] Write a service layer, connect the service layer to the handler
[ ] Write a repository layer, connect the repository to the service layer
[ ] Test/validate the service
[ ] Write unit tests

Acceptance Criteria

When provided the project identifier, return the CLA Member project page details.  Demonstrate locally and in the DEV environment.

References

https://lfx.dev.platform.linuxfoundation.org/foundation/cncf/cla

Summary

We want to leverage the API Gateway Authentication headers to apply role-based access to the Easy CLA services.

Background

The EasyCLA v2 system will reside behind the LF platform API Gateway. The REST services we develop will be protected by the Gateway and the ACS service. We need to leverage this by restricting access based on user roles.

Tasks

• update the swagger specification to include authentication (see org service or project service as an example)
• include the LFX Kit library, see the org service example
• Omit the security protection for api-docs and the health and status
• Auth user object will be included in the API requests
• Bind the API security to the LFX auth model in the server configuration
• Rebuild and ensure the authUser as part of the API request (like request params)
  as a function parameter
• Ensure that only LF Admins can create CLA Groups as defined in issue #12
• Note: you will need to obtain an Authorization Bearer Token for API calls. We have two approaches for this:
  - Open and log into the LXF UI, from the console: https://lfx.dev.platform.linuxfoundation.org/home/dashboard

Example 1:

1. LF Admin logs in
2. Makes an API request to see the audit log
3. Code reviews the roles/access for the user, see that user is the LF Admin role
4. API allows all audit logs in the response

Example 2:

1. A company manager for Google logs in
2. Makes an API request to see the audit log
3. The code reviews the roles/access for the user, see that user is the company
   manager
4. we look up the user’s company id
5. pass the company ID to the backend to filter the audit log based on the company id

Acceptance Criteria

Demonstrate to the stakeholders.

References

See the LF Core Platform organization or the project service as an example.