fabric8-analytics / fabric8-analytics-common Goto Github PK

View Code? Open in Web Editor NEW

6.0 6.0 42.0 3.33 MB

fabric8-analytics core common development

License: Apache License 2.0

Shell 2.14% Gherkin 38.49% Python 55.06% CSS 0.18% HTML 4.05% Dockerfile 0.08%

e2e-tests

fabric8-analytics-common's People

Contributors

Stargazers

Watchers

fabric8-analytics-common's Issues

report license of stack given its components

Given the stack (a list of components/packages) -

Report the license of the stack.
If there's conflict amongst the licenses - report that.
Report the outlier components/packages with respect to license.

History graphs with common issues and dead code detection does not work in case all source files are perfectly ok

This is cause by different status message returned by check scripts. How to fix it: test for "special" status messages saying that every N source files are ok.

Add stack analysis test for pom.xml

We have a test for stack analysis of Python's requirements.txt. It would be great if we could add similar test for pom.xml. I can provide sample pom.xml file, if needed.

Bad HTTP code returned by the /flow-scheduling API endpoint when the job already exist

Given the job with ID=TEST2 already exist:

> POST /api/v1/jobs/flow-scheduling?job_id=TEST2&state=paused HTTP/1.1
> Host: stage_host
> User-Agent: curl/7.51.0
> Content-Type: application/json
> Accept: application/json
> auth-token: _______
> Content-Length: 439
> 
* upload completely sent off: 439 out of 439 bytes
< HTTP/1.1 401 UNAUTHORIZED
< Content-Type: application/json
< Content-Length: 66
< Set-Cookie: _____; path=/; HttpOnly
< 
{
  "error": "Job with the given job id 'TEST2' already exists"
}

The error message is ok, but the HTTP code is a bit misleading

Integration tests failure in scenario Check the Gemini API endpoint 'stacks-report/report

The following scenario Check the Gemini API endpoint 'stacks-report/report failed recently
(see https://ci.centos.org/job/devtools-e2e-fabric8-analytics/937/console)

  Scenario: Check the Gemini API endpoint 'stacks-report/report'                                                   # features/gemini.feature:210

    Given System is running                                                                                        # features/steps/common.py:29

    When I access the /api/v1/stacks-report/report endpoint of Gemini service for STAGE/monthly/201902.json report # features/steps/gemini.py:80

    Then I should get 200 status code                                                                              # features/steps/common.py:90

      Assertion Failed: assert 500 == 200

       +  where 500 = <Response [500]>.status_code

       +    where <Response [500]> = <behave.runner.Context object at 0x7fa079d15e10>.response



    Then I should get a valid report                                                                               # None

How to deploy to OpenShift ?

Deploying on DevCluster ( OpenShift ) can be done using deploy.sh script as described in README.md

However this process hasn't worked for me yet.

As shown in the image, many services fail to run for me.

An updated documentation is required that describes:

what are minimum configuration parameters that must be provided in env.sh file?
how to obtain these parameters?

Do we need the `gemini_fix` branch?

@yzainee do we really need the gemini_fix branch? If not, please delete it or ask me to do so.

https://github.com/fabric8-analytics/fabric8-analytics-common/branches

 Active branches

    gemini_fix
    Updated 3 months ago by Yusuf Zainee

Add the victimsdb-lib repository into the QA Dashboard

URL to the repository: https://github.com/fabric8-analytics/victimsdb-lib

Add the f8a-stack-report repository into the QA Dashboard

URL to the repository: https://github.com/fabric8-analytics/f8a-stacks-report

The link to API examples does not work

In the README.md, there's link to API examples:
https://github.com/fabric8-analytics/examples

Unfortunately the link is not correct (probably caused by renaming repositories?)

Proposal: Analytics for Multi-module Maven Applications

https://docs.google.com/a/redhat.com/document/d/1Ogkn-FGdn34WDHYDWut-5VVJH_7m8rCVPpaAx0iz680/edit?usp=sharing

data-model-importer API fails to run via docker compose

Start the services using Docker Compose as below:

$ ./docker-compose.sh up

Failure output:

data-model-importer_1   | [2017-05-29 06:59:29 +0000] [6] [INFO] Starting gunicorn 19.7.1
data-model-importer_1   | [2017-05-29 06:59:29 +0000] [6] [INFO] Listening at: http://0.0.0.0:9192 (6)
data-model-importer_1   | [2017-05-29 06:59:29 +0000] [6] [INFO] Using worker: sync
data-model-importer_1   | [2017-05-29 06:59:29 +0000] [11] [INFO] Booting worker with pid: 11
data-model-importer_1   | [2017-05-29 06:59:33 +0000] [11] [ERROR] Exception in worker process
data-model-importer_1   | Traceback (most recent call last):
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
data-model-importer_1   |     worker.init_process()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
data-model-importer_1   |     self.load_wsgi()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
data-model-importer_1   |     self.wsgi = self.app.wsgi()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
data-model-importer_1   |     self.callable = self.load()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
data-model-importer_1   |     return self.load_wsgiapp()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
data-model-importer_1   |     return util.import_app(self.app_uri)
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/util.py", line 352, in import_app
data-model-importer_1   |     __import__(module)
data-model-importer_1   |   File "/src/rest_api.py", line 23, in <module>
data-model-importer_1   |     if not BayesianGraph.is_index_created():
data-model-importer_1   |   File "/src/graph_manager.py", line 76, in is_index_created
data-model-importer_1   |     status, json_result = cls.execute(str_gremlin_dsl)
data-model-importer_1   |   File "/src/graph_manager.py", line 48, in execute
data-model-importer_1   |     data=json.dumps(payload))
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/requests/api.py", line 112, in post
data-model-importer_1   |     return request('post', url, data=data, json=json, **kwargs)
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/requests/api.py", line 58, in request
data-model-importer_1   |     return session.request(method=method, url=url, **kwargs)
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
data-model-importer_1   |     resp = self.send(prep, **send_kwargs)
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
data-model-importer_1   |     r = adapter.send(request, **kwargs)
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 502, in send
data-model-importer_1   |     raise ConnectionError(e, request=request)
data-model-importer_1   | ConnectionError: HTTPConnectionPool(host='bayesian-gremlin-http', port=8182): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x26b7510>: Failed to establish a new connection: [Errno 111] Connection refused',))

The issue is that data-model-importer depends on gremlin-http to be up. By the time gremlin-http starts, the importer already had failed trying to connect.

We could use some sort of a delay mechanism to delay starting of data-model-importer until gremlin-http has already started.

Stack analysis 2.0

Current stack analysis is driven with pre-cooked application stacks.

The proposal indicated below is going to understand the intent of the an OSIO user's application stack and derive recommendations based on this intent. Along with this, this proposal covers the 'additional information' for the user's application stack and all the generated recommendations. Goal of providing this additional information is to convince the OSIO user to act up on the generated recommendations.

This proposal covers showing usage based outliers and license analysis of the user's application stack.

https://docs.google.com/document/d/1VM5JLrM1DCd-hKdB2RAXrarT4sZhtliXV1wRgYLi7Tg/

Add the fabric8-analytics-rudra repository into the QA Dashboard

URL to the repository: https://github.com/fabric8-analytics/fabric8-analytics-rudra

Include Camel Fuse artefacts and their dependencies as known dependencies

Apache Camel is a well-known project, one of the biggest Apache project. it would nice to not have it listed as "Unknown dependencies"

Broken Links in the README.

Both the links under API section of the README, are broken.
https://github.com/fabric8-analytics/fabric8-analytics-common/blob/master/README.md#api

TODO: test for the fabric8-analytics-jobs issue #151

fabric8-analytics/fabric8-analytics-jobs#151

Support VSCode tests for VSCode version 1.32.0

The newest VSCode stable version is 1.32.0. Tests needs to be updated to support this version.

RFE: add check if docker service is running into ./runtests.sh

Include set of Camel sample projects for first set of data for analysis

there is a task to group all quickstarts of Camel/Fuse.
For now, it is just a Jira issue https://issues.jboss.org/browse/ENTESB-9131 but we can find several samples.

I copy-paste here for convenience:

Rename test step `should get a valid report` to be more precise

fabric8-analytics-common/integration-tests/features/gemini.feature

Line 207 in f1e3598

Then I should get a valid report

TODO: Test for fabric8-server issue #136

fabric8-analytics/fabric8-analytics-server#136

Document all test steps implemented in the common.py module

Currently just one test step is documented. It should be improved a bit :)

TODO: component analysis - check if correct metadata is written into the bayesian-core-package-data (unrelated to component version)

Entries to check in the bayesian-core-package-data bucket:

{ecosystem}/{package}.json

{ecosystem}/{package}/{analysis}.json for ["git_stats", "github_details", "keywords_tagging", "libraries_io"]

Add README.md to the `ui-tests` subproject

https://github.com/fabric8-analytics/fabric8-analytics-common/tree/master/ui-tests

TODO: Test for fabric8-server issue #135

fabric8-analytics/fabric8-analytics-server#135

Add the f8a-emr-deployment repository into the QA Dashboard

Next steps with deploy script

Automate deployment updates

Currently, every repository inside fabric8-analytics organization can set up automated builds form PRs.
Our aim with deploy script is to get the code from developer to dev cluster ASAP.
The idea is to setup local git hook that will automatically trigger one of the following:

Automated builds locally using docker build

Use docker build command to build images locally on a developers machine.
Either by using CI/CD scripts or manually invoking docker build command.

Automated build using openshift build capabilities

We can use each developers resources on dev cluster and invoke docker build there.
Defining build config and all configuration options are well documented in openshift documentation.

[integration-tests] Failing scenarios: features/component_search

No idea what's the cause, so filling issue here.

From https://ci.centos.org/job/devtools-f8a-master-deploy-e2e-test/814/console

Scenario: Check the component search functionality for existing component from the npm ecosystem  # features/component_search.feature:27

    Given System is running                                                                         # features/steps/common.py:38

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-jobs-bayesian-preview.b6ff.rh-idev.openshiftapps.com

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-gremlin-http-preview-b6ff-bayesian-preview.b6ff.rh-idev.openshiftapps.com

    Given Component search service is running                                                       # features/steps/component_analysis.py:12

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

    When I acquire the authorization token                                                          # features/steps/authorization.py:25

    Then I should get the proper authorization token                                                # features/steps/authorization.py:13

    When I search for component wisp with authorization token                                       # features/steps/component_analysis.py:43

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

    Then I should find the analysis for the component wisp from ecosystem npm                       # features/steps/component_analysis.py:146

      Traceback (most recent call last):

        File "/usr/lib/python3.4/site-packages/behave/model.py", line 1456, in run

          match.run(runner.context)

        File "/usr/lib/python3.4/site-packages/behave/model.py", line 1903, in run

          self.func(context, *args, **kwargs)

        File "/tests/features/steps/component_analysis.py", line 160, in check_component_analysis_existence

          format(component=component, ecosystem=ecosystem))

      Exception: Component wisp for ecosystem npm could not be found

      

      Captured logging:

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-jobs-bayesian-preview.b6ff.rh-idev.openshiftapps.com

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-gremlin-http-preview-b6ff-bayesian-preview.b6ff.rh-idev.openshiftapps.com

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io





  Scenario: Check the component search functionality for existing component from the pypi ecosystem  # features/component_search.feature:35

    Given System is running                                                                          # features/steps/common.py:38

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-jobs-bayesian-preview.b6ff.rh-idev.openshiftapps.com

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-gremlin-http-preview-b6ff-bayesian-preview.b6ff.rh-idev.openshiftapps.com

    Given Component search service is running                                                        # features/steps/component_analysis.py:12

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

    When I acquire the authorization token                                                           # features/steps/authorization.py:25

    Then I should get the proper authorization token                                                 # features/steps/authorization.py:13

    When I search for component clojure_py with authorization token                                  # features/steps/component_analysis.py:43

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

    Then I should find the analysis for the component clojure_py from ecosystem pypi                 # features/steps/component_analysis.py:146

      Traceback (most recent call last):

        File "/usr/lib/python3.4/site-packages/behave/model.py", line 1456, in run

          match.run(runner.context)

        File "/usr/lib/python3.4/site-packages/behave/model.py", line 1903, in run

          self.func(context, *args, **kwargs)

        File "/tests/features/steps/component_analysis.py", line 160, in check_component_analysis_existence

          format(component=component, ecosystem=ecosystem))

      Exception: Component clojure_py for ecosystem pypi could not be found

      

      Captured logging:

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-jobs-bayesian-preview.b6ff.rh-idev.openshiftapps.com

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-gremlin-http-preview-b6ff-bayesian-preview.b6ff.rh-idev.openshiftapps.com

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io





  Scenario: Check the component search functionality for existing component from the maven ecosystem  # features/component_search.feature:43

    Given System is running                                                                           # features/steps/common.py:38

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-jobs-bayesian-preview.b6ff.rh-idev.openshiftapps.com

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-gremlin-http-preview-b6ff-bayesian-preview.b6ff.rh-idev.openshiftapps.com

    Given Component search service is running                                                         # features/steps/component_analysis.py:12

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

    When I acquire the authorization token                                                            # features/steps/authorization.py:25

    Then I should get the proper authorization token                                                  # features/steps/authorization.py:13

    When I search for component vertx with authorization token                                        # features/steps/component_analysis.py:43

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

    Then I should find the analysis for the component io.vertx:vertx-core from ecosystem maven        # features/steps/component_analysis.py:146

      Traceback (most recent call last):

        File "/usr/lib/python3.4/site-packages/behave/model.py", line 1456, in run

          match.run(runner.context)

        File "/usr/lib/python3.4/site-packages/behave/model.py", line 1903, in run

          self.func(context, *args, **kwargs)

        File "/tests/features/steps/component_analysis.py", line 160, in check_component_analysis_existence

          format(component=component, ecosystem=ecosystem))

      Exception: Component io.vertx:vertx-core for ecosystem maven could not be found

      

      Captured logging:

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-jobs-bayesian-preview.b6ff.rh-idev.openshiftapps.com

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): bayesian-gremlin-http-preview-b6ff-bayesian-preview.b6ff.rh-idev.openshiftapps.com

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

      INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): recommender.api.prod-preview.openshift.io

RuntimeError - Failed to initialized graph schema

When running fabric8 analytics, via sh docker-compose.sh up, I encounter following error every time:

data-model-importer_1   | [2017-09-28 08:13:34 +0000] [11] [INFO] Worker exiting (pid: 11)
data-model-importer_1   | [2017-09-28 08:13:34 +0000] [6] [INFO] Shutting down: Master
data-model-importer_1   | [2017-09-28 08:13:34 +0000] [6] [INFO] Reason: Worker failed to boot.
data-model-importer_1   | [2017-09-28 08:13:38 +0000] [6] [INFO] Starting gunicorn 19.7.1
data-model-importer_1   | [2017-09-28 08:13:38 +0000] [6] [INFO] Listening at: http://0.0.0.0:9192 (6)
data-model-importer_1   | [2017-09-28 08:13:38 +0000] [6] [INFO] Using worker: gevent
data-model-importer_1   | [2017-09-28 08:13:38 +0000] [11] [INFO] Booting worker with pid: 11
data-model-importer_1   | [2017-09-28 08:13:39 +0000] [11] [ERROR] Exception in worker process
data-model-importer_1   | Traceback (most recent call last):
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
data-model-importer_1   |     worker.init_process()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
data-model-importer_1   |     super(GeventWorker, self).init_process()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
data-model-importer_1   |     self.load_wsgi()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
data-model-importer_1   |     self.wsgi = self.app.wsgi()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
data-model-importer_1   |     self.callable = self.load()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
data-model-importer_1   |     return self.load_wsgiapp()
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
data-model-importer_1   |     return util.import_app(self.app_uri)
data-model-importer_1   |   File "/usr/lib/python2.7/site-packages/gunicorn/util.py", line 352, in import_app
data-model-importer_1   |     __import__(module)
data-model-importer_1   |   File "/usr/lib64/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
data-model-importer_1   |     result = _import(*args, **kwargs)
data-model-importer_1   |   File "/src/rest_api.py", line 31, in <module>
data-model-importer_1   |     raise RuntimeError("Failed to initialized graph schema")
data-model-importer_1   | RuntimeError: Failed to initialized graph schema

Add the fabric8-analytics-rudra into the QA dashboard

Update registry URL for pushing images

This issue is opened to track progress of this Trello: https://trello.com/c/zLOXKyW1/712-switch-to-pushregistrydevshiftnet

Changes made so far:

Update 2

version can be specified on the CLI (via environment variable)
existence of the version needs to be checked in tests itself (in Given clause)
patterns needs to be splitted by version

component analysis throughput
stack analysis throughput for known components
stack analysis throughput for unknown components

Server API
Gemini API
Recommender API
License analaysis API

Random integration test failure

Feature: Smoke test # features/smoketest.feature:1
  Scenario: Check the /schemas entry point  # features/smoketest.feature:8
    Given System is running                 # features/steps/common.py:43
    When I access /api/v1/schemas/          # features/steps/common.py:266
    Then I should get 200 status code       # features/steps/common.py:601
      Assertion Failed: assert 500 == 200
       +  where 500 = <Response [500]>.status_code
       +    where <Response [500]> = <behave.runner.Context object at 0x7fb27c2b0278>.response
      Captured logging:
Failing scenarios:
  features/smoketest.feature:8  Check the /schemas entry point

Basic checks - start, stop
Test if analytics extension can be found
Test if analytics extension can be installed
Extension + Python project
Extension + Maven project
CVE display

(sub-issues to be opened for each point described above)

fabric8-analytics / fabric8-analytics-common Goto Github PK

fabric8-analytics-common's People

Contributors

Stargazers

Watchers

Forkers

fabric8-analytics-common's Issues

Automate deployment updates

Automated builds locally using docker build

Automated build using openshift build capabilities

Recommend Projects

Recommend Topics

Recommend Org