chaoss / grimoirelab-sirmordred Goto Github PK

What's the goal of the category config in perceval?
Should we either use 'pull_request', 'issue', or both?

.mordred logs

perceval.errors.BackendError: ['pull_request', 'issue'] category not valid for GitHub

.mordred config

category = [pull_request, issue]

Most of the configuration file is about the dashboard itself. But a tiny part has to do with the infrastructure needed to produce it: now, that's mainly links to ElasticSearch and MariaDB. I think it would be better to have that part in a separate file. The first one could even be public, so that anyone seeing the dashboard could have access to it. The second one could be private, so that infrastructure details don't get compromised.

Hi
I checked kibiter new version release (https://github.com/grimoirelab/kibiter/releases)
So i tried to use this kibiter and elasticsearch 6.1.0 with modred.
But it looks like not working well now.

2018-01-26 11:30:07,219 - mordred.mordred - DEBUG -  Waiting for all threads to complete. This could take a while ..
2018-01-26 11:30:07,226 - mordred.task_manager - DEBUG - Executing task <mordred.task_panels.TaskPanels object at 0x7fc416ab3320>
2018-01-26 11:30:07,229 - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 127.0.0.1
2018-01-26 11:30:07,271 - urllib3.connectionpool - DEBUG - http://127.0.0.1:9200 "GET /.kibana/config/_search HTTP/1.1" 503 145
2018-01-26 11:30:07,272 - mordred.task_manager - ERROR - Exception in Task Manager 503 Server Error: Service Unavailable for url: http://127.0.0.1:9200/.kibana/config/_search
2018-01-26 11:30:07,275 - mordred.mordred - DEBUG - No exceptions in threads. Let's continue ..
2018-01-26 11:30:07,276 - mordred.mordred - DEBUG -  Task manager and all its tasks (threads) finished!
2018-01-26 11:30:07,276 - mordred.mordred - DEBUG - Tasks Manager starting ..
2018-01-26 11:30:07,276 - mordred.mordred - DEBUG - backend_tasks = []

Thank you

We get this error when we add the identities.yaml

2017-12-04 11:32:44,565 - mordred.task_identities - ERROR - [sortinghat] Error in command ['grimoirelab2sh', '-i', '/tmp/tmpf88twrz3', '-s', 'cord:manual', '-o', '/tmp/tmphoxmsc1b']
2017-12-04 11:32:44,568 - mordred.task_identities - ERROR - Can not generate the SH JSON file from GrimoireLab yaml file. Do the files exists? Is the API token right?

For bugzilla, is there a way to specify which product to collect information from in the project.json file?

Remove 'unified_releases' directory as this release method has been moved to https://github.com/grimoirelab/grimoirelab/tree/master/releases

It seems that this repository doesn't have any license and README file seems to be lost somewhere.

When launching mordred for CHAOSS, with 18.03-03 packages, I get the following error:

Elasticsearch aliases for pipermail: creating...
Elasticsearch aliases for pipermail: created!
Enrichment for pipermail: finished after 00:00:10 hours
2018-03-20 22:47:28,881 - mordred.task_manager - ERROR - Exception in Task Manager 'ElasticSearch' object has no attribute 'bulk_upload_sync'
Exception in thread pipermail:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_manager.py", line 92, in run
    task.execute()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 257, in execute
    self.__autorefresh()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 178, in __autorefresh
    enrich_backend.elastic.bulk_upload_sync(eitems, field_id)
AttributeError: 'ElasticSearch' object has no attribute 'bulk_upload_sync'

This happens for all the data sources. It seems that something is happening with the ElasticSearch object...

I've been looking at ways of improving the documentation for this project (i.e. #166) and just recently stumbled upon the README that exists under the docker/ subdirectory. Cool! Also noticed that there is auto-generated docs under the doc/ dir. Nice!

TL;DR I think we need to consolidate the docs a little bit, between the various doc locations in this repo, as well as the tutorial repo.

Between the docs under the subdirectories, and the README, I feel like we're getting a bit lost. It would be nice to converge on a single approach.

I personally am a fan of immediate, getting-started type stuff being documented in the README. Things like requirements (needed dependencies, accounts that need to be setup), a quickstart-style usage guide, and how to run the tests are all things I think every project should put in a README.

But then the README-in-a-subdir approach that the docker/ subdir has taken, vs. a single top-level docs/ directory, are approaches that I think are against each other. I think we should choose one.

As an aside, I like the fact that detailed config documentation is auto-generated and the docs live with the code. That is great. Perhaps we could even add some automation to the CI in this project to generate that code on every push to master and publish it to e.g. this project's github page? Perhaps long-term we could document mordred's API as well and publish an API reference? This kind of documentation feels very specific to this project, so I think this information living within the repo (or on this repo's GitHub Pages) makes sense.

It feels like having documentation that is specific around Docker for this project makes sense and is needed as this project is the keystone piece when it comes to the grimoirelab docker container. But does it make sense for the docker/ subdir to have its own README? Maybe. Or maybe some of that information belongs in the docker guide that is part of the grimoirelab tutorial? The main sections I see in the current docker/README are:

• The different configuration and setup files mordred uses, feels like it could belong in the top-level README if we could condense it down. OR, we could move that to a guide in the tutorial under Mordred? I think I prefer the latter option.
• I believe we already have tutorial documentation around the projects.json file. If so, we should simply link to that and avoid duplicating our documentation.
• im not super familiar how docker-compose.yml fits in. but is this still needed?
• again not clear to me what the advanced features are about but sounds like this could be better served as a tutorial guide

Let me know your thoughts! I know it's a lot to take in but I think these kinds of writeups from someone like me, a newcomer with no context around how a lot of this stuff works, can be helpful to smooth out the contribution process and therefore scale the development of GrimoireLab up by getting more committers and contributors involved! As always I am happy to send pull requests with ideas on how we could change stuff, or implementations of ideas that come out of this conversation.

Cheers,
Fil

The way dockerize applications log messages is via sending the logs via stdout and sterr. So, for Mordred it would be great if those logs are redirected that way directly from all.log instead so they could be available via docker logs command.

Now, we have the grimoirelab/grimoirelab for stuff common to all of GrimoireLab. This unified_releases file seems like a clear example of that, since it controls which commits in all repos are a part of a release. So, I propose to move the directory to be the releases directory in grimoirelab/grimoirelab.

If somebody else agrees, I volunteer to produce the corresponding pull requests (here and in grimoirelab/grimoirelab).

Hi, I am following the guide: https://chaoss.github.io/grimoirelab-tutorial/before-you-start/installing-grimoirelab.html

An error occures when typed (grimoirelab) $ pip3 install grimoire-mordred
It shows a long message. In the middle, it says
Failed building wheel for dulwich
and at the end:
Command "/home/emresulun93/venvs/grimoirelab/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip- install-hu_ieezp/dulwich/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n' );f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-6gh_7yc7/install-record.txt - -single-version-externally-managed --compile --install-headers /home/emresulun93/venvs/grimoirelab/include/site/p ython3.5/dulwich" failed with error code 1 in /tmp/pip-install-hu_ieezp/dulwich/

Python 3.6 is installed on the system and I tried to create the virtual environment with Python 3.5 instead of 3.6 but got the same error. Also, the same error occurred while installing the perceval.

The idea of this feature is to have a way to automate the refreshing of the identities file taken from a remote repository, like gitdm or mailmap.

Without having this, if a user wants this data in the identities database she has to:

• download the files
• run the scripts available in sortinghat/misc to convert them to Sorting Hat format
• place the files into an identities folder inside each project
• load them using Mordred and placing the files in the identities_file parameter

When running master/HEAD of all the tools, and launching mordred, after a while I see the following error:

2018-03-07 22:07:24,248 - grimoire_elk.arthur - ERROR - Problem executing study <bound method GitEnrich.enrich_areas_of_code of <grimoire_elk.elk.git.GitEnrich object at 0x7f68fb4a0940>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/arthur.py", line 479, in do_studies
    study(enrich_backend, no_incremental)
  File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/elk/git.py", line 748, in enrich_areas_of_code
    areas_of_code(git_enrich=enrich_backend, in_conn=in_conn, out_conn=out_conn)
  File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/elk/study_ceres_aoc.py", line 194, in areas_of_code
    ndocs = aoc.analyze()
  File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/elk/ceres_base.py", line 80, in analyze
    from_date = self._out.latest_date()
  File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/elk/ceres_base.py", line 264, in latest_date
    raise nfe
  File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/elk/ceres_base.py", line 252, in latest_date
    response = search.execute()
  File "/usr/local/lib/python3.5/dist-packages/elasticsearch_dsl/search.py", line 679, in execute
    **self._params
  File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/__init__.py", line 636, in search
    doc_type, '_search'), params=params, body=body)
  File "/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py", line 314, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/http_urllib3.py", line 163, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')

The dashboard seems to be generated correctly, though. Any idea of what's happening? Just in case it matters, I'm testing this with Elasticsearch 6.1.

Hello,

I tried to follow the instructions found at mordred/docker/readme.md; I followed everything exactly, but only switched the git repos I'm tracking in projects.json.

All of the containers launch successfully, but after about 5 minutes the mordred container crashes and it crashes again immediately if I try to start it with docker-compose. I checked the docker logs and found this at the end:

/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'namespaces' warnings.warn(msg) /home/bitergia/stage: line 79: bin/mordred: No such file or directory

It seems the bin/mordred executable is, in fact, missing.

Here is my docker-compose.yml config:

mordred:
image: bitergia/mordred:latest
volumes:
- devel/mordred/docker/conf/:/home/bitergia/conf/
- devel/logs/:/home/bitergia/logs/
links:
- mariadb
- elasticsearch

mariadb:
restart: "always"
image: mariadb:10.0
expose:
- "3306"
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=
- MYSQL_ALLOW_EMPTY_PASSWORD=yes

elasticsearch:
restart: "always"
image: elasticsearch:2.2
command: elasticsearch -Des.network.bind_host=0.0.0.0 -Dhttp.max_content_length=2000mb
ports:
- "9200:9200"

kibana:
image: bitergia/kibiter:4.4.1
environment:
- PROJECT_NAME=mytest
- NODE_OPTIONS=--max-old-space-size=200
links:
- elasticsearch
ports:
- "8081:5601"

kibana-ro:
image: bitergia/kibiter:4.4.1-public
environment:
- PROJECT_NAME=mytest
- NODE_OPTIONS=--max-old-space-size=200
links:
- elasticsearch
ports:
- "8091:5601"

In order to have a quicker response with the changes applied to production, we need the projects.json file to be read from a remote gitlab file which can be private. It is exactly the same behavior we have for the identities file.

[general]
..
gitlab_api_token = ***

[projects]
projects_file = https://URL/raw/master/projects.json

By including gitlab_api_token we avoid having the token repeated.

I tried to run mordred with the mordred config file in https://grimoirelab.gitbooks.io/tutorial/mordred/a-grimoirelab-dashboard-in-one-step.html, but I get errors like ('Wrong section param:', 'general', 'sleep'), ('Wrong section param:', 'general', 'kibana') and Error while consuming configuration: ('Wrong section param:', 'es_enrichment', 'studies'). It is here in this line:
https://github.com/chaoss/grimoirelab-mordred/blob/master/mordred/config.py#L623 where we compare the params in cfg file and the params_general, which does not have the fields kibana, sleep, etc that raises the error. So either the documentation is old or there is problem is the params_general variable.

when i ran the mordred from cmd line
$ mordred -c infra.cfg dashboard.cfg project.cfg override.cfg

getting the below error. Not clear what is the exact issue
2018-04-03 18:11:33,843 - mordred.mordred - INFO -
2018-04-03 18:11:33,844 - mordred.mordred - INFO - ----------------------------
2018-04-03 18:11:33,844 - mordred.mordred - INFO - Starting Mordred engine ...
2018-04-03 18:11:33,844 - mordred.mordred - INFO - - - - - - - - - - - - - - -
2018-04-03 18:11:33,850 - requests.packages.urllib3.connectionpool - WARNING - Retrying (Retry(total=20, connect=11, read=8, redirect=5, status=None)) after connection broken by 'NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f984b5f4780>: Failed to establish a new connection: [Errno 111] Connection refused',)': /

Recent versions of SortingHat allow us to merge accounts using different algorithms by data sources, this feature is not possible with Mordred yet. By adding this feature, we'll allow our system to get a list of github users and merge that into the identities database without side effects with usernames from other sources like IRC, where the username is not a good match indicator at all.

This is a proposal for the syntax:
matching = [email, username:github]

I would unify by 'email' and by 'username' only among identities where source = github.

I am getting this error with mordred 'bitergia/mordred:18.04-01'

2018-04-13 07:45:52,128 - grimoire_elk.elk - ERROR - Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/elk.py", line 583, in enrich_backend
total_ids = load_identities(ocean_backend, enrich_backend)
File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/elk.py", line 380, in load_identities
identities = enrich_backend.get_identities(item)
File "/usr/local/lib/python3.5/dist-packages/grimoire_elk/enriched/github.py", line 134, in get_identities
user = self.get_sh_identity(item[identity + "_data"])
KeyError: 'assignee_data'

Perceval Jira backend allow user to specify a project:

$ perceval jira 'https://tickets.puppetlabs.com' --project PUP

In this case, only issues for that project will be retrieved from the Jira API.

The current way of specifying this for Mordred is in mordred.cfg:

[jira]
raw_index = jira_test-raw
enriched_index = jira_test
project = PUP
max-issues = 10

But this is weird, and allows only for a single project to be filtered. This should in fact be in projects.json, and should be specified in a way that any list of projects from the Jira API endpoint are specified. For example, it could be like:

"jira": [
     "https://tickets.puppetlabs.com --project project1",
     "https://tickets.puppetlabs.com --project project2",
     "https://tickets.puppetlabs.com --project project3",
     ...
],

See some more details in a recent question, #121.

Hi!

I have recently stood up a grimoirelab deployment via docker. It seems to be up! I can access it on a URL! Yay!

However, the first screen I see when loading it up in the browser is this:

I wonder what I did wrong? I'm not even sure how to start debugging this. Would someone be willing to help me sort this out?

My relevant configs:

-bash-4.2$ cat credentials.cfg
[github]
api-token = supersecret
enterprise-url = https://git.corp.adobe.com

[projects]
projects_file = /projects.json
-bash-4.2$ cat projects.json
{
    "opensource_submission_process": {
        "github": [
            "https://git.corp.adobe.com/OpenSourceAdvisoryBoard/opensource_submission_process"
        ],
        "git": [
            "https://git.corp.adobe.com/OpenSourceAdvisoryBoard/opensource_submission_process"
        ]
    }
}

Here's how the docker container was stood up:

docker run --name grim -d -p 127.0.0.1:9200:9200 -p 127.0.0.1:5601:5601 -v $(pwd)/logs:/logs -v $(pwd)/es-data:/var/lib/elasticsearch -v $(pwd)/credentials.cfg:/override.cfg -v $(pwd)/projects.json:/projects.json -t grimoirelab/full

Thanks in advance for any help!

Cheers,
Filip Maj

I have updated the grimoire-mordred Python package and now, when I run mordred -c mordred.cfg, I get this error message:

$ mordred -c mordred-ubuconeu.cfg 
Traceback (most recent call last):
  File "/home/jsmanrique/grimoirelab/venv/bin/mordred", line 39, in <module>
    from mordred.mordred import Mordred
  File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/mordred/mordred.py", line 52, in <module>
    from mordred.task_report import TaskReport
  File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/mordred/task_report.py", line 35, in <module>
    from manuscripts.report import Report
ImportError: No module named 'manuscripts'

Why?

And if manuscripts Python package is needed, is it not installed through pip install --upgrade grimoire-mordred ??

I have set up my own Grimoirelab instance via the handy Docker container.

I only have a couple of GitHub data sources defined in my projects.json.

When I load Grimoirelab I see the following warnings at the top of the page:

Can I do anything with my configuration to avoid this warning?

Thanks for any info! I am also happy to send a PR to fix stuff up for the tutorial/docs to reflect what I learn :)

When #80 is merged, we will have two separate kinds of information in the panels section: information about the dashboard (such as the default index pattern, or the default time period for the dashboard), and about the infrastructure supporting it (Kibiter url, for example).

I propose to split that into two sections: panels and kibiter. The first one about stuff which is directly related to the dashboard, and the second one stuff which would reflect details of the infrastructure, related to Kibiter, that could change in different deployments of the same dashboard, not affecting the dashboard itself.

For now, the new parameters introduced in #80 would come under kibiter.

My projects.json:

{
    "opensource_submission_process": {
        "github": [
            "https://REDACTED/OpenSourceAdvisoryBoard/opensource_submission_process"
        ],
        "git": [
            "https://REDACTED/OpenSourceAdvisoryBoard/opensource_submission_process"
        ]
    },
    "react-spectrum": {
        "github": [
            "https://REDACTED/React/React-spectrum"
        ],
        "git": [
            "https://REDACTED/React/React-spectrum"
        ]
    }
}

The GitHub data shows up great on the dashboard. The git data gives me a bunch of warnings along the lines of "could not locate that index-pattern-field":

The URLs from my projects.json all point to the same GitHub Enterprise instance my company uses.

Thanks for any help!

Using the release elasticgirl.17 I've seen that we are doing the wrong time calculation when the enrichment lasts more than a day.

See these two lines:

2017-10-26 11:39:37,216 - mordred.task_enrich - INFO - [git] enrichment starts                 
..
2017-10-27 18:26:58,117 - mordred.task_enrich - INFO - [git] enrichment finished in 06:47:20 

Complete log below:

2017-10-26 11:39:35,183 - mordred.mordred - INFO -                                             
2017-10-26 11:39:35,183 - mordred.mordred - INFO - ----------------------------                
2017-10-26 11:39:35,183 - mordred.mordred - INFO - Starting Mordred engine ...                 
2017-10-26 11:39:35,183 - mordred.mordred - INFO - - - - - - - - - - - - - - -                 
2017-10-26 11:39:36,215 - mordred.mordred - INFO - [<class 'mordred.task_projects.TaskProjects'>] will be executed on Fri, 27 Oct 2017 11:39:36                                               
2017-10-26 11:39:37,216 - mordred.task_enrich - INFO - [git] enrichment starts                 
2017-10-27 11:39:37,317 - mordred.task_projects - INFO - Reading projects data from  /home/bitergia/conf/sources/projects.json                                                                
2017-10-27 18:26:58,117 - mordred.task_enrich - INFO - [git] enrichment finished in 06:47:20   
2017-10-27 18:26:59,182 - mordred.task_enrich - INFO - [git] enrichment starts 

Perceval requires that both a JIRA URL as well as a project key is provided:

$ perceval jira 'https://tickets.puppetlabs.com' --project PUP

The projects file documentation is lacking on this topic - the only examples given are for backends that take a single URL as a parameter. What about backends that require multiple parameters? How do I specify that in the projects file?

When launching master/HEAD of mordred, with master/HEAD of GrimoireELK:

Starting Mordred to build a GrimoireLab dashboard
This will usually take a while...
Traceback (most recent call last):
  File "/usr/local/bin/mordred", line 39, in <module>
    from mordred.mordred import Mordred
  File "/usr/local/lib/python3.5/dist-packages/mordred/mordred.py", line 47, in <module>
    from mordred.task_enrich import TaskEnrich
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 33, in <module>
    from grimoire_elk.elk.elastic import ElasticSearch
ImportError: No module named 'grimoire_elk.elk.elastic'
Failed to start Mordred: 1

I look in GrimoireELK and it seems grimoire_elk/elk/elastic.py is not there anymore...

Currently, Mordred tries always to set up a sidebar menu for Kibiter / Kibana.

I don't need that menu, so wouldn't it be nice having that as an optional phase in the Mordred set up file?

After configuring and setting up the dashboard using mordred, my dashboard is unable to retrieve new data from my git repositories.

The approach will be to add new sections per studies to mordred config, and that all params included in those sections, will be passed as a kwargs dict to the study. But all of these must be done in mordred, so opening this ticket in mordred.

Using master version of GrimoireELK+Mordred I've seen this log message:

2017-11-14 13:33:48,247 - mordred.task_enrich - INFO - Refreshing identities fields in enriched index

It does not add any useful information.

So it should contain:

• the name of the index it is refreshing
• the number of identities to be refreshed (wishlist)

I want to use arthur in modred~ when i check the code, i guess below code will activate arthur feature.
But it is not working. Could you let me know usage guide?

[es_collection]
url = http://localhost:9200
user =
password =
arthur = true
arthur_url = http://127.0.0.1:8080
redis_url = http://127.0.0.1:6379

Hey I noticed that studies are now configured in each backend section, but I can't find documentation for it.
I noticed that git backend accept following studies studies = [enrich_demography, enrich_onion,enrich_areas_of_code] but what about other backend (github,...)?
Is mordred taking care of onion and areas of code or do I have to follow this doc doc to create aliases?

[git]
studies = [enrich_demography, enrich_onion,enrich_areas_of_code]

Trying to run some Meetups groups analysis with Mordred, I get exceptions related with "forbidden items" from the Meetup API (depending on group configuration).

These issues cause errors in the threads running in Mordred, and that some following steps, like enriched index creation, fail.

Could Mordred handle that? Or is it a Grimoire ELK issue?

When using mordred and pointing the git urls to Enterprise Github
it is always trying to point to Github only.

How can i point to Enterprise Github?

which api token should be used and also are the orgs_file and identities file needs to be created?

The current behavior of Mordred does not update an existing panel even if the flag panels is set to true. This is confusing because we have fresher available panels that our users are not seeing unless they wipe out the .kibana index.

Mordred should allow user to:

• get fresh panels when they are released
• get the menu updated (as the menu is sometimes manually modified, overwriting it could be a pain the neck if so we must allow users to disable this)

My proposal is to decouple the current panels flag into panels and menu. When they are true the content will be replace based on the standard product, when they are false nor standard panels and menu will be updated.

👋 Hey all, been trying to get your example of Mordred up and running from the example folder. I've made all the directory calls relative to the path and included a copy of the docker-compose file on my branch here.

Once I run it (docker-compose up -d, I get into Kibana but do not have any items in ES as visible here:

I believe it is due to an error with Mordred running to completion. I can see it fails in this gist

Is it possible to use backends that do not use a url like meetup or stackoverflow?

The following configuration for studies raises an error when running Mordred (see below):

studies = []

[enrich_demography]

[enrich_areas_of_code]

The error is raised after retrieval of git data is done:

Enrichment for git: starting...
2018-06-04 15:29:09,565 - mordred.task_enrich - ERROR - Missing config for study :
2018-06-04 15:29:09,565 - mordred.task_manager - ERROR - Exception in Task Manager Missing config for study :
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_manager.py", line 92, in run
    task.execute()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 291, in execute
    raise e
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 281, in execute
    self.__enrich_items()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 126, in __enrich_items
    studies_args = self.__load_studies()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 73, in __load_studies
    raise DataEnrichmentError(msg)
mordred.error.DataEnrichmentError: Missing config for study :
Exception in thread git:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_manager.py", line 92, in run
    task.execute()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 291, in execute
    raise e
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 281, in execute
    self.__enrich_items()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 126, in __enrich_items
    studies_args = self.__load_studies()
  File "/usr/local/lib/python3.5/dist-packages/mordred/task_enrich.py", line 73, in __load_studies
    raise DataEnrichmentError(msg)
mordred.error.DataEnrichmentError: Missing config for study :

This with 18.05-03.

Maybe this could be related to #155?

Apparently, when specifying no studies (eg, studies = []), the corresponding section for specifying the study arguments is still needed. My impression is that only sections for active studies should be in the config file. That is, both the following configurations should be valid:

studies = []

studies = [enrich_demography, enrich_areas_of_code]

[enrich_demography]

[enrich_areas_of_code]

Right now, apparently only the second one is valid.

SirMordred does seem to access SSL Elasticsearch without verifying certificates, which is useful for, for example, accessing Elasticsearch with a self-signed SSL certificate. But this is not the case for studies, which seem to always verify. We need to either:

• make not-verifying certs the behavior for studies, as is now for the rest of the interactions with Elasticsearch, or
• have an option in the corresponding sections in mordred.cfg files (sections on raw and enriched indexes) to declare that access to Elasticsearch is verifying / non-verifying.

I think we could start with the former, and let this or another issue for the latter, which is maybe a bit more complex.

I am deploying a new demo of GrimoireLab and I would like to test affiliations information management.

As first step, I've seen there are dedicated params in mordred.cfg about this. I've been reading the docs. But, it's not clear for me if which params I need to touch. Let me explain why.

If I've understood right, affiliate, load_orgs, and orgs_file, are the main params. And I plan to use this orgs.json file as data source:

And this is what the doc says:

affiliate (bool: True): Affiliate identities to organizations (Required)
load_orgs (bool: False):
orgs_file (str: None): File path with the organizations to be loaded in Sortinghat

So, without any further docs (I've not found anything else about these params anywhere), I think the values should be:

affiliate: True
load_orgs: True
orgs_file: orgs.json

But, then, I check current examples provided in the main GrimoireLab repository (1, 2) and I see this:

affiliate: True
load_orgs: false
orgs_file: orgs.json

Could someone clarify why or how is the right way to work?

My objective is that at least, people committing using a corporate account email domain are rightly affiliated.

Thanks!

If I've understood right, there should be a default menu.yml file as part of grimoire-mordred Python package. But it seems there is not, and mordred command still requires a menu.yml in the folder where it's executed from:

Exception in thread Global tasks:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/mordred/task_manager.py", line 76, in run
    task = tc(self.config)
  File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/mordred/task_panels.py", line 61, in __init__
    with open(TaskPanelsMenu.MENU_YAML, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'menu.yaml'

Wouldn't it be nice if mordred would use the existing menu.yml if no other is provided?

From chaoss/grimoirelab-elk#349

In the original issue you can find more details on installed packages and tests done to try to reproduce it from https://github.com/chaoss/grimoirelab-elk.

@jsmanrique wrote:

I am getting the following error running an areas of code analysis:

Executing for git the studies ['enrich_demography', 'enrich_areas_of_code']
/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/pandas/core/ops.py:816: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
 result = getattr(x, name)(y)
2018-05-11 07:46:03,905 - grimoire_elk.elk - ERROR - Problem executing study <bound method GitEnrich.enrich_areas_of_code of <grimoire_elk.enriched.git.GitEnrich object at 0x7f0f6ccf9c18>>
Traceback (most recent call last):
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/elk.py", line 460, in do_studies
   study(enrich_backend, no_incremental)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/git.py", line 787, in enrich_areas_of_code
   areas_of_code(git_enrich=enrich_backend, in_conn=in_conn, out_conn=out_conn)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/study_ceres_aoc.py", line 194, in areas_of_code
   ndocs = aoc.analyze()
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/ceres_base.py", line 93, in analyze
   process_results = self.process(item_block)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/study_ceres_aoc.py", line 158, in process
   events_df = data_filtered.filter_(["filepath"], "-")
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/cereslib/dfutils/filter.py", line 71, in filter_
   self.data = self.data[self.data[column] != value]
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/pandas/core/ops.py", line 879, in wrapper
   res = na_op(values, other)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/pandas/core/ops.py", line 818, in na_op
   raise TypeError("invalid type comparison")
TypeError: invalid type comparison

Config files:

• projects.json.txt

My mordred.cfg file:

[general]
short_name = Project_Name
update = false
min_update_delay = 10
debug = true
logs_dir = /tmp/logs

[projects]
projects_file = projects.json

[es_collection]
url = http://localhost:9200
user =
password =

[es_enrichment]
url = http://127.0.0.1:9200
user =
password =
autorefresh = true

[sortinghat]
host = localhost
user = root
password = *******************
database = shdb
load_orgs = true
orgs_file = orgs.json
unaffiliated_group = Unknown
affiliate = true
autoprofile = [github,git]
matching = [email-name]
sleep_for = 0
#bots_names = []

[panels]
kibiter_time_from= "now-90d"
kibiter_default_index= "git"

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
raw_index = git-gathered-raw
enriched_index = git-gathered
studies = [enrich_demography, enrich_areas_of_code]

[github]
raw_index = github-gathered-raw
enriched_index = github-gathered
api-token = **************************************************
sleep-for-rate = true
sleep-time = 300
no-archive = true

[*pipermail]
raw_index = mls-gathered-raw
enriched_index = mls-gathered

[*meetup]
raw_index = meetup-gathered-raw
enriched_index = meetup-gathered
api-token = *****************************************
sleep-for-rate = true
no-archive = true

[*stackexchange]
raw_index = stackexchange-gathered-raw
enriched_index = stackexchange-gathered
api-token = *************************************

It would be nice to know how to get set up in this repository, as well as how to run the tests.

My assumptions / attempts so far have included:

1. Get a virtualenv, install/update pip, setuptools and wheel as per https://grimoirelab.gitbooks.io/tutorial/content/before-you-start/installing-grimoirelab.html
2. Run python3 setup.py install
3. Run pip install -r requirements.txt - maybe? Maybe the last command implicitly installs the reqs?

But how do I run the tests? I tried python3 tests/run_tests.py, but that failed completely. I tried to run it from the tests/ directory and got farther, but it looks like I'm missing a ton of dependencies:

src/grimoirelab-mordred/tests on master [?] via grimoirelab
➔ ./run_tests.py
.EEEEEE
======================================================================
ERROR: test_task (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_task
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/maj/src/grimoirelab-mordred/tests/test_task.py", line 33, in <module>
    from mordred.task import Task
  File "../mordred/task.py", line 27, in <module>
    from grimoire_elk.elk import get_ocean_backend
ImportError: cannot import name 'get_ocean_backend'


======================================================================
ERROR: test_task_collection (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_task_collection
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/maj/src/grimoirelab-mordred/tests/test_task_collection.py", line 35, in <module>
    from mordred.task_collection import TaskRawDataCollection
  File "../mordred/task_collection.py", line 42, in <module>
    from grimoire_elk.elk import feed_backend
ImportError: cannot import name 'feed_backend'


======================================================================
ERROR: test_task_enrich (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_task_enrich
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/maj/src/grimoirelab-mordred/tests/test_task_enrich.py", line 36, in <module>
    from mordred.task_projects import TaskProjects
  File "../mordred/task_projects.py", line 36, in <module>
    from mordred.task import Task
  File "../mordred/task.py", line 27, in <module>
    from grimoire_elk.elk import get_ocean_backend
ImportError: cannot import name 'get_ocean_backend'


======================================================================
ERROR: test_task_identities (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_task_identities
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/maj/src/grimoirelab-mordred/tests/test_task_identities.py", line 26, in <module>
    import httpretty
ModuleNotFoundError: No module named 'httpretty'


======================================================================
ERROR: test_task_panels (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_task_panels
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/maj/src/grimoirelab-mordred/tests/test_task_panels.py", line 33, in <module>
    from mordred.task_panels import TaskPanels
  File "../mordred/task_panels.py", line 35, in <module>
    from mordred.task import Task
  File "../mordred/task.py", line 27, in <module>
    from grimoire_elk.elk import get_ocean_backend
ImportError: cannot import name 'get_ocean_backend'


======================================================================
ERROR: test_task_projects (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_task_projects
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/maj/src/grimoirelab-mordred/tests/test_task_projects.py", line 27, in <module>
    import httpretty
ModuleNotFoundError: No module named 'httpretty'


----------------------------------------------------------------------
Ran 7 tests in 1.630s

FAILED (errors=6)

If someone could help me with this, I will happily issue a pull request to update the README.md.

Thanks!

In order to offer a better interaction with the user, the system must keep updated the enriched indexes based on the projects.json file. The update process is not 100% done as data sources are never removed from the enriched indexes even the they are deleted from the file.

These feature request is aimed to make Mordred smart enough to remove repositories when they are deleted from the project.json file.

Hi, I want to know if there any way to check the git commit is pushed to Gerrit. I think it as follows:

1. for each git commit id, query it in gerrit index;
2. if the commit id is found, we can label it reviewed;
3. Or, we take it as non-reviewed;
   Is there any way to get it?