unee-t / bugzilla-customisation Goto Github PK

We have serveral "types" of environments:
1- Prod: the production environment accessible at dashboard.unee-t.com
2- Main Dev: the Main DEV environment for the BZFE accessible at dev.dashboard.unee-t.com
3- Local DEV: an environment that a developer can create on a local machine (or on an dedicated AWS instance).
4- Demo Environment (does not exist yet) where we can showcase what unee-t does today with demo cases and demo users.
5- Test Environment (does not exist yet) where we can show and test the newest features that are about to be rolled out with demo cases and demo users.

We need to be able to differentiate these environment in one glance to avoid silly mistakes.

We will do that by creating different color scheme for each of these 5 types of environment.
This will be done by:
1- creating a slightly different version of the skin for each of the environment
2- assign this skin (initially manually in the BZ back end and then automatically as we deploy the environment) to the correct environment.

This issue track our progresses to achieve that.

The MEFE needs to communicate with the BZFE.
In order to do this, we use the BZ API mechanism.

Each user for each environment needs to have an API key that is properly configured in the MEFE and the BZFE to make sure that the 2 system can communicate.

The first user that we need to worry about it the Admin user (user id 1 in the BZFE).

As of today, the API key for each users are recorded in the table 'user_api_key' in the BZ installation.

We need to make sure that each time we deploy in a new environment (demo, dev, prod, etc...) the correct API is used for the relevant user.

This issue tracks how we will do that.

Context:
The current SQL primer in the Docker container only creates 1 admin user.
This is not a problem for now as we know that we need to run a specific SQL script to create demo users for each environment. The script to create demo user is currently available in https://github.com/unee-t/bz-database/tree/master/demo-test%20environment.
We DO NOT want to store the Administrator API key there for any sensitive (i.e non local) environment as this is a public repo.
The only API key that should be visible there is the generic API key that a dev will use in a "local" installation.

Previous prime has other usernames to test as documented upon:

https://github.com/unee-t/bugzilla-customisation#login-info-set-by-sqldemosqlgz

Now there only seems to be a [email protected] account?

If you want just one Administrator to begin with, can you perhaps document how to create users?

Idea of the demo.sql.gz in my mind is to be able to quickly search for bugs, login as a user, ensure permissions are correct. Starting from a blank slate with just an administrator account would be inconvenient and time waste from a testing POV!

https://s.natalian.org/2017-11-10/1510291332_2558x1406.png

I think this API KEY must have been reset when we updated the SQL.

Just generated a new one at https://dashboard.unee-t.com/userprefs.cgi & updated the key at https://ap-southeast-1.console.aws.amazon.com/ec2/v2/home?region=ap-southeast-1#Parameters:sort=Name

062175f
This commit does not seem to work as intended: the parameters are NOT what is expected.

I've created a branch (https://github.com/unee-t/bugzilla-customisation/tree/db_configuration_for_bz) that tries to fix that also but the parameters are still not changed to what we need even after the
make down
make up
sequence in my "local/aws" machine...
@kaihendry any idea what might be wrong?

When the BZ back end sends an email to a user notifying of a change in a case, the URL is the URL for the dashboard.

It should be the URL for the MEFE instead.

We need to customize the BZ email so that the URL is the URL for the MEFE and NOT the url for the BZFE.

Limit: in order for the URL to work, the user would have to be logged in to the MEFE as himself.
This is NOT a magic link!

This is acceptable in the MVP.

After adding bug group and also attach_data the performances of Unee-T becomes agonizingly slow to the point where we are hitting timeout errors in the BZFE.

We need to address this.
One solution is to add more indexes in the database to speed up some key queries.

This issue tracks our progresses to remove that performance bottleneck and make sure perfs are acceptable with a large number of cases, users and unit.

In this event where to load balancer can't reach a healthy host, we could perhaps have a more user friendly fail whale.

Feature request for the bridge BZFE/MEFE
Once a user has been invited to a claim via a "magic link" it should be possible to send the same link again.

The scenario is:

• We need an input from a user is a role (the occupant needs to give his/her availability so we can enter the unit).
• Invitation is sent once.
• After a certain period of time (ex: 1 week) still no reply from the occupant.
• We send a reminder to the occupant so that we can get the information we need.
  This reminded should also include the magic link.

This should not be a problem IMO as the BZFE will automatically send emails to all the people in CC on the case each time the case is changed...

In order to avoid API errors in the MEFE we need to temporarily change the default configuration of the BZ installation.

This has been done via the BZ back end interface for the DEV, DEMO and PROD environments.
We need to make sure that the docker script uses the same parameters too.

These parameters are in the respective parameters files located here.

Values should be:

• line 27: "commentonchange_resolution" : "0",
• line 28: "commentonduplicate" : "0",
• line 63: "noresolveonopenblockers" : "0",

url: https://demo.dashboard.unee-t.com/admin.cgi

I seem to recall that we had a similar problem in the past...

Relates to the UneeTBridge extension.
As mentioned on a8f6cf0

The logs have no information to help debug this.

BMO's @dylanwh recommends NYTProf and has a tutorial here: https://www.youtube.com/watch?v=1hrdVxI0uFM&t=1s

• Gist of how to enable NYTProf

Nonetheless we don't quite know how to reproduce the 5xx error on production.

db_1 | 2018-04-09 1:12:02 140000618817408 [ERROR] InnoDB: The Auto-extending innodb_system data file './ibdata1' is of a different size 0 pages than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages! db_1 | 2018-04-09 1:12:02 140000618817408 [ERROR] InnoDB: Plugin initialization aborted with error Generic error db_1 | 2018-04-09 1:12:02 140000618817408 [Note] InnoDB: Starting shutdown... db_1 | 2018-04-09 1:12:03 140000618817408 [ERROR] Plugin 'InnoDB' init function returned error. db_1 | 2018-04-09 1:12:03 140000618817408 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. db_1 | 2018-04-09 1:12:03 140000618817408 [Note] Plugin 'FEEDBACK' is disabled. db_1 | 2018-04-09 1:12:03 140000618817408 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded db_1 | 2018-04-09 1:12:03 140000618817408 [ERROR] Unknown/unsupported storage engine: InnoDB db_1 | 2018-04-09 1:12:03 140000618817408 [ERROR] Aborting db_1 | bugzillacustomisation_db_1 exited with code 1

I'm getting this error when trying to get bugzilla running on Windows 10. Any ideas on what might be causing this?

Idea is that if AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY is correctly set, with the right permissions/role whatever, all the keys values will be set via aws ssm get-parameter.

What is the process to release this in the production environment?

Guide creating a user:
https://bugzilla.readthedocs.io/en/5.0/installing/mysql.html#mysql

GRANT SELECT, INSERT,
UPDATE, DELETE, INDEX, ALTER, CREATE, LOCK TABLES,
CREATE TEMPORARY TABLES, DROP, REFERENCES ON bugs.*
TO bugzilla@localhost IDENTIFIED BY '$DB_PASS';

FLUSH PRIVILEGES;

FROM_EMAIL is populated with:

$ aws --profile uneet-dev ssm get-parameters --names FROM_EMAIL --query Parameters[0].Value --output text
DEV Unee-T Case <[email protected]>

I noticed [email protected] on a password reset email & [email protected] elsewhere.

So can we agree that all staging email goes to [email protected] and that it is properly manned?

In the BZFE when we try to use a pre-configured search like this one (on the bottom of the BZEFE

Another example of a pre-saved query generates the following link:
https://dev.dashboard.unee-t.com/buglist.cgi?bug_status=__open__&component=Management%2520Company&list_id=509091&product=A_02-14_1%2520-%2520%2523263

This is incorrect as the query should be
https://dev.dashboard.unee-t.com/buglist.cgi?bug_status=__open__&component=Management%20Company&list_id=509093&product=A_02-14_1%20-%20%23263

The source of the issue is that the '%' seems to be escaped twice and converted into '%25'.
see:

• https://stackoverflow.com/questions/16084935/a-html-space-is-showing-as-2520-instead-of-20
  A possible fix could be to tweak some of the Apache config: as explained here
• https://community.atlassian.com/t5/Confluence-questions/Why-confluence-link-URL-get-modified-with-25-sign/qaq-p/459280

The issue only happened in a clustered AWS environment and does NOT happen on a standard AWS EC2 installation.

Now that we have a lot of users displayed it is impractical to choose assignee and CC with a drop down.

We need to switch the parameter 'usemenuforusers' to OFF to have a text entry field instead

Currently pushing to Docker hub with my own account

We have one field 'Deadline' in the list of available fields for a case.
This field is part of the 'timetracking' family of fields and is made visible to user who are member of the timetracking group.

The issue:
In case 12345, the 'Deadline' field is visible for a user that is a member of the group 'syst_see_timetracking' (the time tracking group)

In case 61885, the 'Deadline' field is NOT visible for a user that is a member of the group 'syst_see_timetracking' (the time tracking group)

even though other time tracking information are visible on the case

We need to understand why this happens so that the 'Deadline' field is available to users that are member of the group 'syst_see_timetracking' (the time tracking group)

The only apparent difference between the 2 cases is that they are for different units...

@kaihendry do you want to have a crack at this one?

We want to capture the MEFE id of the objects that we use to alter the BZ database (ex: when we process an invitation for a new user).

The MEFE ids are NOT integers.
We need to alter the BZ tables that record the MEFE ids of these object to be able to accept the MEFE ids.

in the folder https://github.com/unee-t/bugzilla-customisation/tree/differentiate_envo/params there are 2 params.json files:

• dev-params.json
• prod-params.json

1 is missing:

• local-params.json
  This should set all the parameters that are needed for a local environment except the values that are specific to the local installation:
• urlbase
• maintainer
• email parameters

@kaihendry can you let me know what should be done so we can deploy a local installation with all the other parameters which are not installation dependent correctly configured?

There is an issue with bugzilla.conf #66 (comment)

Related performance issue unee-t/bugzilla#8

https://ap-southeast-1.console.aws.amazon.com/ecs/home?region=ap-southeast-1#/clusters/master/services/ecscompose-service-bugzilla/events does suggest there is an issue.

Can't see the same behaviour on staging 8126-4485-3088. So this is going to be fun.

There are no clues in the bugzilla logs. https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logStream:group=bugzilla;streamFilter=typeLogStreamPrefix

On Mozilla IRC, #bugzilla advised for tracing:

22:55 <@dylan> hendry: if you want to do some profiling -- after you've got mod_perl turned on, you can copy:  https://github.com/mozilla-bteam/bmo/blob/master
/mod_perl.pl#L27-L33, https://github.com/mozilla-bteam/bmo/blob/master/mod_perl.pl#L151-L157, https://github.com/mozilla-bteam/bmo/blob/master/mod_perl.pl#L161
-L164
22:55 <@dylan> this will produce nytprofile files in the data/ directory
22:56 <@dylan> you can turn them into pretty readable reports with nytprofhtml command. you'll need the package Devel::NYTProf for that however.

After 538e8ab the MEFE does not seem to be accessible any more.

What I've done:

• Create a new ECS machine
• clone the bugzilla-customisation repo
• build the application (make up)

Test:

• go to the BZFE: [IP address]:8081 -> BZFE is accessible
• go to to MEFE: [IP address]:8080.
  • Expected result: MEFE is accessible
  • Actual result: Unable to connect.

This can be seen at http://54.169.243.75:8080/

@kaihendry any idea why this is happening?

We have a weird issue in the prod environment:
Timetracking information are not accessible for users even though they are granted membership to the group that allow visibility and editing of the timetracking information.

This is true for user Nast or Franck for example.

It seems that the group membership is ignored by BZ.

A problem that might be related to that is that admin user are not able to impersonate another user in the LIVE anymore.

This problem does NOT happen in the dev/staging environment.

We need to understand what are the differences between both environments to pinpoint the root cause of the issue.

This documents our investigations.

Blocked by https://travis-ci.org/unee-t/bugzilla

@franck-boullier a params.json can be snagged after you've configured it via the Web interface like so: docker cp customisation_bugzilla_1:/opt/bugzilla-5.0.3/data/params.json /tmp/params.json

We'd like to use Mixpanel to display dashboard and relevant data so we can monitor and improve Unee-T

This issue captures our progress to acheive that.

A new parameter 'MIXPANEL_SNIPPET' has been created in the AWS parameter store for the PROD envo to record the Mixpanel JS snippet for this environment.

I think it's saving the query as something unnecessarily complex..
workaround is to bookmark a simple search, e.g. https://dev.dashboard.unee-t.com/buglist.cgi?quicksearch=house&list_id=509068

EFS is not available in the Singapore region.

https://twitter.com/nathankpeck/status/926317004641812486

@nbiton suggested mLab. Will open an account and switch to them ASAP for dev and production.

Btw Mongo is lacking local mounts https://github.com/unee-t/bugzilla-customisation/blob/master/docker-compose.yml#L63

We need to fix the error

bugzilla_1 | Not a reference at Bugzilla/DB/Schema.pm line 2916..

When we run the BZ installation script
This tracks our progresses to do this.

https://s.natalian.org/2018-04-19/latency.mp4 just cared to monitor TargetResponseTime.

Using Aurora should allows us to plot metrics of queries taking a long time. I am assuming this is the cause of long TargetResponseTimes reported by the ECS load balancer on Bugzilla.

Then we can hopefully drill down the troublesome queries.

@kaihendry a weird error happened to some users yesterday.

See the screenshot for more details.
Error message was

Couldn't open pipe to sendmail (/usr/sbin/sendmail): Cannot allocate memory

BG: https://unee-t.slack.com/messages/C871N3LFP/convo/C871N3LFP-1512708819.000007/ & https://s.natalian.org/2017-12-08/504.png

mysqldump -h db.dev.unee-t.com -P 3306 -u root --password=$(aws --profile lmb-dev ssm get-parameters --names MYSQL_ROOT_PASSWORD --with-decryption --query Parameters[0].Value --output text) --ignore-table=bugzilla.attach_data bugzilla  > dev.sql

is ~330M. Full DB with attachments is 17GB: #21 (comment)

Currently RDS configurations between dev and prod are different.

https://aws.amazon.com/premiumsupport/knowledge-center/log-client-ip-load-balancer-apache/

To minimize the risk of time zone issue, the installation should use UTC as the default timezone.

In the BZ back-end, this is done as administrator by going to /editsettings.cgi and change the parameter 'Timezone used to display dates and times ' to UTC.

We need to alter the default parameters when we build Unee-T to reflect that.

https://gist.github.com/kaihendry/5de385d271c6fafde9c9fd317e03ca0f

Currently the prime at https://github.com/unee-t/bugzilla-customisation/blob/master/sql/demo.sql.gz doesn't contain example users and units to act out common use cases with Unee-T.

I order to let user experience Unee-T in a controlled/stable environment we need to have a demo environment up and running.

The existing solution (docker image on a specific EC2 instance) sorta work but is not optimal for several reasons, 1 of these reasons is that each time a new master is released, the Mongo DB is nuked and this creates a few extra steps.

Ideally, we'll need to have the demo environment setup similar to the prod (using demo specific environment variable and auto update each time a new tag is created in the master)?

It does NOT need to be on a cluster though, a standard EC2 instance should work fine.

Since we don't fix on a version of our base, i.e. we always use latest with FROM uneet/bugzilla, this make it very difficult to fix versions and roll back.

We should be fixing versions like FROM mozillabteam/bmo-slim:20180225.1 and furthermore deploying by uneet/bugzilla-customisation:${COMMIT}

The work around is to basically roll master back, which is super awkward.

Unee-T will sponsor this integration, which would probably be accomplished like so Paws->service('SNS')->Publish(Message => $jsonPayload, TopicArn => '...');. Please contact me for a AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY.

This will hopefully enable us to fix:

• unee-t/frontend#186
• unee-t/frontend#185

But not unee-t/frontend#191 (which would probably have to come from the Frontend)

When developing locally, demo.sql.gz seems to be missing some schema. Therefore https://github.com/unee-t/invite/tree/master/sql fails to run and fails to invoke an invitation.

by_environment does not exist * nFKobGdSNewju7DP7 Error 1305: PROCEDURE bugzilla.table_to_list_dummy_user_by_environment does not exist * QxdJo7oebbACkqhRm Err
or 1305: PROCEDURE bugzilla.table_to_list_dummy_user_by_environment does not exist * p43fdzfRG2PsD3KhK Error 1305: PROCEDURE bugzilla.table_to_list_dummy_user_
by_environment does not exist * xSnMusxGBE67XnDFT Error 1305: PROCEDURE bugzilla.table...]
I20180614-13:51:51.665(8)?   response:
I20180614-13:51:51.668(8)?    { statusCode: 500,
I20180614-13:51:51.669(8)?      content: '6 errors occurred:\n\n* HjBiNS9Ravjs9w7TH Error 1305: PROCEDURE bugzilla.table_to_list_dummy_user_by_environment does
 not exist\n* nFKobGdSNewju7DP7 Error 1305: PROCEDURE bugzilla.table_to_list_dummy_user_by_environment does not exist\n* QxdJo7oebbACkqhRm Error 1305: PROCEDUR
E bugzilla.table_to_list_dummy_user_by_environment does not exist\n* p43fdzfRG2PsD3KhK Error 1305: PROCEDURE bugzilla.table_to_list_dummy_user_by_environment d
oes not exist\n* xSnMusxGBE67XnDFT Error 1305: PROCEDURE bugzilla.table_to_list_dummy_user_by_environment does not exist\n* BedKAbRt5osMYxawY Error 1305: PROCE
DURE bugzilla.table_to_list_dummy_user_by_environment does not exist\n',
I20180614-13:51:51.672(8)?      headers:
I20180614-13:51:51.673(8)?       { 'content-type': 'text/plain; charset=utf-8',
I20180614-13:51:51.674(8)?         'x-content-type-options': 'nosniff',
I20180614-13:51:51.675(8)?         'x-robots-tag': 'none',
I20180614-13:51:51.676(8)?         date: 'Thu, 14 Jun 2018 05:51:51 GMT',
I20180614-13:51:51.677(8)?         'content-length': '656',
I20180614-13:51:51.678(8)?         connection: 'close' },
I20180614-13:51:51.679(8)?      data: null } }

Before sending out the invitation, a process on Bugzilla needs to be performed which I am unclear about.

Once it's done, I know I send the invitation off with http://localhost:3000/api/pending-invitations/done?accessToken=blablabla...

This has something to do with https://github.com/unee-t/bugzilla-customisation/blob/master/start#L27

[Mon Jun 18 06:42:51 2018] (9129) Apache2::SizeLimit httpd process too big, exiting at SIZE=417048 KB SHARE=9544 KB UNSHARED=407504 REQUESTS=82 LIFETIME=1188 s
econds
[Mon Jun 18 06:43:03 2018] (9130) Apache2::SizeLimit httpd process too big, exiting at SIZE=419272 KB SHARE=9644 KB UNSHARED=409628 REQUESTS=72 LIFETIME=1011 s
econds
[Mon Jun 18 06:45:11 2018] (9122) Apache2::SizeLimit httpd process too big, exiting at SIZE=424428 KB SHARE=9504 KB UNSHARED=414924 REQUESTS=192 LIFETIME=2485
seconds
[Mon Jun 18 06:45:18 2018] (9124) Apache2::SizeLimit httpd process too big, exiting at SIZE=411324 KB SHARE=9564 KB UNSHARED=401760 REQUESTS=188 LIFETIME=2456
seconds
[Mon Jun 18 06:53:00 2018] (9134) Apache2::SizeLimit httpd process too big, exiting at SIZE=440412 KB SHARE=9564 KB UNSHARED=430848 REQUESTS=89 LIFETIME=1017 s
econds
[Mon Jun 18 06:54:38 2018] (9155) Apache2::SizeLimit httpd process too big, exiting at SIZE=410844 KB SHARE=9544 KB UNSHARED=401300 REQUESTS=62 LIFETIME=689 se
conds
[Mon Jun 18 06:56:08 2018] (9133) Apache2::SizeLimit httpd process too big, exiting at SIZE=423492 KB SHARE=9564 KB UNSHARED=413928 REQUESTS=102 LIFETIME=1229
seconds
[Mon Jun 18 07:12:15 2018] (9157) Apache2::SizeLimit httpd process too big, exiting at SIZE=412612 KB SHARE=9564 KB UNSHARED=403048 REQUESTS=139 LIFETIME=1606
seconds

I'm going to assume the 400MB limit is being exceeded and needs to be tweaked to be a bit higher.

When clicking on a calendar icon (ex Next Step date) the calendar is displayed out of place: too far down.
This is true for all the calendar icons on the pages 'show_bug.cgi'

If something goes wrong with Bugzilla, I don't have access to the administrator user name or pass word to debug settings. I could go straight into the database, but it's easier to look at /admin.cgi in light of the params.json.

Perhaps:

• BZFE_ADMIN_USER
• BZFE_ADMIN_PASS