google / lkml-gerrit-bridge Goto Github PK

View Code? Open in Web Editor NEW

5.0 5.0 10.0 131 KB

License: Apache License 2.0

Python 99.22% Dockerfile 0.78%

lkml-gerrit-bridge's People

Contributors

Stargazers

Watchers

Forkers

wliu0628 dlatypov isabella232 neotim bjh83 jaimehisao lenhard-thomas test-mass-forker-org-1 ghas-results

lkml-gerrit-bridge's Issues

Fix email to message generation

There are a few emails on the Linux-kselftest mailing list that are unable to be generated because they are encoded differently. I am currently unaware of what the specific issue is, but this is something to look into.

[Feature] Add lore link to Gerrit change review description

Add the lore link for each patch uploaded.

Create a service email for uploading patches to Gerrit

Currently, the server uses [email protected] to upload emails to gerrit. However, that isn't sustainable after I leave.

Document the logging criteria and what each level represents in the context of this server

Logging has multiple levels for absl, and currently three are being used (warning, error, info). Write what each of these levels represents within the program.

Fix the patch identification system

Patches are currently being identified as any email with a word surrounded by brackets i.e. [PATCH] or [BUG] etc. However, patches don't necessarily have to follow this subject formatting.

It could be a good idea to try and parse every email as a patch that isn't a reply to a patch in an attempt to determine whether it is a patch or not. If it is parse-able, treat it as a patch and then try to upload it. O/w it should be a comment.

[Feature] Add a SQL Database

Currently, message_dao.py is just a wrapper for a dictionary. This should be changed to use a SQL Database like PostgreSQL.

Find a suitable epoch hash

Currently the epoch hash was an arbitrary choice from ~1 month ago. This should probably be adjusted in the future.

Multiple broken tests

Running all the tests

python3 archive_converter_test.py; python3 archive_updater_test.py; python3 main_test.py; python3 patch_parser_test.py

Causes the following failures:

.F.
======================================================================
FAIL: test_update_return_proper_patches (__main__.ArchiveConverterTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "archive_converter_test.py", line 40, in test_update_return_proper_patches
    compare_message_subject(new_messages, subjects)
  File "archive_converter_test.py", line 30, in compare_message_subject
    self.assertEqual(message.subject, subject)
AssertionError: '[PATCH v2 2/4] kselftests/arm64: add nop checks for PAuth tests' != 'Re: [PATCH] Remove final reference to superfluous smp_commence().'
- [PATCH v2 2/4] kselftests/arm64: add nop checks for PAuth tests
+ Re: [PATCH] Remove final reference to superfluous smp_commence().


----------------------------------------------------------------------
Ran 3 tests in 0.009s

FAILED (failures=1)
.WARNING:absl:There are no commits in git repo: archive_path
......
----------------------------------------------------------------------
Ran 7 tests in 0.008s

OK
.FF
======================================================================
FAIL: test_server_upload_across_batches (__main__.MainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "main_test.py", line 70, in test_server_upload_across_batches
    mock_upload_messages.assert_called_with([messages[2].id,messages[3].id])
  File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: upload_messages(['<1169488670.15515.17.camel@earth>', '<[email protected]>'])
Actual: upload_messages(['<20200827144112.v2.1.I6981f9a9f0c12e60f8038f3b574184f8ffc1b9b5@changeid>', '<[email protected]>'])

======================================================================
FAIL: test_split_parent_and_reply_messages (__main__.MainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "main_test.py", line 51, in test_split_parent_and_reply_messages
    compare_message_subject(replies, expected_replies)
  File "main_test.py", line 49, in compare_message_subject
    self.assertEqual(message.subject, subject)
AssertionError: '[PATCH v2 2/4] kselftests/arm64: add nop checks for PAuth tests' != 'Re: [PATCH] Remove final reference to superfluous smp_commence().'
- [PATCH v2 2/4] kselftests/arm64: add nop checks for PAuth tests
+ Re: [PATCH] Remove final reference to superfluous smp_commence().


----------------------------------------------------------------------
Ran 3 tests in 0.010s

FAILED (failures=2)
...
----------------------------------------------------------------------
Ran 3 tests in 0.006s

OK

Fix unit test action after PR #49

PR #49 added new entries into requirements.txt.
They don't seem to be getting installed, which is causing the action to fail: https://github.com/google/lkml-gerrit-bridge/runs/7921276291?check_suite_focus=true

We have this following bit of code in our config

pip install pytest pytest-mock pytest-cov absl-py pygerrit2
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

yet it somehow doesn't seem to work.
This was copy-pasted from an example somewhere, so we'd hoped it work, but I guess not.

We can probably hard-code more packages in the first line to resolve the issue in the short-term.
In the long-term, I'm not sure what we need to have the action properly support requirements.txt.

GCE does not put stdout and stderr logs in the default bucket

For some reason GCE fails to log things that are printed to std err and std out.

[Feature] Change the base gerrit_git_dir to the appropriate commit as indicated by a patch

Sometimes, patches sent in the mailing list have different base commits than the master branch of Linux. As such, the master branch should be checked out to whatever's specified by the patch.

[Feature] Check whether a patch has been added to Linux

Determine whether a patch has been added to Linux and if it has, remove the change review on Gerrit.

Add in content transfer decoding

Currently, emails with quoted printable content transfer encodings are unable to be parsed by our system. This should be fixed in the future.

[Feature] Email from Gerrit to LKML

Generate email replies from Gerrit back to the original sender of the patch/reply.

Comment get added a few lines below where they should

E.g. https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/6958/1/tools/testing/kunit/kunit_parser.py#347

Source email: https://lore.kernel.org/linux-kselftest/CABVgOSnPpC=j7MrcmDpvvG6i_voiFQe4137ieyYX+-9B4=G39w@mail.gmail.com/

The comment should be right on the last changed line, not one line below that.
91ae2f1 added a reproducer in the unit tests and fixing the TODOs there should resolve the issue.

Another issue to note: patch_parser.py thinks the trailing quoted section

>         if test_result.status == TestStatus.NO_TESTS:
>                 print(red('[ERROR] ') + yellow('no tests run!'))
>         elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
>
> base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
> --
> 2.29.0.rc1.297.gfa9743e501-goog
>

is a pair of empty comments (it picks up the 2 quoted empty lines).

That might be trickier to address and can be looked at separately, in that case.

Add "how to run main.py" and "how to read logs" to README.md

There should be a section about how to run main.py and how to run the docker image.

There should also be information about logs and where to find them. Also, that the --stderrthreshold=info command exists if people want to log to print to stderr as well as to the logs.

[Feature] Identify malformed emails

Be able to identify malformed emails based on our parsing system.