adafruit / adabot Goto Github PK

Now that the libraries have been migrated from Travis to GitHub Actions, remove or change all the infrastructure validators that are Travis related.

Please add two checks to Adabot for circuitpython.org/contributing:

• A list of libraries on Pylint 1.9.2
• A list of libraries on the latest Pylint

Thanks!

It would be nice to have a count of open adafruit/circuitpython issues per milestone, so that we can see how many need to be closed for 4.0, 4.x, how-many are long-term, etc. And also a count of how many have no milestone assigned.

We could also do counts by labels, but that's probably less useful, and we'd probably want only certain labels (nrf52, etc.)

The output is showing the Adafruit CircuitPython BLE library as no release in 107 days:

"https://github.com/adafruit/Adafruit_CircuitPython_BLE (107 days)",

It is currently in pre-release mode, and the pre-releases aren't being picked up by the script.

Is it worth adding pre-releases to this list?

As it stands, the circuitpython_libraries.py file is quite long, and makes edits, additions, and readability difficult.

Going to move the validators and associated code into a new file.

After the initial addition of a library to the bundle, any new release of that library will generate an update to the bundle. However, there are sometimes where a library has received some updates, but no release is done. This could be intentional or accidental.

Add some kind of check that can warn when this situation exists for any library included in the bundle.

Currently the text comes as:

The libraries in each release are compiled for all recent major versions of CircuitPython. Please download the one that matches the major version of your CircuitPython. For example, if you are running 3.0.0 you should download the 3.x bundle.

To install, simply download the matching zip file, unzip it, and copy the lib folder onto your CIRCUITPY drive. Non-express boards such as the Trinket M0, Gemma M0 and Feather M0 Basic will need to selectively copy files over.

However, with the bundle getting larger, copying over the entire lib folder to the CIRCUITPY drive isn't so feasible anymore.

Also, maybe update the example to be 4.0.0 and 4.x bundle since that is current.

During today's meeting, @jepler mentioned that while they conducted a PR review in the past week, they didn't show up as a reviewer in the meeting notes (notes info is generated by adabot/circuitpython_libraries.py).

All reviewers should be listed; celebrating contributions is just the right thing to do!

The issue stems from here, as the script only retrieves the individual that merges a PR. Any other reviewers during the course of the PR are not retrieved.

I actually accomplished this in update_cp_org_libraries.py. Not sure why I didn't transpose it over... But, seeing that the code is already written, this will be a quick fix.

As we've added more boards, the list of downloads broken down by board has begun to become unwieldy in when included in the CircuitPython Weekly notes document. We've made changes in the past to the Libraries section of the library report to slim it down for the CP Weekly notes doc. I feel at this point, we should remove the actual tables from the library report. The information is available on https://circuitpython.org/stats (though the page is currently experiencing the same issues as the library report, and is therefore not displaying anything). I suggest we remove the tables from the library report and replace them with a link to the download stats on circuitpython.org.

Instead we would include the total downloads of the current stable and current unstable releases, followed by a link to the breakdown on circuitpython.org.

@tannewt Thoughts?

validate_labels currently just flags a repo for not having the "standard" set of labels.

Like the Hacktoberfest label updater, validate_labels could actually just update the labels instead of flagging the repo.

I noticed this today. I had wrote some updates (as well as did releases) on 3 display libraries. @siddacious reviewed all 3 of them. I noticed in the email it credits @siddacious and @adafruit-adabot, but not the author. It seems to me that she used to credit the library author as a contributor, but I could be wrong.

For instance, this is what I mean:

April 28, 2019 auto-release

Repository: adafruit/Adafruit_CircuitPython_Bundle · Tag: 20190428 · Commit: 37f8791 · Released by: adafruit-adabot

Updated libraries: ili9341, ssd1331, ssd1351

As always, thank you to all of our contributors: @siddacious, @adafruit-adabot

Add the ability to specify the following via command line when running Adabot:

• 1. Output Log: print, file, or both
• 2. Error list: change depth. This request came out of needing to have a more detailed tracking of the State of the CircuitPython Libraries.
• 3. Add ability to not prompt for Adabot’s password if needed. This would allow for it to run every night, “automated mode” would be really nice, we can automatically run it every night and dump it to the S3 bucket, and always have updated version available.

edited by @sommersoft to add checkboxes and numbers

Navigating to the API docs page, I came across the API v4 pages.

v4 seems to be dumping RESTful and moving to GraphQL.

I've searched and tried to figure out when they might sunset v3, and came up empty.

This is really just a placeholder issue...

In relation to #76, have adabot update the CircuitPython Libraries! issue on the core repo, using the GitHub API.

Currently, the website is not allowing updates to the body of the issue. GitHub support has not responded with a fix.

The Platform Detect repo has been refactored and likely breaks the way that adabot was getting boards. I would recommend looking at circuitpython-org repo. More details in these comments:
#119 (comment)
and
#119 (comment)

The Adafruit_CircuitPython_MCP4728 library is showing up as not in the bundle, and it's definitely in the bundle.

As mentioned in the 9 Dec 2019 weekly meeting, some stats & metrics are desired for things related to Adafruit_Blinka.

PyPI download stats are already gathered daily here. Doesn't include data from piwheels; they still don't have a stats API available. Would be easy to get the PyPI stats into the meeting notes template; its basically a hardcoded one-liner.

What other sorts of information are wanted?

tagging: @makermelissa, @kattni, @tannewt

P.S. If this is a discussion better had on Discord voice, I'm fine with setting up a time for that. I just wanted to get an issue created here.

To-Do List

• Gather Insights (issue/PR events, contribs, etc)
• Blinka downloads (GitHub release & PyPI)
• Count of supported boards

I feel like we already started this or completed it. Or thinking on it more, I think it was to fix it so Pylint checks in folders, not the example name checks.

This is failing the "Missing simpletest example." check - the examples are in folders: Adafruit_CircuitPython_AdafruitIO. We should consider updating the checks to include folderised examples.

The author feels they do make sense in folders, but is willing to rename the files accordingly if this isn't feasible.

https://adafruit-circuit-python.s3.amazonaws.com/adabot/bin/adabot/library_download_stats_20190109.txt

Please sort the lists by download number. Obviously the bundle list is easy to sort mentally, but the PyPI list, not so much.

Thanks!!

Add total download stats from pypistats.org. Can be retrieved with the /overall API request.

This will also push a formatting change. Will tabulate results, like the Arduino script does.

As pointed out by @siddacious, we actually have archived CircuitPython library repositories now.

Update adabot/lib/common_funcs.py::list_repos() to not return any archived repos.

Adding archived:false to the search parameters should work: https://github.com/adafruit/adabot/blob/master/adabot/lib/common_funcs.py#L171

Some CPython built-in modules have been renamed.

Need to add a check to ensure these are properly handled, just like ustruct is.

With the translations, the core download stats have gotten long and eye-soreish. So, in an effort to reduce this, and make the meeting notes a little more mouse-wheel friendly, I figure tabulating the stats will do the trick.

Board        lang1   lang2   lang3
Feather        19      45     32

Add a list of libraries that have been updated in the last 7 days, which kind of coincides with the current insights (issues, PRs, reviews, etc). Also enables an easily accessible list for updating the circuitpython-weekly-newsletter.

"Updated library" is defined as either a newly released library, or a library that has had a new release.

Format will be Markdown, as follows:

Library updates in the last 7 days:
**New Libraries**
* [Library Name](library_url)

**Updated Libraries**
* [Library Name](library_url)

In talking with @kattni, we shared the concern of adding to the length of the already long library report. To address this, we'll remove the list of open library issues and replace it with a link to the CircuitPython Libraries! issue on the core repo. The list is currently 94 lines long.

A secondary effort to have adabot update that issue will be completed, using the GitHub API. This is to replace the above daily reporting of current issues and because the website is not allowing updates on the issue anyway. See #77.

adruino library manager is picky - the release number must match the library.properties file and the prop file must be updated before the release, otherwise stuff breaks. this happens often enough, we should add a check to adabot!

This is a suggested refinement to #18. Add some sort of differentiation based on amount of time between last release and last commit. So for example, repos behind by several weeks can be somehow highlighted over repos that are behind by one day.

Since we've moved drivers.rst from the core to the Bundle, need to fix the CircuitPython drivers page missing driver check.

Changes made to the CI of a library have no effect on the code or the user, and therefore do not require a new release be made. We should update the checks to ignore changes made only to the CI so we can make those changes without requiring a library-wide release sweep every time. This would ensure we do not miss out on releasing code changes etc. when CI changes are applied across the board.

From #119 (comment)

If you'd like download statistics for pypi:

https://pypistats.org/ and https://pypistats.org/api/
https://stackoverflow.com/questions/38102317/why-pypi-doesnt-show-download-stats-anymore (but see the newer answers)

Example: https://pypistats.org/packages/adafruit-blinka

Tagging @brennen @kattni @tannewt

The cron jobs are having failures at a decent percentage. Luckily, the bundle updates are running smoothly, and the only issue seems to be in circuitpython_libraries. There are only two issues I see occurring:

1. Hitting the Travis 10 Minute Idle:

   • There is no output at all for these failures, which leads me to believe its related to the call for the latest pylint version.

2. Getting a KeyError exception:

Running CircuitPython library checks...
$ python -m adabot.circuitpython_libraries -o $LIB_CHECK_CP_FILE

Traceback (most recent call last):
  File "/opt/python/3.6.3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/python/3.6.3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/travis/build/adafruit/adabot/adabot/circuitpython_libraries.py", line 1131, in <module>
    run_library_checks()
  File "/home/travis/build/adafruit/adabot/adabot/circuitpython_libraries.py", line 943, in run_library_checks
    errors = validate_repo(repo)
  File "/home/travis/build/adafruit/adabot/adabot/circuitpython_libraries.py", line 798, in validate_repo
    errors.extend(validator(repo))
  File "/home/travis/build/adafruit/adabot/adabot/circuitpython_libraries.py", line 297, in validate_repo_state
    full_repo = github.get("/repos/" + repo["full_name"])
  File "/home/travis/build/adafruit/adabot/adabot/github_requests.py", line 61, in get
    remaining = int(response.headers["X-RateLimit-Remaining"])
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'x-ratelimit-remaining'
Latest pylint is: 2.2.2
Found 144 repos to check.
Found 129 submodules in the bundle.
Running GitHub checks as adafruit-adabot
Running Travis checks as adafruit-adabot
4900 requests remaining this hour
4800 requests remaining this hour
4700 requests remaining this hour

I'll try and replicate them, and figure out how to mitigate.

As discussed with @sommersoft, these changes would be made to the Libraries section only:

• Collapse list of Open PRs.
• Update "## open issues" line to read "## open pull requests and ## open issues" (still followed by circuitpython.org URL).
• List PRs merged over the previous week with number of days the PR was open before merging.
• Possibly show "oldest" and "newest" data for open pull requests.
• All other data will remain the same.

Repositories that have files in the examples folder that don't have the sensor/module name as a prefix and aren't .py files get added to list of modules missing module/sensor prefixes

i moved the original title here - sommersoft

If an initial library is in work, and nothing is in the repo, almost (if not) all errors will be flagged.

There are a few checks that could still be valuable (Travis enabled, librarians), but the rest could be skipped.

@kattni, add anything to this description I may be missing.

I'd like to publish a beta version of a library as a pre-release. It should not be in the bundle.

We could query github to find the latest release that is not a pre-release, or we could go by tag. A alpha or beta release tag will contain a -, like 3.0.0-beta.0. Regular release tags don't contain a -.

In discord I suggested that the weekly stats could include the number of "good first issue"s that we have for new contributors.

We're only doing one fetch of the issues list which is limiting us to a response of 100 issues. We need to handle pagination to get all relevant issues.

We moved the drivers.rst file to the docs in the Bundle from the core. The description of the validator currently reads CircuitPython drivers page missing driver and could be clearer.

Move off the computer at my house onto Travis.

https://docs.travis-ci.com/user/cron-jobs/

This includes:

• https://github.com/adafruit/Adafruit_CircuitPython_BusDevice
• https://github.com/adafruit/Adafruit_CircuitPython_miniQR

More coming.

Each of the validator functions are nicely broken out, so why not allow the option to run them à la carte.

As you can see here, the validators are already run from this list. Parser just needs the choices set, and the default.

There is reportedly some issue with adding library RTD pages as a subproject to the core's RTD project. As such, some newer libraries are being added to the drivers.rst page with their direct RTD URL.

Update validate_core_driver_page to also check for the direct RTD URL.

In talking with @makermelissa on discord about fixing results returned by validate_core_driver_page, I ran a fresh report using the -v "validate_core_driver_page" flag.

The result had an additional 6 repos listed, compared to a full (non -v) run. Those 6 do not have releases on GitHub, so should be ignored. They are listed because validate_contents was not run, which didn't trigger adding the un-released repo to BUNDLE_IGNORE list.

All of the validators check if a repo is in the BUNDLE_IGNORE list, and skip the check if the repo is in the list.

So, long story short, using the -v flag should not produce different results than a full run.

The list of results under the "CircuitPythonLibrarians team missing or does not have write access" check includes libraries for which this issue is not the case. It appears to be triggering on most libraries. Is it possible the way this check works has changed?

rather than try to keep a learn guide updated, we can create a file that we then embed, which will simply list all the drivers, maybe with a single line of text describing it (from the driver repo itself)

So we don't miss PRs to the Community Bundle.

Failed to fetch ReadTheDocs valid versions returned 127+ repos on the last few successful runs.

Running the RTD list of subprojects on my own, I get 131 subprojects listed in the JSON. Something is amiss!

Need to figure out how to construct reasonable tests to run for PRs.

Notes:

• I'd like to avoid just running every script, due to rate limiting and time. However, testing the rate limit handling is probably a good idea.
• More notes to come...