python / buildmaster-config Goto Github PK

See this result, for example.

-rw------- 1 buildbot buildbot 4.6M Jan  8 15:54 '/data/www/buildbot/test-results/3.x/PPC64 Fedora 3.x/build_145.xml'

This wasn't true previously, see this result:

-rwxr-xr-x 1 buildbot buildbot 4.6M Jan  7 17:05 build_138.xml

This appears to be due to umask being set to 0077 in the current master process:

$ grep Umask /proc/`pgrep -f buildbot.tac`/status
Umask:  0077

My PR python/cpython#109958 was merged on branch main, but a buildbot failure "wasm32-wasi 3.11" was reported on this PR.

The PR for 3.11 is: python/cpython#109962

Yesterday, I upgraded dependencies: 8548bf2

It seems like since this upgrade, buildbot doesn't trigger new jobs anymore. I don't know what's going on.

I reverted the change (5c92579). Let's see if it does repair the server.

this came up several times recently around testing the NoGIL build on PRs - it's currently not possible to do that using the !buildbot nogil comment (example attempt) because the nogil builders are marked as unstable.
considering the NoGIL build is "experimental", keeping these builders as "unstable" is probably a good idea, but then we need some way to be able to request these builders on PRs.

after chatting with @ambv, I propose that we extend the !buildbot command to support unstable builders.
the mechanics can be discussed in this issue. here are the options I am considering:

1. easy but changes existing behavior: include unstable buildbots by default
2. also easy but kinda hacky: include unstable buildbots if they have a certain tag (e.g. "trigger-on-pr-command")
3. introduce new spelling for explicitly allowing unstable builders (e.g. !buildbot_with_unstable ... or !buildbot[unstable] or !buildbot ... #with_unstable - bikeshedding welcome)

AMD64 Arch Linux Asan Debug runs git clean -f -f -d iso. git clean -fdx.

@Rosuav reported on python-buildbots@ not receiving an email from the master that his worker had disconnected. As a worker owner, this is a very handy email and we should make sure it's working correctly.

In python/cpython#92886, Irit noticed that a bunch of tests fail when run with -O:

On main (3.12 branch):

% ./python.exe -Om test -j2
[...]
10 tests failed:

[...]

This seems wrong because
(1) it indicates that some assertions should be exceptions instead.
(2) it implies that we don't have a buildbot testing with -O

I applied 94b4769

When I deployed the config using buildbot reconfig command, I got this error:

...
2019-09-19 14:31:49+0000 [Broker,6121,8.43.85.235] Worker cstratak-RHEL8-x86_64 attached to AMD64 RHEL8 LTO 3.x
2019-09-19 14:31:49+0000 [Broker,6121,8.43.85.235] Worker cstratak-RHEL8-x86_64 attached to AMD64 RHEL8 custom
2019-09-19 14:31:49+0000 [Broker,4173,8.43.85.216] during reconfig:
	Traceback (most recent call last):
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
	    _inlineCallbacks(r, g, status)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	--- <exception caught here> ---
	  File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/master.py", line 397, in doReconfig
	    yield self.reconfigServiceWithBuildbotConfig(new_config)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/util/service.py", line 49, in reconfigServiceWithBuildbotConfig
	    yield svc.reconfigServiceWithBuildbotConfig(new_config)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/process/measured_service.py", line 31, in reconfigServiceWithBuildbotConfig
	    yield super(MeasuredBuildbotServiceManager, self).reconfigServiceWithBuildbotConfig(new_config)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/util/service.py", line 520, in reconfigServiceWithBuildbotConfig
	    yield svc.reconfigServiceWithSibling(config_sibling)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/worker/base.py", line 331, in reconfigServiceWithSibling
	    yield self.updateWorker()
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
	    result = g.send(result)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/worker/base.py", line 538, in sendBuilderList
	    d1 = self.attachBuilder(b)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/worker/base.py", line 543, in attachBuilder
	    return builder.attached(self, self.worker_commands)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
	    return _cancellableInlineCallbacks(gen)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, g, status)
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1421, in _inlineCallbacks
	    status.deferred.callback(getattr(e, "value", None))
	  File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 459, in callback
	    assert not isinstance(result, Deferred)
	builtins.AssertionError: 
	
2019-09-19 14:31:49+0000 [Broker,4173,8.43.85.216] WARNING: reconfig partially applied; master may malfunction
Reconfiguration failed. Please inspect the master.cfg file for errors, correct
them, then try 'buildbot reconfig' again.

I ran stop followed by start buildbot commands: it worked around the issue.

I'm not sure if we should do anything with this error which only occured once?

cc @pablogsal @zware

When the "test-with-buildbots" label is applied, Bedevere posts a comment to notify that builds have been scheduled, but the comment is quite unhelpful as it doesn't say how the results can be accessed.

It should be possible to add a hyperlink to the buildbot results page for the particular commit, or at least the given PR (for example https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F30617%2Fmerge or https://buildbot.python.org/all/#/changes/14341 in the case above).

Hi, Python triagers can add test-with-buildbots label to PRs, but they cannot stop a build if it hangs. Currently, only core devs can do that. This makes it hard for triagers to "unstick" a PR after they test it with buildbots.

I think triagers should be able to stop builds. I will open a discussion at https://discuss.python.org/c/core-workflow/8. Please let me know if I should open a discussion at python-buildbots/python-dev too. Thanks for your time.

Since the old parser is going to be removed in 3.10, the VintageParser worker should not run for PRs against the master branch.

There are now buildbots running reference leaks tests. I would like to extract leaks as warnings. Example of leak:

test_code leaked [3, 4, 4] memory blocks, sum=11

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/46/steps/test/logs/stdio

I added the following regex to master/custom/steps.py:

        # test_os leaked [6, 6, 6] memory blocks, sum=18,
        r'test_.* leaked \[',

But it seems like the regex doesn't work, doesn't match or that the my config was not applied? I'm quite sure that the new config was applied. I tested manually the regex using the code from steps.py. I missed something...

See my commit 0d44a6a.

@zware: Would you mind to take a look?

The four pablogsal-arch-x86_64 builders are marked STABLE, but they live in the unstable section in master/custom/builders.py.

Either mark these builders UNSTABLE or move them to the "stable" section.

Ah, I though this was expected to succeed. In that case I will mark it as UNSTABLE for now

Originally posted by @pablogsal in #289 (comment)

It's been a month living in the UNSTABLE set, and now a breaking change has made its way into the active branches. The responsible PR was checked with test-with-buildbots, but since that only submits builds to STABLE, the faulty PR slipped through.

Please switch the worker back into STABLE to prevent this happening again.

PS: The failure just so happens to be related to the issue that got this worker moved to UNSTABLE in the first place.

Username

itamaro

GitHub username

@itamaro

Name

Itamar Oren

Email address

No response

Password status

I will use an existing owner password

Processor architecture

amd64

Operating System

Windows Server 2022

Anything special about the worker?

AWS hosted Windows worker for NoGIL builds

Example of PR: python/cpython#110458

Multiple Refleaks buildbots were run:

• AMD64 RHEL7 Refleaks PR
• s390x Fedora Refleaks PR
• AMD64 RHEL8 Refleaks PR
• AMD64 Windows11 Refleaks PR
• etc.

We have several workers that haven't been up for some time, and every time the master restarts a message is sent to the status mailing list announcing that the worker is down. If these workers have been abandoned I would like to remove them from the master, or figure out if there's something we can do differently from the master side to improve things.

Here are the currently-troublesome workers:

• @embray
  • bray-win10-cygwin-amd64
    • No data on last build
    • Retired in #356 per #349 (comment)
• @jkloth
  • kloth-win64
  • kloth-win11
    • @jkloth, I know you announced a move and extended downtime some months ago; do you have a timeline for the return of these workers?
    • #375 opened to remove, will merge in mumble days
• @kulikjak
  • kulikjak-solaris-sparcv9
    • Last builds were a month ago after a 2 month disappearance, and those builds did not finish
    • Hoped to return around first week of December 2022 according to #349 (comment)
    • Back in service, connection issues (see #349 (comment))
• @nsait-linaro
  • linaro2-win-arm64
    • Last build was 3 months ago. I seem to have a vague memory of this being a transitional worker anyway? Am I making that up? :)
    • Removed in #355 per #349 (comment)
• @orsenthil
  • skumaran-ubuntu-x86_64
    • Last build was 2 months ago.
    • To be fixed according to #349 (comment)
    • Back in service, connection issues (see #349 (comment))

Honorable mentions:

• @koobs
  • koobs-freebsd-564d
  • Last build was only 6 days ago, but this one still seems like a frequent entry in the list of workers that don't reconnect after a master restart. Are such restarts actually causing a problem for this worker? Do we need to figure out a different strategy for restarting?
  • To be fixed Soon™ according to #349 (comment)

To anyone @-mentioned above, please comment here with the status of your worker(s). I'll give at least a couple of weeks for responses here before I do anything with these (and may forget about it for longer than that :)), I'm not trying to rush anyone out here.

Thank you all for running your workers, whether they're still active or not. They have all served the CPython project well!

Hello! The readme mentions make commands to start the buildbot, and I got flashbacks from a job where we had process issues because of make. I’m afraid I don’t have solid references to share; the general idea is that make is designed to run commands over a graph of targets, but is not made to run a process, forwarding signals and exit codes, restarting, etc. Now I don’t know what bin/buildbot does, maybe it’s a wrapper that handles all that and make is only used to kick that off, not monitor things properly, in which case my worry is unfounded. But if make is really used as process supervisor on servers, I would suggest to use a tool made for that. Cheers!

Username

opsec

GitHub username

No response

Name

Kurt Jaeger

Email address

[email protected]

Password status

I need a new owner password

Processor architecture

amd64

Operating System

FreeBSD 14.0-BETA1

Anything special about the worker?

No response

Username

bcannon

GitHub username

No response

Name

No response

Email address

No response

Password status

I will use an existing owner password

Processor architecture

amd64

Operating System

Debian

Anything special about the worker?

For WASI builds. I will be retiring the bcannon-wasm worker once this new worker is up and running. I still have the password for bcannon-wasm, so I can reuse that.

Hi,

        I use Buildbot ( 0.9.8 ) and Twisted (==17.5.0) to run git poller,

        Config is below.

   c['change_source'].append(changes.GitPoller(
    repourl='[email protected]:/home/git/201701_postrack_swfw_app', branch='master',pollinterval=300, pollAtLaunch=True,
    gitbin="C:\\Program Files\\Git\\cmd\\git.exe",
    encoding="big5"))

   But it always to get the following exception :
  Can anybody tell what’s going wrong ? thanks!

  Jacky

2017-06-26 11:49:08+0800 [-] while polling for changes
Traceback (most recent call last):
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\defer.py", line 459, in callback
self._startRunCallbacks(result)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\defer.py", line 567, in _startRunCallbacks
self._runCallbacks()
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\defer.py", line 1442, in gotResult
_inlineCallbacks(r, g, deferred)
--- ---
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\buildbot\changes\gitpoller.py", line 190, in poll
path=self.workdir)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\buildbot\changes\gitpoller.py", line 357, in _dovccmd
path=encodeArg(path), env=os.environ)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\utils.py", line 174, in getProcessOutputAndValue
reactor)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\utils.py", line 28, in _callProtocolWithDeferred
reactor.spawnProcess(p, executable, (executable,)+tuple(args), env, path)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet\posixbase.py", line 353, in spawnProcess
return Process(self, processProtocol, executable, args, env, path)
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet_dumbwin32proc.py", line 217, in init
doCreate()
File "d:\jacky\buildbot_virtualenv\buildbot\lib\site-packages\twisted\internet_dumbwin32proc.py", line 214, in doCreate
command, cmdline, None, None, 1, flags, env, path, StartupInfo)
builtins.TypeError: Objects of type 'bytes' can not be converted to Unicode.

Username

itamaro

GitHub username

@itamaro

Name

Itamar Oren

Email address

No response

Password status

I will use an existing owner password

Processor architecture

amd64

Operating System

macOS 13.5

Anything special about the worker?

Mac with Intel CPU, hosted in AWS

Username

itamaro

GitHub username

@itamaro

Name

Itamar Oren

Email address

No response

Password status

I will use an existing owner password

Processor architecture

arm64

Operating System

macOS 14.0

Anything special about the worker?

Mac with M1 CPU, hosted in AWS

Bedevere used to comment on PRs that break stable buildbots, e.g. python/cpython#17975 (comment) .

It seems like the feature no longer works. For example PR python/cpython#94549 broke four stable WebAssembly buildbots but bedever-bot did not add a warning to the PR.

Rather, it should be possible to select a branch or specific builder(s) to force.

Based on @gpshead 's indication here, I understand we want to avoid "nogil" as a public moniker, preferring the more wordy "free-threading".

I don't know how "public" we consider our buildbot configs, so I'm posing this as a question - I added a bunch of "nogil" flavored workers and builders - should I go ahead and rename them to "free-threading" everywhere?

@ned-deily requested that the "buildbot went away" emails are not sent to the buildbot-status mailing list. Any objections?

CC: @vstinner @zware

The !buildbot Raspbian command didn't match any buildbot worker, but bedevere-bot decided instead to test the PR on all buildbot workers in this case: python/cpython#109101 (comment)

The !buildbot ARM Raspbian command matched ARM Raspbian PR as expected: python/cpython#109101 (comment)

By the way, it would be nice to match the regex anywork in the worker name, not only at the beginning of the name.

See the comment I left on 7801492

Thanks to #89 we now have a Windows ARM32 buildbot.

Looking at https://buildbot.python.org/all/#/builders though, I see it's attached to 2.7, 3.6 and 3.7 branches, while the support is only coming in for 3.x:

Can/should we restrict the buildbot to only the 3.x branch? I assume it's just going to fail to build against the other branches.

$ date
Fri Aug 25 09:51:55 UTC 2023

$ grep 'starting build ' master/twistd.log
# empty

$ grep 'starting build ' master/twistd.log.1|tail

2023-08-24 16:38:47+0000 [-] starting build <Build AMD64 Fedora Stable Clang Installed 3.x number:4304 results:success>.. pinging the worker <WorkerForBuilder builder='AMD64 Fedora Stable Clang Installed 3.x' worker='cstratak-fedora-stable-x86_64' state=BUILDING>
2023-08-24 16:38:47+0000 [-] starting build <Build AMD64 Fedora Stable LTO + PGO 3.x number:4551 results:success>.. pinging the worker <WorkerForBuilder builder='AMD64 Fedora Stable LTO + PGO 3.x' worker='cstratak-fedora-stable-x86_64' state=BUILDING>
2023-08-24 16:45:35+0000 [-] starting build <Build s390x Fedora Rawhide 3.x number:None results:success> using worker <WorkerForBuilder builder='s390x Fedora Rawhide 3.x' worker='edelsohn-fedora-rawhide-z' state=AVAILABLE>
2023-08-24 16:45:35+0000 [-] starting build <Build s390x Fedora Rawhide Clang 3.x number:None results:success> using worker <WorkerForBuilder builder='s390x Fedora Rawhide Clang 3.x' worker='edelsohn-fedora-rawhide-z' state=AVAILABLE>
2023-08-24 16:45:36+0000 [-] starting build <Build s390x Fedora Rawhide Clang Installed 3.x number:None results:success> using worker <WorkerForBuilder builder='s390x Fedora Rawhide Clang Installed 3.x' worker='edelsohn-fedora-rawhide-z' state=AVAILABLE>
2023-08-24 16:45:36+0000 [-] starting build <Build s390x Fedora Rawhide 3.x number:3548 results:success>.. pinging the worker <WorkerForBuilder builder='s390x Fedora Rawhide 3.x' worker='edelsohn-fedora-rawhide-z' state=BUILDING>
2023-08-24 16:45:36+0000 [-] starting build <Build s390x Fedora Rawhide LTO + PGO 3.x number:None results:success> using worker <WorkerForBuilder builder='s390x Fedora Rawhide LTO + PGO 3.x' worker='edelsohn-fedora-rawhide-z' state=AVAILABLE>
2023-08-24 16:45:36+0000 [-] starting build <Build s390x Fedora Rawhide Clang 3.x number:3538 results:success>.. pinging the worker <WorkerForBuilder builder='s390x Fedora Rawhide Clang 3.x' worker='edelsohn-fedora-rawhide-z' state=BUILDING>
2023-08-24 16:45:37+0000 [-] starting build <Build s390x Fedora Rawhide Clang Installed 3.x number:3692 results:success>.. pinging the worker <WorkerForBuilder builder='s390x Fedora Rawhide Clang Installed 3.x' worker='edelsohn-fedora-rawhide-z' state=BUILDING>
2023-08-24 16:45:37+0000 [-] starting build <Build s390x Fedora Rawhide LTO + PGO 3.x number:3627 results:success>.. pinging the worker <WorkerForBuilder builder='s390x Fedora Rawhide LTO + PGO 3.x' worker='edelsohn-fedora-rawhide-z' state=BUILDING>

Yesterday, a config change was made:

commit 6fed7ed896f67054c89428f0ef048f9f3a6b0098 (HEAD -> main, origin/main, origin/HEAD)
Author: Zachary Ware <[email protected]>
Date:   Thu Aug 24 11:38:26 2023 -0500

    Add issue template for worker addition (#383)

I suppose that the server was restarted to take in account this new config.

I restart the server to try to repair the issue:

 buildbot@buildbot:/srv/buildbot$ make stop-master
(...)
2023-08-25 09:54:04+0000 [-] Ignoring SIGTERM, master is already shutting down.

I have to kill the server :-(

buildbot@buildbot:/srv/buildbot$ ps ax|grep buildbot
1015538 pts/0    S      0:00 sudo -u buildbot -s -H
1016928 pts/0    S+     0:00 grep buildbot
3484334 ?        Sl   894:05 /srv/buildbot/venv/bin/python3.9 -c from twisted.scripts import twistd; twistd.run() --no_save --logfile=twistd.log --python=buildbot.tac
buildbot@buildbot:/srv/buildbot$ kill -9 3484334
buildbot@buildbot:/srv/buildbot$ ps ax|grep buildbot
1015538 pts/0    S      0:00 sudo -u buildbot -s -H
1017004 pts/0    S+     0:00 grep buildbot

I restarted the server:

$ make start-master

I see at least one issue: make stop-master is usually unable to stop the server. It seems like yesterday, deploying a new config triggered make stop-master. The server got SIGINT, stop starting new builds, but the server didn't stop neither, and so was not restarted properly.

Our database (state.sqlite) currently contains quite a lot of cruft, especially defunct workers and their associated builders. It also predates several foreign key relationships added to the buildbot database model due to issues with getting migrations to play nicely with sqlite, so we can't simply delete the old workers without picking through every table to clear out related data.

To remedy this, I'd like to simply remove the old database and start fresh. @pablogsal, @vstinner suggested that you may want to compute some statistics before doing so; do you, and if so do you need the database to be active or just need a copy?

The simplest way to do this (without undoing the restrictions I just put in place) is to add a worker-owners group based on notification emails and/or a GitHub group.

The worker at https://buildbot.python.org/all/#/workers/22

It has gone away, and doesn't seem like it'll ever come back. (I'm not thrilled about it, since I vouched for it being added, but it's out of my control.)

With GitHub auth now enabled, we can safely add force schedulers, including ones that accept repositories other than python/cpython (for instance, to allow testing code in your own or someone else's fork).

Suggested list of events:
'successToWarnings', 'exception', 'exceptionToSuccess', 'failureToWarnings', 'exceptionToFailure', 'successToFailure', 'failureToSuccess'

Seems that the builder cstratak-RHEL7-aarch64 doesn't have the correct credentials. CC: @stratakis

2021-09-21 10:45:54+0000 [Broker,40,10.132.169.157] invalid login from unknown user 'cstratak-RHEL7-aarch64'
2021-09-21 10:45:54+0000 [Broker,40,10.132.169.157] Peer will receive following PB traceback:
2021-09-21 10:45:54+0000 [Broker,40,10.132.169.157] Unhandled Error
        Traceback (most recent call last):
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/cred/portal.py", line 117, in login
            return maybeDeferred(
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 191, in maybeDeferred
            result = f(*args, **kwargs)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1909, in unwindGenerator
            return _cancellableInlineCallbacks(gen)  # type: ignore[unreachable]
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1816, in _cancellableInlineCallbacks
            _inlineCallbacks(None, gen, status)
        --- <exception caught here> ---
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
            result = current_context.run(gen.send, result)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/pbmanager.py", line 198, in requestAvatarId
            raise error.UnauthorizedLogin()
        twisted.cred.error.UnauthorizedLogin:

I have been looking everywhere to understand why I get errors when buildbot runs, but not when building manually.

One thing I found, that I think may be a typo is the argument passed to ./configure for --prefix.

In my local file I see, e.g.:
2019-03-04 02:24:13+0000 [HangCheckProtocol,client] ./configure --prefix '$(PWD)/target' --with-pydebug --with-openssl=/opt/aixtools --without-computed-gotos
2019-03-04 02:24:13+0000 [HangCheckProtocol,client] in dir /home/buildbot/buildarea/3.x.aixtools-aix-power6/build (timeout 1200 secs)
2019-03-04 02:24:13+0000 [HangCheckProtocol,client] watching logfiles {}
2019-03-04 02:24:13+0000 [HangCheckProtocol,client] argv: [b'./configure', b'--prefix', b'$(PWD)/target', b'--with-pydebug', b'--with-openssl=/opt/aixtools', b'--without-computed-gotos']

And I suspect that --prefix $(PWD)/target should be --prefix ${PWD}/target

I'll be digging here to see if I can find the file, on my own, and submit a PR.

Or - you can point me at it! :)

It should be feasible to add pre-merge PR testing, triggered by the addition of the awaiting merge label.

While synchronization may not be hot - I do notice what feel like incorrect names.

I would expect the basename, e.g. "AIXBuild" to be the generic aka starting point and then other names for when arguments are added. This was done - there is a classname AIXBuildWithGcc - except there is no configure argument "--with-gcc". Further, it magically deletes the argument "--without-computed-gotos" that the class AIXBuild has.

So, the coming PR will contain my suggestion:

classnames "AIXBuild", and "AIXBuildWithoutComputedGotos"

Currently, when "git clone" fails, the whole build is marked as an error:
https://buildbot.python.org/all/#/builders/57/builds/266

• job 0 "git" failed because "git clone" failed (network issue)
• job 1 "make distclean" failed because there is no Makefile: this one is fine, Clean step uses warnOnFailure = 1, it's just a warning

I failed to find the proper configuration to say: if git fails, give up on this build, but don't mark is as an error.

(venv) C:\Users\alexw\coding\buildmaster-config>python -c "import master.utils"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\alexw\coding\buildmaster-config\master\utils.py", line 3, in <module>
    from jinja2 import Markup
ImportError: cannot import name 'Markup' from 'jinja2' (C:\Users\alexw\coding\buildmaster-config\venv\Lib\site-packages\jinja2\__init__.py)

jinja.Markup doesn't exist with Jinja2==3.1.2.

Not sure if this submodule is really needed? (I don't really know anything about this codebase.)

Hi! I am planning on adding support for cross compiling and then Emscripten as a platform. However I am unsure how to set up a realistic testing environment.

I can run the current master code, but it seems that the buildbot master relies on a file /etc/buildbot/settings.yaml to configure workers.

I am unsure of what the format of the file should be for testing. Could you give an example of what this file should look like please? (I can guess some of the information from settings.py, but I don't know what the configuration looks like for workers)

The server used for AIX PPC64 builds has 16 core (64 hardware threads) and can easily support more parallelization during the bot tests.

Username

opsec

GitHub username

No response

Name

Kurt Jaeger

Email address

[email protected]

Password status

I will use an existing owner password

Processor architecture

amd64

Operating System

FreeBSD 15-CURRENT

Anything special about the worker?

No response

@paulproteus has offered to run a worker to test the ac_cv_posix_semaphores_enabled=no build that he's trying to fix in bpo-40692. I suggested to fix the chicken-and-egg situation of not wanting a constantly failing builder and not wanting to merge a fix for an untested platform by going ahead and setting up the worker to run a regular build until python/cpython#20239 is ready to merge, at which time we could switch it over to use ac_cv_posix_semaphores_enabled=no.

I've not yet had time and memory coincide to get it set up, so I'm opening this issue in case anyone else does have time :)

Can someone try to propose a PR on https://github.com/python/buildmaster-config/blob/master/master/custom/builders.py to remove Windows 7 from stable buildbots? That would remove them from stable on all Python branches.

The other option would be to hack https://github.com/python/buildmaster-config/blob/master/master/custom/workers.py to exclude 3.x branch from Windows 7 workers.

The last option is to hack https://github.com/python/buildmaster-config/blob/master/master/master.cfg to add a special rule to not schedule jobs on the 3.x branch if "win7" flag is present.

I'm not sure which approach is the best. Maybe @zware or @pablogsal have an opinion on that :-)

Username

itamaro

GitHub username

@itamaro

Name

Itamar Oren

Email address

No response

Password status

I will use an existing owner password

Processor architecture

amd64

Operating System

Ubuntu Linux

Anything special about the worker?

AWS hosted worker for NoGIL builds
see #389