buildbot / buildbot-docker-example-config Goto Github PK

View Code? Open in Web Editor NEW

35.0 15.0 182.0 3.04 MB

example configuration for running buildbot in docker

Python 70.73% Shell 27.95% Dockerfile 1.32%

buildbot-docker-example-config's Introduction

buildbot-docker-example-config

example configuration for running buildbot in docker

buildbot-docker-example-config's People

Contributors

Stargazers

Watchers

Forkers

tardyp digitalgeko capitalaslash sleepyjames blag djalexd erdc giltayar christophert rightisleft pop mvock larver-imvu nicolai86 cabotcomm oftl hashdatainc chandrasekharvv hbousse ppadamcz zanssa sebastiendarocha mrkeiser nordgedanken gyuvarajme hartym pentve violake jessecooper jbarney-sl kleopatra999 francisjen tcrst lfany markb968 codevbus zhangsirui jasongiedymin matt84104 scoday eragnms ggershoni perlun chigraph fchavant jstacoder oprietop rongcong uazw stealth251 stinkyfingers drudi olivier-o a114m gregorydlogan mbacchi jried31 borgified chenkai0567 fabricetriboix tototyty chrisvire carlessanagustin muodov iotile jimmykim1 afcarl caspx sehrich tsanghan scottrnpe riziero bdrydyk junyij yunusem brandonawells sonictt1 kefavn adrienjoly crazyrex arne-w binderdavid whitemoondreamsinc thanhtruongnhu nakyong tim-leblanc twaddell49 viniciusks wkarcher shenry07 sampx cupermind ismailhammounou benjaminhar benharosh ktjooho kolszewskieu cko-bz leaguilar rajgoesout

buildbot-docker-example-config's Issues

Restarting worker fails

Hello,

when my docker-compose is not shutdown gracefully, the buildbot-worker cannot restart : ( "Another twistd server is running, PID 5" "simple_worker_1 exited with code 1" ).

To reproduce :

start a simple docker-example-config : #docker-compose up
check the UI works and the worker is started
force shutdown with a double Ctrl-C
start again : #docker-compose up
I have an error on my worker container :
worker_1 | Another twistd server is running, PID 5
worker_1 |
worker_1 | This could either be a previously started instance of your application or a
worker_1 | different application entirely. To start a new one, either run it in some other
worker_1 | directory, or use the --pidfile and --logfile parameters to avoid clashes.
worker_1 |
simple_worker_1 exited with code 1

To clean :

find the id of the worker container : # docker ps -a | grep buildbot-worker
rm this container : # docker rm 75f48fd8c417
start again : #docker-compose up
it works again

Remark :

Apparently even when shutdown gracefully, I have the same problem

Thanks !

Working docker example

Hey @tardyp!

Just thought I'd share my working version of a minimal docker+buildbot setup. Complete with the new user-defined networks instead of links.

Thanks for carrying the torch for the community. Take care!

https://github.com/jgalvan2/buildbot-docker

When running single buildbot, it fails : download

Hello All,

I did 'docker-compose up' at simple directory.
It gave error like below

buildbot_1 | tar: Child returned status 1
buildbot_1 | tar: Error is not recoverable: exiting now
buildbot_1 | Can't download from $BUILDBOT_CONFIG_URL: https://github.com/buildbot/buildbot-docker-example-config/archive/master.tar.gz
buildbot_1 | gzip: invalid magic
buildbot_1 | tar: Child returned status 1
buildbot_1 | tar: Error is not recoverable: exiting now
buildbot_1 | Can't download from $BUILDBOT_CONFIG_URL: https://github.com/buildbot/buildbot-docker-example-config/archive/master.tar.gz
buildbot_1 | gzip: invalid magic

When i download with curl, it downloaded well.
Could someone help me?

curl -O https://github.com/buildbot/buildbot-docker-example-config/archive/master.tar.gz

[dumb-init] /usr/src/buildbot/contrib/docker/master/start_buildbot.sh: No such file or directory

After a fresh install of Docker and Docker-Compose on Ubuntu 16.04.2 x64 and following the First Run Guide for Docker, i get the following error when trying to spin up all containers.

root@buildbot:~/buildbot-docker-example-config/simple# docker-compose up
Starting simple_db_1
Starting simple_buildbot_1
Starting simple_worker_1
Attaching to simple_db_1, simple_buildbot_1, simple_worker_1
db_1        | LOG:  database system was shut down at 2017-03-03 12:53:22 UTC
db_1        | LOG:  MultiXact member wraparound protections are now enabled
db_1        | LOG:  database system is ready to accept connections
db_1        | LOG:  autovacuum launcher started
buildbot_1  | [dumb-init] /usr/src/buildbot/contrib/docker/master/start_buildbot.sh: No such file or directory
simple_buildbot_1 exited with code 2
worker_1    | 2017-03-03 12:55:08+0000 [-] Loading buildbot.tac...
worker_1    | 2017-03-03 12:55:09+0000 [-] Loaded.
worker_1    | 2017-03-03 12:55:09+0000 [-] twistd 17.1.0 (/usr/bin/python 2.7.12) starting up.
worker_1    | 2017-03-03 12:55:09+0000 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
worker_1    | 2017-03-03 12:55:09+0000 [-] Starting Worker -- version: latest
worker_1    | 2017-03-03 12:55:09+0000 [-] recording hostname in twistd.hostname
worker_1    | 2017-03-03 12:55:09+0000 [-] Starting factory <buildbot_worker.pb.BotFactory instance at 0x7efea0797b48>
worker_1    | 2017-03-03 12:55:09+0000 [-] Connecting to buildbot:9989
worker_1    | 2017-03-03 12:55:09+0000 [-] Connection to buildbot:9989 failed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.DNSLookupError'>: DNS lookup failed: buildbot.

Docker Version: 17.03.0-ce
Docker-Compose Version: 1.11.2, build dfed245

Apparently start_buildbot.sh is not found inside the buildbot-master image. When trying to build a Docker image from the original Dockerfile i get the error:

Directory '/usr/src/buildbot' is not installable. File 'setup.py' not found.

Did i miss any configuration?

multi master configuration tag should be master not latest

There does not appear to be a latest tag on the builedbot. The simple configuration uses master but the multimaster configuration uses latest which is reported as not found. If you change it master it works. I can issue a pull request is you like.

c++ and css builds

write code to different builds to enhance availability

Unknown endpoint type: '8010' while starting simple config

After cloning the repository and starting with docker-compose up I ran into this:

buildbot_1  |   Traceback (most recent call last):
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
buildbot_1  |       current.result = callback(current.result, *args, **kw)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
buildbot_1  |       _inlineCallbacks(r, g, deferred)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
buildbot_1  |       result = result.throwExceptionIntoGenerator(g)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
buildbot_1  |       return g.throw(self.type, self.value, self.tb)
buildbot_1  |   --- <exception caught here> ---
buildbot_1  |     File "/usr/lib/python2.7/site-packages/buildbot/master.py", line 304, in startService
buildbot_1  |       yield self.reconfigServiceWithBuildbotConfig(self.config)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
buildbot_1  |       result = result.throwExceptionIntoGenerator(g)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
buildbot_1  |       return g.throw(self.type, self.value, self.tb)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/buildbot/util/service.py", line 54, in reconfigServiceWithBuildbotConfig
buildbot_1  |       yield svc.reconfigServiceWithBuildbotConfig(new_config)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
buildbot_1  |       result = g.send(result)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/buildbot/www/service.py", line 228, in reconfigServiceWithBuildbotConfig
buildbot_1  |       self.port_service = strports.service(port, self.site)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/application/strports.py", line 40, in service
buildbot_1  |       endpoints.serverFromString(reactor, description), factory)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1703, in serverFromString
buildbot_1  |       nameOrPlugin, args, kw = _parseServer(description, None)
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1621, in _parseServer
buildbot_1  |       getPlugins(IStreamServerEndpointStringParser), endpointType
buildbot_1  |     File "/usr/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1637, in _matchPluginToPrefix
buildbot_1  |       raise ValueError("Unknown endpoint type: '%s'" % (endpointType,))
buildbot_1  |   exceptions.ValueError: Unknown endpoint type: '8010'

simple buildbot_1 fails with ImportError: No module named psycopg2

When running docker-compose up with the simple example, I get the following over and over:

buildbot_1  | checking for running master
buildbot_1  | checking master.cfg
buildbot_1  | upgrading basedir
buildbot_1  | upgrading database (postgresql+psycopg2://buildbot:xxxx@db/buildbot)
buildbot_1  | Can't upgrade master yet. Waiting for database ready?
buildbot_1  | checking basedir
buildbot_1  | /usr/lib/python2.7/site-packages/buildbot/config.py:102: ConfigWarning: [0.9.0 and later] `buildbotNetUsageData` is not configured and defaults to
basic.
buildbot_1  | This parameter helps the buildbot development team to understand the installation base.
buildbot_1  | No personal information is collected.
buildbot_1  | Only installation software version info and plugin usage is sent.
buildbot_1  | You can `opt-out` by setting this variable to None.
buildbot_1  | Or `opt-in` for more information by setting it to "full".
buildbot_1  |
buildbot_1  |   category=ConfigWarning,
buildbot_1  | /usr/lib/python2.7/site-packages/buildbot/config.py:102: ConfigWarning: [0.9.0 and later] NOTE: `status` targets are deprecated and ignored They are replaced by reporters
buildbot_1  |   category=ConfigWarning,
buildbot_1  | problem while upgrading!:
buildbot_1  | Traceback (most recent call last):
buildbot_1  |   File "/usr/lib/python2.7/site-packages/buildbot/scripts/upgrade_master.py", line 121, in _upgradeMaster
buildbot_1  |     yield upgradeDatabase(config, master_cfg)
buildbot_1  | ImportError: No module named psycopg2

I tried installing psycopg2 manually, but it fails because it can't find the pg_config executable.

/var/lib/buildbot # pip install psycopg2
Collecting psycopg2
  Downloading https://files.pythonhosted.org/packages/74/83/51580322ed0e82cba7ad8e0af590b8fb2cf11bd5aaa1ed872661bd36f462/psycopg2-2.7.4.tar.gz (425kB)
    100% |████████████████████████████████| 430kB 4.4MB/s
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).


    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Q8qcHR/psycopg2/

When I try to install psycopg2-binary, it still tries to build from source (I'm guessing because none of the wheels are compatible?).

/var/lib/buildbot # pip install psycopg2-binary
Collecting psycopg2-binary
  Downloading https://files.pythonhosted.org/packages/77/09/4991fcd9a8f4bea1ee3948e1729fa17c184d25bd10809bacc143626361b9/psycopg2-binary-2.7.4.tar.gz (426kB)
    100% |████████████████████████████████| 430kB 6.4MB/s
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2_binary.egg-info
    writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).


    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-2kFAwj/psycopg2-binary/

I believe that libpq-dev needs to be installed: http://initd.org/psycopg/docs/install.html#build-prerequisites for this installation to work properly.

can't fork this example to private repo

Hello,

I have a private repo and I can't find a way to fork this repo to setup the master.cfg. If I fork this repo then my config will be world readable. Can I setup a config variable or some way to keep my master.cfg private and not open on github?

Thanks

start_buildbot.sh fails to master.cfg

The script fails to download throwing the following error, initially, I supplied the GitHub url directly earlier but, the script would download a "page not found" from GitHub instead but both urls actually work from the browser...

buildbot_1  | gzip: invalid magic
buildbot_1  | tar: Child returned status 1
buildbot_1  | tar: Error is not recoverable: exiting now
buildbot_1  | Can't download from $BUILDBOT_CONFIG_URL:  https://github.com/Submitty/submitty-buildbot/archive/master.tar.gz

duplicate masters on container down/up

Bringing containers down and up with a persistent storage directory results in a trail of "master" buildbot's due to the random hostname.