Comments (22)
Or arrange a visit from you friendly neighborhood Linux guy :)
from awx-build.
Hi Anders
Have you followed the instructions here:
https://github.com/MrMEEE/awx-build/blob/master/migrations.md
??
from awx-build.
Yes, though not quite in order, I only noticed the CELERY_ROUTES change somewhat later. And the database change one-liner didn't work for me, but I was able to perform the changes in psql (entering via scl.)
I tried going over it again just now to be sure everything was done, and get this:
[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations --merge
No conflicts detected to merge.
[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage migrate
Operations to perform:
Apply all migrations: auth, conf, contenttypes, djcelery, main, oauth2_provider, sessions, sites, social_django, sso, taggit
Running migrations:
No migrations to apply.
Your models have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
I didn't see that earlier, but it seems to suggesting running makemigrations without --merge? Gave it a try:
[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations
You are trying to change the nullable field 'modified' on oauth2accesstoken to non-nullable without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
2) Ignore for now, and let me handle existing rows with NULL myself (e.g. because you added a RunPython or RunSQL operation to handle NULL values in a previous data migration)
3) Quit, and let me add a default in models.py
Select an option:
So... this seems like a good time to get a coffee...
from awx-build.
hmmm :)..
what happens if you run makemigrations with --merge?
from awx-build.
Same as above, it was maybe hidden in the paste...
[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations --merge
No conflicts detected to merge.
from awx-build.
I would try with option number 1.. :)
from awx-build.
What do I give as the default?
[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations
You are trying to change the nullable field 'modified' on oauth2accesstoken to non-nullable without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
2) Ignore for now, and let me handle existing rows with NULL myself (e.g. because you added a RunPython or RunSQL operation to handle NULL values in a previous data migration)
3) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>>
from awx-build.
timezone.now
will probably do?
from awx-build.
hum during my multiple test i saw this.
If i remember it's because you run the makemigrations after inserting the null=True in the file
i think i have go with the option 1 too
from awx-build.
That seemed to have resolved the awx-channels-worker issue, after restarting services it ran fine for a while...
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> timezone.now
Migrations for 'main':
/opt/awx/embedded/lib/python2.7/site-packages/awx/main/migrations/0048_auto_20180821_1309.py
- Alter field o_auth2_access_token on activitystream
- Alter field o_auth2_application on activitystream
- Alter field created_by on credential
- Alter field modified_by on credential
- Alter field created_by on credentialtype
- Alter field modified_by on credentialtype
- Alter field created_by on custominventoryscript
- Alter field modified_by on custominventoryscript
- Alter field created_by on group
- Alter field modified_by on group
- Alter field created_by on host
- Alter field modified_by on host
- Alter field created_by on inventory
- Alter field modified_by on inventory
- Alter field inventory_update on inventoryupdateevent
- Alter field created_by on label
- Alter field modified_by on label
- Alter field notification_type on notification
- Alter field created_by on notificationtemplate
- Alter field modified_by on notificationtemplate
- Alter field notification_type on notificationtemplate
- Alter field modified on oauth2accesstoken
- Alter field scope on oauth2accesstoken
- Alter field created_by on organization
- Alter field modified_by on organization
- Alter field project_update on projectupdateevent
- Alter field created_by on schedule
- Alter field modified_by on schedule
- Alter field system_job on systemjobevent
- Alter field created_by on team
- Alter field modified_by on team
- Alter field created_by on unifiedjob
- Alter field modified_by on unifiedjob
- Alter field created_by on unifiedjobtemplate
- Alter field modified_by on unifiedjobtemplate
- Alter index_together for inventoryupdateevent (3 constraint(s))
- Alter index_together for projectupdateevent (4 constraint(s))
- Alter index_together for systemjobevent (3 constraint(s))
from awx-build.
And then stopped?
from awx-build.
[root@dc0awx01 ~]# systemctl status awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
● awx-celery-worker.service - AWX Celery worker service
Loaded: loaded (/usr/lib/systemd/system/awx-celery-worker.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
Main PID: 1002 (python2)
CGroup: /system.slice/awx-celery-worker.service
├─1002 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
├─1549 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
├─1550 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
├─1551 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
├─1552 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
└─1553 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
Aug 21 15:14:45 dc0awx01 celery[1002]: [2018-08-21 13:14:45,744: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 16.0 seconds...
Aug 21 15:15:01 dc0awx01 celery[1002]: [2018-08-21 13:15:01,767: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Aug 21 15:15:01 dc0awx01 celery[1002]: Trying again in 18.00 seconds...
Aug 21 15:15:01 dc0awx01 celery[1002]: [2018-08-21 13:15:01,768: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 18.0 seconds...
Aug 21 15:15:19 dc0awx01 celery[1002]: [2018-08-21 13:15:19,794: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Aug 21 15:15:19 dc0awx01 celery[1002]: Trying again in 20.00 seconds...
Aug 21 15:15:19 dc0awx01 celery[1002]: [2018-08-21 13:15:19,795: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 20.0 seconds...
Aug 21 15:15:39 dc0awx01 celery[1002]: [2018-08-21 13:15:39,831: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 22.0 seconds...
Aug 21 15:15:39 dc0awx01 celery[1002]: [2018-08-21 13:15:39,832: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Aug 21 15:15:39 dc0awx01 celery[1002]: Trying again in 22.00 seconds...
● awx-cbreceiver.service - AWX cbreceiver service
Loaded: loaded (/usr/lib/systemd/system/awx-cbreceiver.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
Main PID: 1008 (python2)
CGroup: /system.slice/awx-cbreceiver.service
├─1008 python2 /opt/awx/bin/awx-manage run_callback_receiver
├─1542 python2 /opt/awx/bin/awx-manage run_callback_receiver
├─1543 python2 /opt/awx/bin/awx-manage run_callback_receiver
├─1544 python2 /opt/awx/bin/awx-manage run_callback_receiver
└─1545 python2 /opt/awx/bin/awx-manage run_callback_receiver
Aug 21 15:13:42 dc0awx01 systemd[1]: Started AWX cbreceiver service.
Aug 21 15:13:42 dc0awx01 systemd[1]: Starting AWX cbreceiver service...
● awx-celery-beat.service - AWX Celery beat service
Loaded: loaded (/usr/lib/systemd/system/awx-celery-beat.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
Main PID: 1003 (python2)
CGroup: /system.slice/awx-celery-beat.service
└─1003 python2 /opt/awx/bin/celery beat -A awx -l info --pidfile= -s /var/lib/awx/beat.db
Aug 21 15:13:50 dc0awx01 celery[1003]: [2018-08-21 13:13:50,712: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 4.0 seconds...
Aug 21 15:13:55 dc0awx01 celery[1003]: [2018-08-21 13:13:55,180: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 6.0 seconds...
Aug 21 15:14:01 dc0awx01 celery[1003]: [2018-08-21 13:14:01,194: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 8.0 seconds...
Aug 21 15:14:09 dc0awx01 celery[1003]: [2018-08-21 13:14:09,219: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 10.0 seconds...
Aug 21 15:14:19 dc0awx01 celery[1003]: [2018-08-21 13:14:19,241: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 12.0 seconds...
Aug 21 15:14:31 dc0awx01 celery[1003]: [2018-08-21 13:14:31,265: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 14.0 seconds...
Aug 21 15:14:45 dc0awx01 celery[1003]: [2018-08-21 13:14:45,292: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 16.0 seconds...
Aug 21 15:15:01 dc0awx01 celery[1003]: [2018-08-21 13:15:01,318: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 18.0 seconds...
Aug 21 15:15:19 dc0awx01 celery[1003]: [2018-08-21 13:15:19,344: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 20.0 seconds...
Aug 21 15:15:39 dc0awx01 celery[1003]: [2018-08-21 13:15:39,381: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 22.0 seconds...
● awx-channels-worker.service - AWX channels worker service
Loaded: loaded (/usr/lib/systemd/system/awx-channels-worker.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-08-21 15:15:53 CEST; 2s ago
Main PID: 2076 (python2)
CGroup: /system.slice/awx-channels-worker.service
└─2076 python2 /opt/awx/bin/awx-manage runworker --only-channels websocket.*
Aug 21 15:15:53 dc0awx01 systemd[1]: Started AWX channels worker service.
Aug 21 15:15:53 dc0awx01 systemd[1]: Starting AWX channels worker service...
● awx-daphne.service - AWX daphne service
Loaded: loaded (/usr/lib/systemd/system/awx-daphne.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
Main PID: 1006 (python2)
CGroup: /system.slice/awx-daphne.service
└─1006 python2 /opt/awx/bin/daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer
Aug 21 15:15:08 dc0awx01 daphne[1006]: 2018-08-21 13:15:08,194 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:13 dc0awx01 daphne[1006]: 2018-08-21 13:15:13,211 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:18 dc0awx01 daphne[1006]: 2018-08-21 13:15:18,218 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:23 dc0awx01 daphne[1006]: 2018-08-21 13:15:23,226 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:28 dc0awx01 daphne[1006]: 2018-08-21 13:15:28,234 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:33 dc0awx01 daphne[1006]: 2018-08-21 13:15:33,241 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:38 dc0awx01 daphne[1006]: 2018-08-21 13:15:38,254 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:43 dc0awx01 daphne[1006]: 2018-08-21 13:15:43,265 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:48 dc0awx01 daphne[1006]: 2018-08-21 13:15:48,282 ERROR Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:53 dc0awx01 daphne[1006]: 2018-08-21 13:15:53,295 ERROR Error trying to receive messages: [Errno 111] Connection refused
● awx-web.service - AWX web service
Loaded: loaded (/usr/lib/systemd/system/awx-web.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
Main PID: 1014 (uwsgi)
CGroup: /system.slice/awx-web.service
├─1014 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
├─1088 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
├─1089 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
├─1090 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
├─1091 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
└─1092 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
Aug 21 15:13:42 dc0awx01 systemd[1]: Started AWX web service.
Aug 21 15:13:42 dc0awx01 systemd[1]: Starting AWX web service...
Think I'm getting close to calling my friendly Linux guy...
from awx-build.
check if rabbitmq is running :p
from awx-build.
There we go! rabbitmq-server wasn't enabled in systemd and I hadn't thought to check it. With that started (and enabled!) all services restart without errors and the web page displays "AWX is upgrading."
from awx-build.
wuhuu..
from awx-build.
@anlag could you confirm that you successfully upgraded from 1.0.6.X to 1.0.7.X ?
If possible could you share your final steps to achieve it ?
Thanks.
from awx-build.
I'm not actually sure, my AWX still says it's upgrading after an hour and a half. I'll check back tomorrow, but the short version of what I did was "yum upgrade" from the repo, apply the fixes in
https://github.com/MrMEEE/awx-build/blob/master/migrations.md
and lastly make sure rabbitmq-server is running.
from awx-build.
Ok thanks for your feedback.
One hour and a half seems bad... Hope I'm wrong !
We need to figure out on the cluster version with @Aglidic...
from awx-build.
It shouldn't have the upgrading screen when we have done the manual update with migrate/makemigrations..
Have you tried restarting the services..??
from awx-build.
I had restarted the services. Today I've done so again and then rebooted the server, unfortunately I still get the upgrading screen when I try to access the web UI. Looking at the services, I can see awx-daphne now produces an error:
[root@dc0awx01 ~]# systemctl status awx-daphne.service
● awx-daphne.service - AWX daphne service
Loaded: loaded (/usr/lib/systemd/system/awx-daphne.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-08-22 11:47:55 CEST; 1h 11min ago
Main PID: 1022 (python2)
CGroup: /system.slice/awx-daphne.service
└─1022 python2 /opt/awx/bin/daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer
Aug 22 11:47:55 dc0awx01 systemd[1]: Started AWX daphne service.
Aug 22 11:47:55 dc0awx01 systemd[1]: Starting AWX daphne service...
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,721 INFO Starting server at tcp:port=8051:interface=127.0.0.1, channel layer awx.asgi:channel_layer.
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,722 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,722 INFO Using busy-loop synchronous mode on channel layer
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,722 INFO Listening on endpoint tcp:port=8051:interface=127.0.0.1
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,800 ERROR Error trying to receive messages: [Errno 111] Connection refused
Everything else seems to run okay.
from awx-build.
Again, doublecheck that the rabbitmq is running....
from awx-build.
rabbitmq was running and awx-daphne didn't show that error again after I restarted only that. But as my friendly neighbourhood Linux guy concluded upon closer inspection, I hadn't run "awx-manage migrate" again after setting that new default with "awx-manage makemigrations" as detailed earlier in the thread. Executing "awx-manage migrate" now resolved the issue and the server seems to be running fine now with 1.0.7.3.
from awx-build.
Related Issues (20)
- ar
- Upgrade from 9.1.1 to the latest version HOT 2
- Project awx-build HOT 3
- rpm.miracle.dk unavailable. HOT 8
- install rabbitmq-server - HTTPS Error 403 - Forbidden HOT 1
- No directory for Centos7 on https://rpm.miracle.dk/AWX-RPM HOT 2
- The CentOS_7 repo has x86_64, noarch, and source all in the same repodata HOT 3
- .spec files from SRPMs use "python3-" rather than "python%{python3_pkgversion}" HOT 3
- Stop setting "%release" twice HOT 1
- Name .spec files consistently HOT 1
- .spec files lack %license and %doc files, and lack '%check' or documenting stanzas HOT 1
- Add '%{?dist}' to "Release:" line on all .spec files HOT 1
- AWX Installation on RHEL 8.X HOT 1
- Status of awx-rpm project ? HOT 1
- Repo not working HOT 3
- Unable to update projects in AWX-RPM HOT 1
- Latest release not quite ready for RHEL7 out of the box
- Issues during install "awx-manage not found" HOT 2
- error while install centos8 HOT 2
- Need steps to upgrade awxtower 11.2.0 to latest version HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from awx-build.