Git Product home page Git Product logo

Comments (22)

MrMEEE avatar MrMEEE commented on August 23, 2024 1

Or arrange a visit from you friendly neighborhood Linux guy :)

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

Hi Anders

Have you followed the instructions here:

https://github.com/MrMEEE/awx-build/blob/master/migrations.md

??

from awx-build.

anlag avatar anlag commented on August 23, 2024

Yes, though not quite in order, I only noticed the CELERY_ROUTES change somewhat later. And the database change one-liner didn't work for me, but I was able to perform the changes in psql (entering via scl.)

I tried going over it again just now to be sure everything was done, and get this:

[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations --merge
No conflicts detected to merge.
[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage migrate
Operations to perform:
  Apply all migrations: auth, conf, contenttypes, djcelery, main, oauth2_provider, sessions, sites, social_django, sso, taggit
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

I didn't see that earlier, but it seems to suggesting running makemigrations without --merge? Gave it a try:

[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations
You are trying to change the nullable field 'modified' on oauth2accesstoken to non-nullable without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Ignore for now, and let me handle existing rows with NULL myself (e.g. because you added a RunPython or RunSQL operation to handle NULL values in a previous data migration)
 3) Quit, and let me add a default in models.py
Select an option:

So... this seems like a good time to get a coffee...

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

hmmm :)..

what happens if you run makemigrations with --merge?

from awx-build.

anlag avatar anlag commented on August 23, 2024

Same as above, it was maybe hidden in the paste...

[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations --merge
No conflicts detected to merge.

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

I would try with option number 1.. :)

from awx-build.

anlag avatar anlag commented on August 23, 2024

What do I give as the default?

[root@dc0awx01 log]# sudo -u awx /opt/awx/bin/awx-manage makemigrations
You are trying to change the nullable field 'modified' on oauth2accesstoken to non-nullable without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Ignore for now, and let me handle existing rows with NULL myself (e.g. because you added a RunPython or RunSQL operation to handle NULL values in a previous data migration)
 3) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>>

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

timezone.now
will probably do?

from awx-build.

Aglidic avatar Aglidic commented on August 23, 2024

hum during my multiple test i saw this.

If i remember it's because you run the makemigrations after inserting the null=True in the file

i think i have go with the option 1 too

from awx-build.

anlag avatar anlag commented on August 23, 2024

That seemed to have resolved the awx-channels-worker issue, after restarting services it ran fine for a while...

Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now
Type 'exit' to exit this prompt
>>> timezone.now
Migrations for 'main':
  /opt/awx/embedded/lib/python2.7/site-packages/awx/main/migrations/0048_auto_20180821_1309.py
    - Alter field o_auth2_access_token on activitystream
    - Alter field o_auth2_application on activitystream
    - Alter field created_by on credential
    - Alter field modified_by on credential
    - Alter field created_by on credentialtype
    - Alter field modified_by on credentialtype
    - Alter field created_by on custominventoryscript
    - Alter field modified_by on custominventoryscript
    - Alter field created_by on group
    - Alter field modified_by on group
    - Alter field created_by on host
    - Alter field modified_by on host
    - Alter field created_by on inventory
    - Alter field modified_by on inventory
    - Alter field inventory_update on inventoryupdateevent
    - Alter field created_by on label
    - Alter field modified_by on label
    - Alter field notification_type on notification
    - Alter field created_by on notificationtemplate
    - Alter field modified_by on notificationtemplate
    - Alter field notification_type on notificationtemplate
    - Alter field modified on oauth2accesstoken
    - Alter field scope on oauth2accesstoken
    - Alter field created_by on organization
    - Alter field modified_by on organization
    - Alter field project_update on projectupdateevent
    - Alter field created_by on schedule
    - Alter field modified_by on schedule
    - Alter field system_job on systemjobevent
    - Alter field created_by on team
    - Alter field modified_by on team
    - Alter field created_by on unifiedjob
    - Alter field modified_by on unifiedjob
    - Alter field created_by on unifiedjobtemplate
    - Alter field modified_by on unifiedjobtemplate
    - Alter index_together for inventoryupdateevent (3 constraint(s))
    - Alter index_together for projectupdateevent (4 constraint(s))
    - Alter index_together for systemjobevent (3 constraint(s))

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

And then stopped?

from awx-build.

anlag avatar anlag commented on August 23, 2024 
[root@dc0awx01 ~]# systemctl status awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
● awx-celery-worker.service - AWX Celery worker service
   Loaded: loaded (/usr/lib/systemd/system/awx-celery-worker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
 Main PID: 1002 (python2)
   CGroup: /system.slice/awx-celery-worker.service
           ├─1002 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
           ├─1549 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
           ├─1550 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
           ├─1551 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
           ├─1552 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01
           └─1553 python2 /opt/awx/bin/celery worker -A awx -B -l info --autoscale=50,4 -Ofair -s /var/lib/awx/beat.db -n celery@dc0awx01

Aug 21 15:14:45 dc0awx01 celery[1002]: [2018-08-21 13:14:45,744: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 16.0 seconds...
Aug 21 15:15:01 dc0awx01 celery[1002]: [2018-08-21 13:15:01,767: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Aug 21 15:15:01 dc0awx01 celery[1002]: Trying again in 18.00 seconds...
Aug 21 15:15:01 dc0awx01 celery[1002]: [2018-08-21 13:15:01,768: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 18.0 seconds...
Aug 21 15:15:19 dc0awx01 celery[1002]: [2018-08-21 13:15:19,794: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Aug 21 15:15:19 dc0awx01 celery[1002]: Trying again in 20.00 seconds...
Aug 21 15:15:19 dc0awx01 celery[1002]: [2018-08-21 13:15:19,795: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 20.0 seconds...
Aug 21 15:15:39 dc0awx01 celery[1002]: [2018-08-21 13:15:39,831: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 22.0 seconds...
Aug 21 15:15:39 dc0awx01 celery[1002]: [2018-08-21 13:15:39,832: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Aug 21 15:15:39 dc0awx01 celery[1002]: Trying again in 22.00 seconds...

● awx-cbreceiver.service - AWX cbreceiver service
   Loaded: loaded (/usr/lib/systemd/system/awx-cbreceiver.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
 Main PID: 1008 (python2)
   CGroup: /system.slice/awx-cbreceiver.service
           ├─1008 python2 /opt/awx/bin/awx-manage run_callback_receiver
           ├─1542 python2 /opt/awx/bin/awx-manage run_callback_receiver
           ├─1543 python2 /opt/awx/bin/awx-manage run_callback_receiver
           ├─1544 python2 /opt/awx/bin/awx-manage run_callback_receiver
           └─1545 python2 /opt/awx/bin/awx-manage run_callback_receiver

Aug 21 15:13:42 dc0awx01 systemd[1]: Started AWX cbreceiver service.
Aug 21 15:13:42 dc0awx01 systemd[1]: Starting AWX cbreceiver service...

● awx-celery-beat.service - AWX Celery beat service
   Loaded: loaded (/usr/lib/systemd/system/awx-celery-beat.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
 Main PID: 1003 (python2)
   CGroup: /system.slice/awx-celery-beat.service
           └─1003 python2 /opt/awx/bin/celery beat -A awx -l info --pidfile= -s /var/lib/awx/beat.db

Aug 21 15:13:50 dc0awx01 celery[1003]: [2018-08-21 13:13:50,712: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 4.0 seconds...
Aug 21 15:13:55 dc0awx01 celery[1003]: [2018-08-21 13:13:55,180: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 6.0 seconds...
Aug 21 15:14:01 dc0awx01 celery[1003]: [2018-08-21 13:14:01,194: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 8.0 seconds...
Aug 21 15:14:09 dc0awx01 celery[1003]: [2018-08-21 13:14:09,219: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 10.0 seconds...
Aug 21 15:14:19 dc0awx01 celery[1003]: [2018-08-21 13:14:19,241: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 12.0 seconds...
Aug 21 15:14:31 dc0awx01 celery[1003]: [2018-08-21 13:14:31,265: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 14.0 seconds...
Aug 21 15:14:45 dc0awx01 celery[1003]: [2018-08-21 13:14:45,292: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 16.0 seconds...
Aug 21 15:15:01 dc0awx01 celery[1003]: [2018-08-21 13:15:01,318: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 18.0 seconds...
Aug 21 15:15:19 dc0awx01 celery[1003]: [2018-08-21 13:15:19,344: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 20.0 seconds...
Aug 21 15:15:39 dc0awx01 celery[1003]: [2018-08-21 13:15:39,381: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 22.0 seconds...
● awx-channels-worker.service - AWX channels worker service
   Loaded: loaded (/usr/lib/systemd/system/awx-channels-worker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-21 15:15:53 CEST; 2s ago
 Main PID: 2076 (python2)
   CGroup: /system.slice/awx-channels-worker.service
           └─2076 python2 /opt/awx/bin/awx-manage runworker --only-channels websocket.*

Aug 21 15:15:53 dc0awx01 systemd[1]: Started AWX channels worker service.
Aug 21 15:15:53 dc0awx01 systemd[1]: Starting AWX channels worker service...

● awx-daphne.service - AWX daphne service
   Loaded: loaded (/usr/lib/systemd/system/awx-daphne.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
 Main PID: 1006 (python2)
   CGroup: /system.slice/awx-daphne.service
           └─1006 python2 /opt/awx/bin/daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer

Aug 21 15:15:08 dc0awx01 daphne[1006]: 2018-08-21 13:15:08,194 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:13 dc0awx01 daphne[1006]: 2018-08-21 13:15:13,211 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:18 dc0awx01 daphne[1006]: 2018-08-21 13:15:18,218 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:23 dc0awx01 daphne[1006]: 2018-08-21 13:15:23,226 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:28 dc0awx01 daphne[1006]: 2018-08-21 13:15:28,234 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:33 dc0awx01 daphne[1006]: 2018-08-21 13:15:33,241 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:38 dc0awx01 daphne[1006]: 2018-08-21 13:15:38,254 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:43 dc0awx01 daphne[1006]: 2018-08-21 13:15:43,265 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:48 dc0awx01 daphne[1006]: 2018-08-21 13:15:48,282 ERROR    Error trying to receive messages: [Errno 111] Connection refused
Aug 21 15:15:53 dc0awx01 daphne[1006]: 2018-08-21 13:15:53,295 ERROR    Error trying to receive messages: [Errno 111] Connection refused

● awx-web.service - AWX web service
   Loaded: loaded (/usr/lib/systemd/system/awx-web.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-21 15:13:42 CEST; 2min 13s ago
 Main PID: 1014 (uwsgi)
   CGroup: /system.slice/awx-web.service
           ├─1014 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
           ├─1088 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
           ├─1089 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
           ├─1090 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
           ├─1091 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...
           └─1092 /opt/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/awx/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri...

Aug 21 15:13:42 dc0awx01 systemd[1]: Started AWX web service.
Aug 21 15:13:42 dc0awx01 systemd[1]: Starting AWX web service...

Think I'm getting close to calling my friendly Linux guy...

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

check if rabbitmq is running :p

from awx-build.

anlag avatar anlag commented on August 23, 2024

There we go! rabbitmq-server wasn't enabled in systemd and I hadn't thought to check it. With that started (and enabled!) all services restart without errors and the web page displays "AWX is upgrading."

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

wuhuu..

from awx-build.

powertim avatar powertim commented on August 23, 2024

@anlag could you confirm that you successfully upgraded from 1.0.6.X to 1.0.7.X ?
If possible could you share your final steps to achieve it ?

Thanks.

from awx-build.

anlag avatar anlag commented on August 23, 2024

I'm not actually sure, my AWX still says it's upgrading after an hour and a half. I'll check back tomorrow, but the short version of what I did was "yum upgrade" from the repo, apply the fixes in

https://github.com/MrMEEE/awx-build/blob/master/migrations.md

and lastly make sure rabbitmq-server is running.

from awx-build.

powertim avatar powertim commented on August 23, 2024

Ok thanks for your feedback.
One hour and a half seems bad... Hope I'm wrong !
We need to figure out on the cluster version with @Aglidic...

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

It shouldn't have the upgrading screen when we have done the manual update with migrate/makemigrations..

Have you tried restarting the services..??

from awx-build.

anlag avatar anlag commented on August 23, 2024

I had restarted the services. Today I've done so again and then rebooted the server, unfortunately I still get the upgrading screen when I try to access the web UI. Looking at the services, I can see awx-daphne now produces an error:

[root@dc0awx01 ~]# systemctl status awx-daphne.service
● awx-daphne.service - AWX daphne service
   Loaded: loaded (/usr/lib/systemd/system/awx-daphne.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-08-22 11:47:55 CEST; 1h 11min ago
 Main PID: 1022 (python2)
   CGroup: /system.slice/awx-daphne.service
           └─1022 python2 /opt/awx/bin/daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer

Aug 22 11:47:55 dc0awx01 systemd[1]: Started AWX daphne service.
Aug 22 11:47:55 dc0awx01 systemd[1]: Starting AWX daphne service...
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,721 INFO     Starting server at tcp:port=8051:interface=127.0.0.1, channel layer awx.asgi:channel_layer.
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,722 INFO     HTTP/2 support not enabled (install the http2 and tls Twisted extras)
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,722 INFO     Using busy-loop synchronous mode on channel layer
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,722 INFO     Listening on endpoint tcp:port=8051:interface=127.0.0.1
Aug 22 11:48:03 dc0awx01 daphne[1022]: 2018-08-22 09:48:03,800 ERROR    Error trying to receive messages: [Errno 111] Connection refused

Everything else seems to run okay.

from awx-build.

MrMEEE avatar MrMEEE commented on August 23, 2024

Again, doublecheck that the rabbitmq is running....

from awx-build.

anlag avatar anlag commented on August 23, 2024

rabbitmq was running and awx-daphne didn't show that error again after I restarted only that. But as my friendly neighbourhood Linux guy concluded upon closer inspection, I hadn't run "awx-manage migrate" again after setting that new default with "awx-manage makemigrations" as detailed earlier in the thread. Executing "awx-manage migrate" now resolved the issue and the server seems to be running fine now with 1.0.7.3.

from awx-build.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.