openstack-k8s-operators / nova-operator Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 41.0 50.31 MB

License: Apache License 2.0

Dockerfile 0.25% Makefile 1.87% Go 96.13% Shell 1.75%

nova-operator's People

Contributors

Stargazers

Watchers

nova-operator's Issues

Ensure durable / quorum rabbit queues and auto delete

The current default RabbitMQCluster configuration does not enable durable queues and auto delete features so nova cannot use them and they are disabled by #159. Eventually we want to enable these cluster wide but it needs agreement and adaptation from all the operators using rabbit.

Properly default struc fields

After may failed attempt I think @stuggi found the solution.

type SimpleSpec struct {
        // +kubebuilder:validation:Optional
        // +kubebuilder:default={one:Example1, two:Example2}
        Pair Pair `json:"pair,omitempty"`
        // +kubebuilder:validation:Optional
        // +kubebuilder:default="bar"
        Foo string `json:"foo,omitempty"`
}

type Pair struct {

        // +kubebuilder:validation:Optional
        // +kubebuilder:default=Example1
        One string `json:"one"`

        // +kubebuilder:validation:Optional
        // +kubebuilder:default=Example2
        Two string `json:"two"`
}

Pair needs to have a full default defined so that when pair is missing from the input this default will be used as the value
One and Two needs to have its own default defined at the field level, so that when only one of them is defined in the input the other can use the default value defined at the field level.

[NovaAPI]Expose the nova-api service

expose it via openshift Route or k8s native ingress
expose it in keystone as endpoint

cold migrate between computes is not possible due to missing neutron service token from the super-conductor conf.

2023-03-22 13:05:33.041 20 ERROR nova.network.neutron [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] The [neutron] section of your nova configuration file must be configured for authentication with the networking service endpoint. See the networking service install guide for details: https://docs.openstack.org/neutron/latest/install/
2023-03-22 13:05:33.068 20 WARNING nova.scheduler.utils [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] Failed to compute_task_migrate_server: Unknown auth type: None: neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.069 20 WARNING nova.scheduler.utils [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] [instance: 507bd531-6541-43a2-91a1-5052600cf6cd] Setting instance to ACTIVE state.: neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] Exception during message handling: neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 100, in wrapper
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 331, in migrate_server
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self._cold_migrate(context, instance, flavor,
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 426, in _cold_migrate
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     LOG.info('During %s the instance was deleted.',
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     raise self.value
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 385, in _cold_migrate
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     task.execute()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/base.py", line 26, in wrap
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.rollback(ex)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     raise self.value
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return original(self)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/base.py", line 40, in execute
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return self._execute()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/migrate.py", line 253, in _execute
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.network_api.get_requested_resource_for_instance(
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 2586, in get_requested_resource_for_instance
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     neutron = get_client(context, admin=True)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 247, in get_client
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     auth_plugin = _get_auth_plugin(context, admin=admin)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 227, in _get_auth_plugin
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     _ADMIN_AUTH = _load_auth_plugin(CONF)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 88, in _load_auth_plugin
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     raise neutron_client_exc.Unauthorized(message=err_msg)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server

This is due to the limited service token configuration in https://github.com/openstack-k8s-operators/nova-operator/blob/master/templates/novaconductor/config/nova.conf

We need to move the conductors to use the new config generation pattern as that config already has a templated out neutron neutron section.

[NovaAPI]Implemet proper readiness and liveness probes

Current we use /bin/true but we should use http-get

The nova-metadata agent gets wrong service config

$ oc rsh -c nova-metadata-metadata nova-metadata-0 cat /etc/nova/nova.conf
[snip]
[keystone_authtoken]
www_authenticate_uri = <no value>
auth_url =  <no value>
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = <no value>
password = <no value>
cafile = 
region_name = regionOne
[snip]

This cause metadata request failre:

2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler [None req-ec4090a6-681f-43ca-abe9-8d3c89d92d03 - - - - - -] Failed to get metadata for instance id: 1a261b7b-0fbc-4c6d-8324-95fda749f161: keystoneauth1.exceptions.catalog.EndpointNotFound: Could not find requested endpoint in Service Catalog.
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler Traceback (most recent call last):
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/api/metadata/handler.py", line 324, in _get_meta_by_instance_id
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     meta_data = self.get_metadata_by_instance_id(instance_id,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/api/metadata/handler.py", line 89, in get_metadata_by_instance_id
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     data = base.get_metadata_by_instance_id(instance_id, address)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/api/metadata/base.py", line 694, in get_metadata_by_instance_id
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return InstanceMetadata(instance, address)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/api/metadata/base.py", line 150, in __init__
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     self.security_groups = security_group_api.get_instance_security_groups(
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 584, in get_instance_security_groups
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     sg_bindings = get_instances_security_groups_bindings(
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     ports = _get_ports_from_server_list(servers, neutron)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     ports.extend(neutron.list_ports(**search_opts).get('ports'))
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     ret = obj(*args, **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return self.list('ports', self.ports_path, retrieve_all,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     ret = obj(*args, **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     for r in self._pagination(collection, path, **params):
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     res = self.get(path, params=params)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     ret = obj(*args, **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return self.retry_request("GET", action, body=body,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     ret = obj(*args, **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return self.do_request(method, action, body=body,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     ret = obj(*args, **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 284, in do_request
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     resp, replybody = self.httpclient.do_request(action, method, body=body,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/client.py", line 341, in do_request
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     self._check_uri_length(url)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/client.py", line 334, in _check_uri_length
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     uri_len = len(self.endpoint_url) + len(url)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/neutronclient/client.py", line 348, in endpoint_url
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return self.get_endpoint()
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 291, in get_endpoint
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return self.session.get_endpoint(auth or self.auth, **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1243, in get_endpoint
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return auth.get_endpoint(self, **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     endpoint_data = self.get_endpoint_data(
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     service_catalog = self.get_access(session).service_catalog
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     self.auth_ref = self.get_auth_ref(session)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     self._plugin = self._do_create_plugin(session)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/generic/base.py", line 136, in _do_create_plugin
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     disc = self.get_discovery(session,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 608, in get_discovery
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return discover.get_discovery(session=session, url=url,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 1455, in get_discovery
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     disc = Discover(session, url, authenticated=authenticated)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 535, in __init__
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     self._data = get_version_data(session, url,
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 102, in get_version_data
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     resp = session.get(url, headers=headers, authenticated=authenticated)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     return self.request(url, 'GET', **kwargs)
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler   File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 815, in request
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler     raise exceptions.EndpointNotFound()
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler keystoneauth1.exceptions.catalog.EndpointNotFound: Could not find requested endpoint in Service Catalog.
2023-03-27 12:01:19.803 12 ERROR nova.api.metadata.handler

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository.

WARN: Post-upgrade task did not match any on allowedPostUpgradeCommands list

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update openstack-k8s-operators (github.com/openstack-k8s-operators/infra-operator/apis, github.com/openstack-k8s-operators/keystone-operator/api, github.com/openstack-k8s-operators/lib-common/modules/common, github.com/openstack-k8s-operators/lib-common/modules/database, github.com/openstack-k8s-operators/lib-common/modules/test, github.com/openstack-k8s-operators/mariadb-operator/api)
Update module sigs.k8s.io/controller-runtime to v0.14.4

Detected dependencies

gomod

api/go.mod

go 1.19

github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20230130113944-08eaed348cec@08eaed348cec

k8s.io/api v0.26.1

k8s.io/apimachinery v0.26.1

sigs.k8s.io/controller-runtime v0.14.2

go.mod

go 1.19

github.com/go-logr/logr v1.2.3

github.com/google/uuid v1.3.0

github.com/onsi/ginkgo/v2 v2.8.0

github.com/onsi/gomega v1.26.0

github.com/openshift/api v3.9.0+incompatible

github.com/openstack-k8s-operators/infra-operator/apis v0.0.0-20230203151002-5ca00e898340@5ca00e898340

github.com/openstack-k8s-operators/keystone-operator/api v0.0.0-20230127072022-efdd85d844e1@efdd85d844e1

github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20230130113944-08eaed348cec@08eaed348cec

github.com/openstack-k8s-operators/lib-common/modules/database v0.0.0-20230130113944-08eaed348cec@08eaed348cec

github.com/openstack-k8s-operators/lib-common/modules/test v0.0.0-20230130113944-08eaed348cec@08eaed348cec

github.com/openstack-k8s-operators/mariadb-operator/api v0.0.0-20230118093207-9764a3d19c85@9764a3d19c85

go.uber.org/zap v1.24.0

gopkg.in/yaml.v3 v3.0.1

k8s.io/api v0.26.1

k8s.io/apimachinery v0.26.1

k8s.io/client-go v0.26.1

sigs.k8s.io/controller-runtime v0.14.2

Check this box to trigger a request for Renovate to run again on this repository

Do not render non nova-api specific config for nova-api server

There is a bunch of config rendered for the nova-api service that are not needed for the nova-api and therefore confusing to have it in the config of nova-api. E.g.:

state_path
allow_resize_to_same_host
initial_*allocation_ration
instance_name_template
force_config_drive
disable_fallback_pcpu_query
enable_qemu_monitor_announce_self
reserve_disk_resource_for_image_cache

[fedora@gibi-f35 install_yamls]$ oc rsh -c nova-api-api pod/nova-api-0
sh-5.1# egrep -e "^[^#]" /etc/nova/nova.conf
[DEFAULT]
state_path = /var/lib/nova
allow_resize_to_same_host = true
max_logfile_count=5
max_logfile_size_mb=50
log_rotation_type=size
log_file = /var/log/nova/nova-api.log
debug=true
initial_cpu_allocation_ratio=4.0
initial_ram_allocation_ratio=1.0
initial_disk_allocation_ratio=0.9
instance_name_template="instance-%(uuid)s"
force_config_drive=True
transport_url=rabbit://fixme
osapi_compute_workers=1
metadata_workers=0
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[oslo_messaging_rabbit]
amqp_durable_queues=true
amqp_auto_delete=true
heartbeat_in_pthread=true
[oslo_policy]
enforce_new_defaults=true
[api]
use_forwarded_for = true
auth_strategy = keystone
[oslo_middleware]
enable_proxy_headers_parsing = True
[wsgi]
api_paste_config = /etc/nova/api-paste.ini
[upgrade_levels]
compute = auto
[oslo_messaging_notifications]
driver = noop
[notifications]
[cache]
enabled = True
backend = oslo_cache.dict
[workarounds]
disable_fallback_pcpu_query=true
enable_qemu_monitor_announce_self=true
reserve_disk_resource_for_image_cache=true
[database]
connection = mysql+pymysql://nova_cell0:12345678@openstack/nova_cell0
[api_database]
connection = mysql+pymysql://nova_api:12345678@openstack/nova_api
[keystone_authtoken]
www_authenticate_uri = http://keystone-public-openstack.apps-crc.testing
auth_url =  http://keystone-public-openstack.apps-crc.testing
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = nova
password = 12345678
cafile = 
region_name = regionOne
[placement]
www_authenticate_uri = http://keystone-public-openstack.apps-crc.testing
auth_url =  http://keystone-public-openstack.apps-crc.testing
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = nova
password = 12345678
cafile = 
region_name = regionOne
valid_interfaces = internal
[glance]
www_authenticate_uri = http://keystone-public-openstack.apps-crc.testing
auth_url = http://keystone-public-openstack.apps-crc.testing
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = nova
password = 12345678
cafile = 
region_name = regionOne
valid_interfaces = internal
[neutron]
www_authenticate_uri = http://keystone-public-openstack.apps-crc.testing
auth_url = http://keystone-public-openstack.apps-crc.testing
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = nova
password = 12345678
cafile = 
region_name = regionOne
valid_interfaces = internal
metadata_proxy_shared_secret = 42
service_metadata_proxy = true

Update ceph libvirt secret fails when cluster have multiple users

[root@standalone ~]# ls /etc/ceph
ceph.client.admin.keyring      ceph.client.radosgw.keyring
ceph.client.openstack.keyring  ceph.conf
[root@standalone ~]# ls /etc/ceph/${cluster}.client.*.keyring | grep -v admin | awk -F . '{print $3}'
openstack
radosgw
[root@standalone ~]# echo ${CLUSTER_USER_MAP[$cluster]}
openstack radosgw
[root@standalone ~]# keyring=$(awk '$1 == "key" {print $3}' /etc/ceph/${cluster}.client.${CLUSTER_USER_MAP[$cluster]}.keyring)
awk: fatal: cannot open file `/etc/ceph/ceph.client.openstack' for reading: No such file or directory

openstack-k8s-operators/data-plane-adoption#58 (comment)
/etc/ceph/${cluster}.client.${CLUSTER_USER_MAP[$cluster]}.keyring is expanded to /etc/ceph/ceph.client.openstack radosgw.keyring which breaks the script

NovaAPI should use StatefulSet instead of Deployment

To avoid the need to service de-registration at any Deployment restart event we should use StatefulSet instead to keep the hostnames stable. So that we only need to handle service de-registration at a real scale down event.

api should be useing new default

enforce_new_defaults shoudl be set to true for the api

Multicell nova-metadata follow-up

add validation webhook (add reject cell0 metadata from the validation webhook)

Do not run things in containers as root

nova-operator/pkg/novaapi/dbsync.go

Line 53 in 6e5ff21

runAsUser := int64(0)

we should not be runnign as root

https://sdk.operatorframework.io/docs/best-practices/best-practices/#summary-1

"Does not run as root"

we should be using the nova user here.

the nova user and group is 42436

https://github.com/openstack/kolla/blob/master/kolla/common/users.py#L148-L151

https://github.com/openstack/tripleo-common/blob/0a4ca78500a30e80a1746ac65188350d597a32bc/container-images/kolla/base/uid_gid_manage.sh#L62

Originally posted by @SeanMooney in #84 (comment)

also in

nova-operator/templates/novaapi/config/nova-api-config.json

Line 25 in 6e5ff21

"owner": "root",

Originally posted by @SeanMooney in #84 (comment)

also in

nova-operator/pkg/novaapi/deployment.go

Line 37 in 6e5ff21

runAsUser := int64(0)

Originally posted by @SeanMooney in #84 (comment)

Wait for the Secret CellMessageBusSecretName to exists and include its hash to the InputHash

Current TransportURL Secret handling does not ensure the Secret exists, has a transport_url key, and we don't include it in the InputHash of our Service CRD so we don't detect if the content of the Secret changes.

NovaMetadata service is connected to the service=nova-api pods.

nova-operator/controllers/novametadata_controller.go

Lines 437 to 443 in 165d29f

 apiEndpoints, ctrlResult, err := endpoint.ExposeEndpoints( 

 ctx, 

 h, 

 novametadata.ServiceName, 

 getServiceLabels(), 

 ports, 

 r.RequeueTimeout,

calls

nova-operator/controllers/novaapi_controller.go

Lines 603 to 607 in 165d29f

 func getServiceLabels() map[string]string { 

 return map[string]string{ 

 common.AppSelector: NovaAPILabelPrefix, 

 } 

 }

that makes nova-metadata service connected to nova-api pods instead of the nova-metadata pods.

$ oc describe  pod nova-metadata | grep service=
                  service=nova-metadata

$ oc describe service nova-metadata | grep service=
                          service=nova-api
Selector:                 service=nova-api

This causes connection error when ovn-metadata agent tries to connect to the nova-metadata from the external compute

filter api endpoint in config

nova-api should have
[DEFAULT]
enabled_apis=['osapi_compute']

and the metadtat confg should have

[DEFAULT]
enabled_apis=['metadata']

rename reconcileNormal to doRconsile or a similar more descriptive name.

There are 2 issues with the current naming

reconcileNormal is not partially descriptive in what the function is doing
or differentiating it from the public Reconcile function.

it is a private function used to actually implemented the reconsile called within Reconcile
after reconsile is able to satisfy some basic preconditions like retrieving the instance of the
resource that is to be managed and preparing some commen clients/helpers that will be required to
interact with the k8s API

the other issue is that reconcileNormal is not a particularly PC name scheme
there as some unavoidable uses of normal such as normalisation of import or in vector calculus
where using a different term would be incorrect but we do not need to prop use it to describe the happy path ie. the work to be done if setup succeeds.

Check if we can improve readbility be reducing number of function params

The one param per line comes from the python black style to me. I can let that go but what should we do with long lines? I need to wrap it at some point to fit into a screen :) What would be the place of the split? In your example you split at after some parameters in a single line then switched to one line per param for the rest of the params.

Maybe we have too many params for a single function call and we should wrap the params to a single struct.

Originally posted by @gibizer in #87 (comment)

NovaAPISpec should have Cell0DatabaseHostname and Cell0DatabaseUser

nova-operator/api/v1beta1/novaapi_types.go

Line 100 in 6e5ff21

APIDatabaseHostname string `json:"apiDatabaseHostname"`

The nova-api service depends on cell0 DB to register itself. So when the NovaAPI is created it should get the connection information.

[envtest]Every Update() call should have a retry on conflict

We have some flaky test cases where we update Job or Deployment Status to simulate success. However we can race with the nova-operator, that is also updating those entities in parallel. Which can lead to conflict response to in the test. (In the controller too but that is already handled via requeuing the reconciliation)

Restructure secret handling

Pass Username via the secret
We might also want to pass the keystone auth URL, Database URL, rabbit URLs via Secret
Generate a separate secret for each lower level CR.

Ansible compute followup TODOs

TODOs left for followup:

do not requeu while waiting for the ansible jobs
add controller ref to the ansible jobs we create to not leak them if nova is deleted while the job running
pass NetworkAttachments to the ansiblee CR
create a common ensureAEEDeploy function from the two similar one
blank out nova-compute.conf
add envtest coverage to the negative ansible execution case.

Originally posted by @gibizer in #268 (review)

[sample]Update our samples to use different MariaDB instances for cells

As we want to only support that mode in production we should have the samples reflect that. This though needs install_yamls changes to actually create multiple MariaDB CRs during make nova_deploy

The nova-operator uses the placement logo instead of the nova one

nova-operator/config/manifests/bases/nova-operator.clusterserviceversion.yaml

Line 58 in 1f1f5ae

 - base64data: iVBORw0KGgoAAAANSUhEUgAAAlgAAAJYCAMAAACJuGjuAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAA4VBMVEUAAAAeGRcfGBUfGBUeGRYeGRUeGBYfGRUfGRYeGBYfGRYfGRYfGRYfGBYfGRYfGRYfGRYfGRYfGBYfGRYfGRYfGRYeGBUfGRYfGRYeGRYfGRYfGRYgGhcrJSI8NjQwKyg4MzB1cW+qp6a6ubiamJZmYmBZVFJGQT8mIB1KRUOjoJ/39/f+/v79/f3S0dHd3dzy8fFZVVPp5+e7urmEgX9xFyedFjBHGB6Cf33Ny8uTkY++FjjsFULbFD3DwsHd3Nxzb23wNFv6tMP0b4uvrazT0dHq6uofGRYbFRL////uE0G6F3GlAAAAR3RSTlMAAQgUHio1P0kEU2h+iZejsb5zXdbx/unKDeH4/PXo7/PV09PR3+zx+eLn8fv06+zr3ezp09/g5efY5Nzu393Y5+Tj5OPd1tEMb7gAAAABYktHREmHBeR8AAAAB3RJTUUH5gYcBjcBesujqAAALQJJREFUeNrtnXlD4krWxjvsYV+yAAmbiICALNrdyr1XnTsqYb7/B5qAaGOqskBCUpHnN+/8847SGB5OnXrqnFM/fgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgu8JForF4IpFIJlO2JBOJeCzKB/2WAdvw0VgimUpnsrl8oVAslhxQLORz5UoqEY/yXNBvH7DIJkylMvmiIEor5+x+Vq4Wyik9dNUgLrAHF6lFk+ViXVRUVTpEV5/6UlVFqRez6XitFkHoAu/w0XS+qqvqGEntoSqiXC/lKvEax0FboJbMlgRZdSeqz9ClyPVqKV9OxZDQnzVcrJIv1V2GKkJdYr1UzGVS8UjQfx4IhkgsnRcUb1X1uTDWS4VsJpWAF3F28LFUTvBoCTQLXdVCNp2MxyLIuc4GrpbI1U+pqg8UQRfXxorAwngGcJFYpqp6nFuZhC1J3y7qu8VMIhqBFfHdiVRK8omSK7q6tlZEXhdX0H85OCWx3KlyditUcWtFpLFb/KbUUgWvHQaHcUv/r1KvFnVxJaJYFL8bsUzppHtBO3WpK1WuFnKZVCKG0PV94GJZIUBZfaIqOyuiBqPrO8DH83LQmvqDsjlbTGysCCyM4YZPFA8qizk1GytCXxdhRYQcPlliSlfvqKJc19fFVCzoxwOOhEuVAnAZnLGxIvSkK1EL+iGBg+FSxRPrStM0N7+u7xaL+bKedCGfDxN80rGujhKIpDSazZYsuRPXShSKWysCoSsk8Amn66DaaLVbjYMLStVm56J72bvqb37Xhbg2RpcurgqsiFDAxQqis8910OxdXlx2ho3DbFRVvh6t1+Px5Eb/3XZjoLhxYSVJUmVYEWEglnemq1WjMx1NJpPR7LJ5UMxS59P1lvFkNFp0O8PWQFXdFFBIysaKyJeT0QgPK4JRuGhZdvQZa63OYvwukNlF85AFbdBZrP8wmi2mt3fzw5rJaHIV68LWioCymCSSFuw+YfHnr1+/fv7u/ZHH6F4+RAJ3+8Lahq7F9OF22RvWXW4Wd1ZEBVYEeySLth/ez7/+/vvvf/662FPG4/AAQZDCelfn0+318nnYEl2qS5LfqyJgRbBELG+7Ifz91/82/OffPVHM7t0L6z10PV7e9/ptWXEZupQPKwILIxNEyrbV7drPf7bC+u9sXxDX3ghrvcnaRg+Xnat+uyG6sCI2ewGp/m5FRFFzEzR8smr7kX0Ka7QviGtlMHC6sbMS1k5e48n0YmtFOH5RGtsi+mo+AysiYLiaEwfrQ1hftnb//eevn78HK0fzHOyF9W5FzBaPd/Pm4MgpER//mCqKslDUd4uwIgIjUnFQ2fcRsb7kWP/+53+bfF7fKzqw4R0Ja8tkpm8Xl72mMwPEInTp2qpuJpEgnw+EuJOjHHWXvP+9txaO//v3+/9P19avnw3PhLV97dnGihi6L+JRNlZEOR70Qz5DomUnlrv0+9f/3lX07+Tjs9cD1id///Prt5fCel9qH3ueVN9L9Xwi6Md8fiSqTj48Sfm5U9Df/y4m25Dy73/+/t8ef/+0DnxHCGs9fmh5U3coloN+zGdHNOMsKGi///lQ0H//XSwW//73P//7ik3IOkZY62nTowqxPFx5n0mWHMaEj7Vwq63/fI1WW/5iWFgqhOUztbLTLEb5/UdLpKo2EUv2XFij24Y3S2E9E/SDPjfsDwk/afz6nxX//LTWwBHCWlz2PUneVSGDRgyfKcuOPx7lM8ui8fdPm83lYcKaTB8v7688WAjfK5kxa8RnYoVDvvg/zZX19y/vfKzZ9PFlV/DgSlPb3ossei+CoGJ/SuhIWX//9dsuuDgS1mQxvdmWaEmaJrk405HEulAq5FCiFRC803rkHQ0TZTnQlb2wdkWlTVFaSW4OoT/7WxGogoKLlw790KjK+udXyz7JtiybmUxGi5dtGbyyLX05UlcfHflJdOQHCu/k+Nkgj9+//iHC1c+Bg82bhbAmrx+NO27MBTTuMAO5EmqD3svtS29goY+GLq09H2tTOONoPTUR1uLx8nlT3+eq1XDz8mg1ZAYuSrjujd7NaDx5eraICyvp9++fv/766x+dv3791GXlLMyQwho9dZf3vWFL1jaZugtRbee0pTGnjRUiSeNKqDW7myrhyaON4a3Iv39u+O1UVRth7bd/jWc3ty+b3Z+y6aFwISpMlmSQGumOXr2XtE+HdkmTqmc00kE5kTK82QWqraUwb+qBSjo6TV99WApZzMJljihRkty63yU+HTthSTsO0cFyMfroU1WkleLGp1K2lkI5iUDFItGqwXzShg877/tuYP/hHioLqTG/u/TIUtiOhMfVh4wSNzZ9ac+75q7ZpQNhHS6JQaMhii6PldV6MZfeWArY/bEKnyLMhvtd2fGoexJhSSuX/ajVQnZrKSBQsUw0Q6RY15+O5SmE5QqMfg8NsdzXFEuSho8fdsDUowI7TxA/LIWgnxhwRNw4GVLrPX364UM2ptyqW0sBN/2GigRxy+Xd52CG0V3DXTrkHunTUuBxN3moSBmDkrb8aBlcT7qtIIUlqYoiV/OVOCyF8BEh+75exp9HLm/tAIWlCMUsLIWwEssZ8/Pm7d4BcS+YfeH2mujK+zXRWAFDSaJgEJY6v9k7Jb7wP2Sp9fcqBVxsH2qSVaOw9utaxm+HjIH0IFAJW0shjhr10JOWjR/uxf5QtcmzX2vh1lLIZ3Cg/E3IEB/w7WRPWIdNrj1WU7s7xpM1Hp7CNyGaMwaswc14T1fj0fNphSWpqlLfjEWDpfCtiBtbVdXh9GvnzEvzlLpShAIshe9Iytj59fXmiE318CHDtg+KVfKHpRD0QwDeQ3R+DS5nX4S1ntx6H7IkZVNNnEnBUvi2ZI01M43HyVdhrWdLTzeGW0shl0khUH1n+LzhY9caT2ODsMY3tj0VjgOVDEvhLODIMTNtsp900rUbIeNAU6oiivVSrhKHpXAOJIncfT4jhLWeddwqS1//CuUULIWzwZi7a427ESms8fTKXcGfWtrMUsAohfMhKxuE1e5OSGGtJ499V55DMYXDv/OiYEjLtTaRu7+Xz1y7qHKQhDR2gGcFVzPOtNX6M6qw1qPl0bWkklhG/8N5wSeMIyK1qwlVV+vx4u7YBF4t4QqbM4NPE7l7hx6wdKZO+u1piBUkWGdGJGvorteaF6bCGk87RylLrSJgnRsR45wZrf9gKixdWUc1g4l5ZFjnRs0wyk/Srt7MhaUr65gMvo6V8NzgYsQov6vR2orFsnmwsuoJVDCcGZGUcYCR+jy2FNZ6dt0/1IMXYjDcz4xaRjYErNZybcPk8erAQSECVsJzI2ocwy0Nu3bC2mwOG4dU0ahVCOvciJXM58xYKGvWPeiWNwjr7IjVDYva54xIa0Y39wecHCLHOjf4pOmMSLtEa/HSE51KC7vCc6NWMQprsBw7EpYuraf74cBZEg8f69wwzoj8OmfGdj18eO47khac93MjXjAIS5q/OheWHrUeem0HG0RVwFnheZGoGqv8OgfdAz4ej17vWvZXyYkZhKyzImm8WUlajg4R1np7ydKl7U3gajUR9J8KfCRSMWZI2rWzTaFBWtdz6x2ipGRjQf+xwD9iWeKWwgNy9z1pjW6W84blTfNCBhvD8yGeN66Ew5tjhLWR1uv9VcPqatRSGso6G4g5M2pnepyw1htf67lvdTitKwv++3nAVWSjsC4dHeiYSev1uWVRuixUMK/hHOC4pLHza6V2D8/dvyyIjz1T70FS6zAdzgA+ka0Sd86rN04PdEyUNVm8mN+9o9bL2Bt+b7hoJV+qEwpQm8enWJ9B6+bOrCpej1lwHb4zfLySryqUPHvQO8h3N5HW4vrKtJWnnsPhzndlI6s6PaIMXOXun0xe75sm20NVzsWRwX9H+FgqL5uEE63RPfRAx4TZhWnZg5yPozjr28FF4rm6amo2NV5dbQr3g9bT3KzqQSzgLsvvBhfNVEVzD1NrLdxtCvcYzy5NJi1LYhGr4feCSxbqVnUI6tCTFOtjOeya3JQiiQXUOnwnopmSaCGrzZwZj1Ks3XJ4YRqzoKzvQySRE2wK8loXXqVY7zx1zP4lKOvbUEsWFJvydK394K2wFnem1Q5iHnnWt6CWLq7s0PpvnuXuW17npv+WJOfRaxh+uFqmaqsrXVijsZeMllaTJes5nEiHHa6WtUuvNihXb17y0GlZrr11zLsNO9Fc3cmkBbXR95KmzUgaFW2s4YaLZh3parMYeov1Pyap1RQs+BATy9YPm2blH8Vk0A8HHE20zKyuVkoeRTRhJZIR3AvgZMg5NFiEE77iwGcIECGDK3bCCJ8sMbsOblGrGJ0VQvhYUXT/4Z8UsYAq+PARzclBC8deWVn4pGGjVmF3Q/iJKqSQZoULLlHy6Or5kyKhhCZkxHLsx6sNShlHO2GCq9Tdf+i+UE0H/azAASTtK7AYQS3AgA8PtRzrTsMf6ln476Ehxbbl/pUSbNKQwBG3LzGNnEP+Hg6IO8QZR0jCzAoF0UKYAhZOdsJCJBkWq+EDGf57GAhXhrVBgeUQAvhECA4JvyLJaYQs5omWD70OPHgklCkzDxcPxemzgXoFLinjcGkxbCvhBoQs1glLWYMBIRP0gwPWJEvuP+UgwD2sjJMJm9ewA+UzbBPNBa2QIxHzOIpmmURoCrGMlDCpm2Uq4Tp/3qNehknKMLnwuaM7lBLSd3aJFoLWx/HUUT3DLmE1GzbImB/JLBzT42VsUKsoy2IVLnQVM3tIIsqyGIWLhngl1NP3HEIWm/CJkHTn0GeUqlX027NJJBOKomS10W63KVcaSgqOddikFo4Uq9V5ferOB+T/IGGqEZtEq2EomWktZ5PxZHFFea9FrIUswidCsBKqrevZ5tae8UWTzLPqWAtZpJaRg5aNLZIer95vg3rrkcKScF7IIjHmUyxJbX/oaj25p6yFqFBmkXiV9TYKXVejz9vrXlrkD5RSQT9EQJJkPGBt4tXe/cBPPfJH6ih9Z49ohvGSGT2/2r93erakJFm5oJ8iIIjn2RaWquft+3ewjl8oKzfmg7BHgu0US2p85u07Hppk+l7CnWCswaXY7lRVelPDrdPTDvlNEDJoiWaMWjlo6VgzuJx91dV6dkkKS8xDWIwRzwctnUOFNXkkhaUWYZEyBusDbdXOwiCs8Q3l5uhqDCGLLSqMu1hK88YgrPV0SOxjVSGJ9kKmiGSDVo4dg9uxMcnqkLUzddyOyRZx9hu/LkYGYY0uG8afkeQchMUUacZTLJ07Y5I1eWwQ5rtYxNR3pijLQevGBknqGZOsyRMpLKmKKlKW4Bk3GzaS6d8at4WLFqXYD9tClogzP2VGkhrXxm3hrE36DXVcrcMQXAhSLElbTozZ+xW5LZTTSLLYgcuGoNx9dU9sCzvEtnAll5FksQNfZLtkZot2bzzUGd2RSZaI2SDswMXYXwl1YXXejNvCS1JYCkqy2CGSCsOUGa33ZBTWRZsUVgnCYoZaVg5aNU6ENX8wCqtLCksV0KnDDNEwpFgrbfhoFNYD2bUqyRAWM8TCsBKupGHXKKxXSju0iD57Voik5KBF4wSt/2K03l/7pLCUFBxSRohmGa/F2iJJTaOwJk8UYYkVOKSMEAtFirWSmsYznTFNWEoGRhYjJOpMN3596EpzKCxY74wQSYUiYK3IpZAurCyMLDaIZUMhLKc5looZt4wQL4ZhJdz4WOSukGI3KHkIiw1SAtMt0HvCcmKQrtQCHFIm4CqhWAmpRzqPbYqwihAWE8SYb/x6R6IcQnepwoL1zgShuftS6xFlMxc0YZUgLCZIh+KgcEWrxxpR6rFWEoTFBuWQpFgrjVaaTIlYuPmECaLsN359CItspqDdTwFhsUFoUqyVem0c3jDqUxw4CIsJuPBcL94y+qPrBaWvcDNwJuiHCnRyYSiZ2ULYWGNKi72evENYDMDV2J8y8wHZpPMAYbEKlwjPparErMjRBYTFKnyIrhd/MW4KZ3eUTeFKEZC8Bw+fl4PWi0OkhnHYzHpBcxuwK2SCWikclQ16IJpPjcKatmj1PhAWA/Dx0KyEzsZxr3BWyAS1ShimzLwL65Hw3S+owlIgrOAJyfXiq81VhcYbT9aLO3rEQj1W4HDRalhOoAcd40q4vpnThYUK0sDh4mFZCbVW17gSjm8b1B9VcH9v4NRYv47iD+0FcQJ9qVF/El06wcP+9eIfNDrGWqz1W4cuLPQVBk88LCmW1nw0Bqz17ZD+s2ixD56kHBJ7VL2aGYU1WQ7oP6tgbHLQ1Crh6FRdae2lMV6tZ8/0lVBScP9X0MTyYRHW/I0Q1uOQLiwMXgscPhGWg8LBvdFrWI/v6WYDRkUyQDokKZY2fyVXwiuTgIXhtoETYfx68U8GSzJgdfsmwsI47sBh/XrxT+bEpb3rybPJSrhSihBWwKTCUZUsDYirVdfjN7OAJYl52FgBkwnHQaFKVvjRe+vfwSVNgZMLWjLOdNV4IDKs9aJvemSAa+WCJhaOxq/G3YwMWN2G6c/jIsygCUeKJfWJAr/1ejo3d3YF5O4BUw5DiqW1L8iFcNJtmBpwagkrYbBw+RBUNkgNsnBUD1g9c2NXLEBYgcJFwzBlRqJ47uvJi3mGtZJzkaAf7XnDp8JwqWrzmkyw1k89zfw3hAyEFSh8GFKswXJECVhLi4C1qqJoJlgiBfarktW7Bamr9cPcImCtSlEu6Ed71oThenGlQ3Ea1pPngdXvFBCwAiWSZL23XhvMX0mnwaKsYYucQ8AKlFpZDlo5NijDR5qu6BNmPhEqEFagRFlPsZT+C2UdXI8uG1YBCzPegybG+MVMavOaEq/Wk9e2dZl+AaUNgRJJykFLxxI9XtF0tZ72Bta/mAv6yZ450TLTK6E0vKXqanY5sA609UzQT/a84WMFlg8K1fnjiKar0Uvb5jdLqaAf7ZmTENjtKNQGPbquJrdDzeZ3cblqsHAMXy+uNZ5fafvB9frpyvbbUIY9GijRMrN7QrV9/0aV1Xr23LD75Xo66Cd73nDxAqPCkgb9C+oyuB5b9E98UsTVAcGSqrIpLGnQe53Q49Xowi5x18HItYBh9HpxtX2xGJvoqtu2/y5IFaRYgRJjsvFLa3UeZya6mjz2HegKl+gES4TF+3M08ep6arIM6rqyNRpWm5IZjAMJEC6SZq/vSxoM7x/MZLWePFw50NVKxIzIAOFq7BW763tBE0/0Q1eO9hr1BEpmgqOWYs10Vwft3sNobKErB/nVCmNmgkXXFVs7QlVt3r2OTFdBfT/40Hf2TZDL6CgMjEi6ylS80hq9i5uZhazWo27T4TvG0Ibg0HV1cmdU03H4c6358nZqsQjqLC6bDt+xUkLqHhQ+7Aelxvy5N2xplvLa/I/isHd//TqzVNV6PL1rOlKpTj2LTtVg8MFnkNT2/evb2+Py+WrYbCnaBzsxvSM1msOr3n33zTpWbdL2m2f788EPqlgJg8EPn0Ftv7cujydvDxf3V/1+v91utRo7Wq12u9nvXz1fPLxN1raMR49D5xsNpYCAFQw++AxSe68lfjyejN6eHroXl3d3nV6n07m7vOg+PulxaqxHKrtgtUmvLhwcD34ioCg5GPzwGVrGUQu6uCajPSaTiQNFvf/qU6dxyBcB16oGgw8+g9q6njmVjS2zi75tWd8+YhauexD44DNIHupq8nrfPux7UEXtaBD4sh9ceqWr8dv1vOF4N/gOSvwCwI96hs1+0BtdjWe3nYa0OizAypWgH/IZsvEZTr4Otpcj95rayGr0tHTqte/98wWMbPAfP+oZGvee6Go8WVz2DwxWG5QKzp99J3J6n0HSek+e6Ortsn+Qx7BDxYwZ//GlnkGjDc0+lMnTXb+lHpi0bxFxfY7v6Hm7D51eWu/Npar0lP3+quWsLsKIKsRhYvmMT/XtWvPCweGfhapuru+Hg+NkpQesHAKWv3D6OuiHrnSG3emRdsN4Nr297DWkI1WlIyRx/uwvPvgMnzQ6FzeLkeOjwJ2oJqPF60Wvpa2k49+oiLoGn4n42TchqYPG/PJhoy1n4tpUPywe74YNVXWn/moKhVi+4oPPYGDQaLWvLh/sK/g2Ruhb9+6q3WoobmOqko8gdfeTQPomNGnQavavencXD28zau/NZDTbFAH2rvrNluIisfqkhONnX/HHZ6Bpa1N93GoO573n5/vly0v38fHh4fX14eGx+3K9vH9+7s0/ypa9+OeUHEx3Pwm4j16TVrvy9uZwOJy/M2y2G8qfKnhvwNRRP+EiqYDilQFpi6b/31ZnG7z9B5QsPCwfqSVZ6UvdSkn6g9cvX8XkIh9hro/+ZCgYMOMjkVT1XHSFFgofYW0+wwmR09gS+gYflM/gPyKuZPIPFuf1nQqcPvsGMz6DH8g5HBL6BTs+gw+UMBrSL/w/dw6Qehm68onz8Rk24H4Tv+DPx2fYkIfV4A9n5DNsEMtBP/DzgMl7AU4JxoD4Ahdh9UavU4Guel84K59hC+pl/CCSPKf94AYF82V84Lx8hg0qrpDzAT1vP7N1cKXiCrnTE1jfRJDCwhyQU3NW586fiOhSPTV+9tEzgypgJTwx5+czbFAKGGZ7Wmpn5zNskHBL74k5P5/hHRm1oyeFPz+fYYtaxUp4Ss6snuEPYgElM6fjPH2GLfUMVsLTcZ77wS0CLr08Hed37vyHUg3l7qfinHUl5hGwTgWfOtt1cHObKoR1Is52P7illMRKeBrOsZ5hj0IUwjoFZ+wzbFFyQX8C35Qz9hk2SAKqkk/COe8Ht6AF+iSc67nzJ1IelQ0ngE+d57nzHxS0QJ+A8/YZtghogfaeM/cZtqAF2nPO3Wd4J4taLK85c5/hHbUCd9Rjzt5n2CChBdprzt5n2IIWaK+JnL3PsEVBC7S3wGd4By3Q3oL94DsSWqA9BbragRZoT4HPsEMVkWJ5CHyGT+pogfYO+Ax/ELASegbqGf6AFmjvgM+wR72MldAbcO78BSEJF8sboKsvlNCf4w0R+Az7oAXaI/gE9oP7CLii0BtieehqH7RAewOXELAQ7oNb6z0CwvoCWqA9gsdS+AUhE/Qn8l3gcUi4D1qgPSNyzmOwCPI4KPSK87s41QIJLdDesTnSCfoDZQW0QHvKuV0ibk4RLdCeEkkVkcHrSDmkWN7CJ4siEq2VWsFBocfw8W+tLEUWFfs/T6rDbPCcSKwgBv3xnwpJLmZzJfsvjlpE45f3RGJ5OWgFnEpXmVgtGs/ZfnEUTJk5CbFyNWgNnERX9e2Vlnzc1q4TU6hKPgnR9DfcHKr1XZ9gpGgTsqQ6VsITEUnm69/tfEco765biuRthKUUsRKeCi6W+2Zn0kL24xovW2HJWdRinY5Iuih/n6Al1cufXYK2wqojxTolET2Hd+D5hAI9b//TfWorLCGGquRTwtUSue9hlu72gzvshCUW0QJ9YiKxit0OKgyoX3RlK6x6FivhyYkkysV60MJwiSR80ZWtsAQM8vMBjk9ki/UwL4jSp8+ww0ZYUgkplj/o0qrWQ+s9ELqyE5ZSgK78IhIvl+piOM0HoWxMxW2EVUdVsn9wkWgyV1VUKWxr4hefYYeNsEqpoJ/2eRGJxiv5sLnxar1Cmug2wsJIW7/havFkOhumTaLBZ9hhLSw1hxQrALhoopLNF8ORzBt9hh3WwqqjBToo+FiiUs4XS1WhLrPszJP7wXeshYUUK1j4aCJVKecKpSqrR9W6rugOurWw0AIdMJwOz/OReI7NpEtfB01OZqyFVYbtzgZ8lMnmC5rPsMNSWHW0QLMCXxGCVhEJ1WfYYSUsCVNmmIFPs9d6IQnmuvpRsxIWWqCZgc8yl2Tp+ZVFSZVVxFIrKJlhBD5WYs3T2uwHLVxOq4iFFmhmqJVZC1imPsMOi4iloAWaFbg4a3MA1XrGurTYImIpSLFYIWbfsO4vkp2urCKWmEa5OxtwScYaW618hh3mEQst0MwQzzOmK/q58xfMI5ZSwkrIBpEKW+XwElkvSmIesdACzQqJQtBSMuoqYl9NZR6x6mm4WEzAZ+WgtWTUlYN3bR6x0ALNCGyN77b1GXaYRiwFLdBMwNkOQfBZVxVnujCNWGiBZgM+zVJZg2TvM+ww/T4IadRiMQAfZWrsn5h3uqMzjVhogWaCWoapQ0LnQ63MIpZaQMBiAC7OVlVDPeFUFmYRq54N+pkCnWiZpcz9EGFFTGqpq+jPYQDm7pB2L6wCDgoZIJZjaiH0QlhogWYALs1U5u6FsNACzQJxtg4JvRAWWqAZgGeuHtm9sPJIsYInUQxaR94Ly9EBNjglXCQne6cIVfVke+lWWDJaoAOHT3nXoqqq7X574EG1oEthSSU0fgWOh9Ma1OHFdDG9aLlXltuIhf6cwKl5ZTVoanN5Mxqvx7PLhhawsFSHNTfgdMS9qWrQtHbndjZe64zf2gELS5KxEgZNNOPJQqi1rq4XW1npwpo1AxaWUoLZEDBcouTBLk5q9JfTyXrH5CnoiIUW6MCJZt0n2tKg3Xn6lNV6PboLOscSkWIFjQf1yFKj9zAa/9HV+LUd9K6wHscJdLDE8641IM1fFnvhar1edAaudeVOWEoJnaoB47YeWZOGy5vRvqzWk64HNpY7Yck5CCtY4u4OCTW1ef84G3/R1fpm7l5WLoWFFuiAibgcC9nqXRtltR4tD1sIVblKuxjDjbAkIY4+iiDhE+4OCRt3b2sj48fhQTtCVcilU1nytig3wlKLCFiBEnXX+ixRdLWePR+UYKnVSoTjKJOaXQhLEnMIWEESSblaCKXm7ZgU1vVBprsqvGdDEWJz6iZi1SGsQIm5OyRUL2ekrt6uDvHxFeFjliMxofIAYRHVZGIBS2GA1CqyG1tAmU/JgDW5ax0QsJTq54xQYoCSc2HxGcLjrUbhjwaHy0PCRndE6uq1f8BrbvKrjzfjRljkFkRIImQFRrSsuAlYg/mMDFizzgGHhGo180c7LoTF1QgzTnYwYRKcCHf1yFr/YUIGrMeGc63q8WpvvXIhLEqCJhbgvAdFNOfq3KXxTOpqPe05Xwh1Xe0rx5WwiCRLElA1ExQZN1UNknb1RC6EowvnnvuHz/CBK2GR3Wsykqxg4GIFNwFLa15SAtbt8IB4ZbgzwpWwannj64tlrIWBwLlrfdaeFxTP/c6xLaYYdeVOWD+yxiRLKWAtDAI+XnKVYTVfyIA1fnHsuStV4hpBd8IiLoVVhUTQz/gsqblrfVaWFM99ceVUV198hh3uhEVefKCkYJH6D59wVY8sDW8onvvSqef+1WfY4U5Y0SzxHpFkBUDMXVXD4IX03Mdvfee6ouzY3AmLqxjTO6mAtdB3IilXh4SDOSVzHz079NxVgXqHoDth/UhVDX+RJGA8lu8kCq4OCVuPFM/9wWFjDuEz7HAZseKEe6JgoJ/f1DKijQg0zTz6aI0OZSFczJ15o4qJrtxGrGiZ+JPQT+E3SZurmLTGcD40PfRT+zTP/cWpriomcnEpLC5l/LKoRSRZ/hK1no8sNfr3j0+Pzy36cqm1lhQL62boaHFVaPvBd1wK6wc5TVyoBP2kz4y0VVWDNGj3XnXljEdmBTA9Wp370lG8Us115VpYsYLx6yLiYgo/4aJWh4Tq4OrxvVl+/ES1D7TmBamr8W3Tma4sbuNyKyzKzBzHtzwBD+Ctqhqk4cVHs/x4QRfWPa3OveNkR6hWra5dcissPml8gVUJSZZ/8DHTeuT3ZvmPxNwkYg0fj/XclWrKqqrTrbB+xKpEkgXDwT9qZrc+a9LnPL73fR51EpF4TVoN6ycnh4RK1brr3bWwyEGqYi7op30+8IkqNWBpWmv+OY9vG4Vem5QfVOdTiud+7yBgmfsMO1wLi3LdYrGGg2ifiOWoh4Rao3//1Z2a0SYRqQ2K5z5+cDC9T9eVzTtzLawIebBePfAlwLHwqTolzZYGrS/z+DYB65ZmkA56lMx95sBz1/eDdrHDtbB+1MgmsAyE5Q9x2iGh2ph/mce3gd4V0X4iA9ao23KiK9tP2L2wyBt8xTwK332Bp7U+D4bdhVEvo2tKFNJad5RDwqm9527tM+xwLyzSSJGqcLJ8IUlOWZNazzekXKiTiKSrN0qH6qXtQmjjM+zwQFhJ41qoHvwa4BhqWTJz13qUAQyze8ppota8plhYj007b9TOZ9jhXli0hugMGqJ9IEWpamjeOu2KUJ5pde73djtCtVRxJBD3wkJDdDBwNUo9sklXBCUKacNb8icn123beOWwxsALYRFJllpFE9jJ4SvkIaHSd9wVoS5pnnvPJmA58Bl2eCEsMocUk0iyTgwXpUxZG1xQ9nlPfVra1LuhBKz7hp2uUk4/WC+EFSUbojNYC08M5cRj0xVBKQaldUVIA1qH6oPNGFvFic+wwwth/SgbvztoiD41XJycS6y1uk67ItQO2ZgzHvWsrQZnPsMOT4SFhmjfiZJWg9boUUanTWlqUdqvFAl2rQ8JldIhU/w9ERbREC2JaAI7KVySKAnfjE6jdkWQAUtr3JGbx/HiyjJgKaXKIbUFnggLDdE+w8UoU9Ya95S06XVOWwj7FM99dDmwClhq6bBeBk+ERWuIxoWYp6RCVjVo8ydynzdb0iys9iWtMadpdUjo3GfY4YmwfqQE4/uvYi08IeQwltWqtaRM/3+hdUUoV5RcbHFnrSvn+8F3vBEWWb6BhugTwpVl4qPXOpQ2rrdnilq0YZfiot5aVcsctB98xxthRcvEH4CG6NNBaX2WqIeEVM9dvD/Uc1dKB+vKI2HxKcJUwdSZU0E9JJTuKMbUE/WQ8OqB1NVkaRGwDvMZdngjrB9xYvdbTQf9AXxXeMpAd7VJOSQc0a4rkRrXlDG2DxaNObqujuhh8ChioSHaN7go5S7uxiXtuhKa4Sl1aI05z+YWlq6rY96mRxErWiYbolGTdRJqFfKQUOlTDglnPceNOV3zMbZq6bjhnx4JK0I2RGPqzGmIlchDwnaX1hVBbcyhee6zK9N5Nbqujmtg8EhYP2KYOuMPlLVhc78SqRZqV4TanNI8d9MOVeVYXXkmLHLhl9EQfQoorc9av+uwK0Kj5WLjp75ZwDpeV54Jq1ZGQ7QfxLJkHKIZU+NHynUl0oDmuc/uzDJ3PW8/umDTK2GhIdof0mQ9MtWYonZFaM0XWmOOyay/Y/eD73glLC5KuMFCBX2rXhPPk2pp0GY9XlMPCZ8pnvvUbBiWK115JqwfPGEHy2iI9hzaVUwdSlXDG+2Ehu65X5sshMf6DDu8ExalVwfHhR6ToLQ+N2jV6/e0OndaaFs/zE115SoueCYsjjwZFZBkeQtPuYqJekj4MKQdEvZooc3Ec3exH3zHO2FFyYboCsx3LyGHGWyMKbJ6fTzqUQKW2qKEtrcO3cJyrSvvhEVriMaYW0+JklUNGs1Jn3RpapHuidA2frtrmOjqeJ9hh4fCQkP0aYmkyMydVr1Ov66EEtrGC3NduX63HgqLaIiWxAQsUu+IFSmdhBeUQ0LqFeGk567ryjS/cv9uPRQW2RCtoCHaO2oV8pBQpTjp1K4IbUDWP7yZ6ErfD3qw6fJQWJSG6DzWQs9IkAPdtWaX1hVBG4ZF1j9MO2bxKumF/+ilsNJkkoXSGa+IZhXCQqA56ZMX2iQicozt03PjRPvBd7wUVoJMstAE5hWUemRt+EgaU1Oq5943XEAxfu2dVleeCotsiF6V4WR5A631WaSMuKJ2RUjGJunJg4muRPc+ww4vhUU2RK/yaIj2BI5yFZM2fyUD1iutK0L66rmPR7dzs/zKsxYYL4X1I4mG6BMRIw8JV7QRV9SuCGNjzuzFpJveE59hh6fCIi+9ltEQ7QV8luKNUtptJvSuiK/HiW93Dbqu1JKHgxg9FVaUMBxWWSRZ7uETlENCSrsNvStCbd7u39b02DO5Htojn2GHp8KiNUQjyXJPLW/8lOiHhKMLWkqu7gWs8WI5NJlWJHqqK2+FhYbokxAhc9eVQrl7fvw0pHVFqJ/e6HjWfW6Z1YsWPfIZdngrLLIhWkZDtGtilKqGBuUenNmSmjup3dFuM3izHEomjV5i0VW9KIm3wqJMxkR9slsiKfIqJq3/Sgas2zY1GulZ/ngjq+nF3LQvVY9XHr9rooiq7malJa8el0pxVDi4g3Z3nEZZCd86K/oy11guRqNFd2gxn8FzXdWI0XCym4oEyoWfuLzQJZGMSBlP2ybqZSbXZtP/1cHwrtNvqKbXLylF9z4Dtw8fIQ1dpZrgv/yQg4Dz8ZORVF0l33TM8HoOXxS8Q5myttrcSmis8HudW0wiajQsBkEqxYTrfIWPJVPp1I50JS8Qq66kVLMV/Wc+/pNwEMCiie3rVXK0m6/FUjmd+vKfzYtCWQ6h3R23VdbV9etsL2qNLK8r0Szm9YkF97qKVorVPQSZIgRJrQt7P1PKJ+1eNZUv7V6O/oWoV43oL4qc3hnpqoketOb8eXk7nU22kWvy0re/IpyKnHffTMUlq7b3shKoNgfJXLygHPyiIu5EcUY0b/pwNU1Tmp3l9e3NdDG9HR7+IWyp5zzYXUXLdtdn0hCsz/toB+8OlIX7wZzApaqWj1GSNG3Q7N11WodHjA2KUPbiG04r6rHHxuOkHZA6EBZ6Dp3AO/jWSqrOMR/sJp9Oe/IxxPNHCct60BWtPdeBsNBn4QQuXT3i4Tr+EPS03ZNd1HHCEiGswOCjRySwThHKcY/yEUSs8JHIH5NpOPkEiumYV6YPIlb44GKZonxUBmWJWs15103MxRCxwgcXiedKorfSUuvFipfPHxErnPCJvOChtBSxmvNsFdyCHCuk8LVUQVa8kZaqCLm4N5vBTxCxQkskmsp7kmpJQi4R9dicRo4VZiKxVL7uVlpKNZuMeX/mceRSaOO8545y3iGsQ+H4eDpXFY942B/Ui+WU19FqSzR71Flh2fJFnZw6UITlben+uRBL54rCUZapJFcL5cSJ7nTgU8IRZ5U2ty1xieLhclVQsXwkkUSmUK0fGLdUWSjl0idcI2LlqlA/DKFoOyYiUzziRRGwjoZPlIuCLDrbJUqqKNer+fSJSysjiUr5MBz4/lz84BdFvHIDH6npa2JJVhTLsgZJVRVFFAqZRDTCn/qB85FDcfKODn3Nk/+Z3x6uFo0l0tlCVTZNbiRRKOYqiVi0htI3cBC1WDyRTGey+WKpulkb9filKvrKJ5SKhVy5kkrEY7qo8CUGx8Bv5ZVKVyqZLZVKJZVK6oFKj1PQFHAF2ukAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHMH/ARGGFR0KtXQfAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDIyLTA2LTI4VDA2OjU0OjU0KzAwOjAwfMlSewAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMi0wNi0yOFQwNjo1NDo1NCswMDowMA2U6scAAAAASUVORK5CYII= 

[envtest]Do not use simulated success when run against existing cluster

When envtest run locally the test case simulate Job and Deployment success by directly updating Status fields of CRs. When envtest runs against an external cluster the test should not simulate the success but actually need to wait for the success to happen automatically by the cluster. So make the simulation steps dependent on USE_EXISTING_CLUSTER ENV variable.

Do not persist Status if Reconcile exits due to panic

Today we use a deferred call to always update the Status field when the Reconcile() call exists. We should consider not doing that if the Reconcile() exited due to panic, to avoid persisting potentially incorrect data.

Use db.CreateOrPatchDBByName to select the proper MariaDB CR

As openstack-k8s-operators/lib-common#63 merged we can now support different DB Service instance for different DB schemas by passing the MariaDB CR name to lib-common to indicate which DB Service we want to use. So Nova controller needs to be adapted to do so when create nova_api and cell0 DB.

when deploying with a real cell we can try and create it in the db even if it already exists

i have not had time to dig into this but it looks like our multi cell support is not entirely correct.

[centos@vscode nova-operator]$ oc logs job/nova-cell1-conductor-db-sync
Defaulted container "nova-cell1-conductor-db-sync" out of: nova-cell1-conductor-db-sync, init (init)
INFO:__main__:Loading config file at /var/lib/config-data/merged/nova-conductor-config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/config-data/merged/nova.conf to /etc/nova/nova.conf
INFO:__main__:Setting permission for /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/config-data/merged/custom.conf to /etc/nova/nova.conf.d/custom.conf
INFO:__main__:Setting permission for /etc/nova/nova.conf.d/custom.conf
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/nova
+ '[' cell1 = cell0 ']'
++ crudini --get /etc/nova/nova.conf api_database connection
+ API_DB=mysql+pymysql://nova_api:12345678@openstack/nova_api
+ '[' '!' -z API_DB ']'
++ crudini --get /etc/nova/nova.conf database connection
+ DB=mysql+pymysql://nova_cell1:12345678@openstack/nova_cell1
++ crudini --get /etc/nova/nova.conf DEFAULT transport_url
+ MQ=rabbit://default_user_VPe0WObxmJCkA-XPoHM:[email protected]:5672
+ nova-manage cell_v2 create_cell --name cell1 --database_connection mysql+pymysql://nova_cell1:12345678@openstack/nova_cell1 --transport-url rabbit://default_user_VPe0WObxmJCkA-XPoHM:[email protected]:5672 --verbose
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
The specified transport_url and/or database_connection combination already exists for another cell with uuid 4f99322a-6304-4b1a-a572-3abe98e792c9.

Start using TransportURL CR to provide message bus access to the nova services

Work is ongoing to have a TransportURL CR openstack-k8s-operators/openstack-operator#27 we can use for rabbit access. After that code is stabilized we can start using them

Support cell deletion

When an entry is removed from Nova.cellTemplates map the nova-controller does not delete the corresponding NovaCell CR today. But it should.

Allow passing in placement, neutron, cinder, glance, etc creds to Nova

Nova uses a bunch of other OpenStack services to function. To be able to call those services nova needs their user and password configured.

Extend our CRDs to allow passing in the necessary users and passwords
Pass those values to the different service CRs they need them
Use those values in the config generation of the nova services

Remove the "Requeuing" from the log message when CR instance cannot be read

    we are not actuly seeting the requeue timeout here we proably shoudl do that because otherwise the log message is not correct

to have here. we woudl want to log where we actully reque or just remove Requeuing from the error log above

Originally posted by @SeanMooney in #87 (comment)

server create hangs

openstack --debug server create --flavor c1 --image cirros-0.5.2-x86_64-disk --nic net-id=private vm5 --wait

openstack client got 504:

http://nova-public-openstack.apps-crc.testing:80 "POST /v2.1/servers HTTP/1.1" 504 92
RESP: [504] cache-control: no-cache connection: close content-length: 92 content-type: text/html
RESP BODY: Omitted, Content-Type is set to text/html. Only text/plain, application/json responses have their bodies logged.
Unknown Error (HTTP 504)
Traceback (most recent call last):
  File "/home/gibi/.local/lib/python3.11/site-packages/cliff/app.py", line 410, in run_subcommand
    result = cmd.run(parsed_args)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/cliff/display.py", line 117, in run
    column_names, data = self.take_action(parsed_args)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/openstackclient/compute/v2/server.py", line 1729, in take_action
    server = compute_client.servers.create(*boot_args, **boot_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/novaclient/v2/servers.py", line 1562, in create
    return self._boot(response_key, *boot_args, **boot_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/novaclient/v2/servers.py", line 887, in _boot
    return self._create('/servers', body, response_key,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/novaclient/base.py", line 363, in _create
    resp, body = self.api.client.post(url, body=body)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 401, in post
    return self.request(url, 'POST', **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gibi/.local/lib/python3.11/site-packages/novaclient/client.py", line 78, in request
    raise exceptions.from_response(resp, body, url, method)
novaclient.exceptions.ClientException: Unknown Error (HTTP 504)
clean_up CreateServer: Unknown Error (HTTP 504)
END return value: 1

nova-api-0 pods logs:

2023-03-22 17:23:54.283 12 DEBUG nova.api.openstack.wsgi [None req-1c727214-bb04-45fa-94bf-9214b172635d 1475b41fa12f4ccd917600dbdfab9f4c a715e52df9b14daca69eaa2c93c2add9 - - default default] Returning 404 to user: Flavor c1 could not be found. __call__ /usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py:942
2023-03-22 17:23:54.283 12 INFO nova.api.openstack.requestlog [None req-1c727214-bb04-45fa-94bf-9214b172635d 1475b41fa12f4ccd917600dbdfab9f4c a715e52df9b14daca69eaa2c93c2add9 - - default default] 192.168.130.1 "GET /v2.1/flavors/c1" status: 404 len: 75 microversion: 2.1 time: 0.082676
2023-03-22 17:23:54.287 12 DEBUG nova.api.openstack.wsgi [None req-91a1058c-df04-4435-aa99-c9b176e53e94 1475b41fa12f4ccd917600dbdfab9f4c a715e52df9b14daca69eaa2c93c2add9 - - default default] Calling method '<bound method FlavorsController.index of <nova.api.openstack.compute.flavors.FlavorsController object at 0x7f38522bcc70>>' _process_stack /usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py:513
2023-03-22 17:23:54.290 12 INFO nova.api.openstack.requestlog [None req-91a1058c-df04-4435-aa99-c9b176e53e94 1475b41fa12f4ccd917600dbdfab9f4c a715e52df9b14daca69eaa2c93c2add9 - - default default] 192.168.130.1 "GET /v2.1/flavors" status: 200 len: 333 microversion: 2.1 time: 0.004020
2023-03-22 17:23:54.293 12 DEBUG nova.api.openstack.wsgi [None req-30f1ef57-930c-4aa4-9cde-f9aa5b386912 1475b41fa12f4ccd917600dbdfab9f4c a715e52df9b14daca69eaa2c93c2add9 - - default default] Calling method '<bound method FlavorsController.show of <nova.api.openstack.compute.flavors.FlavorsController object at 0x7f3852246df0>>' _process_stack /usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py:513
2023-03-22 17:23:54.296 12 INFO nova.api.openstack.requestlog [None req-30f1ef57-930c-4aa4-9cde-f9aa5b386912 1475b41fa12f4ccd917600dbdfab9f4c a715e52df9b14daca69eaa2c93c2add9 - - default default] 192.168.130.1 "GET /v2.1/flavors/7c596f09-3ea0-4f77-8cc1-118e624d9243" status: 200 len: 500 microversion: 2.1 time: 0.003312
2023-03-22 17:23:54.457 12 DEBUG nova.api.openstack.wsgi [None req-147455bd-4973-457c-b8d8-243e846be3dc 1475b41fa12f4ccd917600dbdfab9f4c a715e52df9b14daca69eaa2c93c2add9 - - default default] Action: 'create', calling method: <bound method ServersController.create of <nova.api.openstack.compute.servers.ServersController object at 0x7f385208f8b0>>, body: {"server": {"name": "vm5", "imageRef": "fa062cd8-9cc0-458a-9181-c811e005ae44", "flavorRef": "7c596f09-3ea0-4f77-8cc1-118e624d9243", "min_count": 1, "max_count": 1, "networks": [{"uuid": "6e1cf8b3-6aee-4054-804f-074787a4bd43"}]}} _process_stack /usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py:511

at this point nova-api-0 hangs, then after a while k8s kills the pod and starts a new one.

There is no event in the statefulset nova-api about the restart but by watching the statefulset during the reproduction I see that the availableReplicas goes from 1 to 0 then back to 1.

Content of the log of nova-api-api container:

192.168.130.1 - - [22/Mar/2023:17:36:21 +0000] "GET /v2.1/flavors/c1 HTTP/1.1" 404 75 "-" "python-novaclient"
AH01628: authorization result: granted (no directives)
192.168.130.1 - - [22/Mar/2023:17:36:21 +0000] "GET /v2.1/flavors HTTP/1.1" 200 333 "-" "python-novaclient"
AH01628: authorization result: granted (no directives)
192.168.130.1 - - [22/Mar/2023:17:36:21 +0000] "GET /v2.1/flavors/7c596f09-3ea0-4f77-8cc1-118e624d9243 HTTP/1.1" 200 500 "-" "python-novaclient"
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
Timeout when reading response headers from daemon process 'nova-api': /usr/bin/nova-api-wsgi
192.168.130.1 - - [22/Mar/2023:17:36:21 +0000] "POST /v2.1/servers HTTP/1.1" 504 247 "-" "python-novaclient"
AH00470: network write failure in core output filter
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
Timeout when reading response headers from daemon process 'nova-api': /usr/bin/nova-api-wsgi
10.217.0.1 - - [22/Mar/2023:17:36:24 +0000] "GET / HTTP/1.1" 504 247 "-" "kube-probe/1.25"
Timeout when reading response headers from daemon process 'nova-api': /usr/bin/nova-api-wsgi
10.217.0.1 - - [22/Mar/2023:17:36:24 +0000] "GET / HTTP/1.1" 504 247 "-" "kube-probe/1.25"
Timeout when reading response headers from daemon process 'nova-api': /usr/bin/nova-api-wsgi
10.217.0.1 - - [22/Mar/2023:17:36:24 +0000] "GET / HTTP/1.1" 504 247 "-" "kube-probe/1.25"
Timeout when reading response headers from daemon process 'nova-api': /usr/bin/nova-api-wsgi
10.217.0.1 - - [22/Mar/2023:17:36:24 +0000] "GET / HTTP/1.1" 504 247 "-" "kube-probe/1.25"
Timeout when reading response headers from daemon process 'nova-api': /usr/bin/nova-api-wsgi
10.217.0.1 - - [22/Mar/2023:17:36:29 +0000] "GET / HTTP/1.1" 504 247 "-" "kube-probe/1.25"
AH00470: network write failure in core output filter
Timeout when reading response headers from daemon process 'nova-api': /usr/bin/nova-api-wsgi
10.217.0.1 - - [22/Mar/2023:17:36:29 +0000] "GET / HTTP/1.1" 504 247 "-" "kube-probe/1.25"
AH00470: network write failure in core output filter
mod_wsgi (pid=12): Shutdown requested 'nova-api'.

Based on this the nova-api-wsgi just hang and that blocked the kube-probe requests as well.

Fix post merge image build

⏳ Pushing "nova-operator:latest, nova-operator:9d34e43bd0dfa07b8f77804bea77c1cdb937e275" to "quay.io//nova-operator:latest, quay.io//nova-operator:9d34e43bd0dfa07b8f77804bea77c1cdb937e275" respectively as ""
/usr/bin/podman push --quiet --digestfile nova-operator-latest_digest.txt nova-operator:latest quay.io//nova-operator:latest --tls-verify=true --creds=:
Error: writing blob: initiating layer upload to /v2/***/nova-operator/blobs/uploads/ in quay.io: unauthorized: access to the requested resource is not authorized

https://github.com/openstack-k8s-operators/nova-operator/actions/runs/3225689260

support multiple ceph users per cluster

          basically i just need to change the cluster user map form a map of string to string to a map of string to list of string and add an additional loop over the users

Originally posted by @SeanMooney in #301 (comment)

The [keystone_authtoken] www_authenticate_uri option should point public url

Currently the [keystone_authtoken] www_authenticate_uri option points internal url, but this should point public url, because the url is returned to clients when auth request fails.

We probably have to add separate options for public/internal endpoint and use different endpoints accordingly.

[Nova]Consolidate cell0 and nova_api DB syncs

Do the cell DB sync in the NovaConductor. Which means do the cell0 db sync in the super conductor
Mova the nova_api DB sync to the Nova.
Do the orchestration of cell0 and NovaAPI from Nova.
3.1. Nova CR creates nova_api DB and sync it
3.2. Nova CR creates NovaCell(cell0) and wait until cell0 DB sync is ready there
3.3. Nova creates the NovaAPI passing in access to both nova_api and cell0 DB
3.4. NovAPI does not do db sync, it assumes that both nova_api and cell0 DB is ready to be used.

The nova external compute reconsiler is useing the wrong secrete for the config generation

as of ab34bb0

the edpm ssh key secrete is being passed to the config gen function instead of the instance CR secret

ab34bb0#diff-195a08f97364d8db1117085a1fe611eb7a17542c70dd06f224e439bdc2ddd439R161-R194

as a result, the nova keystone user password is not set in the nova.conf in compute nodes.

Register Nova cells from a Job started by the Nova controller

Currently the dbsync run by NovaConductor controller registers the cell to the DB if the cell has API DB access. We should move this logic to a Job in Nova controller to make it work for cells without API DB access.

Another reason to move the cell registration logic to Nova as the top level services (nova-api, nova super conductor, nova-scheduyler) need to be restarted after a new cell is registered to clean up the cell caches in these services.

ExternalCompute controller re-runs the ansible deplyoment steps after each nova-operator restart

It seems the current reconciler re-runs the deployment steps after each nova-operator restart.
To reproduce:

create NovaExternalCompute and wait until it becomes ready after it run the ansible deploy steps
delete the nova-operator-controller-manager pod to trigger a restart of nova-operator

Expected behavior:
If nothing is changed in the NovaExternalCompute resource then the ansible code is not re-run

Actual behavior:
The ansible code is re-run each time.

While re-running ansible should not cause any trouble as we expect that the ansible code is idempotent it will still causes many 2N ansible execution in an environment with N compute for each nova-operator restart.

[envtest]Turn on parallel and randomized execution

To tease out unstable tests run them in parallel and in random order. This might need a change to the make test target to call ginkgo directly instead of via go test

Duplicated Job execution

Our Job (e.g. DB sync) execution logic is the following:

construct the job definition
call job.DoJob() to start or wait for Job to finish
when Job finished successfully persist the hash of the job definition to instance.Status.Hash
delete the Job

Even after fixing the logic error in lib-common and moving the job deletion after the hash is persisted I can still reproduce duplicated Job execution in envtest but with less frequency and with a different root cause.

Note that this can be reproduced infrequently and it needs the compute running the test env to be (IO?) loaded. I used pre-commit run -a in a loop in parallel to generate the load.

The below logs produced with an instrumented version of the nova-operator

The test were run with KUBEBUILDER_ASSETS=/home/gibi/.local/share/kubebuilder-envtest/k8s/1.24.2-linux-amd64/ ginkgo --focus '' -v --fail-fast ./...

There are two types of reproduction:

With just a single NovaConductor CR:

1.6658193073443248e+09	INFO	---Test---	Simulated job success	{"on": "66c980f4-8034-42fb-8121-9090bfa3d3d3/192d77d4-ce9d-4473-8ee8-34049e11ac0f-cell-db-sync"}
1.6658193073443933e+09	INFO	---Test---	ExpectCondition	{"type": "DBSyncReady", "expected status": "True", "on": "66c980f4-8034-42fb-8121-9090bfa3d3d3/192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.665819307349581e+09	INFO	Reconciling 	{"controller": "novaconductor", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaConductor", "NovaConductor": {"name":"cf487061-647c-4a70-ad62-fbcb97f7de2e","namespace":"1c365e00-0f5c-46d0-85b7-15d327e17da6"}, "namespace": "1c365e00-0f5c-46d0-85b7-15d327e17da6", "name": "cf487061-647c-4a70-ad62-fbcb97f7de2e", "reconcileID": "fea4a743-e4fd-4e41-9806-f576d0503f65", "request": "1c365e00-0f5c-46d0-85b7-15d327e17da6/cf487061-647c-4a70-ad62-fbcb97f7de2e"}
1.6658193073497856e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "1c365e00-0f5c-46d0-85b7-15d327e17da6", "ObjectName": "cf487061-647c-4a70-ad62-fbcb97f7de2e"}
1.6658193073630116e+09	INFO	Reconciling 	{"controller": "novaconductor", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaConductor", "NovaConductor": {"name":"30653c02-33ff-403b-9385-7e14a9dad922","namespace":"d5687987-fea3-44d1-bb6a-8d5e3c5856d4"}, "namespace": "d5687987-fea3-44d1-bb6a-8d5e3c5856d4", "name": "30653c02-33ff-403b-9385-7e14a9dad922", "reconcileID": "f41669b2-4019-4894-ad29-615735fa8bec", "request": "d5687987-fea3-44d1-bb6a-8d5e3c5856d4/30653c02-33ff-403b-9385-7e14a9dad922"}
1.6658193073632247e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "d5687987-fea3-44d1-bb6a-8d5e3c5856d4", "ObjectName": "30653c02-33ff-403b-9385-7e14a9dad922"}
1.6658193073699455e+09	INFO	Reconciling 	{"controller": "novaconductor", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaConductor", "NovaConductor": {"name":"192d77d4-ce9d-4473-8ee8-34049e11ac0f","namespace":"66c980f4-8034-42fb-8121-9090bfa3d3d3"}, "namespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "name": "192d77d4-ce9d-4473-8ee8-34049e11ac0f", "reconcileID": "801b971f-08ea-453a-a69b-28b1f81bc898", "request": "66c980f4-8034-42fb-8121-9090bfa3d3d3/192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.6658193073700974e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "ObjectName": "192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.6658193073709254e+09	INFO	controllers.NovaConductor	Job Status Successful
1.6658193073709402e+09	INFO	controllers.NovaConductor	DoJob	{"persisted hash": "", "new job hash": "n55ch64dh76h659h57dh566h54h65dh89h5ddh64ch5d5hbdh98h64h6ch664h8h8dh578h97h64fh8dhd7h655h554h54bhf6h69h9fh5f7h65bq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "ObjectName": "192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.6658193073709626e+09	INFO	controllers.NovaConductor	!@!@!@!@!@!@!@!@!@!About to persist DB sync hash	{"hash": "n55ch64dh76h659h57dh566h54h65dh89h5ddh64ch5d5hbdh98h64h6ch664h8h8dh578h97h64fh8dhd7h655h554h54bhf6h69h9fh5f7h65bq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "ObjectName": "192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.6658193073753877e+09	INFO	controllers.NovaConductor	Deleting Job	{"Job.Namespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "Job.Name": "192d77d4-ce9d-4473-8ee8-34049e11ac0f-cell-db-sync", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "ObjectName": "192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.6658193073759675e+09	INFO	---Test---	ExpectCondition succeeded	{"type": "DBSyncReady", "expected status": "True", "on": "66c980f4-8034-42fb-8121-9090bfa3d3d3/192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.6658193073774097e+09	INFO	controllers.NovaConductor	@@ successfully saved hash	{"hash": "n55ch64dh76h659h57dh566h54h65dh89h5ddh64ch5d5hbdh98h64h6ch664h8h8dh578h97h64fh8dhd7h655h554h54bhf6h69h9fh5f7h65bq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "ObjectName": "192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.665819307377457e+09	INFO	controllers.NovaConductor	@@ instance read back	{"hash": "n55ch64dh76h659h57dh566h54h65dh89h5ddh64ch5d5hbdh98h64h6ch664h8h8dh578h97h64fh8dhd7h655h554h54bhf6h69h9fh5f7h65bq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "66c980f4-8034-42fb-8121-9090bfa3d3d3", "ObjectName": "192d77d4-ce9d-4473-8ee8-34049e11ac0f"}
1.6658193073775167e+09	INFO	Reconciling 	{"controller": "novaconductor", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaConductor", "NovaConductor": {"name":"4c10a882-b51d-4a8d-97d7-560e5151e505","namespace":"dab13b22-73aa-4ee7-b238-f007942d3297"}, "namespace": "dab13b22-73aa-4ee7-b238-f007942d3297", "name": "4c10a882-b51d-4a8d-97d7-560e5151e505", "reconcileID": "7792e8d0-3698-4623-b4ea-e0a7685c97a6", "request": "dab13b22-73aa-4ee7-b238-f007942d3297/4c10a882-b51d-4a8d-97d7-560e5151e505"}
1.6658193073776546e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "dab13b22-73aa-4ee7-b238-f007942d3297", "ObjectName": "4c10a882-b51d-4a8d-97d7-560e5151e505"}
1.665819307385968e+09	INFO	Reconciling 	{"controller": "novaconductor", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaConductor", "NovaConductor": {"name":"cf487061-647c-4a70-ad62-fbcb97f7de2e","namespace":"1c365e00-0f5c-46d0-85b7-15d327e17da6"}, "namespace": "1c365e00-0f5c-46d0-85b7-15d327e17da6", "name": "cf487061-647c-4a70-ad62-fbcb97f7de2e", "reconcileID": "30e6a162-6c99-44f5-a4cc-aba9760af803", "request": "1c365e00-0f5c-46d0-85b7-15d327e17da6/cf487061-647c-4a70-ad62-fbcb97f7de2e"}
1.6658193073861756e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "1c365e00-0f5c-46d0-85b7-15d327e17da6", "ObjectName": "cf487061-647c-4a70-ad62-fbcb97f7de2e"}

The log shows that NovaConductor saved the hash without an error and it was even able to read it back from k8s. Then the next reconciliation of NovaConductor the controller reads the instance again and it has no hash. But the test ends here so the second job creation is not visible in the log but based on the code it would lead to a second DB sync.

Drive the whole thing from Nova CR:

1.665822723452333e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234566615e+09	INFO	controllers.NovaConductor	Job Status incomplete... requeuing
1.665822723456689e+09	INFO	controllers.NovaConductor	DoJob	{"persisted hash": "", "new job hash": "n587h5cfhffh67fhbdh5ffhdch695h67dh677h97h8h689h59dhd9hb9h84h698h55ch598h5cdh5f6h4h5d9hd8h64bh64fh548h57fh699h594h57dq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.665822723457244e+09	INFO	---Test---	Simulated job success	{"on": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c/f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0-cell-db-sync"}
1.6658227234574397e+09	INFO	controllers.NovaCell	NovaConductor f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0 updated.	{"ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234574695e+09	INFO	controllers.NovaCell	After updating conductor	{"hash": "", "ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234736543e+09	INFO	controllers.Nova	Updated status at the end of reconciliation	{"ObjectType": "*v1beta1.Nova", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc"}
1.665822723473751e+09	INFO	Reconciling 	{"controller": "nova", "controllerGroup": "nova.openstack.org", "controllerKind": "Nova", "Nova": {"name":"f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc","namespace":"ff442c24-e9d5-470b-b54b-ce6d48bf8f0c"}, "namespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "name": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc", "reconcileID": "28628b44-6dea-45af-b1f4-30a3ca334114", "request": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c/f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc"}
1.6658227234741585e+09	INFO	controllers.Nova	Reconciling	{"ObjectType": "*v1beta1.Nova", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc"}
1.665822723479182e+09	INFO	Reconciling 	{"controller": "novaconductor", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaConductor", "NovaConductor": {"name":"f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0","namespace":"ff442c24-e9d5-470b-b54b-ce6d48bf8f0c"}, "namespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "name": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0", "reconcileID": "12b8e326-f4e4-4a28-9712-77a6f0490dd4", "request": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c/f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234793909e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234871306e+09	INFO	controllers.NovaCell	Updated status at the end of reconciliation	{"ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234947073e+09	INFO	Reconciling 	{"controller": "novacell", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaCell", "NovaCell": {"name":"f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0","namespace":"ff442c24-e9d5-470b-b54b-ce6d48bf8f0c"}, "namespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "name": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0", "reconcileID": "c65a37d8-3f90-41a4-9c87-14f377e30c74", "request": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c/f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234975305e+09	INFO	controllers.NovaConductor	Job Status Successful
1.6658227234975674e+09	INFO	controllers.NovaConductor	DoJob	{"persisted hash": "", "new job hash": "n587h5cfhffh67fhbdh5ffhdch695h67dh677h97h8h689h59dhd9hb9h84h698h55ch598h5cdh5f6h4h5d9hd8h64bh64fh548h57fh699h594h57dq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.665822723497598e+09	INFO	controllers.NovaConductor	!@!@!@!@!@!@!@!@!@!About to persist DB sync hash	{"hash": "n587h5cfhffh67fhbdh5ffhdch695h67dh677h97h8h689h59dhd9hb9h84h698h55ch598h5cdh5f6h4h5d9hd8h64bh64fh548h57fh699h594h57dq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234992487e+09	INFO	controllers.NovaCell	Reconciling	{"ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227234993303e+09	INFO	controllers.NovaCell	Before updating conductor	{"hash": "", "ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235178542e+09	INFO	controllers.NovaConductor	Deleting Job	{"Job.Namespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "Job.Name": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0-cell-db-sync", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235287838e+09	INFO	controllers.NovaConductor	@@ successfully saved hash	{"hash": "n587h5cfhffh67fhbdh5ffhdch695h67dh677h97h8h689h59dhd9hb9h84h698h55ch598h5cdh5f6h4h5d9hd8h64bh64fh548h57fh699h594h57dq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235288546e+09	INFO	controllers.NovaConductor	@@ instance read back	{"hash": "n587h5cfhffh67fhbdh5ffhdch695h67dh677h97h8h689h59dhd9hb9h84h698h55ch598h5cdh5f6h4h5d9hd8h64bh64fh548h57fh699h594h57dq", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235289252e+09	INFO	Reconciling 	{"controller": "novaconductor", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaConductor", "NovaConductor": {"name":"f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0","namespace":"ff442c24-e9d5-470b-b54b-ce6d48bf8f0c"}, "namespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "name": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0", "reconcileID": "a534f3d5-5df8-4734-9cd2-f4d9de67cb5c", "request": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c/f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235291655e+09	INFO	controllers.NovaConductor	Reconciling	{"hash": "", "ObjectType": "*v1beta1.NovaConductor", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235402277e+09	INFO	controllers.NovaCell	Updated status at the end of reconciliation	{"ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235402787e+09	ERROR	Reconciler error	{"controller": "novacell", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaCell", "NovaCell": {"name":"f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0","namespace":"ff442c24-e9d5-470b-b54b-ce6d48bf8f0c"}, "namespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "name": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0", "reconcileID": "c65a37d8-3f90-41a4-9c87-14f377e30c74", "error": "Operation cannot be fulfilled on novaconductors.nova.openstack.org \"f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/gibi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/gibi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/gibi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
1.6658227235403993e+09	INFO	Reconciling 	{"controller": "novacell", "controllerGroup": "nova.openstack.org", "controllerKind": "NovaCell", "NovaCell": {"name":"f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0","namespace":"ff442c24-e9d5-470b-b54b-ce6d48bf8f0c"}, "namespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "name": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0", "reconcileID": "e09db44e-2ca7-4950-8373-1d5374f83bca", "request": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c/f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.665822723540631e+09	INFO	controllers.NovaCell	Reconciling	{"ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.6658227235406656e+09	INFO	controllers.NovaCell	Before updating conductor	{"hash": "n587h5cfhffh67fhbdh5ffhdch695h67dh677h97h8h689h59dhd9hb9h84h698h55ch598h5cdh5f6h4h5d9hd8h64bh64fh548h57fh699h594h57dq", "ObjectType": "*v1beta1.NovaCell", "ObjectNamespace": "ff442c24-e9d5-470b-b54b-ce6d48bf8f0c", "ObjectName": "f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0"}
1.665822723546473e+09	INFO	controllers.NovaConductor	Job dbsync f5f7bc8c-bf1c-4c0a-a1a5-0c80f0d942cc-cell0-cell-db-sync - created

This also shows that the job finished, the hash is persisted and read back and the Job is deleted. Then the next reconcile of NovaConductor does not see the persisted hash and therefore creates the Job again.

It is almost like the reconciler reads stale data from k8s / etcd. I tried to reproduce this stale read behavior in a simple operator but without luck.

When PreserveJobs set to False from True then the controller should delete the existing failed Jobs.

    im not partically happy with supporting this.

but im glad you at least have tests for it

if we do support this when its set back to false it shoudl delete all jobs that were created while the deleteion is disabeld.

Originally posted by @SeanMooney in #87 (comment)

Make naming of our CRs consistent

    it took me a while to realise we are not prefixing the names here because i assume

we are using the rc name as the name of the db right?

can you add a comment to descibe why the dbs are differnt just to make it clear.

Originally posted by @SeanMooney in #87 (comment)

The nova-api service runs with partial oslo.config due to config validation error

We set metadata_workers to 0 for the nova-api and the nova-api starts but fails to load the full oslo configuration as the config validation fails with:

$ oc logs nova-api-0 -f -c nova-api-api
[snip]
+ exec /usr/sbin/httpd -DFOREGROUND
mod_wsgi (pid=12): Attach interpreter ''.
mod_wsgi (pid=12): Imported 'mod_wsgi'.
mod_wsgi (pid=12): Started thread 0 in daemon process 'nova-api'.
AH01628: authorization result: granted (no directives)
AH01628: authorization result: granted (no directives)
mod_wsgi (pid=12, process='nova-api', application=''): Loading Python script file '/usr/bin/nova-api-wsgi'.
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
mod_wsgi (pid=12): Callback registration for signal 12 ignored.
  File "/usr/bin/nova-api-wsgi", line 52, in <module>
    application = init_application()
  File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/wsgi.py", line 20, in init_application
    return wsgi_app.init_application(NAME)
  File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi_app.py", line 128, in init_application
    init_global_data(conf_files, name)
  File "/usr/lib/python3.9/site-packages/nova/utils.py", line 1122, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi_app.py", line 105, in init_global_data
    gmr.TextGuruMeditation.setup_autorun(
  File "/usr/lib/python3.9/site-packages/oslo_reports/guru_meditation_report.py", line 152, in setup_autorun
    cls._setup_signal(signal.SIGUSR2,
  File "/usr/lib/python3.9/site-packages/oslo_reports/guru_meditation_report.py", line 185, in _setup_signal
    signal.signal(signum,
mod_wsgi (pid=12): Failed to exec Python script file '/usr/bin/nova-api-wsgi'.
mod_wsgi (pid=12): Exception occurred processing WSGI script '/usr/bin/nova-api-wsgi'.
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/oslo_config/cfg.py", line 2716, in _do_get
    return (convert(val), alt_loc)
  File "/usr/lib/python3.9/site-packages/oslo_config/cfg.py", line 2685, in convert
    return self._convert_value(
  File "/usr/lib/python3.9/site-packages/oslo_config/cfg.py", line 2813, in _convert_value
    return opt.type(value)
  File "/usr/lib/python3.9/site-packages/oslo_config/types.py", line 315, in __call__
    raise ValueError('Should be greater than or equal to %g' %
ValueError: Should be greater than or equal to 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/bin/nova-api-wsgi", line 52, in <module>
    application = init_application()
  File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/wsgi.py", line 20, in init_application
    return wsgi_app.init_application(NAME)
  File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi_app.py", line 128, in init_application
    init_global_data(conf_files, name)
  File "/usr/lib/python3.9/site-packages/nova/utils.py", line 1122, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi_app.py", line 114, in init_global_data
    CONF.log_opt_values(
  File "/usr/lib/python3.9/site-packages/oslo_config/cfg.py", line 2603, in log_opt_values
    _sanitize(opt, getattr(self, opt_name)))
  File "/usr/lib/python3.9/site-packages/oslo_config/cfg.py", line 2219, in __getattr__
    return self._get(name)
  File "/usr/lib/python3.9/site-packages/oslo_config/cfg.py", line 2653, in _get
    value, loc = self._do_get(name, group, namespace)
  File "/usr/lib/python3.9/site-packages/oslo_config/cfg.py", line 2732, in _do_get
    raise ConfigFileValueError(message)
oslo_config.cfg.ConfigFileValueError: Value for option metadata_workers from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/nova/nova.conf.d/01-nova.conf') is not valid: Should be greater than or equal to 1

Then the nova-api only logs a subset of the oslo.config parameters so it actually runs with a partial config:

2023-07-05 08:33:50.247 12 DEBUG nova.api.openstack.wsgi_app [-] ******************************************************************************** log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2589
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] Configuration options gathered from: log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2590
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] command line args: [] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2591
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] config files: ['/etc/nova/api-paste.ini', '/etc/nova/nova.conf'] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2592
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] ================================================================================ log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2594
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] allow_resize_to_same_host      = True log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] arq_binding_timeout            = 300 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] block_device_allocate_retries  = 60 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] block_device_allocate_retries_interval = 3 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] cert                           = self.pem log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] compute_driver                 = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.248 12 DEBUG nova.api.openstack.wsgi_app [-] compute_monitors               = [] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] config_dir                     = ['/etc/nova/nova.conf.d'] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] config_drive_format            = iso9660 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] config_file                    = ['/etc/nova/api-paste.ini', '/etc/nova/nova.conf'] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] config_source                  = [] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] console_host                   = nova-api-0 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] control_exchange               = nova log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] cpu_allocation_ratio           = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] daemon                         = False log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] debug                          = True log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] default_access_ip_network_name = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] default_availability_zone      = nova log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] default_ephemeral_format       = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.249 12 DEBUG nova.api.openstack.wsgi_app [-] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'glanceclient=WARN', 'oslo.privsep.daemon=INFO'] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] default_schedule_zone          = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] disk_allocation_ratio          = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] enable_new_services            = True log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] enabled_apis                   = ['osapi_compute', 'metadata'] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] enabled_ssl_apis               = [] log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] flat_injected                  = False log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] force_config_drive             = True log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] force_raw_images               = True log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] graceful_shutdown_timeout      = 60 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] heal_instance_info_cache_interval = 60 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] host                           = nova-api-0 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] initial_cpu_allocation_ratio   = 4.0 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] initial_disk_allocation_ratio  = 1.0 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] initial_ram_allocation_ratio   = 1.0 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.250 12 DEBUG nova.api.openstack.wsgi_app [-] injected_network_template      = /usr/lib/python3.9/site-packages/nova/virt/interfaces.template log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instance_build_timeout         = 0 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instance_delete_interval       = 300 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instance_format                = [instance: %(uuid)s]  log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instance_name_template         = instance-%08x log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instance_usage_audit           = False log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instance_usage_audit_period    = month log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instance_uuid_format           = [instance: %(uuid)s]  log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] instances_path                 = /var/lib/nova/instances log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] internal_service_availability_zone = internal log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] key                            = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] live_migration_retry_count     = 30 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] log_config_append              = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] log_date_format                = %Y-%m-%d %H:%M:%S log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] log_dir                        = None log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] log_file                       = /var/log/nova/nova-api.log log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] log_options                    = True log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] log_rotate_interval            = 1 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.251 12 DEBUG nova.api.openstack.wsgi_app [-] log_rotate_interval_type       = days log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] log_rotation_type              = size log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] logging_context_format_string  = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(user_identity)s] %(instance)s%(message)s log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] logging_debug_format_suffix    = %(funcName)s %(pathname)s:%(lineno)d log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] logging_default_format_string  = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] logging_exception_prefix       = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] logging_user_identity_format   = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] long_rpc_timeout               = 1800 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] max_concurrent_builds          = 10 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] max_concurrent_live_migrations = 1 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] max_concurrent_snapshots       = 5 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] max_local_block_devices        = 3 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] max_logfile_count              = 5 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] max_logfile_size_mb            = 50 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] maximum_instance_delete_attempts = 5 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] metadata_listen                = 0.0.0.0 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.252 12 DEBUG nova.api.openstack.wsgi_app [-] metadata_listen_port           = 8775 log_opt_values /usr/lib/python3.9/site-packages/oslo_config/cfg.py:2602
2023-07-05 08:33:50.254 12 INFO nova.api.openstack.wsgi_app [-] Global data already initialized, not re-initializing.

If the metadata_workers is set to 1 the nova-api dumps the full oslo.config as expected see full.txt

[lint][envtest]Create a rule that all Eventually and Consistently block uses a local Gomega instance for asserts

    if this was nova i woudl wonder if we should have a hacking check to ensure that where we have a Gomega object in scope we use it when adding asserts like this to ensure the failures are captures properly

Originally posted by @SeanMooney in #111 (comment)

EnvTest coverage for ceph integration

The current ceph integration does not have env test coverage or other test in ci.
we should fix that.

Failed AnsibleEE CRs should be deleted before retrying

A failed ansible run leaves the job and ansibleee CRs in place for forensic reasons. However, even deleting the NovaExternalCompute will cause those to not be re-created on the next run, leading to confusion about why changes are not being taken into account.

Any time we're going to do an ansible run, we should delete/recreate/update a previous failed ansibleee CR so that new changes to any of the inputs are used for the subsequent job.

[envtest]Make timeout configurable

When envtest runs agains an existing cluster it can be slower than with running locally, so test timeouts needs to come from ENV variables.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update openstack-k8s-operators (github.com/openstack-k8s-operators/infra-operator/apis, github.com/openstack-k8s-operators/keystone-operator/api, github.com/openstack-k8s-operators/mariadb-operator/api)
Update module github.com/gophercloud/gophercloud to v1.11.0

Detected dependencies

gomod

api/go.mod

github.com/openstack-k8s-operators/lib-common/modules/common v0.3.1-0.20240306153230-dc65ab49ebc0@dc65ab49ebc0

github.com/robfig/cron/v3 v3.0.1

k8s.io/api v0.28.7

k8s.io/apimachinery v0.28.7

k8s.io/utils v0.0.0-20240102154912-e7106e64919e@e7106e64919e

sigs.k8s.io/controller-runtime v0.16.5

github.com/go-logr/logr v1.4.1

github.com/onsi/ginkgo/v2 v2.16.0

k8s.io/apiextensions-apiserver v0.28.7

k8s.io/client-go v0.28.7

k8s.io/component-base v0.28.7

k8s.io/klog/v2 v2.120.1

go.mod

github.com/go-logr/logr v1.4.1

github.com/google/uuid v1.6.0

github.com/gophercloud/gophercloud v1.10.0

github.com/k8snetworkplumbingwg/network-attachment-definition-client v1.4.0

github.com/onsi/ginkgo/v2 v2.16.0

github.com/onsi/gomega v1.31.1

github.com/openstack-k8s-operators/infra-operator/apis v0.3.1-0.20240304123904-038a5ec77a70@038a5ec77a70

github.com/openstack-k8s-operators/keystone-operator/api v0.3.1-0.20240307113114-033a606862c3@033a606862c3

github.com/openstack-k8s-operators/lib-common/modules/common v0.3.1-0.20240306153230-dc65ab49ebc0@dc65ab49ebc0

github.com/openstack-k8s-operators/lib-common/modules/openstack v0.3.1-0.20240306153230-dc65ab49ebc0@dc65ab49ebc0

github.com/openstack-k8s-operators/lib-common/modules/test v0.3.1-0.20240306153230-dc65ab49ebc0@dc65ab49ebc0

github.com/openstack-k8s-operators/mariadb-operator/api v0.3.1-0.20240303091826-438dde8600d3@438dde8600d3

go.uber.org/zap v1.27.0

gopkg.in/yaml.v3 v3.0.1

k8s.io/api v0.28.7

k8s.io/apimachinery v0.28.7

k8s.io/client-go v0.28.7

k8s.io/utils v0.0.0-20240102154912-e7106e64919e@e7106e64919e

sigs.k8s.io/controller-runtime v0.16.5

k8s.io/apiextensions-apiserver v0.28.7

k8s.io/component-base v0.28.7

k8s.io/klog/v2 v2.120.1

	apiEndpoints, ctrlResult, err := endpoint.ExposeEndpoints(
	ctx,
	h,
	novametadata.ServiceName,
	getServiceLabels(),
	ports,
	r.RequeueTimeout,

	func getServiceLabels() map[string]string {
	return map[string]string{
	common.AppSelector: NovaAPILabelPrefix,
	}
	}

openstack-k8s-operators / nova-operator Goto Github PK

nova-operator's People

Contributors

Stargazers

Watchers

Forkers

nova-operator's Issues

Repository problems

Open

Detected dependencies

Open

Detected dependencies

Recommend Projects

Recommend Topics

Recommend Org