pulp / oci_env Goto Github PK

Dockerfile 1.68% Shell 35.25% Python 62.71% Makefile 0.36%

oci_env's Introduction

⚠️ ⛔️ Pulp2 is EOL as of November 30 2022, for more info visit this link https://pulpproject.org/2022/09/19/pulp-2-eol/. ⛔️

Pulp is a platform for managing repositories of content, such as software
packages, and pushing that content out to large numbers of consumers.

For more information, check out the project website:

http://www.pulpproject.org

oci_env's People

Contributors

Stargazers

Watchers

oci_env's Issues

oci-env compose build fails COMPOSE_BINARY=docker

Versions

Ubuntu 23.04
Python 3.11.4
Docker version 24.0.6, build ed223bc

Note

Works with podman

Traceback

oci-env compose build 
2023-11-08 15:27:57,399 - INFO - utils.py:37 - USING /home/workspace/oci_env FOR OCI_ENV_PATH BASED ON GIT CMD OUTPUT
2023-11-08 15:27:57,400 - INFO - utils.py:37 - USING /home/workspace/oci_env FOR OCI_ENV_PATH BASED ON GIT CMD OUTPUT
2023-11-08 15:27:57,401 - INFO - utils.py:37 - USING /home/workspace/oci_env FOR OCI_ENV_PATH BASED ON GIT CMD OUTPUT
2023-11-08 15:27:57,402 - INFO - commands.py:11 - COMPOSE Namespace(is_verbose=False, env_file='', command=['build'], func=<function compose at 0x7f655fca6e80>)
pulp uses an image, skipping
Building _base
[+] Building 22.2s (7/16)                                                                                                                                                      docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                     0.0s
 => => transferring dockerfile: 961B                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                          0.0s
 => [internal] load metadata for ghcr.io/pulp/pulp-ci-centos:latest                                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                        0.0s
 => => transferring context: 811B                                                                                                                                                        0.0s
 => [ 1/12] FROM ghcr.io/pulp/pulp-ci-centos:latest                                                                                                                                      0.0s
 => CACHED [ 2/12] COPY dev_requirements.txt /dev_requirements.txt                                                                                                                       0.0s
 => ERROR [ 3/12] RUN dnf install -y which                                                                                                                                              22.1s
------                                                                                                                                                                                        
 > [ 3/12] RUN dnf install -y which:
22.01 CentOS Stream 8 - AppStream                     1.6 MB/s |  34 MB     00:20    
22.03 Error: Cannot create directory "/var/cache/dnf/appstream-97a634f7b5a0dc54/repodata": Invalid argument
------
Dockerfile:10
--------------------
   8 |     
   9 |     COPY dev_requirements.txt /dev_requirements.txt
  10 | >>> RUN dnf install -y which
  11 |     
  12 |     COPY switch_python /usr/bin/switch_python
--------------------
ERROR: failed to solve: process "/bin/sh -c dnf install -y which" did not complete successfully: exit code: 1
ERROR: Service '_base' failed to build : Build failed

Services fail to start due to "connection in use: ('127.0.0.1', 24817)"

I'm on Fedora-36, and oci-env compose up is hitting the following, forever, trying to bring up services

40 static files symlinked to '/var/lib/pulp/assets'.
[2022-09-12 17:42:33 +0000] [245] [INFO] Starting gunicorn 20.1.0
[2022-09-12 17:42:33 +0000] [245] [ERROR] Connection in use: ('127.0.0.1', 24817)
[2022-09-12 17:42:33 +0000] [245] [ERROR] Retrying in 1 second.
[2022-09-12 17:42:34 +0000] [245] [ERROR] Connection in use: ('127.0.0.1', 24817)
[2022-09-12 17:42:34 +0000] [245] [ERROR] Retrying in 1 second.
[2022-09-12 17:42:35 +0000] [245] [ERROR] Connection in use: ('127.0.0.1', 24817)
[2022-09-12 17:42:35 +0000] [245] [ERROR] Retrying in 1 second.
[2022-09-12 17:42:36 +0000] [245] [ERROR] Connection in use: ('127.0.0.1', 24817)
[2022-09-12 17:42:36 +0000] [245] [ERROR] Retrying in 1 second.
[2022-09-12 17:42:37 +0000] [245] [ERROR] Connection in use: ('127.0.0.1', 24817)
[2022-09-12 17:42:37 +0000] [245] [ERROR] Retrying in 1 second.
[2022-09-12 17:42:38 +0000] [245] [ERROR] Can't connect to ('127.0.0.1', 24817)
Already migrated!
Deleting 'import_export/action_formats.js'
Deleting 'import_export/import.css'
Deleting 'rest_framework/css/bootstrap-theme.min.css'
Deleting 'rest_framework/css/bootstrap-tweaks.css'
Deleting 'rest_framework/css/bootstrap.min.css'

Anyone else seeing this?

Provide a method to rollback the base image

We've had a few instances where changes to the base image (ghcr.io/pulp/pulp-ci-centos:latest) have caused the developer environment to break (in some cases for up to a week with the user permission changes). Developers should have the option to rollback to a previous base image when this happens so that they can continue working while a fix is created.

`oci_env db reset` is recreating the database with a different user than expected

https://github.com/pulp/oci_env/blob/main/base/container_scripts/database_reset.sh#L9

The use of --user postgres is creating the db differently than originally by the oci container.

Auto reload workers.

Galaxy NG has this capability (https://github.com/ansible/galaxy_ng/blob/master/docker/bin/start-worker-reload#L6), and it just needs to be ported over from there.

How to run a single test or test file?

It would be great if there were some docs on how to run a single test or a single test file.

Non-fatal error during compose-up calling postgres-prepare

I see this error (seemingly from dynaconf) during compose up. It doesn't stop the machine from launching so not sure what is going on. Using pulpcore & pulp_file on main.

pulp_1   | Calling /etc/init/postgres-prepare
pulp_1   | 2022-09-28 20:17:15.957 UTC [596] LOG:  starting PostgreSQL 12.11 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-13), 64-bit
pulp_1   | 2022-09-28 20:17:15.958 UTC [596] LOG:  listening on IPv4 address "127.0.0.1", port 5432
pulp_1   | 2022-09-28 20:17:15.958 UTC [596] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
pulp_1   | 2022-09-28 20:17:15.958 UTC [596] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
pulp_1   | 2022-09-28 20:17:15.959 UTC [596] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
pulp_1   | 2022-09-28 20:17:15.961 UTC [596] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
pulp_1   | 2022-09-28 20:17:15.968 UTC [596] LOG:  redirecting log output to logging collector process
pulp_1   | 2022-09-28 20:17:15.968 UTC [596] HINT:  Future log output will appear in directory "log".
pulp_1   | Traceback (most recent call last):
pulp_1   |   File "/usr/local/bin/pulpcore-manager", line 8, in <module>
pulp_1   |     sys.exit(manage())
pulp_1   |   File "/src/pulpcore/pulpcore/app/manage.py", line 11, in manage
pulp_1   |     execute_from_command_line(sys.argv)
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
pulp_1   |     utility.execute()
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
pulp_1   |     settings.INSTALLED_APPS
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
pulp_1   |     self._setup(name)
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
pulp_1   |     self._wrapped = Settings(settings_module)
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
pulp_1   |     mod = importlib.import_module(self.SETTINGS_MODULE)
pulp_1   |   File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
pulp_1   |     return _bootstrap._gcd_import(name[level:], package, level)
pulp_1   |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
pulp_1   |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
pulp_1   |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
pulp_1   |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
pulp_1   |   File "<frozen importlib._bootstrap_external>", line 843, in exec_module
pulp_1   |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
pulp_1   |   File "/src/pulpcore/pulpcore/app/settings.py", line 299, in <module>
pulp_1   |     from dynaconf import DjangoDynaconf, Validator  # noqa
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/dynaconf/__init__.py", line 3, in <module>
pulp_1   |     from dynaconf.base import LazySettings  # noqa
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 17, in <module>
pulp_1   |     from dynaconf.loaders import env_loader
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/env_loader.py", line 8, in <module>
pulp_1   |     from dynaconf.vendor.dotenv import cli as dotenv_cli
pulp_1   |   File "/usr/local/lib/python3.8/site-packages/dynaconf/vendor/dotenv/cli.py", line 16, in <module>
pulp_1   |     @click.option('-f','--file',default=os.path.join(os.getcwd(),'.env'),type=click.Path(exists=_A),help='Location of the .env file, defaults to .env file in current working directory.')
pulp_1   | FileNotFoundError: [Errno 2] No such file or directory
pulp_1   | Operations to perform:
pulp_1   |   Apply all migrations: auth, contenttypes, core, file, python, sessions
pulp_1   | Running migrations:
pulp_1   |   No migrations to apply.

Unable to run tests

pulpcore-manager test pulp_rpm.tests.unit.test_models returns

Creating test database for alias 'default'...
Got an error creating the test database: permission denied to create database

unfortunately no trace back or line in the logs
maybe related to #17

Add a profile for pulp_container

The profile will configure the token server and stuff required to run pulp_container out of the box.

Allow profiles to add to DEV_SOURCE_PATH

It would be super helpful if a profile could hook paths into the install from source mechanics.

Services should depend on "base"

Otherwise, fix-attrs is potentially not run early enough.

just-containers/s6-overlay#476 (comment)

root power being required by some tests

Some tests require root permissions for specific file operations.

The easiest(not the right) way to solve this is adding the pulp user to the sudoers file with NOPASSWD or adding the pulp user to the sudo group.

Using #68 as an epic.

Include pulp_file to the `generate-client` command if using a pulpcore >=`3.40.1`

If you one is using the current main branch of pulpcore where it includes pulp_file now by default the default command oci-env generate-client -i (which installs all clients for the projects included in the DEV_SOURCE_PATH) does not install the pulp_file clients resulting into an error if you wanna run tests.

It works if you explicitly tell it to install the pulp_file client like so: oci-env generate-client -i pulp_file

It would be great if the default command can somehow check if pulp_file clients needs to be included on its own.

Add a scenario for the stream test runner

This will start an sftp storage container and configure the corresponding storage class.

More information on how we configure it in the CI today:
https://github.com/pulp/pulpcore/blob/main/.github/workflows/scripts/install.sh#L86
https://github.com/pulp/pulpcore/blob/main/.ci/ansible/settings.py.j2#L29

The modified file for pulpcore-api is outdated

https://github.com/pulp/oci_env/blob/main/base/s6-rc-modifications/pulpcore-api/run

Does not include changes like PULP_API_WORKERS

In fact, we can probably just follow the advice in the file, and put the missing feature in pulp-oci-images instead.

Fatal error during compose build

Using freshly pulled code

[dalley@thinkpad oci_env]$ oci-env compose build
['podman', '--version', '']
using podman version: 4.2.1
podman build -t localhost/oci_env/pulp:base -f /home/dalley/devel/oci_env/base/Dockerfile /home/dalley/devel/oci_env/base/
STEP 1/11: FROM ghcr.io/pulp/pulp-ci-centos:latest
STEP 2/11: ENV S6_KEEP_ENV=1
--> Using cache f88ba142767b421a71ec163871ac9990e964470c0d75cad411cb5c3f6c871207
--> f88ba142767
STEP 3/11: COPY dev_requirements.txt /dev_requirements.txt
--> Using cache af2c4a414af861793635c379a80bbfc19d0a66f6658be2891c9483a71eb5e005
--> af2c4a414af
STEP 4/11: RUN pip3 install -r dev_requirements.txt
--> Using cache b2b7897542fa116ecbfd3d936fde8c7b29d4a64c3b1470d768eb197eda26e5a2
--> b2b7897542f
STEP 5/11: COPY settings.py /etc/pulp/settings.py
--> Using cache fa33627095ef3ce546746c6519d7f677ba8efa6959ad77c858548e71d2c211d2
--> fa33627095e
STEP 6/11: COPY s6-rc.d/oci-env-prepare/ /etc/s6-overlay/s6-rc.d/oci-env-prepare/
--> Using cache 0e9162c7c4b3fcc062b5f2750c436b74d03af77db434b15a18ce524c00a40ef3
--> 0e9162c7c4b
STEP 7/11: COPY s6-rc.d/oci-env-profiles /etc/s6-overlay/s6-rc.d/oci-env-profiles/
--> Using cache 680a309870fb81681d721803e8a14907f27bfb9df19e0c3cbdf2721c0bf5d9be
--> 680a309870f
STEP 8/11: RUN cd /etc/s6-overlay/s6-rc.d/user/contents.d && touch oci-env-prepare oci-env-profiles
/bin/sh: line 0: cd: /etc/s6-overlay/s6-rc.d/user/contents.d: No such file or directory
Error: error building at STEP "RUN cd /etc/s6-overlay/s6-rc.d/user/contents.d && touch oci-env-prepare oci-env-profiles": error while running runtime: exit status 1
exit code: 1

Exit the container startup if there are errors installing a plugin.

Right now if a plugin can't be installed, the environment will happily start up without installing it. This is confusing and a bad user experience.

S6 can be configured to exit if an init script fails by setting this to 2: https://github.com/pulp/oci_env/blob/main/base/compose.yaml#L38, however the pulp-api init script is designed to fail and exit if it's already been run (https://github.com/pulp/pulp-oci-images/blob/latest/assets/init/pulpcore-api#L10), which blocks the container from starting up.

Once this is fixed in the base images, the containers can be configured to exit on errors.

A mismatch between repositories' names and installed packages

An issue occurs when a user runs tests for pulp-certguard within the environment. The environment should be capable of handling discrepancies between pulp-cerguard and pulp_certguard.

We decided not to rename the root name of the repository because we could not evaluate the impact of this change on sources that depend on pulp-certguard.

Allow profiles to set default variables for use in compose and pulp_config files

Also would be neat if profiles could use the default values of their required profiles.

add a profile for telemetry dev env

The telemetry profile needs to spin up an additional container that serves analytics.pulpproject.org website. It should also modify the /etc/hosts file in the container running pulp to point dev.analytics.pulpproject.org to the container running that site.

compose.env.example is missing

The readme provides a command to copy compose.env.example but the file is not in the repo.

Improve remote debugging tooling

Problem

Couldn't run remote debugging (aka, set breakpoints inside the running server code) with the instructions provided by the README.

Context

I followed the steps in the README for debuging functional tests but it didn't work.

After some research + trial-and-error, I was able to make it work with:

# Setting the breakpoint in the code
#   I've relied on the already exposed 12345, which I don't know if have another purpose
import epdb; epdb.serve(port=12345)

# Attaching pdb from another terminal
#   from here: https://github.com/ionelmc/python-remote-pdb#usage
sudo dnf install rlwrap socat
rlwrap socat - tcp:127.0.0.1:12345

But still, epdb doesn't have a very good documentation and is unmaintained in years.

Proposal

I'd like to have better builtin support for remote debugging out-of-the-box.

The idea I have for this is using remote-pdb, which has slighly better documentation and tooling. For example, it can be configured via envvars to integrate with regular breakpoints(). oci-env could set those envvars by default and we don't have to worry about porting and can use normal breakpoints() syntax.

Alternatives

Continue using epdb and update README instructions
Look another tool with convenient remote-debug capability and update README

pulpcore-worker processes are not reloaded on file changes

Use a single pip install command on startup

oci_env calls pip in multiple places multiple times. Pip is not meant to be used that way wand will happily create conflicts with already installed packages or update otherwise pinned dependencies.

Consolidating all calls to pip install into a single one can be considered a hard blocker for using it in any kind of CI.

generate-client doesn't work with docker & selinux

Problem is that pulp-openapi-generator adds a label to its directory when the bindings are generated and this prevents the oci-env container from being able to read the directory and install the bindings.

Need this change to the pulp-openapi-generator pulp/pulp-openapi-generator#79 and need the generate-client script to account for docker as well when setting the label.

Need to tweak requirements file to get pulp-cli newer versions

pulp-cli is pinned to version ==0.20.2 in dev_requirements.txt, which makes it a little inconvenient to pull updates/bug-fixes into the development environment.

Edit: I don't know why I did not consider using pulp-cli from host (outside oci-env shell), which works fine. So it's not really a significant inconvenient. Still, it seems odd

I've noticed it was done as part of a bugfix (#130), but it's not clear to me if pinning to that version is really part of the fix.
If it's not, we should unpin it again.

generate_client.sh script assumes the container is called base_pulp_1

The container running pulp is called base_pulp_1 only whn COMPOSE_PROJECT_NAME is set to base. The generate_client.sh needs to take COMPOSE_PROJECT_NAME into consideration.

Failed to build on ppc64 : oci-env compose up

Trying to build on ppc64le arch facing below issue while running the command

[root@ibm-p9z-26-lp14 oci_env]# oci-env compose up
Creating network "oci_env_default" with the default driver
Creating volume "oci_env_pulp" with default driver
Creating volume "oci_env_pg_data" with default driver
Creating volume "oci_env_redis_data" with default driver
Building _base
[+] Building 1.7s (7/14)
 => [internal] load build definition from Dockerfile                                                                                                    0.1s
 => => transferring dockerfile: 916B                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                       0.2s
 => => transferring context: 2B                                                                                                                         0.0s
 => [internal] load metadata for ghcr.io/pulp/pulp-ci-centos:3.21                                                                                       0.2s
 => [internal] load build context                                                                                                                       0.0s
 => => transferring context: 1.74kB                                                                                                                     0.0s
 => [ 1/10] FROM ghcr.io/pulp/pulp-ci-centos:3.21@sha256:d830efc3c0c200c597956ba8edaad1f0c37c19b20888c42440592bb05829fa21                               0.0s
 => CACHED [ 2/10] COPY dev_requirements.txt /dev_requirements.txt                                                                                      0.0s
 => ERROR [ 3/10] RUN dnf install -y which                                                                                                              1.1s
------
 > [ 3/10] RUN dnf install -y which:
#0 0.709 exec /bin/sh: exec format error
------
Dockerfile:10
--------------------
   8 |
   9 |     COPY dev_requirements.txt /dev_requirements.txt
  10 | >>> RUN dnf install -y which
  11 |     RUN pip3 install -r dev_requirements.txt
  12 |
--------------------
ERROR: failed to solve: process "/bin/sh -c dnf install -y which" did not complete successfully: exit code: 1
ERROR: Service '_base' failed to build : Build failed
[root@ibm-p9z-26-lp14 oci_env]#

Convert oci_env docs to new stack

The general workflow for this task can be found here.

For this repository, the migration process should involve breaking the readme down into the appropriate content categories.

Run Podman in such a way that it has access to system coredumps

I'm having an issue with createrepo_c 1.0 crashing with segmentation fault during tests. Unfortunately I can't really utilize the coredumps to debug.

The coredump appears when I run coredumpctl on my host machine, but my host machine doesn't have the debug symbols to make the dump useful.

The container on the other hand doesn't have the ability to access the coredump on the host.

There are guides like this for making it work

https://dev.to/mizutani/how-to-get-core-file-of-segmentation-fault-process-in-docker-22ii
https://serverfault.com/questions/1143055/how-to-check-core-dump-inside-a-podman-container

The overall gist is that the container needs loosened security restrictions. Possibly this is something that we wouldn't want to enable all the time, but make it easy to enable when needed?

Container name generated by docker-compose not matching the ones defined for oci-env cli commands

After building the images and starting the services my system creates a container named oci_env-pulp-1. However this apparently does not match the convention used for the oci-env cli commands. I get the following error trying to run them:

$ oci-env test -p pulp_deb functional
Error: No such container: oci_env_pulp_1

For reference I'm using the docker-compose version 2.9.0 with the following .compose.env:

# : separated list of profiles to use. Ex: galaxy_ng/ui:ha
COMPOSE_PROFILE=

# : separate list of python dependencies to include from source
DEV_SOURCE_PATH=pulpcore:pulp_rpm:pulp_deb

# Program to use for compose. This defaults to podman. Uncomment this to use docker-compose.
COMPOSE_BINARY=docker

# Set any custom pulp settings by adding environment variables with the PULP_ prefix
# PULP_MY_SETTING....

# Django admin user credentials that gets created during startup
# DJANGO_SUPERUSER_USERNAME=admin
# DJANGO_SUPERUSER_PASSWORD=password

# Port, hostname and protocol used to configure pulp content origin
# API_HOST=localhost
# API_PORT=5001
# API_PROTOCOL=http

Running a docker ps:

$ docker ps
CONTAINER ID   IMAGE                         COMMAND   CREATED          STATUS          PORTS                                     NAMES
d12999f43c5f   localhost/oci_env/pulp:base   "/init"   30 minutes ago   Up 30 minutes   80/tcp, 0.0.0.0:5001->5001/tcp, :::5001->5001/tcp   oci_env-pulp-1

Auto collect "base" profiles from plugin source

For plugins in DEV_SOURCE_PATH oci-env should attempt to load a profile called "base" in the plugin source.

Instead of -i to install test dependencies, can we have a separate command and/or install them (for all plugins) on startup?

I use bash history a lot and needing to remove the -i the second time i want to excercise the same pulp test is kind of annoying. Leaving it on takes way to much extra time.

`oci-env pulpcore-manager reset_db` fails with not enough permissions

[dkliban@localhost oci_env]$ oci-env pulpcore-manager reset_db

You have requested a database reset.
This will IRREVERSIBLY DESTROY
ALL data in the database "pulp".
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
pulp [None]: root:INFO: Executing... "DROP DATABASE "pulp";"
pulp [None]: root:ERROR: Error: must be owner of database pulp
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django_extensions/management/commands/reset_db.py", line 175, in handle
    cursor.execute(drop_query)
psycopg2.errors.InsufficientPrivilege: must be owner of database pulp

pulp [None]: root:INFO: Executing... "CREATE DATABASE "pulp" WITH OWNER = "pulp"  ENCODING = 'UTF8';"
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/src/pulpcore/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django_extensions/management/utils.py", line 62, in inner
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django_extensions/management/commands/reset_db.py", line 190, in handle
    cursor.execute(create_query)
psycopg2.errors.InsufficientPrivilege: permission denied to create database

Possible workaround is to run oci-env shell and run pulpcore-manager reset_db from there.

As a plugin writer, I want to use the pulp cli

Install and configure the pulp CLI in the oci env container so that it's available from oci-env shell. Additionally it might be nice to provide an oci-env pulp <args> cmd (similar to the pulpcore-manager cmd) that allows users to run pulp cli commands from their host machine.

gunicorn timeout is too short for debugging

When debugging pulpcore-api, the gunicorn process restarts during the debug session.

Unable to start container due to directory permissions

Alongside my pulp development repos I have a directory of various RPM repos - despite not being referenced in the configuration at all, I get the following error when I try to start the environment:

Error: unable to start container aedae1b7800720aeb9bc1053adbb782a422875db8ea2a767078d395c30bc6fb0: lsetxattr /home/dalley/devel/repos: operation not permitted

I'm guessing it wants to recursively make everything under ~/devel available inside the container but doesn't like the permissions on that one directory?

It appears a while back I made it owned by the nginx user to make it easy to symlink to /usr/share/nginx/html/ or something.

-rw-r--r--.  1 dalley dalley    2419 Apr 21 21:48 repo_list.txt
drwxrwxrwx. 15 nginx  nginx   151552 Aug 24 23:53 repos
-rw-r--r--.  1 dalley dalley    4885 Apr 21 14:42 repo_sizes.txt

You can see it has permissions 777 so any user can read it.

After using chown and chgrp on this directory with my own user it all works so this isn't a critical issue, just nice to address if possible.

tests scripts run as root, not user `pulp`

Now that the single container runs pulp services as user pulp, the tests need to be run as user pulp.

oci-env db reset leaves Pulp in a broken state

At commit 2b0de81 if I run:

oci-env compose up
Check pulp is happy at the URL http://localhost:5001/pulp/api/v3/ and it is
Try to reset the db with oci-env db reset
Check again Pulp is happy at http://localhost:5001/pulp/api/v3/ and it isn't it gives a connection reset

What I expected from this command was to:

Stop the Pulp services
Delete the db
Recreate the db
Apply migrations
Start the services

I'm not sure (a) if that's what others expected or (b) how much of that is being done versus not done.

set XDG_CONFIG_HOME=/opt/settings/ in the Containerfile

Every time I want to run tests from inside the container, I have to manually set XDG_CONFIG_HOME=/opt/settings/. It would be better if it was always set.

oci-env can't mount files between host and container

Description

When using the volumes directive on the compose.yaml file of a profile you can't use it to mount a host file into the container.

How to reproduce

Create a new profile and in the compose.yaml file, create a new services entry, add a new volumes directive to it and try to mount or bind a file from the host to the container, like this:

services:
    some_service:
        volumes:
            - ./a_local_file.yml:/etc/a_container_file.yml

What is expected

You're expected to find that same file inside of the container, although you gonna find it mounted as an empty directory.

Additional info

You can find under the ./compiled/oci_env directory the directory with the same name of the file which was intended to be mounted into the container.

No way to run 'performance' tests.

Some plugins (e.g., pulp_rpm) separate 'performance' tests out from 'functional' tests. Need to be able to invoke them from oci-env.

Pulp user password - required by tests

When I run oci-env test -i -p pulp_rpm functional test required pulp password. Is it even set (Tried few without luck)?

===================================================== test session starts ======================================================
platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0
rootdir: /var/lib/pulp
plugins: pulpcore-3.22.0.dev0, xdist-3.0.2, custom-exit-code-0.3.0, pulp-smash-1!0.13.0
collected 294 items                                                                                                            

../../var/lib/pulp/api/test_acs.py ..                                                                                    [  0%]
../../var/lib/pulp/api/test_advisory_conflict.py .                                                                       [  1%]
../../var/lib/pulp/api/test_advisory_upload.py ....                                                                      [  2%]
../../var/lib/pulp/api/test_auto_publish.py ..                                                                           [  3%]
../../var/lib/pulp/api/test_character_encoding.py ..                                                                     [  3%]
../../var/lib/pulp/api/test_comps.py ......                                                                              [  5%]
../../var/lib/pulp/api/test_consume_content.py 
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for pulp:

generate-client should build ruby-client (or other languages)

Currently hardcoded to python.

PR will follow shortly 😉

Make docs building easier

Right now you can build docs with:

oci-env shell
pip install -r /src/pulpcore/doc_requirements.txt
cd /src/pulpcore/docs
make diagrams html

It would be great if:

There was an easy command to install the doc_requirements
There was an easy way to call the make file command (passing through the args to make would be ideal)