Comments (17)
I just copied your Dockerfile (with some adjustments, as we dont have a requirements.txt, etc) and ran this repo's tests inside a container (locally) and it appears to have all ran fine.
I could definitely imagine our use of localhost
being an issue, though it's not obvious to me why I can't reproduce. I could also imagine gevent being the problem, but i dont know enough about gevent to produce a minimal example which was known to fail.
I'd be interested to know:
- if you for-sure don't have something bound to port 5532?
- if you monitor
docker ps
and do you see the container spinning up? - is there any way you can access the spun-up postgres from inside the container's commandline, though timing might be somewhat difficult.
from pytest-mock-resources.
Interesting. What command are you running in terminal? Maybe I'm mixing up commands. The goal is to set it up on GitLab using dind, Also, planning to do a PR for the CI section of the documentation once I have a working setup.
from pytest-mock-resources.
Also, planning to do a PR for the CI section of the documentation once I have a working setup.
Very cool, thanks!
For running pytest-mock-resources's own tests:
FROM python:3.6.8-slim
RUN apt-get update
RUN apt-get install -y apt-file
RUN apt-file update
RUN apt-get install -y libpq-dev gcc
RUN mkdir /project
WORKDIR /project
RUN pip install --upgrade pip
RUN apt-get install -y wget
RUN wget -O get-docker.sh https://get.docker.com
RUN chmod +x get-docker.sh && ./get-docker.sh
ENV PATH="/root/.poetry/bin:${PATH}"
RUN curl -sSL https://raw.githubusercontent.com/sdispater/poetry/1.0.0/get-poetry.py | python \
&& poetry config virtualenvs.create false
COPY . /project
RUN poetry install -E postgres -E mongo -E redshift -E redis
CMD ["pytest", "-x", "tests"]
docker build -f Dockerfile -t foo .
docker run -it -v /var/run/docker.sock:/var/run/docker.sock foo
If that works for you, then it kind of makes me think it's related to gevent (given that it was oddly in your stacktrace?)?
from pytest-mock-resources.
Hmm. I still can't get it to work. Here's a minimal example:
example.py
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column(sa.String)
tests/conftest.py
from pytest_mock_resources import create_postgres_fixture
pg = create_postgres_fixture(scope='module')
tests/test_example.py
from example import User
def test_user_is_created_successfully(pg):
pg.add(User(name='test'))
pg.flush()
assert pg.query(User).count()
Dockerfile
FROM python:3.6.8-slim
RUN apt-get update
RUN apt-get install -y apt-file
RUN apt-file update
RUN apt-get install -y libpq-dev gcc
RUN mkdir /project
ADD requirements.txt /project
WORKDIR /project
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN apt-get install -y wget
RUN wget -O get-docker.sh https://get.docker.com
RUN chmod +x get-docker.sh && ./get-docker.sh
COPY . /project
CMD ["pytest", "-x", "tests"]
requirements.txt
psycopg2-binary==2.8.5
SQLAlchemy==1.3.16
pytest==5.2.0
pytest-mock-resources==1.2.2
The error I'm getting is:
E sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
E Is the server running on host "localhost" (127.0.0.1) and accepting
E TCP/IP connections on port 5532?
E could not connect to server: Cannot assign requested address
E Is the server running on host "localhost" (::1) and accepting
E TCP/IP connections on port 5532?
E
E (Background on this error at: http://sqlalche.me/e/e3q8)
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:127: OperationalError
... # More errors
def check_postgres_fn():
try:
get_sqlalchemy_engine(config["root_database"])
except sqlalchemy.exc.OperationalError:
raise ContainerCheckFailed(
"Unable to connect to a presumed Postgres test container via given config: {}".format(
> config
)
)
E pytest_mock_resources.container.ContainerCheckFailed: Unable to connect to a presumed Postgres test container via given config: {'username': 'user', 'password': 'password', 'port': 5532, 'root_database': 'dev', 'image': 'postgres:9.6.10-alpine'}
/usr/local/lib/python3.6/site-packages/pytest_mock_resources/container/postgres.py:56: ContainerCheckFailed
from pytest-mock-resources.
I did have to change your conftest to
from pytest_mock_resources import create_postgres_fixture
from example import Base
pg = create_postgres_fixture(Base, scope="module", session=True)
(which make perfect sense), but otherwise this passes for me š. you definitely don't have a zombie container running on that port, or otherwise have something bound to that port on the host?
from pytest-mock-resources.
I've made the changes. But it's still not working. What platform are you using? I'm using linux.
from pytest-mock-resources.
I'm currently on osx, but i've run it on linux and our CI (circle CI) runs (in docker) on linux as well.
The changes I made would only apply after you've successfully instantiated the fixture (i.e. in the test body), whereas you're failing to start up the container before the tests start.
It's still not clear to me whether you're seeing the container spin up at all (and then be torn down). I.e. your error is caused by it not being able to connect for some reason, vs there being a service which we're trying to connect to with incorrect credentials.
In addition to clarity on the above, i'd be interested in seeing if you could run pmr postgres
(which is largely a glorified docker run -v 5532:5432 postgres
). If that's successful, then try running the tests.
from pytest-mock-resources.
It runs successfully, here's output:
foo | 781999bfdc843c985d045ab817e637f7d230d14f89a596b8a55ffacb840670ea
foo | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
foo | 781999bfdc84 postgres:9.6.10-alpine "docker-entrypoint.sā¦" 1 second ago Up Less than a second 0.0.0.0:5532->5432/tcp ecstatic_burnell
foo | 67a7d9d135d3 gitlab-pmr_foo "./run.sh" 2 seconds ago Up 1 second foo
One thing I've noticed is that I can't access any service on localhost
from within the container. I've verified this with a http server running on the host machine.
from pytest-mock-resources.
We have an environment variable which can be used to configure the host that it looks at export PYTEST_MOCK_RESOURCES_HOST=foo
. Looking at https://stackoverflow.com/questions/31324981/how-to-access-host-port-from-docker-container/31328031#31328031, i wonder if it needs to be docker0
, or 127.0.0.1
, or one of the workarounds in docker/for-linux's linked github issue thread.
It does feel odd to me that we wouldn't have this problem in our other linux environments though.
from pytest-mock-resources.
Hmm. Let me look into it. I'm considering running a dind
version instead of mounting the socket. Maybe, my internal setup is broken so that's why it's not working. What linux versions are you running, ubuntu?
from pytest-mock-resources.
the only linux machines i can confidently use as comparisons are CI (circleci), which would be Ubuntu, afaik
from pytest-mock-resources.
Hey @michaelbukachi
This problem makes sense - docker in *nix systems does not seem to support host.docker.internal
- which is used by pytest-mock-resources to find the docker host when running tests from within a container on Mac or Windows OS.
Following this issue:
docker/for-linux#264 (comment)
Could you do the following and get back to us? -
- Find the docker host on your ubuntu host machine - this is the ip listed with docker0 when you run
ip config
(usually172.17.0.1
or172.17.0.2
) or listed in/etc/hosts
- Start up your python container
- Export the ip from step 1 to an env var called
PYTEST_MOCK_RESOURCES_HOST
- Try running your tests
If this works then it confirms the issue and means we could add something to that "get_docker_host" function to do the above automatcially.
You could also run your docker container with the --net=host
option instead of the above and it should work, although I'm not sure if that could be implemented in gitlab.
from pytest-mock-resources.
Passing 172.17.0.1
to PYTEST_MOCK_RESOURCES_HOST
causes the test to hang indefinitely.
from pytest-mock-resources.
Got it working using dind
. Had to used a compose file so as not worry about network issues.
version: '3.4'
services:
docker:
image: docker:dind
privileged: true
environment:
DOCKER_TLS_CERTDIR: ''
foo:
container_name: foo
environment:
DOCKER_HOST: tcp://docker:2375
PYTEST_MOCK_RESOURCES_HOST: docker
build:
context: .
dockerfile: Dockerfile
Next up, testing on gitlab!:smile:
from pytest-mock-resources.
Here's a working gitlab configuration:
services:
- docker:dind
variables:
DOCKER_TLS_CERTDIR: ''
stages:
- testing
testing-job:
image: python:3.6.8-slim
stage: testing
variables:
DOCKER_HOST: tcp://docker:2375
PYTEST_MOCK_RESOURCES_HOST: docker
before_script:
- apt-get update && apt-get install -y wget libpq-dev gcc
- wget -O get-docker.sh https://get.docker.com
- chmod +x get-docker.sh && ./get-docker.sh
script:
- pip install -r requirements.txt
- pytest -x tests
Do I need to do a PR for the documentation?:smiley:
from pytest-mock-resources.
hah, fantastic, yes please š
from pytest-mock-resources.
Given the now merged doc update on working in dind/gitlab, closing this issue. Feel free to suggest I reopen if that's not true
from pytest-mock-resources.
Related Issues (20)
- Make use of built-in sqlite json and strict support
- Add support for MySQL 8 HOT 5
- Exception trying to unlink pmr.json when when ran with --pmr-multiprocess-safe HOT 3
- Add Redis the CLI HOT 3
- $round operation for MongoDB HOT 2
- SQLAlchemy 2.0 Support HOT 7
- Certain schema DDL is issued only once per test but should be once per metadata
- Dependabot warning for docker subdependency HOT 3
- [Support] Tests hang when using fixture HOT 7
- Error opening connection from redshift_connector HOT 5
- Add in Left/Right UDF's for Redshift HOT 2
- No such event 'before_execute' for target '<sqlalchemy.orm.session.Session object ... HOT 4
- Support SQLAlchemy 1.4 future mode HOT 4
- Dependency on attrs package should be explicit HOT 3
- get_sqlalchemy_engine() not compatible with asyncpg HOT 5
- Allow defining run_args HOT 1
- Getting errors when using `create_redis_fixture` HOT 3
- Missing delete function in redis fixture HOT 1
- Redis Container dont start after update Docker for Desktop on windows. HOT 6
- `create_postgres_fixture` doesn't clean up after running tests. HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytest-mock-resources.