Merge pull request pytest-docker-compose#24 from RmStorm/develop

Change how fixtures supply containers to solve dynamic ports issue Closes pytest-docker-compose#8

Author: RmStorm

.pycodestyle

@@ -0,0 +1,2 @@
+[pycodestyle]
+max-line-length = 119

README.rst

@@ -1,5 +1,8 @@
 pytest-docker-compose
 =====================
+.. image:: https://circleci.com/gh/pytest-docker-compose/pytest-docker-compose/tree/master.svg?style=svg
+    :target: https://circleci.com/gh/pytest-docker-compose/pytest-docker-compose/tree/master
+
 This package contains a `pytest`_ plugin for integrating Docker Compose into your automated integration tests.
 
 Given a path to a ``docker-compose.yml`` file, it will automatically build the project at the start of the test run, bring the containers up before each test starts, and tear them down after each test ends.
@@ -37,11 +40,8 @@ See `Installing and Using Plugins`_ for more information.
 
 To interact with Docker containers in your tests, use the following fixtures:
 
-``function_scoped_containers``
-    A dictionary of the Docker ``compose.container.Container`` objects
-    running during the test. These containers each have an extra attribute
-    called ``network_info`` added to them. This attribute has a list of
-    ``pytest_docker_compose.NetworkInfo`` objects.
+``function_scoped_container_getter``
+    An object that fetches containers of the Docker ``compose.container.Container`` objects running during the test. The containers are fetched using ``function_scoped_container_getter.get('service_name')`` These containers each have an extra attribute called ``network_info`` added to them. This attribute has a list of ``pytest_docker_compose.NetworkInfo`` objects.
 
     This information can be used to configure API clients and other objects that
     will connect to services exposed by the Docker containers in your tests.
@@ -64,18 +64,18 @@ To interact with Docker containers in your tests, use the following fixtures:
 
 To use the following fixtures please read `Use wider scoped fixtures`_.
 
-``class_scoped_containers``
-    Similar to ``function_scoped_containers`` just with a wider scope.
+``class_scoped_container_getter``
+    Similar to ``function_scoped_container_getter`` just with a wider scope.
 
-``module_scoped_containers``
-    Similar to ``function_scoped_containers`` just with a wider scope.
+``module_scoped_container_getter``
+    Similar to ``function_scoped_container_getter`` just with a wider scope.
 
-``session_scoped_containers``
-    Similar to ``function_scoped_containers`` just with a wider scope.
+``session_scoped_container_getter``
+    Similar to ``function_scoped_container_getter`` just with a wider scope.
 
 Waiting for Services to Come Online
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The fixtures called ``'scope'_scoped_containers`` will wait until every container is up before handing control over to the test.
+The fixtures called ``'scope'_scoped_container_getter`` will wait until every container is up before handing control over to the test.
 
 However, just because a container is up does not mean that the services running on it are ready to accept incoming requests yet!
 
@@ -95,15 +95,15 @@ Here's an example of a fixture called ``wait_for_api`` that waits for an HTTP se
 
 
     @pytest.fixture(scope="function")
-    def wait_for_api(function_scoped_containers):
+    def wait_for_api(function_scoped_container_getter):
         """Wait for the api from my_api_service to become responsive"""
         request_session = requests.Session()
         retries = Retry(total=5,
                         backoff_factor=0.1,
                         status_forcelist=[500, 502, 503, 504])
         request_session.mount('http://', HTTPAdapter(max_retries=retries))
 
-        service = function_scoped_containers["my_network_my_api_service_1"].network_info[0]
+        service = function_scoped_container_getter.get("my_api_service").network_info[0]
         api_url = "http://%s:%s/" % (service.hostname, service.host_port)
         assert request_session.get(api_url)
         return request_session, api_url
@@ -120,11 +120,11 @@ Here's an example of a fixture called ``wait_for_api`` that waits for an HTTP se
 
 Use wider scoped fixtures
 ~~~~~~~~~~~~~~~~~~~~~~~~~
-The ``function_scoped_containers`` fixture uses "function" scope, meaning that all of the containers are torn down after each individual test.
+The ``function_scoped_container_getter`` fixture uses "function" scope, meaning that all of the containers are torn down after each individual test.
 
 This is done so that every test gets to run in a "clean" environment. However, this can potentially make a test suite take a very long time to complete.
 
-There are two options to make containers persist beyond a single test. The best way is to use the fixtures that are explicitly scoped to different scopes. There are three additional fixtures for this purpose: ``class_scoped_containers``, ``module_scoped_containers`` and ``session_scoped_containers``. Notice that you need to be careful when using these! There are two main caveats to keep in mind:
+There are two options to make containers persist beyond a single test. The best way is to use the fixtures that are explicitly scoped to different scopes. There are three additional fixtures for this purpose: ``class_scoped_container_getter``, ``module_scoped_container_getter`` and ``session_scoped_container_getter``. Notice that you need to be careful when using these! There are two main caveats to keep in mind:
 
 1. Manage your scope correctly, using 'module' scope and 'function' scope in one single file will throw an error! This is because the module scoped fixture will spin up the containers and then the function scoped fixture will try to spin up the containers again. Docker compose does not allow you to spin up containers twice.
 2. Clean up your environment after each test. Because the containers are not restarted their environments can carry the information from previous tests. Therefore you need to be very carefull when designing your tests such that they leave the containers in the same state that it started in or you might run into difficult to understand behaviour.

setup.py

@@ -10,7 +10,7 @@
     version="2.0.0",
     author="Roald Storm",
     author_email="[email protected]",
-    url="https://github.com/RmStorm/pytest-docker-compose",
+    url="https://github.com/pytest-docker-compose/pytest-docker-compose",
     packages=find_packages(where="src"),
     package_dir={"": "src"},
     install_requires=["docker-compose", "pytest >= 3.4"],

src/pytest_docker_compose/__init__.py

@@ -1,4 +1,4 @@
-import typing
+from typing import List
 from pathlib import Path
 import warnings
 from datetime import datetime
@@ -23,8 +23,7 @@ class ContainersAlreadyExist(Exception):
 
 
 class NetworkInfo:
-    def __init__(self, container_port: typing.Text,
-                 hostname: typing.Text, host_port: int,):
+    def __init__(self, container_port: str, hostname: str, host_port: int,):
         """
         Container for info about how to connect to a service exposed by a
         Docker container.
@@ -39,7 +38,7 @@ def __init__(self, container_port: typing.Text,
         self.host_port = host_port
 
 
-def create_network_info_for_container(container):
+def create_network_info_for_container(container: Container):
     """
     Generates :py:class:`NetworkInfo` instances corresponding to all available
     port bindings in a container
@@ -48,7 +47,7 @@ def create_network_info_for_container(container):
                         hostname=port_config["HostIp"] or "localhost",
                         host_port=port_config["HostPort"],)
             for container_port, port_configs in
-            container.get("HostConfig.PortBindings").items()
+            container.ports.items()
             for port_config in port_configs]
 
 
@@ -57,10 +56,10 @@ class DockerComposePlugin:
     Integrates docker-compose into pytest integration tests.
     """
     def __init__(self):
-        self.function_scoped_containers = self.generate_scoped_containers_fixture('function')
-        self.class_scoped_containers = self.generate_scoped_containers_fixture('class')
-        self.module_scoped_containers = self.generate_scoped_containers_fixture('module')
-        self.session_scoped_containers = self.generate_scoped_containers_fixture('session')
+        self.function_scoped_container_getter = self.generate_scoped_containers_fixture('function')
+        self.class_scoped_container_getter = self.generate_scoped_containers_fixture('class')
+        self.module_scoped_container_getter = self.generate_scoped_containers_fixture('module')
+        self.session_scoped_container_getter = self.generate_scoped_containers_fixture('session')
 
     # noinspection SpellCheckingInspection
     @staticmethod
@@ -140,7 +139,7 @@ def docker_project(self, request):
         return project
 
     @classmethod
-    def generate_scoped_containers_fixture(cls, scope):
+    def generate_scoped_containers_fixture(cls, scope: str):
         """
         Create scoped fixtures that retrieve or spin up all containers, and add
         network info objects to containers and then yield the containers for
@@ -153,20 +152,19 @@ def generate_scoped_containers_fixture(cls, scope):
         def scoped_containers_fixture(docker_project: Project, request):
             now = datetime.utcnow()
             if request.config.getoption("--use-running-containers"):
-                containers = docker_project.containers()  # type: typing.List[Container]
+                containers = docker_project.containers()  # type: List[Container]
             else:
                 if any(docker_project.containers()):
                     raise ContainersAlreadyExist(
                         'pytest-docker-compose tried to start containers but there are'
                         ' already running containers: %s, you probably scoped your'
                         ' tests wrong' % docker_project.containers())
-                containers = docker_project.up()  # type: typing.List[Container]
+                containers = docker_project.up()
                 if not containers:
                     raise ValueError("`docker-compose` didn't launch any containers!")
 
-            for container in containers:
-                container.network_info = create_network_info_for_container(container)
-            yield {container.name: container for container in containers}
+            container_getter = ContainerGetter(docker_project)
+            yield container_getter
 
             for container in sorted(containers, key=lambda c: c.name):
                 header = "Logs from {name}:".format(name=container.name)
@@ -177,13 +175,27 @@ def scoped_containers_fixture(docker_project: Project, request):
             if not request.config.getoption("--use-running-containers"):
                 docker_project.down(ImageType.none, False)
         scoped_containers_fixture.__wrapped__.__doc__ = """
-            Spins up the containers for the Docker project and returns them in a
-            dictionary. Each container has one additional attribute called
-            network_info to simplify accessing the hostnames and exposed port
-            numbers for each container.
+            Spins up the containers for the Docker project and returns an
+            object that can retrieve the containers. The returned containers
+            all have one additional attribute called network_info to simplify
+            accessing the hostnames and exposed port numbers for each container.
             This set of containers is scoped to '%s'
             """ % scope
         return scoped_containers_fixture
 
 
 plugin = DockerComposePlugin()
+
+
+class ContainerGetter:
+    """
+    A class that retrieves containers from the docker project and adds a
+    convenience wrapper for the available ports
+    """
+    def __init__(self, docker_project: Project) -> None:
+        self.docker_project = docker_project
+
+    def get(self, key: str) -> Container:
+        container = self.docker_project.containers(service_names=[key])[0]
+        container.network_info = create_network_info_for_container(container)
+        return container

tests/pytest_docker_compose_tests/my_network/docker-compose.yml

@@ -4,12 +4,12 @@ services:
   my_api_service:
     build: ./a_buildable_container
     ports:
-      - 5000:5000
+      - "5000:5000"
     depends_on:
       - my_db
     restart: on-failure
 
   my_db:
     image: postgres:11.2-alpine
     ports:
-      - 5432:5432
+      - "5432:5432"

tests/pytest_docker_compose_tests/test_function_scoping_fixtures.py

@@ -8,15 +8,15 @@
 
 
 @pytest.fixture(scope="function")
-def wait_for_api(function_scoped_containers):
+def wait_for_api(function_scoped_container_getter):
     """Wait for the api from my_api_service to become responsive"""
     request_session = requests.Session()
     retries = Retry(total=5,
                     backoff_factor=0.1,
                     status_forcelist=[500, 502, 503, 504])
     request_session.mount('http://', HTTPAdapter(max_retries=retries))
 
-    service = function_scoped_containers["my_network_my_api_service_1"].network_info[0]
+    service = function_scoped_container_getter.get("my_api_service").network_info[0]
     api_url = "http://%s:%s/" % (service.hostname, service.host_port)
     assert request_session.get(api_url)
     return request_session, api_url

tests/pytest_docker_compose_tests/test_module_scoping_fixtures.py

@@ -8,15 +8,15 @@
 
 
 @pytest.fixture(scope="module")
-def wait_for_api(module_scoped_containers):
+def wait_for_api(module_scoped_container_getter):
     """Wait for the api from my_api_service to become responsive"""
     request_session = requests.Session()
     retries = Retry(total=5,
                     backoff_factor=0.1,
                     status_forcelist=[500, 502, 503, 504])
     request_session.mount('http://', HTTPAdapter(max_retries=retries))
 
-    service = module_scoped_containers["my_network_my_api_service_1"].network_info[0]
+    service = module_scoped_container_getter.get("my_api_service").network_info[0]
     api_url = "http://%s:%s/" % (service.hostname, service.host_port)
     assert request_session.get(api_url)
     return request_session, api_url

tests/pytest_docker_compose_tests/test_wrong_scoping.py

@@ -3,10 +3,10 @@
 
 
 @pytest.mark.should_fail
-def test_read_all_module(module_scoped_containers):
-    assert module_scoped_containers["my_network_my_api_service_1"].network_info[0]
+def test_read_all_module(module_scoped_container_getter):
+    assert module_scoped_container_getter.get("my_api_service").network_info[0]
 
 
 @pytest.mark.should_fail
-def test_read_all_function(function_scoped_containers):
-    assert function_scoped_containers["my_network_my_api_service_1"].network_info[0]
+def test_read_all_function(function_scoped_container_getter):
+    assert function_scoped_container_getter.get("my_api_service").network_info[0]

tox.ini

@@ -6,9 +6,14 @@ deps =
     pytest3: pytest>=3,<4
     pytest4: pytest>=4,<5
     pytest5: pytest>=5,<6
+    docker-compose==1.24.1
+    pycodestyle
+    mypy
 whitelist_externals=
     bash
 commands=
+    pycodestyle --config .pycodestyle src
+    mypy --namespace-packages --ignore-missing-imports src
     bash -c '! pytest -m should_fail'
     pytest
     docker-compose -f tests/pytest_docker_compose_tests/my_network/docker-compose.yml up -d