Add executable scripts to start certbot and acme server in certbot-ci (certbot#7073)

During review of certbot#6989, we saw that some of our test bash scripts were still used in the Boulder project in particular. It is about `tests/integration/_common.sh` in particular, to expose the `certbot_test` bash function,  that is an appropriate way to execute a local version of certbot in test mode: define a custom server, remove several checks, full log and so on.

This PR is an attempt to assert this goal: exposing a new `certbot_test` executable for test purpose. More generally, this PR is about giving well suited scripts to quickly make manual tests against certbot without launching the full automated pytest suite.

The idea here is to leverage the existing logic in certbot-ci, and expose it as executable scripts. This is done thanks to the `console_scripts` entry of setuptools entrypoint feature, that install scripts in the `PATH`, when `pip install` is invoked, that delegate to specific functions in the installed packages.

Two scripts are defined this way:
* `certbot_test`: it executes certbot in test mode in a very similar way than the original `certbot_test` in `_common.sh`, by delegating to `certbot_integration_tests.utils.certbot_call:main`. By default this execution will target a pebble directory url started locally. The url, and also http-01/tls-alpn-01 challenge ports can be configured using ad-hoc environment variables. All arguments passed to `certbot_test` are transferred to the underlying certbot command.
* `acme_server`: it set up a fully running instance of an ACME server, ready for tests (in particular, all FQDN resolves to localhost in order to target a locally running `certbot_test` command) by delegating to `certbot_integration_tests.utils.acme_server:main`. The choice of the ACME server is given by the first parameter passed to `acme_server`, it can be `pebble`, `boulder-v1` or `boulder-v2`. The command keeps running on foreground, displaying the logs of the ACME server on stdout/stderr. The server is shut down and resources cleaned upon entering CTRL+C.

This two commands can be run also through the underlying python modules, that are executable.

Finally, a typical workflow on certbot side to run manual tests would be:
```
cd certbot
tools/venv.py
source venv/bin/activate
acme_server pebble &
certbot_test certonly --standalone -d test.example.com
```

On boulder side it could be:
```
# Follow certbot dev environment setup instructions, then ...
cd boulder
docker-compose run --use-aliases -e FAKE_DNS=172.17.0.1 --service-ports boulder ./start.py
SERVER=http://localhost:4001/directory certbot_test certonly --standalone -d test.example.com
```

* Configure certbot-ci to expose a certbot_test console script calling certbot in test mode against a local pebble instance

* Add a command to start pebble/boulder

* Use explicit start

* Add execution permission to acme_server

* Add a docstring to certbot_test function

* Change executable name

* Increase sleep to 3600s

* Implement a context manager to handle the acme server

* Add certbot_test workspace in .gitignore

* Add documentation

* Remove one function in context, split logic of certbot_test towards capturing non capturing

* Use an explicit an properly configured ACMEServer as handler.

* Add doc. Put constants.

Author: adferrand

.gitignore

@@ -44,3 +44,6 @@ tests/letstest/venv/
 
 # docker files
 .docker
+
+# certbot tests
+.certbot_test_workspace

certbot-ci/certbot_integration_tests/certbot_tests/context.py

@@ -1,12 +1,10 @@
 """Module to handle the context of integration tests."""
 import os
 import shutil
-import subprocess
 import sys
 import tempfile
-from distutils.version import LooseVersion
 
-from certbot_integration_tests.utils import misc
+from certbot_integration_tests.utils import misc, certbot_call
 
 
 class IntegrationTestsContext(object):
@@ -30,11 +28,6 @@ def __init__(self, request):
         # is listening on challtestsrv_port.
         self.challtestsrv_port = acme_xdist['challtestsrv_port']
 
-        # Certbot version does not depend on the test context. But getting its value requires
-        # calling certbot from a subprocess. Since it will be called a lot of times through
-        # _common_test_no_force_renew, we cache its value as a member of the fixture context.
-        self.certbot_version = misc.get_certbot_version()
-
         self.workspace = tempfile.mkdtemp()
         self.config_dir = os.path.join(self.workspace, 'conf')
         self.hook_probe = tempfile.mkstemp(dir=self.workspace)[1]
@@ -60,71 +53,18 @@ def cleanup(self):
         """Cleanup the integration test context."""
         shutil.rmtree(self.workspace)
 
-    def _common_test_no_force_renew(self, args):
-        """
-        Base command to execute certbot in a distributed integration test context,
-        not renewing certificates by default.
-        """
-        new_environ = os.environ.copy()
-        new_environ['TMPDIR'] = self.workspace
-
-        additional_args = []
-        if self.certbot_version >= LooseVersion('0.30.0'):
-            additional_args.append('--no-random-sleep-on-renew')
-
-        command = [
-            'certbot',
-            '--server', self.directory_url,
-            '--no-verify-ssl',
-            '--http-01-port', str(self.http_01_port),
-            '--https-port', str(self.tls_alpn_01_port),
-            '--manual-public-ip-logging-ok',
-            '--config-dir', self.config_dir,
-            '--work-dir', os.path.join(self.workspace, 'work'),
-            '--logs-dir', os.path.join(self.workspace, 'logs'),
-            '--non-interactive',
-            '--no-redirect',
-            '--agree-tos',
-            '--register-unsafely-without-email',
-            '--debug',
-            '-vv'
-        ]
-
-        command.extend(args)
-        command.extend(additional_args)
-
-        print('Invoke command:\n{0}'.format(subprocess.list2cmdline(command)))
-        return subprocess.check_output(command, universal_newlines=True,
-                                       cwd=self.workspace, env=new_environ)
-
-    def _common_test(self, args):
-        """
-        Base command to execute certbot in a distributed integration test context,
-        renewing certificates by default.
-        """
-        command = ['--renew-by-default']
-        command.extend(args)
-        return self._common_test_no_force_renew(command)
-
-    def certbot_no_force_renew(self, args):
+    def certbot(self, args, force_renew=True):
         """
         Execute certbot with given args, not renewing certificates by default.
         :param args: args to pass to certbot
+        :param force_renew: set to False to not renew by default
         :return: output of certbot execution
         """
         command = ['--authenticator', 'standalone', '--installer', 'null']
         command.extend(args)
-        return self._common_test_no_force_renew(command)
-
-    def certbot(self, args):
-        """
-        Execute certbot with given args, renewing certificates by default.
-        :param args: args to pass to certbot
-        :return: output of certbot execution
-        """
-        command = ['--renew-by-default']
-        command.extend(args)
-        return self.certbot_no_force_renew(command)
+        return certbot_call.certbot_test(
+            command, self.directory_url, self.http_01_port, self.tls_alpn_01_port,
+            self.config_dir, self.workspace, force_renew=force_renew)
 
     def get_domain(self, subdomain='le'):
         """

certbot-ci/certbot_integration_tests/certbot_tests/test_main.py

@@ -229,8 +229,8 @@ def test_graceful_renew_it_is_not_time(context):
 
     assert_cert_count_for_lineage(context.config_dir, certname, 1)
 
-    context.certbot_no_force_renew([
-        'renew', '--deploy-hook', 'echo deploy >> "{0}"'.format(context.hook_probe)])
+    context.certbot(['renew', '--deploy-hook', 'echo deploy >> "{0}"'.format(context.hook_probe)],
+                    force_renew=False)
 
     assert_cert_count_for_lineage(context.config_dir, certname, 1)
     with pytest.raises(AssertionError):
@@ -250,8 +250,8 @@ def test_graceful_renew_it_is_time(context):
     with open(join(context.config_dir, 'renewal', '{0}.conf'.format(certname)), 'w') as file:
         file.writelines(lines)
 
-    context.certbot_no_force_renew([
-        'renew', '--deploy-hook', 'echo deploy >> "{0}"'.format(context.hook_probe)])
+    context.certbot(['renew', '--deploy-hook', 'echo deploy >> "{0}"'.format(context.hook_probe)],
+                    force_renew=False)
 
     assert_cert_count_for_lineage(context.config_dir, certname, 2)
     assert_hook_execution(context.hook_probe, 'deploy')

certbot-ci/certbot_integration_tests/conftest.py

@@ -86,7 +86,9 @@ def _setup_primary_node(config):
 
     # By calling setup_acme_server we ensure that all necessary acme server instances will be
     # fully started. This runtime is reflected by the acme_xdist returned.
-    acme_xdist = acme_lib.setup_acme_server(config.option.acme_server, workers)
-    print('ACME xdist config:\n{0}'.format(acme_xdist))
+    acme_server = acme_lib.setup_acme_server(config.option.acme_server, workers)
+    config.add_cleanup(acme_server.stop)
+    print('ACME xdist config:\n{0}'.format(acme_server.acme_xdist))
+    acme_server.start()
 
-    return acme_xdist
+    return acme_server.acme_xdist

certbot-ci/certbot_integration_tests/nginx_tests/context.py

@@ -2,7 +2,7 @@
 import subprocess
 
 from certbot_integration_tests.certbot_tests import context as certbot_context
-from certbot_integration_tests.utils import misc
+from certbot_integration_tests.utils import misc, certbot_call
 from certbot_integration_tests.nginx_tests import nginx_config as config
 
 
@@ -33,11 +33,14 @@ def certbot_test_nginx(self, args):
         """
         Main command to execute certbot using the nginx plugin.
         :param list args: list of arguments to pass to nginx
+        :param bool force_renew: set to False to not renew by default
         """
         command = ['--authenticator', 'nginx', '--installer', 'nginx',
                    '--nginx-server-root', self.nginx_root]
         command.extend(args)
-        return self._common_test(command)
+        return certbot_call.certbot_test(
+            command, self.directory_url, self.http_01_port, self.tls_alpn_01_port,
+            self.config_dir, self.workspace, force_renew=True)
 
     def _start_nginx(self, default_server):
         self.nginx_config = config.construct_nginx_config(

certbot-ci/certbot_integration_tests/nginx_tests/test_main.py

@@ -30,6 +30,7 @@ def context(request):
     ('nginx6.{0}.wtf,nginx7.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': False}),
 ], indirect=['context'])
 def test_certificate_deployment(certname_pattern, params, context):
+    # type: (str, list, nginx_context.IntegrationTestsContext) -> None
     """
     Test various scenarios to deploy a certificate to nginx using certbot.
     """
@@ -45,10 +46,7 @@ def test_certificate_deployment(certname_pattern, params, context):
 
     assert server_cert == certbot_cert
 
-    command = ['--authenticator', 'nginx', '--installer', 'nginx',
-               '--nginx-server-root', context.nginx_root,
-               'rollback', '--checkpoints', '1']
-    context._common_test_no_force_renew(command)
+    context.certbot_test_nginx(['rollback', '--checkpoints', '1'])
 
     with open(context.nginx_config_path, 'r') as file_h:
         current_nginx_config = file_h.read()

certbot-ci/certbot_integration_tests/utils/acme_server.py

@@ -1,11 +1,11 @@
+#!/usr/bin/env python
 """Module to setup an ACME CA server environment able to run multiple tests in parallel"""
 from __future__ import print_function
 import tempfile
-import atexit
+import time
 import os
 import subprocess
 import shutil
-import stat
 import sys
 from os.path import join
 
@@ -14,33 +14,52 @@
 import yaml
 
 from certbot_integration_tests.utils import misc
+from certbot_integration_tests.utils.constants import *
 
-# These ports are set implicitly in the docker-compose.yml files of Boulder/Pebble.
-CHALLTESTSRV_PORT = 8055
-HTTP_01_PORT = 5002
 
+class ACMEServer(object):
+    """
+    Handler exposing methods to start and stop the ACME server, and get its configuration
+    (eg. challenges ports). ACMEServer is also a context manager, and so can be used to
+    ensure ACME server is started/stopped upon context enter/exit.
+    """
+    def __init__(self, acme_xdist, start, stop):
+        self.acme_xdist = acme_xdist
+        self.start = start
+        self.stop = stop
+
+    def __enter__(self):
+        self.start()
+        return self.acme_xdist
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.stop()
 
-def setup_acme_server(acme_server, nodes):
+
+def setup_acme_server(acme_server, nodes, proxy=True):
     """
     This method will setup an ACME CA server and an HTTP reverse proxy instance, to allow parallel
     execution of integration tests against the unique http-01 port expected by the ACME CA server.
-    Instances are properly closed and cleaned when the Python process exits using atexit.
     Typically all pytest integration tests will be executed in this context.
-    This method returns an object describing ports and directory url to use for each pytest node
-    with the relevant pytest xdist node.
+    An ACMEServer instance will be returned, giving access to the ports and directory url to use
+    for each pytest node, and its start and stop methods are appropriately configured to
+    respectively start the server, and stop it with proper resources cleanup.
     :param str acme_server: the type of acme server used (boulder-v1, boulder-v2 or pebble)
     :param str[] nodes: list of node names that will be setup by pytest xdist
-    :return: a dict describing the challenge ports that have been setup for the nodes
-    :rtype: dict
+    :param bool proxy: set to False to not start the Traefik proxy
+    :return: a properly configured ACMEServer instance
+    :rtype: ACMEServer
     """
     acme_type = 'pebble' if acme_server == 'pebble' else 'boulder'
     acme_xdist = _construct_acme_xdist(acme_server, nodes)
-    workspace = _construct_workspace(acme_type)
+    workspace, stop = _construct_workspace(acme_type)
 
-    _prepare_traefik_proxy(workspace, acme_xdist)
-    _prepare_acme_server(workspace, acme_type, acme_xdist)
+    def start():
+        if proxy:
+            _prepare_traefik_proxy(workspace, acme_xdist)
+        _prepare_acme_server(workspace, acme_type, acme_xdist)
 
-    return acme_xdist
+    return ACMEServer(acme_xdist, start, stop)
 
 
 def _construct_acme_xdist(acme_server, nodes):
@@ -49,10 +68,10 @@ def _construct_acme_xdist(acme_server, nodes):
 
     # Directory and ACME port are set implicitly in the docker-compose.yml files of Boulder/Pebble.
     if acme_server == 'pebble':
-        acme_xdist['directory_url'] = 'https://localhost:14000/dir'
+        acme_xdist['directory_url'] = PEBBLE_DIRECTORY_URL
     else:  # boulder
-        port = 4001 if acme_server == 'boulder-v2' else 4000
-        acme_xdist['directory_url'] = 'http://localhost:{0}/directory'.format(port)
+        acme_xdist['directory_url'] = BOULDER_V2_DIRECTORY_URL \
+            if acme_server == 'boulder-v2' else BOULDER_V1_DIRECTORY_URL
 
     acme_xdist['http_port'] = {node: port for (node, port)
                                in zip(nodes, range(5200, 5200 + len(nodes)))}
@@ -82,10 +101,7 @@ def cleanup():
 
         shutil.rmtree(workspace)
 
-    # Here with atexit we ensure that clean function is called no matter what.
-    atexit.register(cleanup)
-
-    return workspace
+    return workspace, cleanup
 
 
 def _prepare_acme_server(workspace, acme_type, acme_xdist):
@@ -136,7 +152,6 @@ def _prepare_traefik_proxy(workspace, acme_xdist):
     print('=> Starting traefik instance deployment...')
     instance_path = join(workspace, 'traefik')
     traefik_subnet = '10.33.33'
-    traefik_api_port = 8056
     try:
         os.mkdir(instance_path)
 
@@ -159,12 +174,12 @@ def _prepare_traefik_proxy(workspace, acme_xdist):
       config:
         - subnet: {traefik_subnet}.0/24
 '''.format(traefik_subnet=traefik_subnet,
-           traefik_api_port=traefik_api_port,
+           traefik_api_port=TRAEFIK_API_PORT,
            http_01_port=HTTP_01_PORT))
 
         _launch_command(['docker-compose', 'up', '--force-recreate', '-d'], cwd=instance_path)
 
-        misc.check_until_timeout('http://localhost:{0}/api'.format(traefik_api_port))
+        misc.check_until_timeout('http://localhost:{0}/api'.format(TRAEFIK_API_PORT))
         config = {
             'backends': {
                 node: {
@@ -178,7 +193,7 @@ def _prepare_traefik_proxy(workspace, acme_xdist):
                 } for node in acme_xdist['http_port'].keys()
             }
         }
-        response = requests.put('http://localhost:{0}/api/providers/rest'.format(traefik_api_port),
+        response = requests.put('http://localhost:{0}/api/providers/rest'.format(TRAEFIK_API_PORT),
                                 data=json.dumps(config))
         response.raise_for_status()
 
@@ -195,3 +210,35 @@ def _launch_command(command, cwd=os.getcwd()):
     except subprocess.CalledProcessError as e:
         sys.stderr.write(e.output)
         raise
+
+
+def main():
+    args = sys.argv[1:]
+    server_type = args[0] if args else 'pebble'
+    possible_values = ('pebble', 'boulder-v1', 'boulder-v2')
+    if server_type not in possible_values:
+        raise ValueError('Invalid server value {0}, should be one of {1}'
+                         .format(server_type, possible_values))
+
+    acme_server = setup_acme_server(server_type, [], False)
+    process = None
+
+    try:
+        with acme_server as acme_xdist:
+            print('--> Instance of {0} is running, directory URL is {0}'
+                  .format(acme_xdist['directory_url']))
+            print('--> Press CTRL+C to stop the ACME server.')
+
+            docker_name = 'pebble_pebble_1' if 'pebble' in server_type else 'boulder_boulder_1'
+            process = subprocess.Popen(['docker', 'logs', '-f', docker_name])
+
+            while True:
+                time.sleep(3600)
+    except KeyboardInterrupt:
+        if process:
+            process.terminate()
+            process.wait()
+
+
+if __name__ == '__main__':
+    main()