container will not start #313

blackknight36 · 2018-08-16T14:37:33Z

There appears to be an issue starting containers as a systemd service. I've defined a docker::run resource and puppet runs successfully however the container fails to start as shown below.

[root@docker-srv1 ~]# systemctl start docker-portainer

[root@docker-srv1 ~]# systemctl status docker-portainer
● docker-portainer.service - Daemon for portainer
   Loaded: loaded (/etc/systemd/system/docker-portainer.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2018-08-16 10:30:05 EDT; 1s ago
  Process: 23945 ExecStop=/usr/bin/docker rm portainer (code=exited, status=0/SUCCESS)
  Process: 23879 ExecStop=/usr/bin/docker stop --time=0 portainer (code=exited, status=0/SUCCESS)
  Process: 23783 ExecStart=/usr/local/bin/docker-run-portainer.sh (code=exited, status=0/SUCCESS)
  Process: 23772 ExecStartPre=/usr/bin/docker rm portainer (code=exited, status=1/FAILURE)
  Process: 23762 ExecStartPre=/usr/bin/docker kill portainer (code=exited, status=1/FAILURE)
 Main PID: 23783 (code=exited, status=0/SUCCESS)

Aug 16 10:30:04 docker-srv1.example.com systemd[1]: Starting Daemon for portainer...
Aug 16 10:30:04 docker-srv1.example.com docker-portainer[23762]: Error response from daemon: Cannot kill container: portainer: No such container: portainer
Aug 16 10:30:04 docker-srv1.example.com docker-portainer[23772]: Error: No such container: portainer
Aug 16 10:30:04 docker-srv1.example.com systemd[1]: Started Daemon for portainer.
Aug 16 10:30:04 docker-srv1.example.com docker-portainer[23783]: a4374cf64b53025091cff2008a1f2c4207dbfcf4c39a6bae874a921c773ba5a2
Aug 16 10:30:05 docker-srv1.example.com docker-portainer[23879]: portainer
Aug 16 10:30:05 docker-srv1.example.com docker-portainer[23945]: portainer`

I am able to start the container by running the /usr/local/bin/docker-run-portainer.sh script however the container is killed the next time that puppet runs.

Here is the resource definition.

docker::run { 'portainer':
        image            => 'portainer/portainer',
        detach           => true,
        ports            => ['9000:9000'],
        volumes          => ['/var/run/docker.sock:/var/run/docker.sock', 'portainer_data_vol:/data'],
        privileged       => true,
        pull_on_start    => false,
        extra_parameters => [ '--add-host dc01.example.com:192.168.0.70' ],
    }

Does anybody know how to resolve this?

The text was updated successfully, but these errors were encountered:

davejrt · 2018-08-19T21:41:07Z

Hi

Can you please provide a little more information by filling out the template?

blackknight36 · 2018-08-20T12:57:38Z

No problem. I've filled out the template as much as I can.

## What you expected to happen?

Container is started by puppet.

## What happened?

Service fails to start.  systemctl shows an error as follows.

docker-portainer.service - Daemon for portainer
   Loaded: loaded (/etc/systemd/system/docker-portainer.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2018-08-16 10:30:05 EDT; 1s ago
  Process: 23945 ExecStop=/usr/bin/docker rm portainer (code=exited, status=0/SUCCESS)
  Process: 23879 ExecStop=/usr/bin/docker stop --time=0 portainer (code=exited, status=0/SUCCESS)
  Process: 23783 ExecStart=/usr/local/bin/docker-run-portainer.sh (code=exited, status=0/SUCCESS)
  Process: 23772 ExecStartPre=/usr/bin/docker rm portainer (code=exited, status=1/FAILURE)
  Process: 23762 ExecStartPre=/usr/bin/docker kill portainer (code=exited, status=1/FAILURE)
 Main PID: 23783 (code=exited, status=0/SUCCESS)
 
## How to reproduce it?
I defined a service resource in the puppet manifest which is configured to run the portainer image.

docker::run { 'portainer':
        image            => 'portainer/portainer',
        detach           => true,
        ports            => ['9000:9000'],
        volumes          => ['/var/run/docker.sock:/var/run/docker.sock', 'portainer_data_vol:/data'],
        privileged       => true,
        pull_on_start    => false,
        extra_parameters => [ '--add-host dc01.example.com:192.168.0.70' ],
}

After pushing this change to our control repo I ran the puppet agent to apply changes.  Puppet applies the changes but the service fails to start.

[root@docker-srv1 ~]# puppet agent -t --tags=docker
Info: Using configured environment 'docker_test'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for docker-srv1.example.com
Info: Applying configuration version '640bb807141f0bf50e2a37662f8331c4e9fe7ec7'
Notice: /Stage[main]/Main/Node[docker-srv1.example.com]/Docker::Run[portainer]/Service[docker-portainer]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Main/Node[docker-srv1.example.com]/Docker::Run[portainer]/Service[docker-portainer]: Unscheduling refresh on Service[docker-portainer]
Info: Node[docker-srv1.example.com]: Unscheduling all events on Node[docker-srv1.example.com]
Notice: Applied catalog in 17.22 seconds

[root@docker-srv1 ~]# systemctl status docker-portainer
● docker-portainer.service - Daemon for portainer
   Loaded: loaded (/etc/systemd/system/docker-portainer.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Mon 2018-08-20 08:52:15 EDT; 38s ago
  Process: 12481 ExecStop=/usr/bin/docker rm portainer (code=exited, status=0/SUCCESS)
  Process: 12415 ExecStop=/usr/bin/docker stop --time=0 portainer (code=exited, status=0/SUCCESS)
  Process: 12316 ExecStart=/usr/local/bin/docker-run-portainer.sh (code=exited, status=0/SUCCESS)
  Process: 12303 ExecStartPre=/usr/bin/docker rm portainer (code=exited, status=0/SUCCESS)
  Process: 12235 ExecStartPre=/usr/bin/docker kill portainer (code=exited, status=0/SUCCESS)
 Main PID: 12316 (code=exited, status=0/SUCCESS)

Aug 20 08:52:10 docker-srv1.example.com systemd[1]: Starting Daemon for portainer...
Aug 20 08:52:11 docker-srv1.example.com docker-portainer[12235]: portainer
Aug 20 08:52:12 docker-srv1.example.com docker-portainer[12303]: portainer
Aug 20 08:52:12 docker-srv1.example.com systemd[1]: Started Daemon for portainer.
Aug 20 08:52:12 docker-srv1.example.com docker-portainer[12316]: f9c7cade071565644631d72011826494e3a3b5396937712d8222412d69345f7a
Aug 20 08:52:14 docker-srv1.example.com docker-portainer[12415]: portainer
Aug 20 08:52:15 docker-srv1.example.com docker-portainer[12481]: portainer

[root@docker-srv1 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
60494ed6d6e7        m4ce/puppetboard    "/usr/bin/supervisor…"   3 days ago          Up 3 days           8000/tcp, 0.0.0.0:32769->80/tcp, 0.0.0.0:32768->4000/tcp   puppetboard

<!-- Specific steps, as well as manifests and puppet configuration that may influence reproducing the issue. -->

## Anything else we need to know?


## Versions:
<!-- Please paste in the output of these commands-->

$ puppet --version

 4.10.12
 
$ docker version

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:08:18 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:10:42 2018
  OS/Arch:          linux/amd64
  Experimental:     true

$ facter os
{
  architecture => "x86_64",
  family => "RedHat",
  hardware => "x86_64",
  name => "CentOS",
  release => {
    full => "7.5.1804",
    major => "7",
    minor => "5"
  },
  selinux => {
    config_mode => "enforcing",
    config_policy => "targeted",
    current_mode => "enforcing",
    enabled => true,
    enforced => true,
    policy_version => "31"
  }
}

$ puppet module list

/etc/puppetlabs/code/environments/production/modules
├── garethr-docker (v5.3.0)
├── puppetlabs-apt (v3.0.0)
├── puppetlabs-stdlib (v4.22.0)
└── stahnma-epel (v1.3.0)

mccartjm · 2018-08-20T20:20:32Z

I had this same issue, but adding:
remove_container_on_start => false,
and removing:
detach => true,

The reason this worked for me was, by adding the remove_container_on_start => false, it changed the template for the service file to use the create command instead of the run command. The create command for docker is one step in the docker run command and does not allow the detached flag. Further reading here:
https://stackoverflow.com/questions/37744961/docker-run-vs-create

I'm a little lost on what the remove_container_on_start parameter does or why it's important.

davejrt · 2018-08-22T00:12:19Z

On an os using systemd you don't need the detatch => true param set. If you remove it the example you've provide will run without any issues.

acurus-puppetmaster · 2018-08-22T04:08:51Z

Your not even using this forge module it seems your using garethr-docker?

blackknight36 · 2018-08-22T15:47:00Z

@acurus-puppetmaster Our puppetfile is configured to use the puppetlabs-docker module, I'm not sure why puppet module list is showing garethr-docker.

blackknight36 · 2018-08-22T15:48:54Z

This is the module list on our master.

[root@mdct-puppetmaster docker_test]# puppet module list --environment docker_test
/etc/puppetlabs/code/environments/docker_test/forge_modules
├── blackknight36-chrony (v0.2.3)
├── camptocamp-kmod (v2.2.0)
├── doubledog-cron (v1.1.1)
├── doubledog-ddolib (v0.0.0)
├── doubledog-iscsi (v1.1.0)
├── doubledog-koji_helpers (v1.2.0)
├── doubledog-mirrmaid (v2.0.0)
├── doubledog-openssl (v1.0.0)
├── doubledog-sigul (v1.0.1)
├── doubledog-systemd (v2.2.0)
├── doubledog-tuned (v1.1.0)
├── mdct-apache (v1.0.0)
├── mdct-authconfig (v1.0.1)
├── mdct-autofs (v1.0.0)
├── mdct-bacula (v0.6.2)
├── mdct-bacula_web (v1.0.0)
├── mdct-clamav (v1.0.0)
├── mdct-collectd (v1.0.0)
├── mdct-cvsweb (v1.0.0)
├── mdct-dhcpd (v0.1.0)
├── mdct-dhcpd_driven (v1.0.0)
├── mdct-filesystem (v1.0.0)
├── mdct-firewall_driven (v1.0.0)
├── mdct-flock_herder (v1.0.0)
├── mdct-git_daemon (v1.0.0)
├── mdct-graphite (v1.0.0)
├── mdct-ipmi (v1.0.0)
├── mdct-jenkins (v1.0.0)
├── mdct-jetbrains (v1.0.0)
├── mdct-koji (v1.0.0)
├── mdct-lm_sensors (v1.0.0)
├── mdct-lvm_snapshot_tools (v1.0.0)
├── mdct-mdct_puppeteer (v1.0.0)
├── mdct-megaraid (v1.0.0)
├── mdct-mock (v1.0.0)
├── mdct-network (v1.0.0)
├── mdct-nfs (v1.0.0)
├── mdct-nginx (v1.0.0)
├── mdct-openldap (v1.0.0)
├── mdct-openssh (v1.0.0)
├── mdct-ovirt (v1.0.0)
├── mdct-pagure (v1.0.0)
├── mdct-plymouth (v1.0.0)
├── mdct-prophile (v1.0.0)
├── mdct-puppet (v1.0.1)
├── mdct-rpm_build_tools (v1.0.0)
├── mdct-rsync (v1.0.0)
├── mdct-samba (v1.0.0)
├── mdct-selinux (v1.0.0)
├── mdct-sendmail (v1.0.0)
├── mdct-subversion (v1.0.0)
├── mdct-sudo (v1.0.0)
├── mdct-test_automation (v1.0.0)
├── mdct-tftpd (v1.0.0)
├── mdct-tsocks (v1.0.0)
├── mdct-viewvc (v1.0.0)
├── mdct-vsftpd (v1.0.0)
├── mdct-xdg_desktop (v1.0.0)
├── mdct-xinetd (v1.0.0)
├── mdct-yum (v1.0.0)
├── mosen-cups (v1.4.1)
├── puppet-extlib (v1.0.0)
├── puppet-logrotate (v3.0.0)
├── puppet-staging (v2.2.0)
├── puppet-zypprepo (v2.0.0)
├── puppetlabs-apt (v4.4.1)
├── puppetlabs-concat (v4.2.1)
├── puppetlabs-docker (v2.0.0)
├── puppetlabs-firewall (v1.12.0)
├── puppetlabs-mysql (v3.10.0)
├── puppetlabs-postgresql (v5.7.0)
├── puppetlabs-powershell (v2.1.4)
├── puppetlabs-puppetserver_gem (v0.2.0)
├── puppetlabs-reboot (v2.0.0)
├── puppetlabs-sqlite (v0.0.1)
├── puppetlabs-stdlib (v4.25.1)
├── puppetlabs-tagmail (v2.4.0)
├── puppetlabs-translate (v1.1.0)
├── puppetlabs-vcsrepo (v1.3.2)
└── razorsedge-snmp (v3.8.1)
/etc/puppetlabs/code/environments/docker_test/modules
└── mdct-dart (v1.0.0)
/etc/puppetlabs/code/modules (no modules installed)
/opt/puppetlabs/puppet/modules (no modules installed)

davejrt · 2018-08-22T20:55:22Z

In any case if you're using systemd, remove the detach. I used your code without any issues are removing this.

blackknight36 · 2018-08-23T14:30:39Z

@davejrt Thanks, everything is working now.

blackknight36 · 2019-01-21T21:24:47Z

@davejrt Just ran into this problem again. It would be really nice if the documentation in the README.md file would mention something about removing the detach => true parameter if you actually want your containers to run.

davejrt · 2019-02-11T00:59:48Z

Closing, documented detach param

davejrt self-assigned this Aug 21, 2018

blackknight36 closed this as completed Aug 23, 2018

MWilsonPuppet added this to the V3.0.0 milestone Sep 26, 2018

mconway mentioned this issue Jan 7, 2019

Puppet 6.1.0 with puppetlabs-docker 3.1.0 fails to start containers #402

Closed

blackknight36 reopened this Jan 21, 2019

davejrt mentioned this issue Jan 22, 2019

readme udpate #416

Merged

davejrt closed this as completed Feb 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

container will not start #313

container will not start #313

blackknight36 commented Aug 16, 2018

davejrt commented Aug 19, 2018

Uh oh!

blackknight36 commented Aug 20, 2018 •

edited

Loading

Uh oh!

mccartjm commented Aug 20, 2018

Uh oh!

davejrt commented Aug 22, 2018

Uh oh!

acurus-puppetmaster commented Aug 22, 2018

Uh oh!

blackknight36 commented Aug 22, 2018

Uh oh!

blackknight36 commented Aug 22, 2018 •

edited

Loading

Uh oh!

davejrt commented Aug 22, 2018

Uh oh!

blackknight36 commented Aug 23, 2018

Uh oh!

blackknight36 commented Jan 21, 2019

Uh oh!

davejrt commented Feb 11, 2019

Uh oh!

container will not start #313

container will not start #313

Comments

blackknight36 commented Aug 16, 2018

davejrt commented Aug 19, 2018

Uh oh!

blackknight36 commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mccartjm commented Aug 20, 2018

Uh oh!

davejrt commented Aug 22, 2018

Uh oh!

acurus-puppetmaster commented Aug 22, 2018

Uh oh!

blackknight36 commented Aug 22, 2018

Uh oh!

blackknight36 commented Aug 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davejrt commented Aug 22, 2018

Uh oh!

blackknight36 commented Aug 23, 2018

Uh oh!

blackknight36 commented Jan 21, 2019

Uh oh!

davejrt commented Feb 11, 2019

Uh oh!

blackknight36 commented Aug 20, 2018 •

edited

Loading

blackknight36 commented Aug 22, 2018 •

edited

Loading