Another instance of Certbot is already running

I’ve written before about Certbot and how wonderful it is. Recently I’ve come across a situation where certbot renewals aren’t going thru:

root@host:~# certbot renew --force-renewal
Another instance of Certbot is already running.

root@host:~# ps waux | grep certbot
root     20947  0.0  0.0   4500   744 ?        Ss   16:52   0:00 /bin/sh -c /usr/local/sbin/certbot-renew.sh
root     20949  0.0  0.0   4500   736 ?        S    16:52   0:00 /bin/sh /usr/local/sbin/certbot-renew.sh
root     20953  0.3  4.4 148704 45440 ?        S    16:52   0:00 /usr/bin/python3 /usr/bin/certbot renew --post-hook touch /var/lib/letsencrypt/updated
root     21049  0.0  0.0  12940  1012 pts/4    S+   16:54   0:00 grep --color=auto certbot

root@host:~# certbot --version
certbot 0.31.0

If I check the logfile /var/log/letsencrypt/letsencrypt.log I can see that it keeps on trying and trying to do renewals.

grep "random delay" /var/log/letsencrypt/letsencrypt.log

2020-04-14 16:16:02,526:INFO:certbot.renewal:Non-interactive renewal: random delay of 5 seconds
2020-04-14 16:17:02,135:INFO:certbot.renewal:Non-interactive renewal: random delay of 409 seconds
2020-04-14 16:24:01,684:INFO:certbot.renewal:Non-interactive renewal: random delay of 290 seconds
2020-04-14 16:29:01,744:INFO:certbot.renewal:Non-interactive renewal: random delay of 190 seconds
2020-04-14 16:33:02,261:INFO:certbot.renewal:Non-interactive renewal: random delay of 283 seconds
2020-04-14 16:38:01,813:INFO:certbot.renewal:Non-interactive renewal: random delay of 350 seconds
2020-04-14 16:44:01,613:INFO:certbot.renewal:Non-interactive renewal: random delay of 117 seconds
2020-04-14 16:46:02,187:INFO:certbot.renewal:Non-interactive renewal: random delay of 309 seconds
2020-04-14 16:52:01,834:INFO:certbot.renewal:Non-interactive renewal: random delay of 453 seconds

Here’s what I do next:

  1. find any running certbot processes.
  2. kill them (with -9) one by one
  3. find and lock files
  4. remove them
root@host:~# ps waux | grep cert
root     20947  0.0  0.0   4500   744 ?        Ss   16:52   0:00 /bin/sh -c /usr/local/sbin/certbot-renew.sh
root     20949  0.0  0.0   4500   736 ?        S    16:52   0:00 /bin/sh /usr/local/sbin/certbot-renew.sh
root     20953  0.1  4.4 148704 45440 ?        S    16:52   0:00 /usr/bin/python3 /usr/bin/certbot renew --post-hook touch /var/lib/letsencrypt/updated
root     21152  0.0  0.0  12940   936 pts/4    S+   16:57   0:00 grep --color=auto cert

root@host:~# kill -9 20947
root@host:~# kill -9 20949
root@host:~# kill -9 20953
root@host:~# find / -type f -name ".certbot.lock"
/var/lib/letsencrypt/.certbot.lock
/var/log/letsencrypt/.certbot.lock
/etc/letsencrypt/.certbot.lock

root@host:~# rm /var/lib/letsencrypt/.certbot.lock  /var/log/letsencrypt/.certbot.lock /etc/letsencrypt/.certbot.lock

Next, I want to uninstall it.

root@host:~# apt list --installed | grep certbot

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

certbot/xenial,now 0.31.0-1+ubuntu16.04.1+certbot+1 all [installed]
python3-acme/xenial,now 0.31.0-2+ubuntu16.04.6+certbot+2 all [installed,automatic]
python3-asn1crypto/xenial,now 0.22.0-2+ubuntu16.04.1+certbot+1 all [installed,automatic]

Next, I want to uninstall it:

root@host:~#  apt-get remove certbot
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  python3-acme python3-certbot python3-configargparse python3-funcsigs python3-future python3-icu python3-josepy
  python3-mock python3-parsedatetime python3-pbr python3-requests-toolbelt python3-rfc3339 python3-tz
  python3-zope.component python3-zope.event python3-zope.hookable python3-zope.interface
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  certbot
0 upgraded, 0 newly installed, 1 to remove and 9 not upgraded.
After this operation, 39.9 kB disk space will be freed.
Do you want to continue? [Y/n]

Type in Y to continue, then after that, get rid of code you no longer need

root@mhs02:~#  apt-get remove certbot
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  python3-acme python3-certbot python3-configargparse python3-funcsigs python3-future python3-icu python3-josepy
  python3-mock python3-parsedatetime python3-pbr python3-requests-toolbelt python3-rfc3339 python3-tz
  python3-zope.component python3-zope.event python3-zope.hookable python3-zope.interface
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  certbot
0 upgraded, 0 newly installed, 1 to remove and 9 not upgraded.
After this operation, 39.9 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 158315 files and directories currently installed.)
Removing certbot (0.31.0-1+ubuntu16.04.1+certbot+1) ...
Processing triggers for man-db (2.7.5-1) ...
root@host:~# apt autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  python3-acme python3-certbot python3-configargparse python3-funcsigs python3-future python3-icu python3-josepy
  python3-mock python3-parsedatetime python3-pbr python3-requests-toolbelt python3-rfc3339 python3-tz
  python3-zope.component python3-zope.event python3-zope.hookable python3-zope.interface
0 upgraded, 0 newly installed, 17 to remove and 9 not upgraded.
After this operation, 6,352 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 158306 files and directories currently installed.)
Removing python3-certbot (0.31.0-1+ubuntu16.04.1+certbot+1) ...
Removing python3-acme (0.31.0-2+ubuntu16.04.6+certbot+2) ...
Removing python3-configargparse (0.11.0-1+certbot~xenial+1) ...
Removing python3-mock (1.3.0-2.1ubuntu1) ...
Removing python3-funcsigs (0.4-2) ...
Removing python3-parsedatetime (2.4-3+ubuntu16.04.1+certbot+3) ...
Removing python3-future (0.15.2-4+ubuntu16.04.1+certbot+3) ...
Removing python3-icu (1.9.2-2build1) ...
Removing python3-josepy (1.1.0-2+ubuntu16.04.1+certbot+1) ...
Removing python3-pbr (1.8.0-4ubuntu1) ...
Removing python3-requests-toolbelt (0.8.0-1+ubuntu16.04.1+certbot+1) ...
Removing python3-rfc3339 (1.0-4+certbot~xenial+1) ...
Removing python3-tz (2014.10~dfsg1-0ubuntu2) ...
Removing python3-zope.component (4.3.0-1+ubuntu16.04.1+certbot+3) ...
Removing python3-zope.event (4.2.0-1) ...
Removing python3-zope.hookable (4.0.4-4+ubuntu16.04.1+certbot+1) ...
Removing python3-zope.interface (4.3.2-1+ubuntu16.04.1+certbot+1) ...

Check to see if that removed the process

root@host:~# ps waux | grep cert
root     21304  0.0  0.0   4500   748 ?        Ss   17:01   0:00 /bin/sh -c /usr/local/sbin/certbot-renew.sh
root     21306  0.0  0.0   4500   776 ?        S    17:01   0:00 /bin/sh /usr/local/sbin/certbot-renew.sh
root     21310  0.2  4.7 157436 48228 ?        S    17:01   0:00 /usr/bin/python3 /usr/bin/certbot renew --post-hook touch /var/lib/letsencrypt/updated
root     22017  0.0  0.1  12940  1092 pts/4    S+   17:08   0:00 grep --color=auto cert
root@host:~# ls /usr/local/sbin
certbot-renew.sh

root@host:~#  mv /usr/local/sbin/certbot-renew.sh   /usr/local/sbin/certbot-renew.sh.hide

Triple check nothings running

root@mhs02:~# ps waux | grep cert
root     22118  0.0  0.1  12940  1020 pts/4    S+   17:09   0:00 grep --color=auto cert

Follow the instructions at:
https://certbot.eff.org/instructions on how to install certbot for your installation.

sudo apt-get install certbot python-certbot-apache

Finally, let’s check the server uptime, check the expiration date of a site running on this server, gracefully restart apache, then check the expiration date again:

root@host:~# apachectl status | grep uptime
   Server uptime: 11 days 22 hours 45 minutes 34 seconds

root@host:~# echo | openssl s_client -servername www.example.org -connect www.example.org:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Mar  3 16:27:55 2020 GMT
notAfter=Jun  1 16:27:55 2020 GMT

root@host:~# apachectl graceful
[Tue Apr 14 17:17:24.937566 2020] [so:warn] [pid 26644] AH01574: module security2_module is already loaded, skipping

root@host:~# apachectl status | egrep 'uptime|Restart'
   Restart Time: Thursday, 02-Apr-2020 18:29:10 UTC
   Server uptime: 11 days 22 hours 52 minutes 3 seconds


root@host:~# echo | openssl s_client -servername www.example.org -connect www.example.org:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Apr 14 16:13:15 2020 GMT
notAfter=Jul 13 16:13:15 2020 GMT

Success! We can see that the new renewal dates are in place.

To make sure that renewals happen automatically, I like to have this crontab in place:

# Let's keep let's encrypt ssl certs up to date.
31 3,15 * * * /usr/bin/certbot renew --quiet

Leave a Reply

Your email address will not be published. Required fields are marked *