I’ve written before about Certbot and how wonderful it is. Recently I’ve come across a situation where certbot renewals aren’t going thru:
root@host:~# certbot renew --force-renewal
Another instance of Certbot is already running.
root@host:~# ps waux | grep certbot
root 20947 0.0 0.0 4500 744 ? Ss 16:52 0:00 /bin/sh -c /usr/local/sbin/certbot-renew.sh
root 20949 0.0 0.0 4500 736 ? S 16:52 0:00 /bin/sh /usr/local/sbin/certbot-renew.sh
root 20953 0.3 4.4 148704 45440 ? S 16:52 0:00 /usr/bin/python3 /usr/bin/certbot renew --post-hook touch /var/lib/letsencrypt/updated
root 21049 0.0 0.0 12940 1012 pts/4 S+ 16:54 0:00 grep --color=auto certbot
root@host:~# certbot --version
certbot 0.31.0
If I check the logfile /var/log/letsencrypt/letsencrypt.log I can see that it keeps on trying and trying to do renewals.
grep "random delay" /var/log/letsencrypt/letsencrypt.log
2020-04-14 16:16:02,526:INFO:certbot.renewal:Non-interactive renewal: random delay of 5 seconds
2020-04-14 16:17:02,135:INFO:certbot.renewal:Non-interactive renewal: random delay of 409 seconds
2020-04-14 16:24:01,684:INFO:certbot.renewal:Non-interactive renewal: random delay of 290 seconds
2020-04-14 16:29:01,744:INFO:certbot.renewal:Non-interactive renewal: random delay of 190 seconds
2020-04-14 16:33:02,261:INFO:certbot.renewal:Non-interactive renewal: random delay of 283 seconds
2020-04-14 16:38:01,813:INFO:certbot.renewal:Non-interactive renewal: random delay of 350 seconds
2020-04-14 16:44:01,613:INFO:certbot.renewal:Non-interactive renewal: random delay of 117 seconds
2020-04-14 16:46:02,187:INFO:certbot.renewal:Non-interactive renewal: random delay of 309 seconds
2020-04-14 16:52:01,834:INFO:certbot.renewal:Non-interactive renewal: random delay of 453 seconds
Here’s what I do next:
- find any running certbot processes.
- kill them (with -9) one by one
- find and lock files
- remove them
root@host:~# ps waux | grep cert
root 20947 0.0 0.0 4500 744 ? Ss 16:52 0:00 /bin/sh -c /usr/local/sbin/certbot-renew.sh
root 20949 0.0 0.0 4500 736 ? S 16:52 0:00 /bin/sh /usr/local/sbin/certbot-renew.sh
root 20953 0.1 4.4 148704 45440 ? S 16:52 0:00 /usr/bin/python3 /usr/bin/certbot renew --post-hook touch /var/lib/letsencrypt/updated
root 21152 0.0 0.0 12940 936 pts/4 S+ 16:57 0:00 grep --color=auto cert
root@host:~# kill -9 20947
root@host:~# kill -9 20949
root@host:~# kill -9 20953
root@host:~# find / -type f -name ".certbot.lock"
/var/lib/letsencrypt/.certbot.lock
/var/log/letsencrypt/.certbot.lock
/etc/letsencrypt/.certbot.lock
root@host:~# rm /var/lib/letsencrypt/.certbot.lock /var/log/letsencrypt/.certbot.lock /etc/letsencrypt/.certbot.lock
Next, I want to uninstall it.
root@host:~# apt list --installed | grep certbot
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
certbot/xenial,now 0.31.0-1+ubuntu16.04.1+certbot+1 all [installed]
python3-acme/xenial,now 0.31.0-2+ubuntu16.04.6+certbot+2 all [installed,automatic]
python3-asn1crypto/xenial,now 0.22.0-2+ubuntu16.04.1+certbot+1 all [installed,automatic]
Next, I want to uninstall it:
root@host:~# apt-get remove certbot
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
python3-acme python3-certbot python3-configargparse python3-funcsigs python3-future python3-icu python3-josepy
python3-mock python3-parsedatetime python3-pbr python3-requests-toolbelt python3-rfc3339 python3-tz
python3-zope.component python3-zope.event python3-zope.hookable python3-zope.interface
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
certbot
0 upgraded, 0 newly installed, 1 to remove and 9 not upgraded.
After this operation, 39.9 kB disk space will be freed.
Do you want to continue? [Y/n]
Type in Y to continue, then after that, get rid of code you no longer need
root@mhs02:~# apt-get remove certbot
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
python3-acme python3-certbot python3-configargparse python3-funcsigs python3-future python3-icu python3-josepy
python3-mock python3-parsedatetime python3-pbr python3-requests-toolbelt python3-rfc3339 python3-tz
python3-zope.component python3-zope.event python3-zope.hookable python3-zope.interface
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
certbot
0 upgraded, 0 newly installed, 1 to remove and 9 not upgraded.
After this operation, 39.9 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 158315 files and directories currently installed.)
Removing certbot (0.31.0-1+ubuntu16.04.1+certbot+1) ...
Processing triggers for man-db (2.7.5-1) ...
root@host:~# apt autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
python3-acme python3-certbot python3-configargparse python3-funcsigs python3-future python3-icu python3-josepy
python3-mock python3-parsedatetime python3-pbr python3-requests-toolbelt python3-rfc3339 python3-tz
python3-zope.component python3-zope.event python3-zope.hookable python3-zope.interface
0 upgraded, 0 newly installed, 17 to remove and 9 not upgraded.
After this operation, 6,352 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 158306 files and directories currently installed.)
Removing python3-certbot (0.31.0-1+ubuntu16.04.1+certbot+1) ...
Removing python3-acme (0.31.0-2+ubuntu16.04.6+certbot+2) ...
Removing python3-configargparse (0.11.0-1+certbot~xenial+1) ...
Removing python3-mock (1.3.0-2.1ubuntu1) ...
Removing python3-funcsigs (0.4-2) ...
Removing python3-parsedatetime (2.4-3+ubuntu16.04.1+certbot+3) ...
Removing python3-future (0.15.2-4+ubuntu16.04.1+certbot+3) ...
Removing python3-icu (1.9.2-2build1) ...
Removing python3-josepy (1.1.0-2+ubuntu16.04.1+certbot+1) ...
Removing python3-pbr (1.8.0-4ubuntu1) ...
Removing python3-requests-toolbelt (0.8.0-1+ubuntu16.04.1+certbot+1) ...
Removing python3-rfc3339 (1.0-4+certbot~xenial+1) ...
Removing python3-tz (2014.10~dfsg1-0ubuntu2) ...
Removing python3-zope.component (4.3.0-1+ubuntu16.04.1+certbot+3) ...
Removing python3-zope.event (4.2.0-1) ...
Removing python3-zope.hookable (4.0.4-4+ubuntu16.04.1+certbot+1) ...
Removing python3-zope.interface (4.3.2-1+ubuntu16.04.1+certbot+1) ...
Check to see if that removed the process
root@host:~# ps waux | grep cert
root 21304 0.0 0.0 4500 748 ? Ss 17:01 0:00 /bin/sh -c /usr/local/sbin/certbot-renew.sh
root 21306 0.0 0.0 4500 776 ? S 17:01 0:00 /bin/sh /usr/local/sbin/certbot-renew.sh
root 21310 0.2 4.7 157436 48228 ? S 17:01 0:00 /usr/bin/python3 /usr/bin/certbot renew --post-hook touch /var/lib/letsencrypt/updated
root 22017 0.0 0.1 12940 1092 pts/4 S+ 17:08 0:00 grep --color=auto cert
root@host:~# ls /usr/local/sbin
certbot-renew.sh
root@host:~# mv /usr/local/sbin/certbot-renew.sh /usr/local/sbin/certbot-renew.sh.hide
Triple check nothings running
root@mhs02:~# ps waux | grep cert
root 22118 0.0 0.1 12940 1020 pts/4 S+ 17:09 0:00 grep --color=auto cert
Follow the instructions at:
https://certbot.eff.org/instructions on how to install certbot for your installation.
sudo apt-get install certbot python-certbot-apacheFinally, let’s check the server uptime, check the expiration date of a site running on this server, gracefully restart apache, then check the expiration date again:
root@host:~# apachectl status | grep uptime
Server uptime: 11 days 22 hours 45 minutes 34 seconds
root@host:~# echo | openssl s_client -servername www.example.org -connect www.example.org:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Mar 3 16:27:55 2020 GMT
notAfter=Jun 1 16:27:55 2020 GMT
root@host:~# apachectl graceful
[Tue Apr 14 17:17:24.937566 2020] [so:warn] [pid 26644] AH01574: module security2_module is already loaded, skipping
root@host:~# apachectl status | egrep 'uptime|Restart'
Restart Time: Thursday, 02-Apr-2020 18:29:10 UTC
Server uptime: 11 days 22 hours 52 minutes 3 seconds
root@host:~# echo | openssl s_client -servername www.example.org -connect www.example.org:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Apr 14 16:13:15 2020 GMT
notAfter=Jul 13 16:13:15 2020 GMT
Success! We can see that the new renewal dates are in place.
To make sure that renewals happen automatically, I like to have this crontab in place:
# Let's keep let's encrypt ssl certs up to date.
31 3,15 * * * /usr/bin/certbot renew --quiet