This blog post is split into two parts. Part 1 is the journey to the solution, Part 2 is how to setup our Let’s Encrypt tool on your Sophos firewall. If you are only interested in the latter you can skip ahead.
Journey to a working prototype
For those that dont know what a Sophos XGS is: it’s a firewall product offered by Sophos and its a “replacement” product for the previous Sophos XG and Sophos UTM.
The Sophos UTM used to have excellent support as a web application firewall with Let’s Encrypt, but Sophos removed that feature in the XG and XGS version of their firewall offering. As we use Let’s Encrypt for most of our customers we set out to work around that.
If you Google for solutions you are either offered to use Sophos Factory or to use Let’s Encrypt on another host (windows/linux) that pushes the certificates to the Sophos firewall. It was not satisfying for us to run the certificate renewal process on different hosts so we decided to try to do the whole process on the firewall itself.
A XGS runs a Linux Kernel 4.14 and allows ssh login and a real shell via 5. Device Management -> 3. Advanced Shell. We figured this could be used to run the required programs. We decided to use the excellent acme.sh client to try to get a working http acme process and then see if we can use this to get a working setup.
For a working HTTP acme process you need port 80 to serve static files that are provided by acme.sh. To achieve that we tried getting the httpd that is running on the XGS to serve those files on any port for a start. This turned out to be rather tricky as the httpd config on a XGS requires ports and file locks that are already in use by the Sophos web interface. Luckily the XGS includes a python3 version, with a http module. So we got a working webserver that was serving files on a port of our choice like 8000 but not 80. Port 80 is either used by httpd when the “waf http to https redirection” is enabled or the Sophos iptables firewall rules drop the pakets somewhere between the prerouting and input filter (See Link Layer part of the Diagramm, our python webserver is running on the red application layer).
Our initial reaction was to disable the http to https forwarding and add iptable rules to get the packets to our python webserver running on port 80. But we could not figure out which rules that are responsible for dropping those packets and it was quite ugly approach as it required to deactivate the http to https redirection.
After some consideration we had the idea to just redirect the port to our python webserver during the acme process, with
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination :8000
This approach allowed us to not touch any Sophos settings and redirect the port for the time being and worked like a charm.
With having a webserver that serves our static files generated by acme.sh we were all set to extend this proof of concept to a script that automatically renews Let’s Encrypt certificates running fully on a Sophos XGS.
To have a real prototype we would need a storage path that is permanent between reboots of the XGS and mechanism to run our script every day.
The storage path problem was solved quite easily but just creating an acme folder in /config/ and reboot the firewall to test if our folder was still there, as it was after a reboot so we moved to the next problem.
Running a script everyday on a linux machine is usually done by cronjob or a systemd timer. Both mechanisms are not available on a Sophos XGS. So we tried the little more obscure command at which was also not available. So we opted to use a while loop with sleep to run every day. Which solved the timing problem, but we still needed something in the system to start our script after each reboot.
As the sophos xgs is a busy box system rc.d init scripts are supported and after mounting the root file system read/writable we where able to add a short script to it S01acme which just launches our renew.sh and does not wait for completion. After we where done we switchted the root filesystem back to read only.
With all those building blogs we where able to implement a working lets encrypt client directly on the sophos.
How to setup sophos-xgs-letsencrypt
First you need to login to your sophos via ssh and open an advanced shell.
Then you can run
sh -c "$(curl https://raw.githubusercontent.com/helsinki-systems/sophos-xgs-letsencrypt/main/setup.sh)"
This should take care of the installation and setup your letsencrypt account. For more info head over to the sophos-xgs-letsencrypt GitHub
After that you need to configure which domains and certificates that should be renewed, this is done by editing the /var/acme/config.csv (; semicolon seperated) with vim. Each row contains the name from the certificate in sophos webinterface in the first field and in the second field a list of domains to be requested for that certificate (seperated with a comma ,)
If you don’t have a current certificate name just pick one and use it later in the web interface (in our example that’s OWA)
To get the name for the first field you need to upload a certificate with the pem format to the Sophos web interface (under Certificates).
Example:
OWA;owa.helsinki-systems.de,exchange.helsinki-systems.de
After the configuration is done you need to run /var/acme/renew.sh to request the certificates for the first time.
Depending on if you are adding an already existing certificate to Let’s Encrypt, as in it already existed in your Sophos XGS, you are done now congrats. Otherwise there is another step.
If you do not have the certificate in your XGS yet you need to copy it to your computer using scp on linux or WinSCP on Windows.
scp admin@yoursophosip:/config/certificate/OWA.pem ./scp admin@yoursophosip:/config/certificate/private/OWA.key ./
and reupload them using the sophos web interface under certificates.
Upgrading the Firewall
When you upgrade your firewall you have to rerun the /var/acme/setup.sh to readd the scripts to the sophos system.
Future features and open questions
Firmware Upgrade persistence
Currently we don’t know what happens to our folder in /var/acme after a firmware update if somebody knows the answer or is willing to test it and share their results, we would be happy to know (sophos@helsinki-systems). We ran a quick test upgrading from 19.5 to 20.0 and our old config folder was gone after the upgrade as well as our init script. So we switchted back to /var/acme
Old experiy stats in the sophos webinterface
Currently the expiry stats in the sophos webinterface are not updated after a certificate is renewed. If we find out where those information is stored this could be updated as well. Or handeld via the sophos firewall api
Redirect all other requests to https while running
Currently while the acme is running we are not redirecting the requests to https, but this could easily be added by a little python code.
Warranty
As it has been pointed out by the excellent LuCar Toni in the sophos community forum. Thinkering with your firewall could void the warranty. So be careful of what you do.