After moving to the new blog host and scoring only a B on the SSL report, I wanted to make sure I was doing everything I could do secure the server from exploits. Part of the reason for moving was exactly that - increased security! On my journey to improve the score, I learned a fair bit about SSL, server and browser based attacks, and how to mitigate them.

I also did a bit of digging into how to secure my personal connection to the VPS over ssh. I’ll cover some information on that at the end of this post.

Weakness is unacceptable

Turns out that by default, nginx allows weaker versions of cryptographic technology that can be more easily penetrated by nefarious users. Specifically, an attacker can eventually intercept traffic between a browser and server. More reading, including a technical paper and mitigation, can be found here.

For now, we’ll focus on mitigation. Luckily enough, the same site has some information for sysadmins. Let’s get started.

Generating a better DH group

The above group is convinced that state-level actors are capable of penetrating 1024-bit groups. It wasn’t clear whether that is by sheer computing power or exploits. Nevertheless, the group recommends using at least a 2048-bit length group.

I figured that doubling wouldn’t hurt anything, so chose 4096 bits as the length. Running this command generates the .pem file we need to give to nginx.

openssl dhparam -out dhparams.pem 4096
sudo mv dhparams.pem /path/inside/your/jail

While we’re editing nginx configuration, I’ll also suggest that you enable Strict Transport Security (HSTS). It is helpful in preventing the same encryption downgrade attacks.

I added both HSTS and dhparams info to the nginx configuration file in the jail I created before. Mostly taken straight from their sysadmin suggestion page:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
add_header Strict-Transport-Security "max-age=31536000";

ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

ssl_prefer_server_ciphers on;
ssl_dhparam /path/inside/your/jail/dhparams.pem;

Afterwards, restart nginx:

sudo service nginx reload

This was enough to get me to an A+ on the SSL test. Skeptical readers can run it themselves. Again, I’m not entirely sure how well qualified they are to give that grade - but as long as the grade is improving, things can’t be getting worse… right?

Hardening ssh

Now that the https part of the server was mitigated against most attacks, I wanted to see if there was anything I was missing on the ssh side of things. Knowing that OpenSSL is basically full of vulnerabilities and odd nuggets (here’s one of the latest ones I’ve seen), I figured the defaults weren’t good enough. And, according to at least one guy on the ‘net, I was right!

The information there is great. There’s also a very brief justification for most of the changes, enough to satisfy a sysadmin without needing a ton of security background. Those paying attention will notice the same information about weaknesses in DH that was covered earlier, in the https part of the post.

A small note: in the section about enabling Protocol 2, you’ll be asked to accept the new fingerprint (as the blog says). Even if you skip this step, however, you’ll likely be asked to accept the new fingerprint. The scary warning that pops up might make you think your server has been compromised, but it’s just using the ED25519 keys instead of the old, normally RSA keys.

That’s not to guaranteed your server wasn’t compromised, of course, but you’re probably fine if it happened in between updating your configuration.

Other avenues

Right now I can’t add too much to the linked resources, due to time constraints. I think a few things left unsaid, such as port knocking, are also worth considering. I’m considering adding that to my setup - expect a future blog post on its pros and cons to come.

Important: This blog post contains an affiliate link for DigitalOcean. To try to prove I’m not fishing for your dollars, you’ll have to find it yourself!

While I’m not in it for the money, I stand to gain a small credit (as do you) if you click the affiliate link located on this page. If you choose to get started with DigitalOcean through that link, I’d certainly be very appreciative.

TL;DR link to how-to

Little Green Locks

Today I decided to investigate what it would take to add an SSL certificate to the blog.

I had been using GitHub Pages to host the blog. This is a great service offered by GitHub that allows an individual to commit files to a git repository and serve these out to the web. Because this blog is powered by Hugo, which generates .html files, essentially the preferred file format of a web browser, it is naturally very easy to generate these files and have GitHub Pages serve them up.

Normally, when hosted in the github.io domain (the default), GitHub Pages provides an https connection automagically - here’s an example. Unfortunately, this blog is hosted under my custom domain, which prevents me from using GitHub’s SSL certificate. Obviously, this is completely unacceptable for a security-conscious web site. More on why in a later post.

As far as I could tell, this could be remedied in at least two ways:

1. Purchase an SSL certificate through my registrar, Gandi (or via many other companies)
2. Move to a server under my (remote) control, e.g. a VPS on DigitalOcean, allowing me to set up Let’s Encrypt for free

I chose option 2 because it would allow me to something I’ve always wanted an excuse to work on: learning how to use FreeBSD, jails, and nginx. Plus, it’ll be way more fun.

FreeBSD and its philosophy

I’ve always been intrigued by the BSD family of operating systems. For the less technical audience, you can think of BSD as yet another operating system, similar to Windows, OS X, or GNU/Linux. In fact, Apple’s OS X shares a lot of code with BSD.

More technical readers might appreciate a core difference between BSD and GNU/Linux, which are often lumped together in the *nix-like domain. As far as I can tell, there are two main idealistic differences between GNU/Linux and BSD, and several quasi-myths that I could find little or no evidence for:

1. BSD seeks to provide both the kernel and the initial application layer that actually makes the kernel useful. Thus, GNU/Linux: one cannot really have “just linux” or “just GNU”, but one can clearly have, say, just FreeBSD or just OpenBSD.
2. Licensing. There’s a lot of e-ink spilled on this topic, and there may be more from me about this in future posts; for now, the TL;DR is that GNU/Linux has chosen the GPL, and FreeBSD has generally not.

Less clear differences include:

1. BSD is faster, more performant, etc.: Seems impossible to test this claim on its own merits - how would one set up such a test? - but many anecdotes of less memory usage have cropped up around the tubes.
2. BSD does releases better: given that BSD is both the kernel and a core set of tools, it behooves them to have a bit more polished releases - the viability of the OS depends on it. Again, though, these broad claims require a bit of careful analysis and thought.
3. Security: Many will claim BSD is more secure. Again, a somewhat wild claim, not really empirically testable apples to apples without some serious thought. What is clear, however, is that OpenBSD really prioritizes security. It’s less clear that FreeBSD does the same, though they may benefit from OpenBSD’s attitude (and thus, security updates).

Throw your apps in jail

One major similarity between the two exists in their approach to virtualization. FreeBSD uses a concept of jails to provide virtualization to applications. GNU/Linux has a similar feature, LXC, utilized by tools like Docker. Many think these two are distinct, but in reality, the mechanisms are quite similar. More on this in a future post as well.

The FreeBSD approach to this problem is the use of jails. You can think of jails as Docker-containers built inside of the FreeBSD operating system itself. It’s a bit more nuanced, obviously, but good enough for now.

In any event, software/systems engineers at large have started to appreciate things like virtualization at the level of applications. Docker’s popularity has absolutely exploded, now containing a massive ecosystem of images, applications, and support from higher-level applications like Marathon. And for good reason: isolated environments prevent entire servers from going down, provide an additional layer of security between application exploits and root access on the OS, and provide modular ecosystems for applications.

The awesome benefit of added security is one of the main selling points for jails. If, say, there’s a security hole in the program you’re using to serve up web pages to browsers, the attacker will also have to penetrate the virtual environment that you’ve (hopefully) set up. While this is not impossible, it is far superior to the non-partitioned alternative, where compromising the web server would result in an attacker owning the box.

nginx, the new King of the Hill?

In a bygone era - about 10 years ago - the Apache web server was the silent workhorse of the internet. Serving up LAMP stacks (and even WAMP stacks!!) all over the internet, the Apache web server project was arguably one of a few flagship projects that helped power some of the world’s most popular web pages.

Nowadays, there’s a new kid on the block: nginx. One of the project’s main goals was to beat up on the Apache web server’s apparent poor performance numbers, especially with regards to memory usage.

As someone interested in new technologies - especially more performant ones - I decided trying out nginx would be something worth doing. Due to its popularity, nginx is now becoming the target of many security researchers and hackers alike. It’ll be interesting and useful to learn a bit more about this super-popular web server.

Free SSL from Let’s Encrypt

To obtain the actual SSL certificate for the server, I’ll be using Let’s Encrypt. It has emerged as an organization with great interest in promoting https, aka “the green lock”, across much of the internet: essentially anyone with their own server can register and get an SSL certificate for their web site - for free! They are also making openness a priority. As someone interested in security and privacy, that is something I’m absolutely on board with. Interested readers can get more information here.

Action

In this section I’ll document the process I used to go from GitHub Pages to a FreeBSD Digital Ocean droplet, running nginx inside of a jail.

I’ll treat this as a dumb server serving up .html files. Therefore, the server won’t have hugo, go, or anything else used to actually generate the site. Think of this droplet/server as a Docker container that serves up images from a directory via nginx. Less software on the server means less opportunity for exploits. Simple and clean.

I’m doing this all from Arch Linux. I’ll assume you’ve got SSH keys and are comfortable with the command line. If you aren’t, just read through the docs on DigitalOcean - which, by the way, has excellent documentation (so far, anyway).

Start the FreeBSD droplet

First, you need to sign up for DigitalOcean. If you want to support me, you can use this link, or you can use their web site directly.

Start a droplet with the FreeBSD operating system. You’ll need to add your SSH public key, as the root user will not have password authentication/SSH authentication enabled. All options are optional; the only one I checked was Enable IPv6. Put blog in the name and/or tag it appropriately if you expect to use DigitalOcean a lot.

Once running, you can test your setup like so:

ssh freebsd@your-droplet-ip

For now, close the connection after you ensure it works.

Handling DNS

Next up is transferring DNS from your current provider to D.O. If you’re using Gandi, or a host of other registrars, you’ll certainly find this link useful. Gandi allows you to postpone this to a date in the future, which is probably wise - otherwise, you might have a brief outage in email delivery, web site/blog being unavailable, and things like that.I recommend saving this until after completing the DigitalOcean setup, but because the next document suggests doing it now, I thought I’d cover it here.

In any event, take a look at this document to map the proper DNS entries to DigitalOcean. Make sure you’ve gotten all of the ones from your current DNS provider into the droplet.

Do not forget to add your mail records if you use a custom mail server for your domain’s email address! Forgetting this can seriously hose your email, and you may have delayed/bounced emails from others sending mail to your domain. For anyone running a business from their domain and relying on email, this is crucial to get right. Double check the mx records, especially before making the switch.

Setup FreeBSD with a new user, jails, and a firewall

We’ll now switch gears again and head back to the command line. ssh into your droplet.

Optional: Install modern shell

One interesting note is that FreeBSD defaults to the use of csh. That is absolutely insane! I also installed zsh (yes, I’m a hypocrite about installing stuff):

sudo pkg install zsh

I then made this the default shell of my own user (see below).

Optional: non-default user

Using the freebsd user is OK, but it’s a small security issue as it gives an attacker a (perhaps) guaranteed user-name to target. I recommend creating a new user that you’ll use to interact with your droplet. Whether or not you decide to do this is up to you.

Copy-pasta for doing this:

sudo adduser
# follow prompts
# when prompted, add user to wheel group, OR:
# wait until after and run:
sudo pw groupmod wheel -m max      # replace 'max' with your user in next commands
sudo mkdir /home/max/.ssh
sudo cp ~/.ssh/authorized_keys /home/max/.ssh/
sudo chmod 700 /home/max/.ssh
sudo chmod 600 /home/max/.ssh/authorized_keys
sudo chown -R max:max /home/max
sudo vim /usr/local/etc/sudoers    # ugh, vim

After opening that file, uncomment this line towards the bottom:

%wheel ALL=(ALL) NOPASSWD: ALL

If your vim is rusty, remember: ctrl-p/ctrl-n to navigate, hit i to start to edit (??!?!?), then esc-wq! to save-exit.

Set up firewall, jail, nginx

Follow this excellent post to set up this stack. Huge props to that author for that post.

Get the SSL cert set up

Let’s Encrypt recommends this tool for getting up and running. I gave it a shot as of course I have shell access.

This site is the version we’ll want to use. There’s no plug and play for FreeBSD. Bummer. For some reason I’m not surprised.

I chose to install the python stuff via pkg as the ports thing didn’t work for me. I think it was because I didn’t have ports configured - or something like that - or maybe it’s straight up broken. Unfortunately, this installed a bunch of python2.7 gunk. I’ll be looking into how to do this better soon.

Note: I’m running these commands outside the jail. The firewall prevents running these commands inside the jail, I think; the software will time out, leaving you wondering WTF is going on (it did for me, anyway).

Here we go:

pkg install py27-certbot
sudo certbot certonly --webroot \
    -w /usr/jails/YOUR_JAIL/usr/local/www/blog \     # or whatever
    -d maxthomas.io \
    -m 'your@email.com' \
    --staging \                                      # you can easily get rate limited on 'prod'
    --config-dir /usr/jails/YOUR_JAIL/usr/local/etc/letsencrypt

If that works, run without the --staging flag. Note the locations of your stuff; you’ll need them later when editing the nginx config.

Editing nginx config

After heading back into your jail:

vim /usr/local/etc/nginx/nginx.conf

This post claims that the best way to do redirects is to have two server blocks. I followed along and create the two blocks, one for the redirect and the other for my actual site.

Add the following lines for your ssh cert in the second block; obviously, your domain path will differ, so update it accordingly:

  ssl_certificate /usr/local/etc/letsencrypt/live/my.domain.com/fullchain.pem;
  ssl_certificate_key /usr/local/etc/letsencrypt/live/my.domain.com/privkey.pem;

I also changed the location/root from /usr/local/www/nginx to something else as I was not sure if the former was potentially going to go away with package updates. There is, hilariously, a file called EXAMPLE_DIRECTORY-DONT_ADD_OR_TOUCH_ANYTHING, so I followed its instructions and didn’t do anything in there.

Back to the shell to copy error pages and the index.

mkdir /usr/local/www/blog
cp /usr/local/www/nginx/*.html /usr/local/www/blog/
service nginx restart          # if all went well, no config errors

You should now be able to hit http://yourdomain and get auto-redirected to the https version. Slick! Unfortunately, this only got my website a B grade on the SSL report. I have no idea if that grade is worth the disk it’s stored on; more on that in the future as I try to improve the grade.

I didn’t yet try to automate the renewal with cron. I’ll likely update this post with information about how to do that, but at present, I’m ready to get this thing over with.

Move, Move, Move

Well, we’ve now got the server up and running, but our .html files are conspicuously absent.

As stated earlier, I want this to be a dumb server that just serves up .html files. As a result I’ll be rsyncing the contents of the generated page over to my node, then moving those into the jail. I’ll probably automate this in the future, but for now:

# on the computer with hugo installed
hugo new post/blahfooblah.md
# write something half-decent for all of our sakes
hugo
# now have a bunch of .html files and so forth
rsync -raz --progress doc/ server:blog/                                   # pick your own directories
ssh server
sudo rsync -raz ~/blog /usr/jails/YOURS/usr/local/www/yours/blog/blah     # fix paths yourself

Assuming all your paths line up, you should now be able to hit the link to your blog and have it served up. Now mix yourself a Black Russian a cup of ginger tea and enjoy your new droplet, SSL certificate, and blog host.

Well, there you have it: about a day’s work learning about DigitalOcean, DNS, nginx, FreeBSD, jails, firewalls, Lets Encrypt, and a bunch of other stuff. I’m pretty satisfied with the setup, in all honesty: it’s exactly what I want, a jailed nginx that serves up content I’m producing on another machine. Look out for future updates about automating the certificate renewal as well as automating content movement.