Wednesday, June 22, 2016

Elegant iptables rules for your Linux web server

You've just upgraded to your first VPS or dedicated server and you think you've got all the software bits in place (LAMP stack and all that) OR you've moved from a hosting provider with easily configured hardware firewalls. Now you've read something somewhere that says you need 'iptables' rules with your new host. If you have any friends who manage Linux systems, you've also heard that "iptables is hard." In my experience, the only thing hard about iptables is that no one seems to publish decent rulesets and users are left to figure it all out on their own. It doesn't have to be that way!

Ubuntu/Debian: apt-get install iptables-persistent

RedHat/CentOS: chkconfig iptables on

That installs the iptables persistent package/enables storing iptables so that they load during the next boot.

Those editable configuration files are where the IPv4 and IPv6 iptables rules are stored respectively and are loaded from on boot with the previous bit. Here is a good set of starter IPv4 rules:
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp --syn --dport 80 -j ACCEPT
-A INPUT -p tcp --syn --dport 443 -j ACCEPT
-A INPUT -p tcp --dport 22 -j ACCEPT
-A INPUT -p icmp --fragment -j DROP
-A INPUT -p icmp --icmp-type 3 -j ACCEPT
-A INPUT -p icmp --icmp-type 4 -j ACCEPT
-A INPUT -p icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp --icmp-type 11 -j ACCEPT
And a good set of starter IPv6 rules:
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp --syn --dport 80 -j ACCEPT
-A INPUT -p tcp --syn --dport 443 -j ACCEPT
-A INPUT -p tcp --dport 22 -j ACCEPT
-A INPUT -p icmpv6 -j ACCEPT
You should run:
ifconfig -a
To figure out what interface is the local loopback interface. The rules above default to the 'lo' interface, which is probably correct unless you've got a weird host.

After that, you should change the rules to reflect the ports that you need open. To determine what ports are currently open, you can run:
netstat -plntu | grep -v | grep -v ::1: | grep -v dhclient | grep -v ntpd
That set of commands returns all running TCP/UDP servers that are not exclusively localhost and aren't the standard DHCP client or the NTP daemon (by the way, you should have ntpd installed to avoid severe clock drift). That is, it will show all the ports that probably need to be firewalled properly. Use Google to search for any port numbers you don't recognize. (Hint: Port 22 is SSH/SFTP - it's included above and you probably want to leave that rule alone!) For each port you decide to allow, adjust the rules accordingly - probably by adding new lines that mostly mirror other lines except the --dport option will be different.

After the TCP rules, you should put any UDP rules you need. Since UDP is generally rarer to see except if you are hosting a multimedia or game server, I didn't include any above, but they look like this:
-A INPUT -p udp --dport 2933 -j ACCEPT
Just replace the 'tcp' bit with 'udp' and drop the --syn option. Keep in mind that a lot of mobile technology (e.g. smartphones) don't support UDP over wireless networks. To accommodate mobile devices, it is a good idea to enable TCP mode alongside any UDP servers and set up firewall rules for both.

Once you are ready to fire up the new rules, run commands similar to these:

iptables-restore < /etc/iptables/rules.v4
ip6tables-restore < /etc/iptables/rules.v6
iptables-restore < /etc/sysconfig/iptables
ip6tables-restore < /etc/sysconfig/ip6tables
That's it! You are now a master of iptables rules. And it was just as easy to set up, if not easier than, Ubuntu ufw or other system-specific solutions!

Let's say you get it in your head that you want to restrict access to a single IP address or an IP address range. IMO, if you can, leave your clean and elegant rules as-is and use either the Web Knocker Firewall Service or fail2ban instead of messing around with iptables. For static IP addresses that will never, ever change (really?) you can use the --src option (e.g. -A INPUT -p tcp --dport 22 --src -j ACCEPT) but don't do that unless you really know what you are doing.

One other thing to consider doing is to make changes to your kernel and network stack. The file to edit is /etc/sysctl.conf and here are some relevant options (read the Internets before making such changes):
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
The rest of this post is a quick overview of how the iptables rules work. The default policies of iptables is ACCEPT with no rules, which means all packets are accepted. So the first thing that happens in the rules is to switch both INPUT (packets coming in) and FORWARD (only relevant for redirecting packets - e.g. a router) policies to DROP (i.e. ignore all packets). The OUTPUT (packets headed out) policy is left as ACCEPT (i.e. accept all outbound packets). In my experience, there's never a valid reason to switch OUTPUT to DROP unless you actually want to create headaches for yourself. Keep it simple. Keep it elegant.

Now that we're past the policies, let's look at the rules themselves. The first rule says to have the 'state' kernel module for iptables check to see if an incoming packet is part of a RELATED or ESTABLISHED connection. If so, ACCEPT it and skip the rest of the rules. This is a great rule to have first because nearly all packets will hit this rule and immediately pass through the firewall. It's performance-friendly! It also shows that the ordering of the rules can be quite important for maximizing system performance.

The next rule lets all new connections to all ports on the 'lo' interface (localhost) through. Again, another nice, performance-friendly rule. After that, new connections to TCP ports 80, 443, and 22 are let through. The --syn option checks TCP flags for a valid SYN packet. Since most port 22 connections are extremely long-lived and, depending on the client, --syn might cause frequent disconnects, it is excluded from the rules.

After the TCP and optional UDP rules are the rules for ICMP packets. For IPv4, I drop fragmented ICMP packets since those types of packets are only ever used in a Denial of Service attack. ICMP types 3 and 4 are essential/required, type 8 is for ping (optional), and type 11 is for traceroute (also optional). IPv6 utilizes ICMP heavily, so blocking ICMPv6 traffic is currently considered bad practice. I've also not seen any particular firewall rulesets worth using for more strict ICMPv6 that don't look overly complicated. So I'm simply accepting all ICMPv6 traffic until someone points out issues with doing so (e.g. a confirmed CVE).

The last line simply COMMITs all of the the changes and enables the new rules. If a rule fails to apply for some reason, iptables-restore will roll back all of the changes since the last COMMIT line. This is a really nice feature because you don't want to get through half of the rules, encounter an error, and be locked out of the system.

By the way, Linux nerds, did you see how easy this was? This is totally what you should be doing. Useful things first such as complete iptables rulesets that people can copy and paste. Then slightly less useful, more technical things after that such as describing how those rules work for the people who really want to know.

Saturday, June 04, 2016

The most interesting bug in PHP

The most interesting bug in PHP is the showstopper bug in the core of PHP you finally run into after a month of software development just as you are getting ready to ship a brand new product out the door. Specifically, PHP bug #72333, which is in all current versions of PHP. If you aren't familiar with reading C code, it can be extremely hard to follow along with that bug report especially since PHP streams behind-the-scenes are ugly beasts to try to wrap your head around (mine's still spinning and I wrote the bug report). In short, the problem is a combination of non-blocking mode with SSL sockets when calling SSL_write() with different pointers in 'ext\openssl\xp_ssl.c'.

The temporary patch in userland is to disable non-blocking mode when writing data - if you can - I'm not so sure I can/should. The correct solution is to fix PHP itself by altering how it interfaces with OpenSSL, which could be as simple as altering a couple of lines of code. I'd submit a patch, but I'm not entirely sure what the correct course of action should be since the problem happens so deep in the code and even my suggested fix might cause the more common case (i.e. blocking sockets) to break. It's kind of rare to need the ability to write tons of data to non-blocking SSL sockets in PHP, so it is not surprising that very few people have run into the issue.

Once you've started reading the actual C source code to PHP, it becomes rather frustrating to see how few people actually read the source code to PHP. This is no more self-evident than the comments section on every documentation page on, GitHub, Stack Overflow, forums, and mailing lists where people make uninformed guesses and subsequently pollute issue trackers and Google search results. I blame a combination of laziness and...wait, no, it's pretty much laziness. You can actually download the source code to PHP here [mind blown]. Instead of just blindly compiling and running PHP, you can actually read the source code [mind blown again].

Of course, that doesn't mean the PHP source code is easy to follow - it is written in C and 80% of the code is basically a hodgepodge of horribleness that exists to deal with cross-platform and third-party library integration issues and various bits of ancient cruft that have stuck around from the very beginning of the language. It would probably look a lot cleaner though if the PHP documentation itself linked to the source code (I opened that ticket too but missed proofreading one sentence - sigh). After all, most people tend to spruce things up when they know guests are coming over to visit.