Note for Rick Moen's SVLUG talk of Wednesday, Dec. 5, 2012 on Real-World Linux Security Slide 2 Two severe problems tends to swamp most discussions of Linux security: One is that the subject attracts gadget freaks, whose idea of improvement is to swaddle your Linux system in baroquely complex layers of extra software that is often of dubious merit or is addressing the wrong problem to begin with. As I intend to illustrate further down, complexity is your enemy as a Linux system owner seeking good security, for a number of reasons including your need to understand your system and not have it behave in odd and borderline-indeterminate ways. Also, software that is overfeatured is an actual source of security problems. The other problem is the role played by many (not all) of the spokesmen for and employees of security and antimalware firms, who frankly have a pronounced tendency to throw away perspective and bend or break the truth (mostly scaremongering and sending readers down wrong paths) in order to generate business. During this talk, I'll discuss some examples. To be fair, they are not in the business of helping people arrive at a reasonable and balanced understanding of security; they're in the business of selling software services. This talk aims to be refreshingly free of both gadget-freak technophile jargon and of security-industry scaremongering, and help ordinary non-expert Linux users understand the basics without needing to get deeply into anything. Slide 3 Here, I remind the audience of the vital architectural separation of privileges built into Unix systems, especially that between ordinary user accounts and privileged accounts (mainly the 'root' user), plus the mechanisms for group rights. Each process has an owning user and group IP associated with it. (Those are integer numbers, but mapped with alpha names via system files /etc/passwd and /etc/group.) If you want to become comfortable with Unix systems (such as Linux), the scheme for owning users, owning groups, and rights that applies to all files/directories and all processes is one superb place to start. However, anyway, it's really important to know that a process started by user FOO will normally have only the privileges that user FOO does -- so among other things that process if misbehaving or buggy cannot do any damage beyond what user FOO can. By contrast, a process that runs with the authority of a privileged user can do vastly more, and by the same token can do vastly more damage. Slide 4 Here again, I extend my gentle introduction to what really matters as the bones of any Unix system: processes, rights, ownership -- and mention the extremely useful fact that processes tend to utter some diagnostic output sent to the default STDOUT text output stream and in many cases some other diagnostic output to logfiles. If something is going wrong on a Linux system, about 80% of the time it's some issue with ownership/rights, and clues can be found somewhere in /var/log. Slide 5 Security issues derive primarily from bugs. It's a fact of life, and a large part of real-world security is dealing with that. I mention that 'Coders are optimists.' This is just my observed generality: Programmers tend to be surprised when things go badly wrong. Sysadmins (my own esteemed profession) tend to be surprised when they don't. My warning about 'Your user's processes can do limited damage (deliberately). Don't mess with this' has in mind the tendency of newcomers to Linux to change security-sensitive subtrees' ownership for their convenient editing and their tendency to overuse the root-user account (which includes such use via sudo). If you're wielding root-user authority quite a bit, you're doing something wrong. Ask SVLUG's main discussion mailing list for advice, and you'll get it. Last, this slide marks the surprise appearance of the worst security villain: you, the administrative user. More and worse damage is done to Linux systems by you and me wielding superuser (root and similar) access than by anything else, by far. This is a really vital fact. Learn it now, or learn it the hard way. Or both. ;-> While writing this presentation, I happened to be doing some work on my system -- ominously, before my morning coffee -- and needed to delete some files under /usr/local/src that had somehow gotten stored with root authority (probably stored that way in a tar.gz archive). Let's say I wanted to delete /usr/local/src/stuff/* . [rick@linuxmafia] /usr/local/src/stuff $ su - Password: linuxmafia:~# rm * Oops. My fatigued mind missed that I was no _longer_ in /usr/local/src/stuff, but rather in the root user's home directory. And the two or three files I had in /root are now just GONE. No getting them back. rm is forever in Unix, absent odd measures to compensate -- or backups. Which is one point I'm heading directly towards. Slide 6 The point of this slide is that a balanced approach to security is broader than just 'Keep bad things out and detect if they've gotten in' -- which not coincidentally is all the security firms aspire to do, and typically don't do that very well. Consequently, that is also the near-total focus of the IT press, who sadly seem to do little more than crib from press releases, a lot of the time. To expand on those a bit -- and you may want to skim this excessive detail to get just the major ideas: Limit exposure: At the risk of introducing a bit more jargon, security people talk about a system's 'attack surface': the set of software routines a potential attacker might probe to find exploitable bugs. If it's an attacker who's remote from the system (hasn't found a way to get inside), then the attack surface is small: just the kernel's network layer and any externally reachable network services you are running, which on a desktop system are few-to-none. (Possible exceptions include the CUPS network printing service and the Avahi daemon for ZeroConf casual networking.) If hypothetically an outside attacker finds a way to run local processes on your system, i.e., has broken in, then the 'attack surface' becomes larger, and the key issue is whether he/she is able to find a way to 'escalate privilege' to run processes as the root user or equivalent, by attacking bugs in privileged software. In very general terms, you limit exposure by eliminating bugs and needless complexity in code that is likely part of an attack surface, and especially by hastening to eliminate bugs that are likely to permit privilege escalation. Defend in depth: This is a very general notion that you don't want to rely on only one layer of protection to defend you. An example of doing it wrong is the common corporate error of lazily relying on the corporate firewall to protect everyone from the Big Bad Internet, and doing nothing whatsoever to make machines behind the firewall more difficult to exploit _if_ an intruder is able to reach past the firewall -- derisively called the 'nougat' security model, that is, a hard shell with a yummy soft inside. Harden: Speaking of the nougat model, this is what a smart corporation does to avoid that error (and so should you). Details are mostly beyond the scope of this introductory discussion, but, for example: Distros often compile their kernels using protective patches like ExecShield or PaX that foil many types of attack even against buggy application code, and many compile application software using AppArmor, which is a method for per-application sandboxing. Some discussion of that: http://www.outflux.net/blog/archives/2011/02/11/shaping-the-direction-of-research/ Identify attackers: In theory, one part of a well-balanced security policy is attempting to determine who's attacking you and do something with that information, e.g., file police reports. However, seriously, most of the time it's not worth your time to try, and the authorities won't care. Most of the time, your 'attack' was by an automated script running on an innocent person's computer who had lax security, and often the ultimate bad guy (in some distant jurisdiction with questionable law enforcement) compromised the innocent person's computer using another automated script such that the bad guy doesn't even know where the latter script's running exactly. The reality of Internet security threats is much less dramatic and personal than you might imagine. For one of the extremely rare (but now ancient) exceptions, read Cliff Stoll's book _The Cuckoo's Egg_. Recover: Consider various forms of the worst-case scenario. Either you goof with root-user authority and delete huge swathes of your important files, or a key hard drive / SSD fails, or (least likely by far) the Bad Guys break into your system, issue an Evil Overlord cackle, and delete or scramble everything. If you had a recent backup plus the means to reinstall your OS system + apps, that contingency would be a nuisance rather than a catastrophic horror, right? That's why I say making sure you have the means to reinstall + restore is your #1 priority. It really outweighs all other security concerns. This will be discussed further. Slide 7 A backup scheme doesn't have to be particularly good or clever. The most important bit is that you _do it_. Perfectionism is the enemy of adequacy. On the other hand, thinking your backups are adequate and finding out the hard way that they aren't is tragic. So, you need something easy and simple enough that you _actually do_ it, but which you have reason to think will save you in one of the above worst-case scenarios. Many smart but foolish people put off backup because they haven't worked out how to do it cleverly and well. I've been that guy -- and lucky enough that once-in-a-blue-moon measures got me back when hard drives failed. A couple of years ago, I decided to do better, and spent $100 at the now-closed MicroCenter in Santa Clara on a USB-enclosed 2 TB hard drive. Now, I can connect the drive to my server, mount it to /backup, use ordinary file operations to back up essentials to it, dismount /backup, and store the drive somewhere safely distant. Here's my overall scheme for backing up / restoring and rebuilding my Debian-based server: http://linuxmafia.com/faq/Admin/linuxmafia.com-backup.html Here are some commands that automate copying stuff to the detachable drive: mysqldump -uroot -pMYPASSWORD --all-databases > /backup/alltables-$(date +%F).sql fdisk -l /dev/sda > /backup/partitions-sda-$(date +%F) fdisk -l /dev/sdb > /backup/partitions-sdb-$(date +%F) fdisk -l /dev/sdc > /backup/partitions-sdc-$(date +%F) dpkg --get-selections "*" > /backup/selections-$(date +%F) tar cvzf /backup/etc-$(date +%F).tar.gz /etc Here's my very crude but good-enough script that copies the rest of what matters: #!/bin/sh cd /backup rsync -ax /var/lib var-lib/ ; \ rsync -ax /home/ home/ ; \ rsync -ax /var/spool/ var-spool/ ; \ rsync -ax /usr/local/ usr-local/ The Web page details how those saved trees and snapshots would be used to quickly rebuild the system in the case of sudden failure. Slide 8 As I say on the slide, the best way to know your rebuild + restore scheme is adequate is to try it. The best way to try it is to pretend your hard drive just died, and see what happens if you need to rebuild onto a new hard drive. I personally haven't done this in years, which is arrogance on my part -- but at least I keep skeptically re-studying my backup methods to catch anything I might have missed. What I call Big Dumb Copy methods are things like software that images your entire hard drive to somewhere, e.g., as is done by Symantec Ghost or PartImage. Novices love Big Dumb Copy technology, because it eliminates the need to think. There are a number of drawbacks, including pointlessly including man gigs of distro-furnished files, bloating backup sets, making backups take much longer than they should, and delaying the day you actually bother to understand where your files are that need backup versus those that don't. Slide 9 Novices to Linux are often slow to understand that there is a distro way of doing things (somewhat idiosyncratic to your preferred distro), and that being clued to the Distro Way has huge and compelling advantages. Ignoring your distro's 'way' -- especially its software maintenance regime -- is a particularly bad novice error, but understandable in newcomers because there's nothing quite like it elsewhere. Distros have people who maintain the software they package and make available. The package maintainers are quality checkers, assurers of security, and general gatekeepers. Having them work for you is a godsend, compared to having to manage everything yourself. You want to work with this system, and not be at cross-purposes with it. Every distro has a regime and toolset for the purpose. In the Red Hat family (including CentOS, Mageia/Mandriva, OpenSUSE, Oracle Linux), the toolset involves the 'rpm' program and package format, and some management front-end tool that's most often the one called 'yum'. In the Debian family (including Ubuntu and offshoots), it's the 'dpkg' program and package format, and some front-end tool that's most often the one called apt-get (and optionally a graphical front-end shell for that). It's extremely worth your while to get very familiar with the ins and outs of your distribution's maintenance scheme and the tools you will work with. Do that right after getting serious about backups. You should also subscribe to your distro's announcements-only security announcements mailing list. Seriously. No, you won't get a lot of mail. It'll typically be a few per month. And you won't be reading it in detail; you'll be skimming it, as detailed later. Do not make the ghastly error of sticking to a distro release for years after it's been orphaned (after updates for it cease). That is one of the quick routes to a security catastrophe, not to mention getting stuck using ancient versions of everything. Slide 10 The flip-side implication of having an extremely reliable distro maintenance scheme is that, having observed that the scheme very reliably gets you vetted, software-signed updates that smoothly upgrade your system, you should become intensely wary -- downright suspicious of any suggestion that you install software in other, lesser, less safe ways, especially if the suggestion involves root-user authority (including use of sudo). This cultural and technical norm is very distinctive to the Linux world. It is A Very Good Thing. You should pick up that prejudice and learn to love it. That prejudice will be instrumental in protecting you from doing dumb, system-damaging things. Slide 11 There are things desktop users want to install that distro maintainers cannot or do not package. Most often, it's for various legal reasons, like Nvidia not permitting distros to distribute its binary-only video driver sets. This slide acknowledges the inevitability of people feeling compelling needs to go outside the distro package regime to install some 'exception' software -- and reminds readers of the built-in drawbacks to be aware of. To explain the bit about 'gatekeeping': The distro package maintainer's job includes skeptically watching new releases of the software and deciding that some new ones are too buggy, excessively break old setups, or may have other problems warranting the distro sticking with an older version (or one with modifications by the distro maintainer) a little longer. The package maintainer does other essential tasks, too, such as making double-sure that revisions actually came from the programmer and are not a forgery by someone who broke into the programmer's site (which has happened occasionally). I detail gatekeeping advantage here: http://linuxmafia.com/~rick/weatherwax.html#1 Slide 12 _If_ you need to install a 'local' (non-distro) software package, there is a hierarchy of places to look for the least-bad source. This slide details those in order. By 'a reputable third-party repo', I mean a place like the Dag repo for Red Hat-family distributions, and deb-multimedia.org for Debian-family ones -- large repositories with a long-standing credible record for high quality and security. By 'a reputable outsider's repo', I mean a repo maintained by an individual to serve at least temporarily because there is not (yet) an official distro package. By 'distro-wrap', I mean you've used a software tool such as debhelper for Debian-family, or rpmbuild for Red Hat-family distros. This slide also mentions the worst-case outcome that sometimes happens when you just grab a binary or a tarball or a binary-only package that you have no reason to trust: Trojaned (imposter) software. For example, in 2009, a supposed GNOME screensaver showed up on Web site gnome-look.org, packaged as a .deb file. Impliedly, downloaders were expected to download, then do 'sudo dpkg -i packagename.deb' or equivalent, i.e., install the package from nobody-in-particular using root (sudo) authority. Doing so was of course a blunder of the first order, as the post-install scripts in the .deb deliberately did nasty things to the target system, compromising its security. Newcomers said 'But we didn't know!' OK, noted, and now you're being told: Don't be gullible. If something unexpectedly calls upon you to wield root authority and go outside the normal chain of package tools (and the regular package collections), then be intensely suspicious. The bit about 'check to see if there's a package' merely means that, if you installed something locally because your distro doesn't yet have a package for it, you should occasionally re-check to see if a package has been introduced, and switch to it. Slide 13 There is certain software that is notorious as the Typhoid Mary of security problems on desktop systems. These are the ones. Adobe's software may have many virtues, but these particular ones have a history of being unsafe to expose to the Internet, e.g., because your Web browser is configured to hand off Flash and PDF files to them as 'helper apps' to read them. What makes Adobe Acrobat such a particularly serious problem is its default Preferences configuration that makes it process Javascript included inside the PDF. Unfortunately, Javascript makes a glorious vehicle for automating malware and other mischievous operations, so it's pretty much the last thing you want enabled in a handler for arbitrary PDFs receive off the Internet. The functional purpose for Javascript-handling inside some PDFs is for automation of form-entry when you are entering data _into_ a PDF online, e.g., having the software tell you 'No, that's not correct data that you put into the Zip Code field, because Zip Codes consist only of numbers.' _However_, just viewing or printing PDFs doesn't require this functionality at all. So, it is best to use a different PDF viewing/printing application for your Web browsing such as Evince, Okular, or xpdf -- which also are fully open source whereas Adobe Acrobat is proprietary. If you wish to keep Adobe Acrobat around, that's defensible, but having it handle Internet content is just too dangerous. My mention of muPDF (for MS-Windows) is a reminder that pretty much all the principles of security I'm mentioning in this lecture apply equally to all OSes, not just Linux -- with the exception of my points about distro packages. Flashblock is a slick extension for Firefox that shows a 'play control' window where there would otherwise be a running Flash animation -- such that the animations are handed off to Adobe Flash and run only if you say to play them. NoScript is a much more ambitious Firefox extension that inverts Firefox's default of being willing to run any Javascript specified on any page from any domain -- making it so that Firefox runs no Javascript on any page without your saying that page's Javascript from a given domain _should_ be run. NoScript is a very thorough fix to many Web browser problems, but has the drawback that it's initially a little difficult to get used to. Slide 14 As Marcus Ranum has eloquently pointed out (http://www.ranum.com/security/computer_security/editorials/master-tzu/), a policy of fixing software becomes counterproductive if it's _really bad_ software -- where 'really bad' implies that it has systemic defects that never get really fixed. This is fortunately not too common, but more than you might hope. About 'input validation': Software that handles anything off the Internet, including even URL strings, cannot just naively assume that data is safe and hand it off to brittle, optimistic-coder-written parsing routines. Basically, the data must be combed over to make sure it is valid, otherwise mischievous people on the Internet will poke at it using misshapen input data to see if it can be made to break in ways they can manipulate, hoping to cause security risks. This is Programming 101, yet there are naive programmers all the time who roll out software for Internet exposure -- or that is repurposed by others to handle Internet content -- without bothering to validate input data at all. Ranum's point is that if you're repeatedly 'patching' (fixing, updating) software that the patching doesn't actually fix, you are playing the wrong game. That is software to get rid of, or at least make sure never handles public data. Slide 15 Here, I will be going through some real-world examples of recent security advisories explaining how to quickly go through them and decide which to ignore and why. Soon enough, the task will take you really no time at all. By 'software you omit', I mean software you've not actually installed. Here, it becomes really useful to be fluent with your distro's package tools, so you can quickly determine whether you've installed something or not. The point is: If you aren't running it, you needn't fix it. By 'software you installed but weird configs', I mean security advisories said to apply only if far-fetched optional configurations or customisations are present that are not default and (more to the point) you haven't gone out of your way to implement. Again, you will learn to spot these quickly and ignore them. By 'could make the software crash/hang', I mean what are also called Denial of Service (DoS) problems. These merit a bit of explanation. Let's say the advisory concerns a problem in package libtiff where a malicious person on the Internet might talk you into loading a carefully crafted TIFF image that causes libtiff to crash / hang / segfault / whatever (in other words, croak). Annoying, yes, but not a regular security issue in the sense of intrusion or unauthorised modification of your system, nor of causing it to flood Amazon.com with ping floods, or anything like that. These are in the category of 'upgrade to fix this when it's convenient, but no hurry.' The phrase 'potentially execute arbitrary code' (or similar) is very common in security advisories. This, too, merits a bit of explanation. A bug will often put the program in question into an error mode where it'll probably just do nothing exploitable or die -- but in theory the attackers _might_ find some way to exploit the bug. So, you end up with a definite bug that has no known exploit, but someone _might_ someday be able to create one. Or not. As you might imagine, in the general case, exploits are unlikely, as most bugs that _might_ in theory be exploitable prove too difficult figure out a way to exploit. So, as a general rule these are a bit less trivial than the prior 'could make the software crash/hang', but still not urgent. 'Privilege escalation' is a worrisome phrase to hear in conjunction with any other bug information, because that makes it a real system-level threat, at least potentially -- especially if it's a buffer overflow with likelihood of such elevated privilege. By 'anticipatory', I mean that Unix bugfixes tend to try to patch holes as soon as they're confirmed and long before anyone has found an exploit. Slide 16 I've already explained why Denial of Service (DoS) attacks are just annoying not security compromises per se. Cross-site scripting (XSS) is a similar case but more subtle. An XSS is a tricky sort of situation where a flaw in, say, your Web browser permits the proverbial malicious person, operating a Web site, to send your Web browser HTML that triggers the bug in your browser and coaxes it into sending data to a second Web site that is in some way injurious to the second site. Sucks to be the second Web site's operator, but (as with DoSes), it's not actually a threat against the security of your system in any of the usual senses of the word. You want to fix the XSS problems in your desktop software in order to be a more-polite Internet citizen, but they are not threats against _you_. Rootkits: I'm covering these because I've gotten tired of erroneous reporting in the IT press abetted by distortive quotations from security and antimalware industry figures who know better and should be ashamed of themselves. A rootkit is a set of software that can be installed by an intruder who gains access to a system and who cracks the root-user account by _other means entirely_. Classical rootkits provide specially gimmicked replacements for several standard administrative utilities (netstat, ps, ls, df, du, sar, etc.) that silently fail to report the intruder's processes and files, to avoid revealing the intruder's presence to the system operator. By extension, 'rootkit' can also encompass sets of other _post-attack_ tools to do other things. The key point is that these are post-attack kits installed _after_ the intruder cracked root. They are not themselves in any way useful for attacking the system. This is why I compare them to 'muddy tracks left behind by the burglar'. The uncovering of a rootkit on a system is indeed proof of a problem, but the problem is not the rootkit, but rather whatever means of entry and escalation to root was used by the intruder -- just as the muddy footprints in your living room are not the real problem, but rather the burglar's access that enabled him/her to put them there. Slide 17 This is a return to the temptation towards gadget-freakery. Some people's idea of how to deal with 'attacks' (which are not necessarily credible attacks -- often they mean just automated checks to see if weak username/password combinations are accepted by an ssh server, for example) is to automatically build up elaborate active defences such as adding the remote IP address to an iptables ban list. I question the wisdom of such apparatuses, but that's not the larger point, which is system complexity. System overcomplexity introduces bugs by itself, and also makes it more difficult and a longer task to learn to know your system's software well. My larger point is that it's usually wiser to err on the side of simplicity, especially until you are more expert. I mention logwatch / logcheck (two implementation of the same idea) and OSSEC (www.ossec.net) as things to consider _later_ but not initially, with the putting-off-to-later on complexity grounds. Both are system-monitoring tools that are long-term useful but require significant adjustment ('tuning') to reduce the percentage of trivial babble in their reports. Slide 18 Read them. Some are actually really entertaining, especially Ranum's.