If I Do My Job Well, You Can't Tell I'm Here
A Tale of an Erowid Sysadmin
v1.1 Nov 2015 (v1.0 May 2015)
Originally published in Erowid Extracts #27
Citation: Erowid. "If I Do My Job Well, You Can't Tell I'm Here: A Tale of an Erowid Sysadmin". Erowid Extracts. May 2015;27:4-5. (v1.1) Online edition: Erowid.org/general/about/about_article16.shtml
My original home base is in Germany. As I write this I am in the Dominican Republic. When these lines went to print, I was on Curaçao. When subscribers received the May 2015 issue of Erowid Extracts, I logged in from Colombia. During November 2015, as this story went live on the site, I'm self-stationed in Colonia del Sacramento, Uruguay. But for most of 2014, I lived in a small fishing village in the south of Morocco. Every few weeks I rented a car to explore the country. I've had tea with Sahrawi fishermen; attended couscous parties at friends' houses (the big family lunch that is taken after prayer on Fridays in Islam); given hitchhiking cops a lift to their checkpoint in Western Sahara; visited the "forbidden city" of Smara; and spent countless hours learning traditional healing methods from a Berber herbalist in Fez. All without ever leaving my office.
Public-facing site? Check. Backend machines 1 through 3? Check. The 12 virtual machines? Check.
A constant stream of robots accesses Erowid, from the friendly Googlebot (checking for updated content), to the out-of-control scraper spiders (run by the student who wants to grab a "copy" of Erowid without realizing their bot can't handle it), to the hostile attackers (distributed denial of service attacks, or DDoS). It's important to keep a tab on heavy-hitting traffic to make sure it doesn't cause Erowid.org to become slow or unresponsive. A single minute of downtime means seventy people can't access the information they were looking for. Existing at the same domain name for twenty years means that our servers are subject to constant attack from around the world. Few appear to be targeted at Erowid specifically, and instead are just attacks against any popular web service.
Next, it's on to the more complex projects, all the while keeping an eye on the monitoring systems that dutifully inform us of problems in real time.
Some of these bigger sysadmin tasks of the last twelve months have included:
- Migrating all our backend installations (such as our email management and site statistics systems) into "jails", which is FreeBSD-lingua for "virtual machines". One can think of them as private cloud setups — private so that we don't hand data over to third parties.
- Moving all WordPress instances (the software used for the Erowid Review and columns like Teatime) into a dedicated virtual machine so that the inevitable next WordPress exploit will not endanger data on Erowid.org itself. WordPress is notorious for security breaches that allow malicious software to be run on the exploited machine.
- Updating Erowid.org to the current version of Apache, the webserver we use. The webserver is the main software that delivers pages to readers. This might not sound like much of a task, but Erowid's complex structure made it a big deal.
- Establishing better monitoring for Erowid's systems, which has served us well when dealing with performance degradations or a recent (small) DDoS attack.
- about 200 software security updates
- two machines that were unresponsive because one of our hosting providers screwed up
- one hard disk failure
- lots of script kiddies trying their automated exploit tools on us (here's a hint: stop looking for win.ini — this file does not exist on a FreeBSD server)
- innumerable misconfigured scrapers that hit Erowid.org with way too many requests in way too short intervals, thereby overloading the server.