Anatomy of a Digg

Yesterday morning I woke up to find that one of the posts on my Geek Etiquette blog had been Dugg.

Within half an hour, my site was offline.

I spent a good chunk of the day dealing with my first “digg effect”, and thought it might be interesting to write up.


All times are in Australian east coast time, and approximate.

* Wednesday afternoon, I posted Dresscodes: Geek vs Non-Geek to my blog.
* Thursday, during the day, I checked my website stats and noticed links coming in from Reddit. These increased my blog traffic from the usual 400 a day to about 1100.
* Around the same time, the site was also submitted by the same person — a commenter on GE — to Digg. The traffic was slower to take off there, though, and I didn’t initially see any traffic coming through from there.
* This morning, a bit after 8am, I checked my email. There was a slew of comments on GE, so I checked the stats. Both the GE comment moderation panel and my stats site (on a different domain, but hosted on the same server) were slow to load, but I managed to see that I was being dugg. I found the entry on Digg’s site, and found that I had around 750 diggs (“thumbs up” votes) and rising.
* 8:30am: While moderating comments, suddenly I was locked out of my site. I was getting HTTP 403 (“Access Denied”) errors. I ssh’d into my server and took a look around. I saw that at least one file in my blog installation had a modification time of today — and not by me. My first thought was perhaps that getting dugg had attracted attention to my site and made it the target of malicious intruders. I called work to let them know I’d be late while I dealt with the problem.
* 8:35am: copied the file tree of my GE site to a safe spot, just in case. Took a closer look at what had been changed. Just the `.htaccess` — which had had “Order deny, allow… allow from…” and two IP numbers. Ran `nslookup`, determined that those IPs were a) something at, and b) my work’s gateway. Huh?
* 8:36am: email from Dreamhost support, telling me that my site was bringing the server to its knees, and that they’d cut off access for now. Ah! Stopped worrying about script kiddies. Support suggested enabling the wp-cache plugin. I thought I had already, but no, for some reason it wasn’t activated. Duh! Activated it, and checked that it was active on my other blogs too (it was). Just to be on the safe side, logged in to the Dreamhost panel and checked that my WordPress install was at the latest version, with no outstanding security patches. All was well.
* 8:45am: headed off to work, phoning my boss to let him know I wouldn’t be so late after all.
* 9:15am: arrived at work, checked email. DH support thanked me for enabling cache, but said the load was still high, and could I do anything else — perhaps disabling unnecessary plugins — to help with that. I tweaked a bunch of stuff, then remembered I’d seen a potentially useful plugin the other day. I found Digg Defender, installed and activated it, and the load on DH’s server dropped back to normal.
* Saturday morning, at time of posting, the number of Diggs is approaching 1200.


Unfortunately my usual web stats package was busted by the digg experience (see below), but I can still see a number of first and second level effects:

* My Reinvigorate web stats, for which I signed up for a beta program, claim I had about 30,000 page views (from 28,000 unique visitors) on the day of the Digg and around 3000/2800 so far today. My usual traffic is 300-400 visitors a day. The following graph only shows the last few days, since I activated Reinvigorate stats on the GE site, but it gives you some idea of the traffic spike:

Daily visitors

* My Feedburner stats show RSS subscriptions rising from around 370 to 622 on the day of the Digg.
* My Technorati authority score rose from 27 to 38, based on a number of other blogs linking my post. It looks like they’re about 50% “real” posts and 50% blogs that automatically link anything that comes through digg.
* The post was subsequently picked up by StumbleUpon; right now, a day after the peak of the Digg traffic, I’m seeing most referrals coming through there, rather than Digg itself.
* There have been more than 20 comments on the post; posts on GE more often get single-digit comments. The increased comments definitely aren’t linearly related to the number of visitors, though.

I don’t have any Google ads on GE, so no stats from there, sorry.

Technical issues and solutions

Server load: WordPress configuration solutions

Apparently the load average on my server peaked around 150. Yow! WordPress runs on PHP and MySQL, and that makes for a pretty resource-hungry site if it’s getting hit hard.

If you run a WordPress blog and have even the slightest expectations of any traffic at all, you *must* install and activate wp-cache. As far as I can tell there is no excuse for not doing this. In fact, I thought I *had* done it; perhaps I just mis-clicked somewhere at some point and didn’t notice.

Unfortunately wp-cache only managed to bring the server load down to 20ish: still too high. Next step was to deactivate any unnecessary plugins. I believe any active plugin (i.e. green in the “Plugins” admin screen) will be loaded into the PHP interpreter each time you get a page hit on the site, so it’s generally good practice not to leave anything activated if you’re not using it. I also deactivated a handful of things like the comment preview plugin, which I could live without for a little while.

I also simplified my sidebar. The main thing I removed was the “related posts” section, which wasn’t really working for me anyway. I suspect this may have been generated dynamically and not cached, but I’m not sure; either way, no harm in getting rid of it. If I’d had any other dynamic sidebar cruft, I also would’ve removed it at this point.

By now the load was hovering between 10 and 20 and I’d soaked up some caffeine. Finally I recalled the Digg Defender plugin I’d seen while cruising around the other day. This plugin uses the Coral Cache distributed caching network to protect against sudden spikes in load.

I grabbed it and installed it, and the load dropped back to 3 or 4. Support tells me this is normal for that server at that time of day, so we’re all happy now. I can’t recommend this plugin highly enough; I reckon you should install and activate it “just in case” on any blog that has the slightest chance of being dugg, slashdotted, or otherwise hit by a social networking DDOS.

My stats problems

Dreamhost’s web stats are kind of manky, so for a long time I’ve supplemented them with awstats. I run the awstats update script regularly from cron, and can then view the reports via a website I’ve set up for the purpose.

Unfortunately, getting dugg meant my httpd logs were so huge that the awstats script took a long time to run. That wouldn’t be a problem except that Dreamhost kills any long-running scripts. And the longer I wait to run this thing, the more backlog there is in my access log and the longer it’s *going* to take. It’s currently sitting at half a million lines.

I’ve contacted DH support to see whether there’s any way round the script-killer. I suspect that running it at a quiet time of day, might help. I’ve tried `nice` and it does nothing; I had hoped the script-killer might be smart enough to look for that, but no luck.

If I can’t get it sorted out by the time the logs are rolled, I’ll probably save off a copy of the enormous access.log, cut it up into chunks, and pass them to awstats one by one. At least that will make sure I don’t lose the history of this event.

Finally, I’m actually finding that my Reinvigorate stats are pretty good, and I might just increase my use of, and reliance on, those external stats, rather than relying on my own awstats installation. The stats report images in this post are all taken from my Reinvigorate stats.

Wiki review: DokuWiki

I’ve been looking for a simple, lightweight wiki application for my website since my recent redesign. Here’s what I wanted:

* quick, no-nonsense install and configuration
* easily restrict editing access (me, and perhaps a small handful of friends)
* clean interface and pleasant markup that doesn’t make me want to scream (TWiki, I’m looking at you!)
* easily themeable to look like the rest of my site
* runs on Perl or PHP for preference

Someone on IRC last week (sorry, I forget who) suggested DokuWiki so that’s what I’ve gone with, and it’s looking great so far.

I particularly liked the simple, one-step install procedure. Check this out:

DokuWiki install

The three access control levels are simple and just what I needed, and the filesystem-based storage mechanism — no database needed — meant that the whole thing was done in a single step. No mucking around with MySQL!

I set a few configuration options in the admin screen after that, most importantly URL rewriting to make the page names neater and more search-engine friendly, but such configuration is optional. (Bonus point: configuration is entirely web-based, not requiring the editing of PHP files.) One item that I think should’ve been in the configuration and wasn’t, however: the Creative Commons license. I changed mine from the default CC-BY-NC-SA to CC-BY-SA by editing `dokuwiki/lib/tpl/default/footer.html`.

The user interface and look-and-feel are customisable via templates (i.e. themes or skins), and there are a number of attractive ones available. I suppose I’ll probably be developing my own, but for now I’m using the default template until I get a good feel for the interface and what I want.

There are also a number of plugins available, but again, I’m holding back on those til I see what I need. I suspect that the tag plugin will come in handy, along with one of the improved code markup ones.

From a user perspective, DokuWiki’s syntax seems to be quite similar to MediaWiki‘s. In fact, the whole thing has the feel of a lightweight MediaWiki. If anyone ever tells me, “I tried to install MediaWiki but gave up,” I’ll definitely point them in this direction.

In summary: if you want something that looks clean and professional, is easy to install, and doesn’t need to be massively scalable, DokuWiki looks like a great choice.

(If you’re interested, you can take a look at my embryonic Infotrope Wiki in action.)

OS switching and the XY problem

An XY problem is one where you have a problem X which you want to solve with tool or technique Y, but Y isn’t necessarily the right thing for the job. You ask, “How do I solve X with Y?” and everyone looks at you blankly.

*How do I hammer in this nail with a screwdriver?*

*How do I use Outlook Express under Linux?*

*How do I configure OSX Finder to [do anything useful]?*

I’ve switched (either wholly or partially) from Windows to Linux in 1993-1995, Linux to Mac around Christmas 2004, and Mac to Windows — not very happily, as you’ll see — in 2006.

Each time I’ve found myself asking X/Y questions, and I hear other people doing it all the time as well. Now I’m going to try and pre-empt some of them.

Windows to Linux: Does {MS Word, Outlook, iTunes} run on Linux?

The problem you think you have: you want to run Word, Outlook, or iTunes.

The problem you really have: you want to write documents, read your email, or play mp3s.

The question you should’ve asked: “Can I write documents, read my email, or play mp3s under Linux? What app do I need to use for that?”

The fundamental point: Microsoft apps (or those from many other major commercial vendors) don’t generally run on Linux, but you don’t need Microsoft apps to be productive; that’s just brainwashing and/or force of habit.

See also:

Linux to Mac: How do I configure $APP to do what I want?

The problem you think you have: You just have to find the right configuration flag for the app, so it will do what you want. This is Unix under the hood, after all!

The problem you really have: You want an app to behave in a way that it doesn’t do out of the box (or perhaps at all).

The question you should’ve asked: “Is there a third-party tool that will let me do this?”

The fundamental point: Steve knows best. Trust Steve. He dictates how the Mac will operate. Life will be shiny and lovely if you just think like Steve does. (But everyone else disagrees, and writes shareware apps to fix it.)

See also: VersionTracker.

Linux to Windows: Surely, *surely* there must be a way to do X?

The problem you think you have: Somewhere, in this nest of menus and wizards, there must be an option to do the blindingly simple thing you want.

The problem you really have: You want to do something that should be quite simple, like turn off annoying popups or configure the network.

The question you should’ve asked (choose any):

* what’s the phone number for corporate desktop support?
* can someone provide me with a stiff drink?
* where the hell is that Ubuntu install disk?

The fundamental problem: Quite often you just can’t do the simple things you expect to be able to do. Chances are that you’ll need to reboot, contact someone with Admin privileges (my personal bugbear, in a corporate environment), or just numb your brain with meditation exercises and/or hard liquor until you don’t care any more.

See also: Fatalism (Wikipedia)

Open Source Developers’ Conference: Last day week (or so) to submit papers!

I just submitted my paper proposals for the Open Source Developers’ Conference, an annual event held here in Australia which brings together developers from a range of different open source communities and languages, including PHP, Python, Ruby, Java, Javascript, Mono, and I don’t even know what else. Even though I usually spend most time in the Perl track, it’s good to have options.

Here are the titles of papers I submitted:

* CPAN meets Web2.0: 20 modules in 20 minutes
* Packaging Perl Applications: From CPAN To Your Project
* Better Than DBI: Why writing SQL sucks and how CPAN can help
* Test Driven Development (tutorial)

(The TDD one is the only one that’s not Perl-specific.)

If you have the urge to speak at OSDC, today’s the last day we’re approaching the last days of the Call For Papers, so hurry up and get your proposals in!

Why are the intertubes like a catflap?

A couple of weeks ago my Dad and I were on our way to visit my Grandma, who’s moved into a retirement village down at Torquay. As we were driving through Geelong, I thought I’d see whether Dad had caught any of the buzz about this web2.0 thing all the kids seem to be into these days.

Turns out he knows about blogs, wikis, and Google Maps, and has heard of YouTube and MySpace though he hasn’t visited either. I was pretty impressed; Dad hasn’t really understood the technologies I’ve been using since I was in secondary school.

And so, emboldened, I started to talk about mashups, APIs, RSS feeds, and more. I tried to explain how people can take the photos off Flickr, cross-reference them geographically against Google Maps, and come up with Flickrvision. Or that people can aggregate content and embed rich media from all around the place into other websites, without having to painstakingly collaborate with the content providers at the other end on a case-by-case basis.

“Oh,” said Dad, “that sounds pretty obvious to me. I’m surprised you think it’s newsworthy.”

That’s when my jaw hit the floor. The man who’d been completely bemused by email a few years ago was now claiming that Flickrvision was the obvious and inevitable outcome of this whole “Internet thing”. The guy who’d made me explain again and again how anyone could ever make money off free software now considers it a matter of course that there should be open APIs all over the place.

I’ve been thinking about this for weeks now, and all I can come up with is: “That’s easy for you to say, you smug bastard.” If it were really that easy to come up with this stuff, Dad wouldn’t have been shuffling punch cards in blissful oblivion for Big Blue while Ted Nelson and Doug Engelbart were inventing Hypertext and the people at Xerox PARC were coming up with computer networks and GUIs; he’d be a billionaire by now.

Or, as Douglas Adams put it in Dirk Gently’s Holistic Detective Agency:

> “St Cedd’s,” [Dirk Gently] pronounced, “the college of Coleridge, and the college of Sir Isaac Newton, renowned inventor of the milled-edge coin and the catflap.”
> “The what?” said Richard.
> “The catflap! A device of the utmost cunning, perspicuity and invention. It is a door within a door, you see, a …”
> “Yes,” said Richard, “there was also the small matter of gravity.”
> “Gravity,” said Dirk with a slightly dismissive shrug, “yes, there was that as well, I suppose. Though that, of course, was merely a discovery. It was there to be discovered.” He took a penny out of his pocket and tossed it casually onto the pebbles that ran alongside the paved pathway.
> “You see?” he said. “They even keep it on at weekends. Someone was bound to notice sooner or later. But the catflap… ah, there is a very different matter. Invention, pure creative invention.”
> “I would have thought it was quite obvious. Anyone could have thought of it.”
> “Ah,” said Dirk, “it is a rare mind indeed that can render the hitherto nonexistant blindingly obvious. The cry ‘I could have thought of that’ is a very popular and misleading one, for the fact is that they didn’t, and a very significant and revealing fact it is too.”

And that is why the intertubes are like a catflap.

Incidentally, if Wikipedia is to be believed — and why not, since it’s supposed to be as accurate as Encyclopedia Britannica on scientific matters, though whether Isaac Newton’s invention counts as such is anyone’s guess — Sir Isaac *did* invent the catflap. He did not, however, attend the fictional St Cedd’s College, Cambridge; he was at Trinity.

I was POSH before you were.

Via Stennie on #perlnet, the Microformats Wiki lists me as the originator of the term “Plain Old Semantic HTML”:

> The earliest references to “plain old semantic HTML” that have been found so far are:
> 1998-05-01: Kirrily ‘Skud’ Robert on – “The site is implemented in HTML 4.0 Transitional with style sheets; this means that anyone with an older browser won’t get presentational markup, just plain old semantic HTML. ”

How cool is that? I wish I’d acronymed it at the time.

Things I Wish I’d Known About Relocating

I’ve been thinking for a while about what advice I’d give people wanting to relocate internationally for work. I’ve done it twice now for myself (from Australia to Canada and then back again), and each time it’s pretty much been a lesson in what *not* to do. So I thought I’d record some of the things I wish I’d known before I went.

1. It costs more than you think

Chances are your new employer is paying some of your relocation expenses. At minimum they’re probably sorting out your visa and buying the plane ticket. If you’re lucky they’re also putting you up for a few weeks when you arrive, and/or paying for shipping of some of your stuff.

Anything they don’t cover, you’ll need to cover yourself. In addition to the obvious expenses listed above, that can include:

* time off work between jobs, including time to pack
* costs of vacating your current accommodation (cleaning, advertising, etc)
* costs of moving into new accommodation (bond, rent in advance, etc)
* temporary accommodation in your home country between vacating your home and flying out
* vehicle-related costs, if you have a car at either end
* storage costs (for stuff you leave behind)
* replacement of household and personal goods in new location (but see “shipping is cheaper” below)
* new clothes (especially in different cultures/climates)
* contract cancellation and account setup/membership/connection fees for all kinds of things (phone, Internet, gym, etc)
* paying off debts before you leave (you don’t want them hanging around)
* and more (kids, pets, ??? — not my problem, but might be yours)

Do the maths on all this, thinking of every possible thing you can. It’s almost certain to be in the thousands of dollars. Whatever the cost comes to, you’re going to have to amortize that over the period you intend to spend overseas. Let’s say you’re going for 2-4 years and it’s going to cost you around $10k — most of it in the first year. You’re going to have to make sure you have that money up front to handle the expenses, and that the money you earn overseas over the 2-4 years you stay there is enough to make it worth your while.

2. Simplify before you go

One of the things I’d advise anyone who’s considering international relocation is to start simplifying their lives *now*. The less stuff you have — both physical and intangible — the easier and cheaper the move will be. You’ll probably start thinking about an international move 3-6 months before you actually do it, so here are some things you can do during that lead-up time:

* Clean out your garage, closets, and spare room. Have a garage sale, freecycle it, or send it to charity.
* Start boxing up things that you want to keep long term, but definitely don’t want to take overseas. This includes things like family keepsakes or bulky collectibles. Put the boxes out of the way — perhaps in your now-emptier garage — and then keep paring down your stuff til only the stuff you use regularly is left out.
* Clean out your wardrobe. Discard anything that’s not in good condition or that doesn’t fit well. Put aside anything you haven’t worn in 6-12 months. Box up any seasonal or special-occasions wear that you won’t need to take. Consider the workplace culture and the broader culture where you’re going. It’s not worth taking clothes overseas if you aren’t going to wear them.
* Eat your way through the pantry. Use up tinned food and dry goods. Throw out anything that’s too old. Get to the point where, when you’re ready to leave, there’s only about a box full of stuff to throw out or give away. Drink all your booze (and don’t buy more).
* Cancel accounts. Get rid of your cable TV, Netflix, magazine subscriptions, gym membership, or anything else you don’t need. Consolidate your bank accounts. Consolidate your debts. Rollover your superannuation (retirement savings) into one good fund. Try to get to the point where you could enumerate all your accounts from memory.
* Minimise paperwork. Clean out your filing cabinet and get rid of anything you don’t need to be keeping. Consider scanning documents and backing them up digitally. Get yourself off as many snail-mail mailing lists as possible. Get your bank to send paper statements less often, and use online banking instead.
* Clean up your digital life. Dispose of secondary hardware, both extra computers and seldom-used peripherals. Make sure your email is all in one place, and that it’s web accessible. Consolidate your bookmarks, likewise. Make good backups. Get yourself into a position where you could, if necessary, dump your whole digital life onto a laptop and go.

See Unclutterer for more tips.

3. Lower your living standards

If you simplified in the previous step, chances are you’ve realised you can do without a lot of stuff you previously assumed was necessary. The good news is, this can now save you a heap of money. The trick is to drop your standard of housing when you move.

* If you were living in a house, move into an apartment.
* If you were living in an apartment, move into a smaller one, perhaps a studio or bedsit.
* Consider sharing a house.

Apart from the obvious savings in rent, you’ll also save a heap of money on shipping, furnishing your new abode, an so on. You’ll also have less to worry about if and when you decide to move home again.

Even if you can’t bring yourself to live in a tiny apartment for long, it’s not a bad idea to do it at first, while you get used to your new country and start to find your way around. If you over-commit to a large home, you’ll spend too much of your time maintaining it and furnishing it when you could be doing better things with your time.

4. Shipping is cheaper than buying

Assuming you’ve simplified and minimised your physical possessions, and you’re planning to move into a smaller place when you get overseas, you now have to decide what to ship and what to buy when you get there. The simple fact of the matter is, shipping is almost always cheaper than buying.

You’ll need to contact a shipping agency to find out what shipping costs will actually be. Your choices are between full-service removalists, who’ll come to your house at either end and handle the customs paperwork, or freight companies who’ll deliver your gear to a warehouse somewhere near the airport in your destination country and leave you to figure out customs on your own. The costs vary wildly, of course.

For the full service removalists, I’ve paid — or my employer has paid — around $1000 per cubic metre of stuff to get it all the way across the world. What you might not realise is that that’s about 20 book boxes, or $50/box. In such a book box, I could fit a several hundred dollars worth of books or DVDs, or a few hundred dollars worth of household materials such as kitchen gear or soft furnishings. (Don’t believe me? I have good quality kitchen knives that would fit in a shoebox but cost $500 to replace.) Even small, inexpensive items like facecloths or plastic food containers can add up when you think about it. And if you’ve simplified right down to the basics, you *know* you’re going to need that stuff at the other end.

In my opinion, there are only a handful of things you *shouldn’t* ship overseas, if you can afford the up-front expense. They are:

* electrical equipment, including household appliances, desktop computers, etc, unless you are *sure* that the power systems in the countries are compatible
* large furnishings, especially if inexpensive (eg IKEA bookshelves)
* items which might cause difficulty at customs on either end

Anything you choose not to ship will either have to be stored in your country of origin, or disposed of and then replaced if/when you get back. It’s up to you to figure out which will cost you more:

1. Ship item overseas; use it; ship it home in a few years’ time.
2. Store current item; buy new item overseas; get rid of second item before returning.
3. Get rid of current item; buy new item overseas; ship new item home.
4. Get rid of current item; buy new item overseas; get rid of second item; buy yet another item when you get home.

In most cases, especially for small items, option 1 is best. For furniture, option 2 (buying cheap/secondhand overseas, or renting a furnished apartment) might be best. Option 3 isn’t a bad choice if all your current gear is shoddy, perhaps if you’re just emerging from a stint as a poor student. Option 4 is seldom a good idea.

(Regarding customs: Australian customs is probably one of the strictest on the planet, and can give you trouble over things like wooden items, bedding, or anything that was once a living animal or plant, such as items made of fur or horn. Check the rules for both the country you’re moving to, *and* those for moving home again afterwards, and leave anything behind that will cause drama.)

5. Pack your favourites

There are certain things you should stock up on before leaving home, because you won’t be able to find your preferred variety when you get to where you’re going. These include:

* toiletries – everything other than basic soap and toothpaste, which are fairly generic throughout the Western world
* over-the-counter medications – for aches and pains, cold and flu, and gastro-intestinal problems
* underwear – if you have a favourite brand/style

There’s nothing worse than being in a foreign country with a nasty head-cold or stomach bug and not being able to find your favourite meds. It’ll happen to you eventually, but at least save yourself that stress until you’ve settled in a bit. Take a full bottle/tube/packet of each of your favourite OTC drugs. The same goes for toiletries, especially anything that’s not generic. If you’ve got a favourite scent-free low-allergy deodorant, or a single kind of chapstick that really works for you, or some kind of hair product you just can’t do without, make sure you take a fresh one of each (and perhaps a spare). You’ll probably find something equally good when you get over there, but it might take you a while.

As for underwear, I’m a big believer in this. Buy up a full stock of comfortable underwear, socks, etc in your favourite brands and styles, and you won’t have to deal with it again for at least 6 months. If you don’t have a particular favourite, this is not such a big deal, but as far as I’m concerned, a stock of Bonds tshirts, Holeproof Explorer socks, and my favourite Target bras are vital.

Don’t buy more than you need, though; you don’t need a lifetime’s supply of cough syrup or undies, just enough to see you through the first season or two until you get acclimatised and develop favourites in your new country.

Weaning yourself off LJ: Images and video

So far we’ve been talking about posting fan fiction to a WordPress-based archive. Now let’s look at how you can add images (such as fanart) and video (such as fanvids).

Uploading images

This could hardly be easier. In the “Write Post” interface, you’ll see an “upload” area. Simply upload the image you wish to include in a post. Choose a short title — this will become the ALT tag on the image, seen when someone’s browsing with a text-only browser — and optionally a description.

(Note: if you have a multi-user blog, a user must be an “Author” at least before she can upload images. To allow mere “Contributors” to upload images, you’d need something like the Role Manager plugin (thanks to Smuffster for putting me on to that one.)

Next you’ll see the “Browse” interface showing what you’ve just uploaded. Here’s an example using some Hornblower fan art done by Black Hound:

Browsing images

This is where you choose how you want the picture to display in your post: full size, thumbnail, or just a link. Choose the options you want, then click “Send to editor”, and a chunk of appropriate HTML will wind up in the text area where you edit your post, at the point where your cursor was.

In the case of “Young Bush”, I chose to show him full-size, since the image isn’t too large to fit in my layout. You can see the results here if you like.

Posting videos

First I’m going to deal with videos hosted elsewhere — on YouTube or imeem for example and then with self-hosted ones.

On imeem, YouTube, and most other video hosting sites, you’ll need an account on their system to upload videos. If you haven’t done this before, I’d recommend imeem, which seems to have a strong established following among vidders. (Sure, YouTube’s got more vids, but the *good* ones are on imeem, as far as I can tell.) How you upload your vids will vary depending on the system you use, so you should follow their instructions.

Once your vid’s uploaded, you can view it. Next to the video will be some code for “embedding” the video. Simply copy and past that code into your WordPress post, and Bob’s your uncle.

Here’s one I uploaded to YouTube (because imeem was timing out on me today):

**51st Century Guy: A Doctor Who fanvid starring Captain Jack Harkness**

But what if you want to keep your vids on your own site? The good thing about hosting your own vids is that you get to track how many people are downloading and watching it, collect comments on your own site, and so forth. The downside is that it can be very bandwidth hungry; you’ll need a web hosting provider that gives you plenty of bandwidth.

Either way, you’ll need to set up your WordPress installation to permit the uploading of video files. Here’s how:

1. Install the MIME Config Plugin.
2. Go to “Options” then “Mime types”. You’ll see FLV is already on the list. You’ll may also want to add other types for different file formats, eg. “zip” for zipped download files, “avi” or “mov” for unzipped movie files, etc. A list of file extensions and their associated mime types can be found here.

You should now be able to upload video files.

Your next choice in regard to videos is whether to let people stream it, or only download it and watch it offline.

Downloadable vids

First, let’s talk about download-only vids. These will cost you less in bandwidth, but require a greater investment in download time from your visitors. On the other hand, many vid-watching fans prefer to be able to download files to watch offline. So you probably want to consider providing a downloadable format, whether you make it the only way to watch the vid or just one of multiple options.

To upload a downloadable vid to your WordPress site:

1. You’ll probably want to zip it first. This means it can’t be played over the web.
2. Next, go to “Write post” and upload your zip file using the upload tool as described in images, above; note that it will take AGES and won’t give you any kind of status report or ETA; just leave it ticking for as long as it takes. (Also see note below, re: max file upload sizes.)
3. Once uploaded, choose “Show: Title” and “Link to: File” when including it in your post.
4. You may also want to include a static “teaser” image, in which case just treat it as any other image, and follow the instructions in the previous section.

(Note: Depending on your web hosting provider, there might be a maximum file upload size in place, and that size may or may not be a reasonable one for vid uploads. On Dreamhost they’ve got it set to 8MB by default, which is too small for my needs. You can override this setting by editing the “php.ini” file (if you have access to it) and setting “upload_max_filesize” to something larger; on Dreamhost, follow these instructions to make special PHP configurations for your own website.)

Streamable vids

If you want your vid to stream in-place from your WordPress archive, here’s how to do it.

1. Install the WP-FLV plugin. If you’re on Dreamhost this comes pre-installed, otherwise follow these instructions for installing and configuring plugins.
2. Next, you’ll need to convert your existing vid to FLV. I use FFMpegX on my Mac, but if you use Windows or another operating system you’ll need to find another suitable tool. FLVs are the standard format for online video sharing sites like YouTube or imeem; if you want to save a vid from those sites, check out KeepVid.
3. Upload your FLV file using WP’s file upload interface. It’ll probably take a while — though not so long as the zip file would’ve, as FLVs are usually done at lower frame-rates and are, overall, smaller. My downloadable vid is 38.5MB but my FLV is only 6.6MB.
4. Using the instructions on the WP-FLV page, use the quasi-HTML <flv> tag to include your FLV file in your wordpress post. To get the appropriate URL for the “href” attribute, you can use the file uploader to include a link to the file and then copy the URL out of that.

Here’s the result over on a streaming FLV and a download link for offline viewing.

Protecting yourself from bandwidth theft

When you upload images or video to your own site, you run the risk of other people embedding those images/videos on their own webpages. Every time someone views that external page, it’ll leach the images/video from your own site, costing you bandwidth. You might think that it wouldn’t be that big a deal, but if someone starts linking your stuff from their MySpace profile, all of a sudden it might use a ton of bandwidth and — potentially — shut down your site or cost you lots of money.

You can easily protect your fanart (i.e. images) from that kind of hotlinking. Just install the Hotlink Prevention Plugin and activate it.

Unfortunately, this will only protect images, not videos or other file types you might upload. If you’re concerned about video bandwidth, your best bet is to zip your vids and make them unstreamable, or else only upload them to external sites like YouTube or imeem. (If you’re technically inclined, of course, you can write your own Apache Rewrite rules in .htaccess, but I’m not going to go there for now.) Since so far I’ve been using about 1% of the 3.4 *terabytes* of bandwidth Dreamhost give me each month, I’m going to leave streaming enabled for now and see what happens.


Heard of BookMooch? I was recently introduced to it via the London Perl Mongers mailing list. It’s one of several book-sharing services around, the best-known of which, I guess, is BookCrossing. But unlike BookCrossing, which centres around people dropping off books in public places, BookMooch matches up people who’ve got books and people who want them, and the transfer happens by good old snailmail.

Some of my recently listed books:

Get free books from

I listed* a small handful of books a few weeks ago, and fairly soon someone popped up asking to mooch one of them. I mailed it off to Helena, Montana about a week ago, and today I got notification that it had arrived. I even got karma* for promptness and the book being in good condition.

(* Links may require login.)

The whole system works on points. You get 0.1 points merely for listing a book as available, 3 points for sending a book, and you spend 2 points to mooch a book from someone else. As you can see, for every 2 books you give away, you can receive three, but I suspect the imbalance is to counteract people dropping out of the system over time.

Another thing I like about it is that they actually realise that the world’s a big, round planet with people spread far afield. One of the preferences you can set is whether you’ll ship overseas: yes, no, or maybe — if the book’s rare or lightweight.

The slim genre paperback I shipped to Montana cost 8 AUD, which I don’t begrudge, but I don’t think I’d want to ship anything much heavier. At that rate, books I mooch will set me back around $5 on average. That’s about what I’d pay in a secondhand bookshop anyway. On the upside, I can do it from the comfort of my own home and not be tempted to over-spend or impulse buy. Downside: I love a good secondhand bookshop!

If any Australians are reading this and want to sign up, I strongly encourage you. Doesn’t seem like there are many of us so far, based on what the system told me when I searched for books in Australia, and it’d be nice to have a few more locals to swap with.

I want this office.

Via O’Reilly Radar, check out Three Rings’ cool pirate-themed office!

Pirate office

We’re going through an office re-decoration at the moment. Wonder if I can talk the boss into this? And hammocks instead of desks — I hear the Royal Navy could fit each person into 18″ of space that way, which sounds like about what they’ve got planned for us.

Why Fanfic, Open Source, and Creative Commons Belong Together

In my head this post had the newspaper-style title: **Fan archive must use GPL, Creative Commons, says Skud.** Please feel free to mentally substitute it, if you prefer.

Anyway, I’ve been letting this tumble around in my brain since Astolat‘s request for volunteers the other day. First, I’d better explain some of the background for those who haven’t been following it all.

Fanlib, and why everyone hates the idea

A couple of weeks ago — right before “Strikethrough ’07″, meaning there were two fannish upsets in such quick succession that we all got whiplash — a corporate endeavour called “Fanlib” was launched. The idea was that it would be “the home for fan fiction”, but — as eny fule no — you can’t just set yourself up as such and expect fandom to be impressed. Turns out this mob had seen the dollar signs connected to any kind of user-generated content in this web2.0 world, and had rushed in without really having the faintest idea what they were doing.

Anyway, Teresa over at Making Light posted a good roundup that will provide background better than I can, and Henry Jenkin’s post is also highly informative reading. But the important point is that a bunch of fandom’s highly respected members said “Right, that’s it, obviously we need to create our own archive so that this sort of nonsense needn’t happen again.”

The result is variously known as “fanarchive” or “An Archive Of Our Own”, and it has a LiveJournal community where the organisers are posting regular updates. If you’re interested, you should read:

* Astolat’s call to arms
* Summary of original discussion
* Organisational structure
* Call for volunteers

I volunteered. I have a suspicion that it’ll lead to craziness, but in any case, I’ve put my name down for the website development. And that’s where we get back to the original topic.

Why fanarchive must be open source and use creative commons

I said this in my volunteer email, but I’ll reiterate it here: I strongly believe that the only way to make this archive work is to design it in such a way that, in the case of emergency, anyone can pick it up and keep it going.

We need to keep in mind that while the fanarchive won’t go looking for legal trouble, there’s a good chance that it’ll occur. Currently the fanarchive is being set up in the United States of America, but contributors will be from all over the world, and — a fact easily forgotten in US-centric fan communities — the world does not have a consistent legal environment. If, for some reason, the fanarchive gets shut down in the US, we want to be able to keep it running elsewhere.

We need to decentralise.

We need to be able to replicate the archive, anywhere in the world, at short notice.

We need to be able to copy the entire archive to our hard drives for personal use or backup.

We need people to be able to do this *even if the fanarchive non-profit disappears*.

We also want create the best possible archive we can, and bring in contributions from the most talented people we can get hold of. We want the archive — both the collection of fiction and the archive itself — to grow and improve over time, rather than stagnating. We want to build the archive in a way that encourages input from the whole community. We want to create a website — and an infrastructure under that website — that stimulates creativity and encourages innovation in all directions. (At least, I assume we do.)

The current best way I know of to achieve this is to ensure that both the software and the content of the archive are distributed under licenses that allow people to use it in all kinds of ways, but not to make it commercial. The current best licenses for this, to my knowledge, are:

1. The GNU General Public License for software, and
2. The Creative Commons license for written works.

Continue reading

Yet another 5 uses for a wiki at work

I just can’t stop.

1. Induction materials

We’ve got one wiki page called “NewDeveloperInduction”, and on the day a new hire starts, they quickly learn that that page contains — or links to — all the basic information they’re going to need to know in their first week or two. We also have a policy that it’s the responsibility of any new person still on their probation (i.e first 3 months) to keep the induction pages up to date.

2. Orientation

I mean this in quite a literal, physical sense. Years ago I left Australia to work in Canada, and though we didn’t have a wiki in the workplace there, we did have a shared webpage where all the expats maintained a list of information for new immigrants, including where to find the local public transit timetables, weather forecasts, tax information, and so forth. At my current job, we have a similar page for immigrants (with information on everything from radio stations to recommended banks), as well as one more focussed on the immediate geographical area around our office: car parking, places to get lunch, and so on.

3. Polls

Many wikis have plugins for polls/votes among users, but even if yours doesn’t, you can do it informally by saying “I think X. Who agrees/disagrees?” and letting people add their names to either list. We do it the informal way, and it works pretty well for us.

4. Soliciting help

Traditionally, if you didn’t know how to use something you’d either interrupt someone to ask them, or send round an email asking whether anyone could help you out. At my current work, we have a series of wiki pages called “FooTips” for various values of Foo, eg. “Linux”, “Perl”, etc. If you’re wondering how to achieve some particular thing and don’t really want to interrupt anyone, just add your question to the “Tips wanted” at the bottom of the page, and someone will probably pick it up via RecentChanges and answer it.

5. Professional development

Seems that companies never really send you on useful training courses, but at least if people add useful resources to your work wiki, you can go peruse them whenever you’ve got some downtime and take control of your own learning. On my work wiki, the “Javascript” page is full of links to resources like this series of videos, and on another page, one of my team members has shared links to his favourite tech-related podcasts.