Never one to turn down a nonpaying job, I studiously burned through hundreds of those accursed files in the hopes of finding something shocking -- could some large company or government agency be discriminating against some poor spambot? My sense of social justice is already outraged!
Surprisingly, the breadcrumbs led right to the birthplace of American bureaucracy, the IRS. Take a look at their robots.txt file; notice anything? All search engines are denied access to certain directories. I can understand that, they've got private data that they don't want web crawlers getting their sticky fingers into.
But one engine in particular is singled out, denied access to the entire site. And that web crawler's name... is Discobot.
Discobot is a component of the Discovery Search Engine, a joint venture between Google, Stanford, God, the Illuminati, Big Tobacco, Big Cardboard, Big Boy Restaurants, and Zombie Elvis to create a next generation search engine that breathes methane and sees in color.
Why are my hard-earned tax dollars being used to fight disco? Does the IRS know something I don't about Google and its (potentially) shady business practices? Or is this a prelude to an all-out war on bubblegum pop?
And why isn't the media pressing the presidential candidates on this?