Attack of the Bots

I'm noticing a huge increase in traffic to my blog, and I don't think it's because I'm a scintillating writer. The logs we keep for blogs are pretty specific, so it's not hard to see what's going on.

The bots are taking over.

By 'bots' I mean a range of automatic systems including search engines, feed readers, and more nefarious systems like tools designed to post bogus comments on your blog or worse yet, find a security weakness to hack or crack the server.

Some bots are better behaved than others -- Google drops by every now and then and grabs a copy of the site for their search engine. MSN seems to be indexing the information constantly. I'd say the bulk of the search engine traffic, and a big part of the overall blog traffic, is from MSN.

Then there are the dozen or so Twitter addons which are constantly watching when people post links on Twitter -- guaranteed I'll get a few dozen hits on the server every time a post a link. And these seem to have spawned a new generation of blog search engines which are constantly grabbing the RSS feed and also indexing the HTML version of the page, giving a double hit to the server.

Of course, the structure of the site causes a little extra traffic simply because I present my blog in three categories, Professional, Personal, and Combined. If it gets posted in either Professional or Personal, it shows up automatically in Combined, giving the bots a little more to chew on.

More disturbing are the bots trying to hijack the site in one way or another. There are bots which try to post their own content to the site via the comment system. Over the past few days the traffic has really picked up on that one, and it looks like it's coming from personal computers with viruses -- no rhyme or reason, the postings come from all over the world and all times of day.

We've stooped the spamming with a simple CAPTCHA (which stands for 'Completely Automated Public Turing test to tell Computers and Humans Apart.'); that's why you have to enter that word when you post a comment. The traffic is still there, they just aren't getting their postings through.

Then there are the bots trying to find security weaknesses -- it's amusing to see Windows exploits being tested on our LINUX environment, but it's annoying to know that the barbarians are constantly at the gate trying to get in.

The real problem with this is tracking the effectiveness of marketing Conquent. The blog is one way people find the site, and knowing what kind of traffic we have coming to the blog helps us to understand who's visiting and how we might want to get some traction with those people. But we're constantly spending time filtering out things like 'internetserviceteam.com' from real people who appreciate the information and insights they get from this blog.

At the end of the day it probably doesn't matter. Real interactions with real human beings are what make a company like Conquent succeed, and if someone likes what they when they visit the site, the next step is usually an email or phone call.

And that's the best result to track.