Log crawling

Once in a while, I wander through the web server logs, to see who’s a bot and who’s not, and whether the bots are well-behaved. I’m not all that strict—I don’t even have a robots.txt file (yet)—but I don’t like bots that suck all the pages down extremely quickly or display other anti-social tendencies. One of the bots I have banned is from an outfit called NameProtect, since any mentioning of trademarks anybody on this site will be doing will be entirely within the bounds of fair use. I’ve left the bot from TurnItIn alone, since I don’t have any particular objection to plagiarists who are using our stuff getting caught (if my understanding of how TurnItIn works is flawed, please let me know). Every so often, though, I’ve noticed a bot that’s pretending not to be a bot. Frequently, these are spam address harvesters, but I’ve noticed occasionally that the IP range from one of the spoofers is owned by these assholes. Today, I finally looked in to who they are and whom they work for, and I’m sorry I didn’t ban them a long time ago.