I don't think this is a particularly big deal, but I am fascinated by crawler behavior and the wheres and whys of crawlers not honoring sites' specific robots directives. And it makes it even more interesting when the robot and the site belong to the same company. A few weeks ago, I was trying to find out exactly ...