This morning I received a curt email from my web-hosting provider that Export Law Blog was being crawled by a number of robots making so many file requests that they had to shut down my site temporarily and then add robot blocking code to my system files. Apparently the volume was sufficient that they were worried that the out-of-control robots would not just take down my site but all the other sites on the shared server.

So I asked my web host where the robots were coming from. Get this: one group of them was coming from the Department of State. Judging from some activity I saw on my logs, the State robots were trying to scrape the entire site with simultaneous requests. Bad robot! (Brownie points to anyone on whom that last reference is not lost.) The hosting administrators have now blocked all access from that IP address from the State Department.

As the people at the State Department are our friends, it seems harsh to impose this permanent time out on them just because one of their robots ran amok this morning. If someone from State knows how the robots got loose this morning and swarmed the blog, please let me know and I’ll try to get the IP address unbanned by the folks at my web host. (I could go in an change the commands they put in the .htaccess file, but if I do, and there’s another recurrence of an early morning unsanctioned State Department robot party at my site, I’m likely to rendered homeless and in search of a new web host.)

The other robot(s) came from JPMorgan. Any word from JPMorgan on what caused their robots to go into a feeding frenzy at my site this morning, would also be welcome.

We now return you to your regularly scheduled programming.

Permalink