FrontPage SiteMap RecentChanges RecentNearChanges HowTo Blog RSS
Monaco, National Day

2007-04-10

Some misbehaved bot was swamping our wiki with requests:

alex@mirabel:~$ leech-detector < ~/websites/web_logfiles/communitywiki.org/access.log | head
     208.109.254.194      22811     3K  71%    2.3s  503 (75%), 200 (22%), 404 (1%), 302 (0%), 301 (0%), 400 (0%), 501 (0%), 500 (0%)
      64.208.172.178        901    11K   2%   49.1s  200 (72%), 503 (16%), 404 (7%), 302 (2%), 301 (0%), 400 (0%)
        66.111.62.88        301     2K   0%  180.0s  200 (87%), 503 (9%), 302 (1%), 400 (1%)
      217.20.118.146        299     0K   0%  183.1s  200 (100%)
       71.224.253.50        277     1K   0%   11.7s  200 (64%), 304 (19%), 302 (14%), 400 (0%), 404 (0%)
        72.14.199.67        256     0K   0%  215.5s  304 (59%), 503 (27%), 200 (12%)
        194.0.179.48        153     1K   0%  347.6s  200 (92%), 503 (7%)
       90.31.195.154        139     2K   0%   31.3s  304 (54%), 301 (25%), 200 (18%), 302 (0%), 404 (0%)
        65.214.44.29        136     2K   0%  398.6s  200 (91%), 503 (8%)
      213.186.60.163        121     3K   0%  447.0s  200 (88%), 503 (11%)

Needless to say, 208.109.254.194 is now banned. As you can see, that IP number caused nearly 23’000 requests (71% of all the requests) in the current access logfile, 75% of his requests returned a 503 error code, and still the bot kept on hammering our site.

Sorry it took me a while to understand what was going on. I hope we’re back to normal now.

I noticed $ whois 208.109.254.194 returns some ‘ABUSE’ related telephone contact numbers somebody (not me;) might call.

I usually don’t report these incidents. First, because I don’t think that the kind of consequences I’ll see will deter any future such incidents (and what else could be our goal?), and second, because there’s always the chance that this is some computer science wanna-be student who failed at the creation of his first web crawler. Readers with “fetch ahead” browsers also fall into this second category.

By “fetch ahead” I mean people using browser extensions allowing them to “surf faster” because a background processes fetches all the pages linked to from their current page while they browse. Thus, if looking at RecentChanges, they would retrieve the author’s page, the page’s history, and the page’s last – for every page listed. Something like that.

Those are good points. And I agree about deterrence.

I thought I saw a FireFox extension for FetchAhead? but can’t find it now.

That kind of browsing is wasteful. I wonder if we could ever structure pricing to weight against real costs so lighter users pay less.

This should be true for the ISP connection the user must pay for, and also could be true of a website where a paying user had access to more content, but is otherwise limited to some number of pages or bytes per day.

I’m really disappointed with the semi-social web services available such as MySpace, GMail, Yahoo, MSN, etc… and also most wikis - that restrict the user in arbitrary ways beyond what should be the real concern of preserving Freedom for all users.

We should create a virtual world where users may pay just cost (such as part of the labor and purchase price of installing a new hard drive) to increase their ownership (such as 100 Gigs of encrypted storage) so that growth is not stymied by artificial limitations.

EditNearLinks: PatrickAnderson

Languages: