Some misbehaved bot was swamping our wiki with requests:
alex@mirabel:~$ leech-detector < ~/websites/web_logfiles/communitywiki.org/access.log | head
208.109.254.194 22811 3K 71% 2.3s 503 (75%), 200 (22%), 404 (1%), 302 (0%), 301 (0%), 400 (0%), 501 (0%), 500 (0%)
64.208.172.178 901 11K 2% 49.1s 200 (72%), 503 (16%), 404 (7%), 302 (2%), 301 (0%), 400 (0%)
66.111.62.88 301 2K 0% 180.0s 200 (87%), 503 (9%), 302 (1%), 400 (1%)
217.20.118.146 299 0K 0% 183.1s 200 (100%)
71.224.253.50 277 1K 0% 11.7s 200 (64%), 304 (19%), 302 (14%), 400 (0%), 404 (0%)
72.14.199.67 256 0K 0% 215.5s 304 (59%), 503 (27%), 200 (12%)
194.0.179.48 153 1K 0% 347.6s 200 (92%), 503 (7%)
90.31.195.154 139 2K 0% 31.3s 304 (54%), 301 (25%), 200 (18%), 302 (0%), 404 (0%)
65.214.44.29 136 2K 0% 398.6s 200 (91%), 503 (8%)
213.186.60.163 121 3K 0% 447.0s 200 (88%), 503 (11%)
Needless to say, 208.109.254.194 is now banned. As you can see, that IP number caused nearly 23’000 requests (71% of all the requests) in the current access logfile, 75% of his requests returned a 503 error code, and still the bot kept on hammering our site.
Sorry it took me a while to understand what was going on. I hope we’re back to normal now.
I noticed $ whois 208.109.254.194 returns some ‘ABUSE’ related telephone contact numbers somebody (not me
might call.
I usually don’t report these incidents. First, because I don’t think that the kind of consequences I’ll see will deter any future such incidents (and what else could be our goal?), and second, because there’s always the chance that this is some computer science wanna-be student who failed at the creation of his first web crawler. Readers with “fetch ahead” browsers also fall into this second category.
By “fetch ahead” I mean people using browser extensions allowing them to “surf faster” because a background processes fetches all the pages linked to from their current page while they browse. Thus, if looking at RecentChanges, they would retrieve the author’s page, the page’s history, and the page’s last – for every page listed. Something like that.
Those are good points. And I agree about deterrence.
I thought I saw a FireFox extension for FetchAhead? but can’t find it now.
That kind of browsing is wasteful. I wonder if we could ever structure pricing to weight against real costs so lighter users pay less.
This should be true for the ISP connection the user must pay for, and also could be true of a website where a paying user had access to more content, but is otherwise limited to some number of pages or bytes per day.
I’m really disappointed with the semi-social web services available such as MySpace, GMail, Yahoo, MSN, etc… and also most wikis - that restrict the user in arbitrary ways beyond what should be the real concern of preserving Freedom for all users.
We should create a virtual world where users may pay just cost (such as part of the labor and purchase price of installing a new hard drive) to increase their ownership (such as 100 Gigs of encrypted storage) so that growth is not stymied by artificial limitations.