Thanks to all for the links, ideas
Quote from: BobRay at Mar 02, 2010, 10:04 PM
.....I’ve thought about bad spider and email-harvester blocking, but just never had the time.
I think an essential, proper bot swatting algo, is coding a way to block a few different kinds of unwanted bot traffic:
1. the fast, ’suck down pages as quickly as they can’ bots, (easy to id, kill)
2. the patient, slow, insidious, ’crawl every page’ type of bot (much harder to monitor / kill)
3. the really, outright creepy, surveillance bots. ( some gubment subsidized )
Then you have the ’ever changing IP’ bots who use Tor or Open Proxies to scrape your content who render IP tables all but useless. (which could be any of the above and are much harder if not impossible to id/kill)
Im thinking to ban by ’page consumption behavior’, rather than ban completely by IP, as part of a more intelligent algo to kill most things scraper-like on a site. Of course that would also include the unusual auto-banning via honeypots, manual ip tables, and full country ban capability (does anyone speaking Jingpho or Qabiao from an Asian IP really need to access an english only website? etc, etc...
The algorithm is definitely a complex W.I.P./ set of rules so far.
