[vox-tech] how to modify .htaccess to prevent wget or the likes from downing my site?

[Chanoch (Ken) Bloom]

On Wed, 2011-05-25 at 13:51 -0700, Tony Cratz wrote:
> On 05/25/2011 11:50 AM, Hai Yi wrote:
> > I first asked this question to the support of my web host, and they
> > redirected me to this link:
> > http://www.webhostingtalk.com/showthread.php?t=437549
> > 
> > and the snippet on that page looks like:
> > 
> > 
> > SetEnvIfNoCase User-Agent "^Wget" bad_bot
> > 
> > <Limit GET POST>
> >    Order Allow,Deny
> >    Allow from all
> >    Deny from env=bad_bot
> > </Limit>
> 
> 
> 	From reading this message and latter ones I have to ask, would
> 	it be better to do a deny from all and allow only the browser
> 	you want? At times it is easier to do things the reverse way
> 	of doing things as it can be must simplier to set up.

While allowing only user agents beginning with "Mozilla" would be an
attractive option, there are two big problems:

First, this wouldn't allow search engines to index his site. He could
allow specific search engines based on their user-agents, but then he'd
risk missing the next new big search engine if its indexing bot had a
brand new user-agent that wasn't on his white list.

He'd probably forget to allow text-based browsers like lynx, Links, and
w3m.

--Ken