full-disclosure - Re: Re: Re: Re: Links to Google's cacheof626FrSIRTexploits

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <dvuu0d$dgj$1@sea.gmane.org>
Date: Thu Mar 23 19:47:50 2006
From: davek_throwaway at hotmail.com (Dave Korn)
Subject: Re: Re: Re: Re: Links to Google's
	cacheof626FrSIRTexploits

Valdis.Kletnieks@...edu wrote:
> On Thu, 23 Mar 2006 15:15:00 GMT, Dave Korn said:
>
> > difference?  robots.txt is enforced (or ignored) by the client.  If a 
> > server
> > returns a 403 or doesn't, depending on what UserAgent you specified, 
> > then
> > how could making the client ignore robots.txt somehow magically make the
> > server not return a 403 when you try to fetch a page?
>
> It *can*, however, make the client *issue* a request it would otherwise 
> not have.
>
> If the client had snarfed a robots.txt, and it said "don't snarf anything
> under /dontlook/here/", and a link pointed there. it wouldn't follow the 
> link.
>
>If you tell it 'robots=off', then it *would* follow the link.

  Yes, these are all extremely obvious truisms, but I think now you need to 
go back and read the thread, because you haven't noticed that they're 
utterly irrelevant to the matter at hand.

> Remember - robots.txt *isn't* for the pages that would 403.

  See, thing is, "pages that would 403" is /exactly/ what we were talking 
about.  So saying "switch off robots.txt" is a completely irrelevant 
response.  And the fact that doing so _would_ have /an/ effect in /other/ 
circumstances doesn't make it any less irrelevant, at least not according to 
any definition of the word "relevant" that I've ever seen!

    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....