full-disclosure - Re: Google's robots.txt handling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <6FBCE5B3EB40334992BFA49270064D2957CB9639@swpma1exmail02.idco.intdata.com>
Date: Wed, 12 Dec 2012 16:34:37 +0000
From: "Lehman, Jim" <jim.lehman@...eractivedata.com>
To: "full-disclosure@...ts.grok.org.uk" <full-disclosure@...ts.grok.org.uk>
Subject: Re: Google's robots.txt handling

Yes I think you misunderstood or more likely I poorly worded the post. White listing is better than black listing. Black listing something you don't want googlebot to index just makes it easier for someone to find something you don't want indexed. If that content is sensitive, it probably should not be publicly accessible in the first place. But people never put sensitive content on web server (weak attempt at humor, my apologies).  I am beating the dead horse here, but robots.txt is not a security control. 
Most of the time robots.txt is great for recon sense and not Amy measure of defense. 

White listing just helps in not exposing too much information, a speed  bump if anything security related. I think this falls under the 'defense in depth' heading.   

-----Original Message-----
From: full-disclosure-bounces@...ts.grok.org.uk [mailto:full-disclosure-bounces@...ts.grok.org.uk] On Behalf Of Christoph Gruber
Sent: Wednesday, December 12, 2012 3:19 AM
To: full-disclosure@...ts.grok.org.uk
Subject: Re: [Full-disclosure] Google's robots.txt handling

On 12.12.2012 at 00:23 "Lehman, Jim" <jim.lehman@...eractivedata.com> wrote:

> It is possible to use white listing for robots.txt. Allow what you want google to index and deny everything else. That way google doesn't make you a goole dork target and someone browsing to your robots.txt file doesn't glean any sensitive files or folders. But this will not stop directory bruting to discover your publicly exposed sensitive data, that probably should not be exposed to the web in the first place. 

Maybe I misunderstood something, but do you really think that "sensitive" can be hidden in "secret" directories on publicly reachable web servers?
-- 
Christoph Gruber
By not reading this email you don't agree you're not in any way affiliated with any government, police, ANTI- Piracy Group, RIAA, MPAA, or any other related group, and that means that you CANNOT read this email.
By reading you are not agreeing to these terms and you are violating code 431.322.12 of the Internet Privacy Act signed by Bill Clinton in 1995.
(which doesn't exist)

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

*******************************************************
This message (including any files transmitted with it) may contain confidential and/or proprietary information, is the property of Interactive Data Corporation and/or its subsidiaries, and is directed only to the addressee(s). If you are not the designated recipient or have reason to believe you received this message in error, please delete this message from your system and notify the sender immediately. An unintended recipient's disclosure, copying, distribution, or use of this message or any attachments is prohibited and may be unlawful. 
*******************************************************

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/