lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAH8yC8n5BrNXMdK8VT1LGjpx=TOWR8hw19mn0WCV-LGUDyDFHQ@mail.gmail.com> Date: Tue, 11 Dec 2012 17:57:31 -0500 From: Jeffrey Walton <noloader@...il.com> To: Christian Sciberras <uuf6429@...il.com> Cc: full-disclosure <full-disclosure@...ts.grok.org.uk> Subject: Re: Google's robot.txt handling On Tue, Dec 11, 2012 at 5:53 PM, Christian Sciberras <uuf6429@...il.com> wrote: > If you ask me, it's a stupid idea. :) > > I prefer to know where I am with a service; and (IMHO) I would prefer to > query (occasionally) Google for my CC instead of waiting for someone to > start taking funds off it. > Hiding it only provides a false sense of security - it will last until > someone finds the service leaking out CCs. Agreed. How about search engine data by other crawlers that was not sanitized? > This is especially the case with robots.txt. Can someone on the list please > define a "good web crawler"? Haha! Milk up the nose. > I think the problem here is that people are plain stupid and throw in direct > entries inside robots.txt, whereas they should be sending wildcard entries. > Couple that with actually protecting sensitive areas, and it's a pretty good > defence. We now know you don't need a robots.txt for exclusion. Just ask Weev. Jeff > On Tue, Dec 11, 2012 at 10:38 PM, Jeffrey Walton <noloader@...il.com> wrote: >> >> On Tue, Dec 11, 2012 at 4:11 PM, Mario Vilas <mvilas@...il.com> wrote: >> > I think we can all agree this is not a vulnerability. Still, I have yet >> > to >> > see an argument saying why what the OP is proposing is a bad idea. It >> > may be >> > a good idea to stop indexing robots.txt to mitigate the faults of lazy >> > or >> > incompetent admins (Google already does this for many specific search >> > queries) and there's not much point in indexing the robots.txt file for >> > legitimate uses anyway. >> I kind of agree here. The information is valuable for the >> reconnaissance phase of an attack, buts its not a vulnerability per >> se. But what is to stop the attacker from fetching it himself/herself >> since its at a known location for all sites? In this case, Google >> would be removing aggregated search results (which means the attacker >> would have to compile it himself/herself). >> >> Google removed other interesting searches, such as social security >> numbers and credit card numbers (or does not provide them to the >> general public). >> >> Jeff >> >> > On Tue, Dec 11, 2012 at 2:01 PM, Scott Ferguson >> > <scott.ferguson.it.consulting@...il.com> wrote: >> >> >> >> > If I understand the OP correctly, he is not stating that listing >> >> > something >> >> > in robots.txt would make it inaccessible, but rather that Google >> >> > indexes >> >> > the robots.txt files themselves, >> >> >> >> <snipped> >> >> >> >> Well, um, yeah - I got that. >> >> >> >> So you are what, proposing that moving an open door back a few >> >> centimetres solves the (non) problem? >> >> >> >> Take your proposal to it's logical extension and stop all search >> >> engines >> >> (especially the ones that don't respect robots.txt) from indexing >> >> robots.txt. Now what do you do about Nutch or even some perl script >> >> that >> >> anyone can whip up in 2 minutes? >> >> >> >> Security through obscurity is fine when couple with actual security - >> >> but relying on it alone is just daft. >> >> >> >> Expecting to world to change so bad habits have no consequence is >> >> dangerously naive. >> >> >> >> I suspect you're looking to hard at finding fault with Google - who are >> >> complying with the robots.txt. Read the spec. - it's about not >> >> following >> >> the listed directories, not about not listing the robots.txt. Next >> >> you'll want laws against bad weather and furniture with sharp corners. >> >> >> >> Don't put things you don't want seen to see in places that can be seen. >> >> >> >> > >> >> > >> >> > On Mon, Dec 10, 2012 at 8:19 PM, Scott Ferguson < >> >> > scott.ferguson.it.consulting () gmail com> wrote: >> >> > >> >> > >> >> > /From/: Hurgel Bumpf <l0rd_lunatic () yahoo com> >> >> > /Date/: Mon, 10 Dec 2012 19:25:39 +0000 (GMT) >> >> > >> >> > >> >> > ------------------------------------------------------------------------ >> >> > Hi list, >> >> > >> >> > >> >> > i tried to contact google, but as they didn't answer my email, i >> >> > do >> >> > >> >> > forward this to FD. >> >> > >> >> > This "security" feature is not cleary a google vulnerability, but >> >> > >> >> > exposes websites informations that are not really >> >> > >> >> > intended to be public. >> >> > >> >> > Conan the bavarian >> >> > >> >> > Your point eludes me - Google is indexing something which is publicly >> >> > available. eg.:- curl http://somesite.tld/robots.txt >> >> > So it seems the solution to the "question" your raise is, um, >> >> > nonsensical. >> >> > >> >> > If you don't want something exposed on your web server *don't publish >> >> > references to it*. >> >> > >> >> > The solution, which should be blindingly obvious, is don't create >> >> > the >> >> > problem in the first place. Password sensitive directories (htpasswd) >> >> > - >> >> > then they don't have to be excluded from search engines (because >> >> > listing >> >> > the inaccessible in robots.txt is redundant). You must of missed the >> >> > first day of web school. _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Powered by blists - more mailing lists