lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Tue, 11 Dec 2012 09:19:19 +1100
From: Scott Ferguson <>
Subject: Re: Google's robots.txt handling

> /From/: Hurgel Bumpf <l0rd_lunatic () yahoo com>
> /Date/: Mon, 10 Dec 2012 19:25:39 +0000 (GMT)
> ------------------------------------------------------------------------
> Hi list,
> i tried to contact google, but as they didn't answer my email,  i do forward this to FD.
> This "security" feature is not cleary a google vulnerability, but exposes websites informations that are not really 
> intended to be public.
> (Additionally i have to say that i advocate robots.txt files without sensitive content and working security mechanisms.)
> Here is an example: 
> An admin has a public webservice running with folders containing sensitive informations. Enter these folders in his 
> robots.txt and "protect" them from the indexing process of spiders. As he doesn't want the /admin/ gui to appear in the 
> search results he also puts his /admin in the robots text and finaly makes a backup to the folder /backup.
> <snipped>
> This shouldn't be a discussion about bad practice but the google feature itself. 
> Indexing a file which is used to prevent indexing.. isn't that just paradox and hypocrite?
> Thanks,
> Conan the bavarian

Your point eludes me - Google is indexing something which is publicly
available. eg.:- curl http://somesite.tld/robots.txt
So it seems the solution to the "question" your raise is, um, nonsensical.

If you don't want something exposed on your web server *don't publish
references to it*.

The solution, which should be blindingly obvious,  is don't create the
problem in the first place. Password sensitive directories (htpasswd) -
then they don't have to be excluded from search engines (because listing
the inaccessible in robots.txt is redundant).  You must of missed the
first day of web school.

Kind regards.

Download attachment "signature.asc" of type "application/pgp-signature" (263 bytes)

Full-Disclosure - We believe in it.
Hosted and sponsored by Secunia -

Powered by blists - more mailing lists