Date: Tue, 6 Dec 2011 15:40:36 -0500
From: Charles Morris <cmorris@...odu.edu>
To: Paul Schmehl <pschmehl_lists@...rr.com>
Cc: full-disclosure@...ts.grok.org.uk
Subject: Re: one of my servers has been compromized

Sorry paul, Gage is right here!

Instead of "silly" maybe more like "correct" :(

On Tue, Dec 6, 2011 at 2:42 PM, Paul Schmehl <pschmehl_lists@...rr.com> wrote:
> Don't be silly.  You can run static binaries off a thumb drive without
> taking the system down.  And that includes md5sum.  You can put everything,
> including the script, on a thumb drive and be perfectly comfortable that
> the results are reliable, because statically compiled binaries don't use
> system libraries.  As a quick check to see if system binaries have been
> altered, it's hard to beat.
>
> And I wasn't responding to the OP.  I already did that the other day, when
> I told him he had a web app that got hacked and the scripts were run with
> perl, so noexec, nosuid for /tmp wouldn't have helped.
>
> I will agree with you that we don't typically see the old style hacks any
> more, but some hackers are still installing trojaned sshd binaries.
> Nowadays it's mostly app hacks and primarily (as it's always been) to
> Windows boxes.
>
> --On December 6, 2011 10:49:14 AM -0800 Gage Bystrom
> <themadichib0d@...il.com> wrote:
>
>>
>> But the problem with that is it is a mentality roughly a little more then
>> a decade old. What you described is a userland rootkit detector. Problem
>> is no one uses userland rootkits anymore! Sure there was some recent
>> development in managed code rootkits but it really hasn't home anywhere
>> and is Windows centric. Not to mention your plan is totally flawed. You
>> assume md5sum is safe to begin with. Meaning that to be remotely safe
>> with this you have to run the script for a livecd. Meaning you have to
>> bring down the server everytime you suspect you MAY have been
>> compromised. Completely unacceptable for anyone other than a home user.
>> The only way to circumvent such issues is to recreate tripwire, in which
>> you still have the same fundemental problems tripwire has always has.
>>
>> I know ya mean well, but your first block of advice isn't pratical or
>> effective. The second one the OP already did so alls well for that.
>>
>> :)
>> On Dec 6, 2011 10:19 AM, "Paul Schmehl" <pschmehl_lists@...rr.com> wrote:
>>
>> A "poor man's" root kit detector is to take md5sums of critical system
>> binaries (you'd have to redo these after patching), and keep the list on
>> an inaccessible media (such as a thumb drive).  If you think the system
>> is compromised, run md5sum against those files, and you will quickly
>> know. You could even keep statically compiled copies on the thumb drive
>> to use in an investigation.
>>
>> Start with things you use to check for problems; ls, ps, fstat, sockstat,
>> netstat, wtmp, nc, sshd, etc.
>>
>> It would be fairly trivial to create a simple shell script that would
>> compare the md5sums of system binaries to the saved copies and flag
>> anomalies.
>>
>> And, of course, if you can take a system offline, there are a number of
>> bootable security distros that allow you to do extensive analysis of
>> systems.
>>
>> <http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-t
>> est-forensics-recovery/>
>>
>> In general, on Unix systems, look for oddly named directories in odd
>> places (like /tmp, /dev, etc. and review logs that have been syslogged
>> elsewhere for telltale signs of compromise.
>>
>> It's surprising how few times the shell history logs get wiped, but there
>> are some kits out there that do that for you.  Web apps and improper
>> permissions (world writeable) are the two most frequent causes of
>> compromises that I've seen on Unix systems.
>>
>> --On December 5, 2011 1:53:21 PM +0000 Dan Ballance
>> <tzewang.dorje@...il.com> wrote:
>>
>>
>> Thanks for the heads-up on rkhunter Gage.
>>
>>
>> Is there anything else out there atm that works as a reasonable root kit
>> detector or is such a thing considered impossible now? I realise a
>> skilled attack will be able to bury itself without a trace, but I'm
>> thinking of something that can be used in less skilled breaches such as
>> the one thought to have been identified in this thread. Sometimes
>> something imperfect is still better than nothing I think.
>>
>>
>> Also, am I correct to think that using something like tripwire is the
>> best way to detect root kits properly, but that it obviously needs
>> installing when the box is fresh and before it has been physically
>> connected to a network?
>>
>>
>> thanks to everyone for their valuable contributions here - much
>> appreciated!
>>
>>
>> dan :)
>>
>>
>>
>> On 5 December 2011 11:13, Gage Bystrom <themadichib0d@...il.com> wrote:
>>
>>
>> If it was a rootkit then trying to run the outdated rkhunter would be a
>> moot point. Whatever seizes the kernel first wins, hands down.
>>
>> Fortunately for him, since the bot was so easy to find in the first place
>> and such a simple way of maintaining it, the box was clearly seized by
>> someone who didn't give a rats ass about it. Probably a skiddie or an
>> automated attack to begin with.
>>
>> As for plugging any security holes, check your httpd error logs. If you
>> noted down the time of the bot files creation date you would look around
>> the same time for suspicious log entries. If they were as careless in
>> scrubbing the logs as they were holding the box it would give you a look
>> into how it may have been compromised. If you're getting things like
>> ../.../../../../etc/passwd then some sort of lfi vuln was likely
>> exploited, start grepping your php files for stuff like include(), or if
>> you're getting something like into outfile then check your mysql user
>> permissions and don't let it have file perms, and then start grepping
>> down for sql vulns.
>>
>> If it comes down to being too much of a hassle to get all the obvious
>> vulns at least then go to your boss, admit there is an issue and that
>> time needs to be taken to remove such legacy code as this could have been
>> a far worse incident if it had been more targetted and the end goal
>> wasn't a botnet.
>>
>>
>> On Dec 5, 2011 3:02 AM, "Dan Ballance" <tzewang.dorje@...il.com> wrote:
>>
>> I'm no expert, but here's something to get you started while you wait
>> for more experienced replies. Check for root kits:
>>
>>
>> sudo apt-get install rkhunter
>> sudo rkhunter --update
>> sudo rkhunter --check
>>
>> On 5 December 2011 10:44, Lucio Crusca <lucio@...web.org> wrote:
>>
>> Hello *,
>>
>> I'm not new here, but I've mostly lurked all the time through gmane. I
>> never
>> believed it could happen to me until it actually happened: they
>> compromized
>> one of my servers. It's a Ubuntu 10.04 server with all security patches
>> regularly applied. I'm inclined to believe they used some hole in the web
>> application, which is a old customized Virtuemart version (1.1.3), which
>> is
>> not upgradable because of the invasive code customizations (I'm not the
>> author of that code, so I have no clue about what had been changed back
>> then).
>>
>> Now the problem for me is to track down the security hole. Here is the
>> email
>> my provider received and forwarded to me:
>>
>>
>> Subject: ISP Report; botnet activity on irc.undernet.org
>> [...]
>>
>> Hello, I am an operator on the irc chat network,
>> irc.undernet.org and i would like you to investigate the
>> owner of the Ip addresses that are listed at the foot of this
>> email.
>>
>> This/These host(s) have likely been compromised, and had an
>> altered/rogue process installed on it, and was part of a botnet
>> that was found on our network.
>>
>> The exploit or compromise running on this system is likely
>> to be an irc bot. Can you please alert the person who is
>> responsible, for its security to patch/upgrade, remove the
>> irc process and secure their system.
>>
>> = Unix System owners =
>> A favourite place for hiding the bot(s) is in tmp
>> and in /var/tmp/ or /dev/shm/ or in a users home directory
>> sometimes it may be hidden like /tmp/".  ."/ or similar.
>>
>> The bot files can usually be found by running these one line
>> commands as the root user.
>>
>> find / -exec grep -l "undernet" {} +
>> find / -exec grep -l "sybnc" {} +
>> find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>> find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>>
>> netstat -tanp
>> lsof -i tcp:<Port number>
>>
>> *netstat looking for connections to remote port 6667 or the
>> range of ports between 6660-7000 once you find the port you
>> can use the command, lsof -i tcp:portnumber to determine
>> which process/user it is running under, and terminate it.
>>
>> = Windows System Owners =
>> most windows bots are mIRC scripted bots and generally
>> need a file called mirc.ini to run, you should search for
>> this file. Run a good antivirus scanner and firewall.
>>
>> This Ip/host may be removed from our Irc network due to the
>> risks it presents to our users.
>>
>> Should you need any help with removing the files or bot
>> process, feel free to contact me by mail or on our network,
>> which you connect to using any irc client and issuing
>> /server irc.undernet.org
>>
>> I look forward to your reply
>> Scot
>>
>> * Affected host/IPs, capture time is GMT+1: United kingdom
>> and servers they were connected to.
>>
>> Please note: when resolving server names to IP Addresses
>> that all our servers end with .undernet.org (for example)
>> Tampa.FL.US. is actually  Tampa.FL.US.undernet.org
>>
>> Important: If you reply to this mail needing further
>> information, please leave this mail intact, or supply us
>> with the IP Address(es) in question, as we reference these
>> mails by the unique IP Address
>>
>> Time of Capture: DECEMBER 3, 2011 10:03:48 PM
>>
>> List of IP address(es) and server it connected to:
>> my.server.ip.address (CHICAGO.IL.US
>>
>> BUDAPEST.HU.EU
>>
>> MONTREAL.QC.CA.undernet.org)
>>
>>
>>
>> I've run the "find" commands and found a number of file with the first
>> "find", under /tmp/.m
>>
>> Deleted them, looked up remote connections with netstat, killed perl
>> processes that where trying to connect to port 6959 (only trying because
>> I've now set up iptables so that they actually can't), but those processes
>> kept spawning. Checked crontab of www-data, found the launcher, removed
>> it.
>>
>> Now the problem is: how do I pervent further abuse? What should I search
>> in
>> the logs (if anything) to spot the security hole?
>>
>> TIA
>> Lucio.
>>
>>
>>
>>
>>
>> _______________________________________________
>> Full-Disclosure - We believe in it.
>> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
>> Hosted and sponsored by Secunia - http://secunia.com/
>>
>>
>>
>> _______________________________________________
>> Full-Disclosure - We believe in it.
>> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
>> Hosted and sponsored by Secunia - http://secunia.com/
>>
>>
>> _______________________________________________
>> Full-Disclosure - We believe in it.
>> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
>> Hosted and sponsored by Secunia - http://secunia.com/
>
>
>
> --
> Paul Schmehl, Senior Infosec Analyst
> As if it wasn't already obvious, my opinions
> are my own and not those of my employer.
> *******************************************
> "It is as useless to argue with those who have
> renounced the use of reason as to administer
> medication to the dead." Thomas Jefferson
> "There are some ideas so wrong that only a very
> intelligent person could believe in them." George Orwell
>
> _______________________________________________
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Powered by blists - more mailing lists