Message-ID: <200308030308.h7338TC6021840@ns2.mmicman.com>
From: support at mmicman.com (Edward W. Ray)
Subject: Reacting to a server compromise

 Seems like the standard way to do it.  Also appears plagiarized from SANS
(http://www.sans.org) "Computer Security Incident Handling Step-by-Step,"
but since they are an open organization, I do not think they will mind :)

You also might want to check out Rob Lee's site at
http://www.incident-response.org for good articles and examples of incident
handling.

Regards,

Edward W. Ray
GCIA, GCIH

-----Original Message-----
From: full-disclosure-admin@...ts.netsys.com
[mailto:full-disclosure-admin@...ts.netsys.com] On Behalf Of SecuresDotComs
Sent: Saturday, August 02, 2003 11:08 AM
To: full-disclosure@...ts.netsys.com
Subject: Re: [Full-Disclosure] Reacting to a server compromise


I'll probably get flamed for this, but this is a doc I plagiarized from
multiple sources and slapped together to form the basis of our IRP. One of
the key points I like to make with it is the loss of data is secondary to
the loss of credibility and damage to the public perception of the company.
Comments appreciated, I may even get something out of a flame too.


Incident Response Procedures

Company XYZ Confidential

Overview

Computer security incidents are occurring at an ever-increasing rate on the
Internet. Since we, Company XYZ, depend on the Internet for our livelihood,
we must be prepared to respond to and investigate computer security
incidents. This document contains definitions of terms related to incident
response, goals for the Company XYZ Incident Response Team, and procedures
for handling reported security issues.

Purpose
This policy and procedures statement establishes the responsibilities of XYZ
staff in responding to and reporting computer security problems.

Coverage
This policy and procedures statement applies to all permanent and contract
employees within XYZ.

Definitions
XYZ computer system: A computer system that is owned or managed by XYZ. This
includes the full range of computer systems, from desktop and internal
fileservers to geographically distributed web farms, networking equipment,
etc.

Security problem: A breach, or attempted breach, in the security of a
computer system. This includes, but is not limited to, unusual or apparently
malicious break-in attempts (either local or over a network), virus or
network worm attacks, file or data tampering, unauthorized information
access or disclosure, network router or gateway attacks, or any incident
where a user, either directly or by using a program, performs functions for
which they do not have authorization.

Event: An observable occurrence; an aspect of an investigation that can be
documented, verified, and analyzed.

Incident: An adverse event or series of events that impact the security or
ability to do business of Company XYZ or its clients.

Incident Severity Levels
SEVERE - Incident could have long-term effects on business or incident
affects critical systems.
HIGH - Incursion on non-critical system; detection of precursor to a focused
attack; believed threat of an imminent attack.
MEDIUM - Threat of a future attack; detection of reconnaissance.
LOW - Unsubstantiated rumor of security incident


Evidence: Data on which to base proof or to establish truth or falsehood.

Forensic Analysis: Examination of material and/or data to determine its
essential features and their relationship in an effort to discover evidence
in a manner that is admissible in a court of law; post-mortem examination

Computer Incident Response Team: A group of technical investigators and
security engineers that respond to and investigate security incidents

Policy
A. All XYZ staff and contractors are responsible for helping to ensure the
security of the computer systems that they use and operate. Part of this
responsibility is the duty to report any confirmed or suspected security
problem in a timely manner.

B. It is the responsibility of the Security Officer and the Chief Operations
Officer to coordinate communications with entities outside of XYZ (such as
COAST, Carnegie Mellon CERT, or law enforcement agencies) about a suspected
or confirmed security problem on any XYZ system, and to coordinate
communications in response to external inquiries about such incidents.

Goals
The goals of the Incident Response Team are to:

Maintain or restore business continuity
Defend against further attacks
Deter attackers through investigation and prosecution Perform
counter-intelligence/intelligence activities where appropriate


Reporting: XYZ Staff or Contractors

If you suspect that there may be a security problem on a XYZ computer system
that you use or administer, you are responsible for doing the following:

Immediately contact one of the Information Systems Department Network
Administrators or IT Production Manager:
When sending an e-mail or leaving a message it is important to include the
following details:
Identify yourself and the project you are working on.
Note the time and date you discovered the suspected breach of security.
Provide in as much detail as possible your observations.
Leave a phone number where you may be reached.

In the event that an administrator is not immediately available, contact the
offices of the COO or VP of Engineering.  Do not send a message to
monitors@....com or any other public folder. Do not attempt to fix or
diagnose the system.

Procedures for Initial Reporting Documentation A security incident begins
when a security related event is reported to the Company XYZ Network
Administrators or member of the Executive Management Team. The following
series of procedures should start as the event is being first reported to an
Administrator or member of the Executive Management Team.

Document Event(s)
Event documentation is a critical aspect of incident response and
investigation. Without an accurate and verifiable trail of events, incident
investigation becomes difficult and often ends without conclusion. Likewise,
the response to the incident will not be complete if we cannot prove that
the response will stop the adverse event(s) from occurring in the future.

Document the event thoroughly. DO NOT RUSH. Write down the following
information:

Date of event
Time of event including time zone (as a relative offset from GMT, for
example GMT-8 for Pacific Standard Time) relative to XYZ Who or what
reported the event. If a person reported the event include their full name,
location, telephone number, and e-mail. If an automated system (SiteScope,
IDS, etc.) reported the event include the name of software package, name of
the host where the software is installed, physical location of the host,
host or CPU id of the host, network address of the host, and MAC address of
the host if possible.
A detailed description of the event. Include as much information as
possible.
Identification of the host(s) that the event is related to/occurred on.
Include the hardware manufacturer, operating system type and version, name
of the host, Company XYZ property number, physical location of the host,
host or CPU id of the host, network address of the host, and MAC address of
the host if possible. If the host has a modem connected to it, document the
telephone number of the connected line or the wall jack number.
If the event involves suspicious modifications or behavior of a computer
that is accessible to many people and a person is reporting the incident,
then ask the person for the names or descriptions of others in the area
prior to and just after the event. If there is limited physical access to
the computer, document the physical security controls that limit access.
If a person is reporting the event, then instruct the person not to discuss
the event with anyone other than a member of the Executive Management Team,
IT Production Manager or Incident Handler who is assigned to the incident,
unless the incident handler instructs otherwise.

Assign the Incident to a Handler
The IT Production Manager or member of the Executive Management Team will
immediately assign the incident to an on-call Incident Handler (IH) and
contact the IH immediately.

When an Incident Handler has been confirmed, contact the person who reported
the event and give them the name and contact information for the assigned
IH.
Assign Severity Level to Incident
The IH is responsible for determining the initial severity level. If it is a
new incident, or the new event related to an open incident is more severe
than previous events, assign a severity level to the incident. Use good
judgment, but err on the side of caution. If you think the severity is
between two levels, then assign the incident the higher severity level. The
following questions are intended to help classify the most serious risks,
but are only meant as specific examples of applying the severity levels to
security incidents:
Incident Severity Levels

SEVERE - Incident could have long-term effects on business or incident
affects critical systems.
 HIGH - Incursion on non-critical system; detection of precursor to a
focused attack; believed threat of an imminent attack.
MEDIUM - Threat of a future attack; detection of reconnaissance.
LOW - Unsubstantiated rumor of security incident


Is confidential data at risk?
If there is imminent danger (the act is in progress) that confidential
information can be read, modified, or destroyed by an unauthorized entity or
the disclosure or access already occurred, then assign the incident severity
level SEVERE.
Is business continuity at risk?

If there is imminent danger of disruption of business due to security issues
or malicious acts or the disruption is in progress, then assign the incident
severity level SEVERE.
Is public perception of the company at risk?

If there is imminent danger of modification of the public's perception of
the company due to security reasons other than disclosure of confidential
information or disruption of service (i.e. main web page has been modified
in an unauthorized manner, but orders can still be processed), then assign
the incident severity level SEVERE.

Someone identifies a security problem in Company XYZ systems in a public
forum

If the security problem could lead directly to the compromise of publicly
accessible Company XYZ hosts or customer information, then assign the
incident severity level HIGH. Otherwise, assign the incident severity level
MEDIUM if there is no imminent threat to Company XYZ systems or confidential
data.

FOR SEVERITY LEVEL SEVERE INCIDENTS, THE OWNER(S) OR OPERATOR(S) OF THE
AFFECTED HOSTS SHOULD BE DIRECTED NOT TO USE OR MODIFY THE SYSTEM IN ANY WAY
UNTIL THE INCIDENT HANDLER CONTACTS THEM AND INSTRUCTS THEM TO DO SO.

For severity HIGH incidents where unauthorized use has occurred, the same
rule should be applied.



Coordinate Incident Response Team
The Incident Handler assigned to the incident is responsible for
coordination of the response and investigation, and therefore will be the
Primary Incident Handler (PIH) for the remainder of the investigation. The
first task of the PIH is to review the incident documentation and associated
event reports. The PIH should verify as much information as possible from
the event report(s), verify the assigned severity level based on the
available information, and acquire the resources necessary to respond to the
incident.  The PIH should then go to the location of the incident if
appropriate.

Law enforcement may be informed at client's discretion. There are many
factors to weigh including the severity of the incident, the scope of the
compromise, cost to the company of supporting a criminal investigation, and
the proprietary and confidential company information that would become
public if a criminal investigation occurs. It will be up to the Computer
Incident Response Management Team (CIRMT) to decide whether the incident
warrants legal action. The PIH will be responsible for communicating the
ongoing status of the response and investigation to the CIRMT. It is
recommended that Company XYZ legal counsel be present in all meetings with
law enforcement relevant to ongoing investigations.


Contain and Eradicate
The primary goal of the Incident Response Team is to maintain/restore
business continuity, so containment of a security incident is vital.
Containment and eradication methods are highly dependent on the type and
scope of the security incident, therefore only sample scenarios and methods
are provided. Ideally, the appropriate method can be extrapolated from the
sample methods. The containment phase is also where the bulk of evidence
will be preserved. Always keep the following in mind:

Do not rush. Sacrifice time for thoroughness.
Preserve as much evidence in its original form as possible.
Take detailed notes on your actions and the actions of those around you,
including the time of the action. Include your reason for taking the action.
 Sign and date the bottom of each page of notes.
Record each piece of evidence you find, including a description, location,
time found, and other distinguishing attributes. If it is physical evidence,
record who handled the evidence before it came into your possession. If it
is electronic evidence, record any processing of the evidence that occurred
prior to your possession of the evidence. This data will help maintain an
evidentiary chain and record of possible modification.
If it is possible that a person who physically accessed the system caused
the incident, preserve the physical evidence. Wear white cotton gloves when
using the computer and handling physical evidence. Do not allow
non-investigative personnel to enter the crime scene. Record the names and
contact information of all people present when you entered the area. Make a
record of all physical security controls in the area.
Restrict information about the incident on a need to know basis. Only
management and technical personnel (Network Administrators, Programming
Manager etc.) that can significantly contribute to the resolution or
investigation of the incident should be informed. Only disclose information
that is immediately needed to solve the problem or task at hand.
If there are strong indications that the host has been compromised, then
disconnect the host from the network. If possible, do not physically
disconnect the host from the network; instead, assign the switch port the
host is connected to an unassigned VLAN that does not provide network
connectivity (this way link stays up on the hosts network interface, and a
backup host can connected to the assigned VLAN for evidence gathering).
There may be exceptions to this policy, but the exceptions should be few and
well thought out. The longer an attacker has access to or control of a host,
the higher the risk of long-term negative impact to the company.



Some forensic analysis of the running host needs to be done prior to
shutting the host down to perform a disk copy.

Make a list of processes running on the system (i.e. "ps -auexww" on Unix
and "pstat.exe" or "pview.exe" on NT).
Check the status of the network interface(s) to see if they are in
promiscuous mode ("ifconfig -a" and look for PROMISC flag on Unix and
"querynic" on NT).
Make a list of all listening ports and active network connections ("netstat
-an"). If possible, make a list of processes that own the ports and network
connections ("lsof" on Unix).
Make copies of the executable files associated with every running process on
the system or dump the contents of system memory to a file (??? How
practical is this-probably not very on systems with large amounts of RAM) It
is important to remember that even mundane commands on a host can destroy
valuable forensic evidence. Executing "ls" will change the access time of
the current directory for example. Perform as few operations as possible
that access or modify the file system prior to making a bit-for-bit copy of
the file system on the compromised host. The preferred process is:

Make two bit-for-bit copies of the compromised host's hard drive Remove the
original hard drive from the system and secure it as evidence Use one copy
to aid the creation of a new system disk (copy only data that is known to be
safe) then erase the disk using a secure wipe utility Use the second copy
for forensics Use only executables with verified integrity ("known good")
for these tasks to avoid Trojan Horses and modified binaries. Use an
Incident Response CD-ROM that contains the appropriate binaries and
documentation necessary for the system, but if not, use binaries from the
original installation media. Be aware that many utilities rely on shared
object libraries (libc-2.1.2.so etc.) that could be modified by an attacker,
so sometimes just running "known good" copies of utilities may not be enough
to protect from Trojan Horses. Clear any LD environment variables and then
use the LD_LIBRARY_PATH environment variable on Unix systems to specify the
use of integrity verified shared libraries ("man loader" for more
information) and put integrity verified DLLs in the same directory as the
executable used on Windows NT systems. The Incident Response CD-ROM should
be configured to use these mechanisms, but the configuration should be
manually verified before use.

In order to bring the host back online, the entrance point the attacker used
to compromise the system must be determined. If the entrance point is clear,
the scope of the breach minimal, and the integrity of the system provable,
then it may be safe to patch the entrance point the attacker used and
remove/restore any files they modified to bring the host back online. If the
host can mount a NFS or SMB volume that is big enough to hold an image of
the system hard drive, then make a binary copy (using "dd" for example) of
the system drive on the remote volume so forensic analysis can be performed.
Perform the copy prior to making the modifications necessary to bring the
system back online.

In cases where the integrity of the system is not provable and/or the
entrance point used is not clear, the system should be rebuilt from
vendor-supplied media and backups, all known security patched should be
applied, and unnecessary services should be disabled. Create a baseline for
file integrity checks (Tripwire or equivalent), so the system can be closely
monitored in case the entrance point has not been eradicated. Likewise, if
the system has the CPU and disk resources necessary, enable system auditing
for a period of time after the rebuilt system is deployed. Ideally, the
forensic analysis will bring to light the entrance point used, and at that
time the system can be positively secured against the attack and monitoring
can be de-escalated to the normal level.

If there are signs of compromise on a system in a cluster, then all other
member systems of the cluster should be immediately checked for similar
signs. If the compromise is isolated to a subset of the systems, then the
unaffected systems can be kept online, but they must be closely monitored
until the entrance point is determined and a fix can be applied to the
systems. If all the systems are compromised, then the standard containment
and eradication procedures need to be applied to all systems in the cluster.
If a system has a standby system to take its place, then the standby system
should either have the entrance point fixed or be instrumented for
aggressive monitoring before being brought online.



Forensic Analysis
Forensic analysis will vary greatly from incident to incident, but the
methodology should be consistent. The goal of forensic analysis is to
discover evidence that proves:

What happened
Where it happened
When it happened
Who did it
How they did it
In particular cases, forensic evidence will be used in criminal or civil
legal cases against the perpetrator. Since it may not be apparent at the
beginning of an incident investigation that the outcome will be a legal
case, we must treat every investigation as if it will lead to a court case.
Essentially, we must establish and maintain an evidentiary chain for all
electronic and physical evidence collected during the investigation. We must
also keep detailed logs of our actions and findings as investigators. Most
computer crime investigations lack an evidentiary chain and detailed
investigative logs, and that is a primary reason why it has been difficult
to gain convictions in criminal cases. Also, be aware that this information
will be available to the defense counsel through the information discovery
process and may become public. Do not include company confidential
information unless it is necessary.

To maintain an evidentiary chain the following information needs to be
recorded:

Where, when, and who discovered the evidence Who has handled or examined the
evidence and when Who has had custody of the evidence, during what time
period, and where it was stored/secured If the evidence has changed custody,
how and when the transfer occurred (include shipping numbers etc.) The
relevant person should sign and date each entry. If the investigation leads
to a court case, we must be able to prove that the evidence we discovered
has been securely handled and not been tampered with.



All digital data analysis should be performed on trusted systems that can
only be accessed by incident investigators.  Every precaution must be taken
to not contaminate or co-mingle digital data from separate investigations.

It may be months or years before a case is brought to trail. As an incident
investigator, you will be expected to recount your investigation in minute
detail. Keeping a detailed log throughout an investigation will allow you to
accurately recall all facets of the investigation. It is also important to
document the investigation to establish a credible basis for any action the
company may take as a result.

Document your hypothesis, how evidence supports or contradicts it, the
actions you take to discover evidence or test the hypothesis, important or
influential interactions with other people, relevant thoughts at the time,
and anything else that will allow you to recall the investigation
accurately. Include the time and date for each entry in your notes, and sign
every page.


Follow-up With External Organizations
In cases that involve attacks coming from the Internet, contacting Internet
Service Providers (ISP) and other companies may be necessary to trace the
source of the attack. Many ISPs and companies will only provide information
if a subpoena is issued, but we have found that you will have better luck
talking directly with their security staff managers than speaking with their
helpdesk or customer support. Play up the "one security person to another"
aspect-it will get you further.

If the source address of the attack is obviously spoofed, the ISP may or may
not keep enough information to determine where the attacker's traffic
entered their network. In cases that involve high volumes of traffic from a
spoofed source address, the chance that the ISP can determine where the
traffic entered their network is much greater.

In most cases, the procedure will require contacting a series of ISPs and
companies to determine the source (if possible at all). An important aspect
to remember is to not disclose proprietary information in the process of
talking to external companies, unless doing so is necessary and has
management approval.

Create an Executive Report and a Technical Report

After the incident has been contained, eradicated, and analyzed, create an
executive level report (1-2 pages) for all severity SEVERE-HIGH incidents
that contains:

A high-level description of the incident and its scope The impact on the
company Actions taken to prevent further occurrences Recommendations for
further action The technical report should contain detailed information
about the event, investigation, and conclusions.  All data used in the
report should reference evidence collected and be verifiable.  The report
should be suitable for use in court for either civil or criminal cases.

The reports will be given to the CIRMT and executive management for their
review and response.





SAMPLE

Company XYZ Incident Response Report

CONFIDENTIAL

Incident Number: CL-High-070901.1 CASE REMAINS OPEN

Date: July 9, 2001

Incident Handler: Bob Smith

Client: CLIENT XYZ



Initial Report:



XYZ received notification of a potential security hole that existed within
the CLIENT XYZ web site through an e-mail from the client forwarded by the
Associate Producer.





Subject:    VERY URGENT !!!! SECURITY HOLE IN YOUR SERVER!!!

Author:

Date:       7/8/01 8:30 AM



Dear Sir,



I'm sorry to inform you that your server at http://www.CLIENT.com/ has a
dangerous security hole that can let any hacker delete, manipulate your data
and even take control of your server.



the following is the content of your server :



 Directory of c:\



06/29/00  01:58p        <DIR>          4Demo

02/03/00  04:49p        <DIR>          Acrobat3

08/19/99  08:06p                     0 AUTOEXEC.BAT

08/19/99  07:51p                     0 BOOT.BAK

08/19/99  07:20p                   321 boot.ini



              38 File(s)    940,814,481 bytes

                          1,338,262,528 bytes free



accessible by the following URL:

http://www.CLIENT.com/scripts/..%255c..%255cwinnt/system32/cmd.exe?/c+dir+c:
\

As you can guess, a hacker can execute nearly whatever he wants, moreover he
can change the content of your web site and put any pictures or redirect the
hole url to another web site (hackers, sex....)!!!

the patch to this problem is to uninstall any IIS sample programs or to
unauthorized access to SCRIPTS Directory!!

I'm giving you the solution with no obligations of payement, but if you
insist here is my bank account information:

It's important to inform you that NT server with IIS web server has many
many security holes, and I'm at your service for further cooperation to fix
all of them with the latest technologies for control, test and auditing
techniques.

Moreover, if I can be of service in any technical web advanced
programming/design don't hesitate to contact me.

With warmest regards and hoping to hear from you favourably soon, I am

faithfully yours,

------------------------------------------------


Confirmation of the security hole was made and an incident tracking number
assigned.

Exploit:

Microsoft IIS Web Server Allows Remote Users to Execute Commands on the
Server Due to CGI Decoding Error

Impact:  Disclosure of system information, Disclosure of user information,
Execution of arbitrary code via network

Version(s): IIS 5.0, IIS 4.0 (except when on NT 4 with SP6/SP6a[without any
new hotfix])

Description:  NSFOCUS announced discovery of a vulnerability in Microsoft
Internet Information Server that allows remote users to execute commands on
the server.

NSFOCUS reports that when loading an executable CGI program, the IIS web
server will perform two successive decode operations. The first is to decode
the CGI filename and determine if it is an executable file. The second is to
determine CGI parameters. However, the web server will (improperly) decode
the file name on the second pass. As a result, a remote user can create a
malformed CGI filename to circumvent normal IIS filename security filtering
(such as ".." filtering) and traverse directories.

For example, NSFOCUS reports that the following URL, if the target host has
a virtual executable directory called scripts, will provide a directory
listing of the C:\ directory:

http://[targethost]/scripts/..%255c..%255cwinnt/system32/cmd.exe?/c+dir+c:\

Malformed URLs can be used to run commands with the privileges of the
IUSER_machinename account.

Impact:  A remote user can execute commands on and retrieve files from the
server.

Solution:  Microsoft has prepared a fix (which will be posted shortly).

Vendor URL:  www.microsoft.com/technet/security/bulletin/MS01-026.asp (Links
to External Site)

Cause:  State error

Underlying OS:  Windows (NT), Windows (2000)

Reparations: Available

Microsoft has posted a fix for this on its web site for machines with
Service Pack 5 and above. Application of this patch would completely repair
this hole. It is also possible to defeat this exploit by removing the
execute access from the scripts directory.

Risk Assessment: POOR

It is unknown whether it is possible to repair this hole on the CLIENT
servers due to limitations in both methods of reparations. The servers are
configured with Service Pack 4; the highest Service Pack the code is
compatible with. According to Tech A, attempts to use SP5 or 6a have been
met with failure. We cannot restrict the directory, as it requires
executable access to run the Servlet java engine.
The site will remain unprotected until confirmation of a workable solution
using SP5 and above can be made.

Store Incident File and Evidence

All evidence, logs, and data associated with the incident should be placed
in tamper resistant containers, grouped together, and put in limited access
secure storage. Only incident investigators, executive management, and legal
staff should have access to the storage facility. If and when evidence is
turned over to law enforcement, create an itemized inventory of all the
items, verify the inventory with the law enforcement representative, and
have the representative sign and date the inventory list for our records.
Give the original records to Legal, and save a copy for the security
records. It is recommended that Company XYZ legal counsel be present in all
meetings with law enforcement relevant to ongoing investigations.







_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.netsys.com/full-disclosure-charter.html

Powered by blists - more mailing lists