Arcsight

Security Operation Center

An information security operations center is a dedicated site where enterprise information systems (web sites, applications, databases, data centers and servers, networks, desktops and other endpoints) are monitored, assessed, and defended.

SOC monitors suspicious activity across the network by using logs through SIEM (Security information and event Management) tool.

SIEM Tools:

Arcsight :

ArcSight Enterprise Security Management

ESM collects, normalizes, aggregates, and filters millions of events from thousands of assets across your network into a manageable stream that is prioritized according to risk, vulnerabilities, and the criticality of the assets involved. These prioritized events can then be correlated, investigated, analyzed, and remediated using ESM tools, giving you situational awareness and real-time incident response time.

   Correlation—Many interesting activities are often represented by more than one event. Correlation is a process that discovers the relationships between events, infers the significance of those relationships, prioritizes them, then provides a framework for taking actions.

   Monitoring—Once events have been processed and correlated to pinpoint the most critical or potentially dangerous of them, ESM provides a variety of flexible monitoring tools that enable you to investigate and remediate potential threats before they can damage your network.

   Workflow—The workflow framework provides a customizable structure of escalation levels to ensure that events of interest are escalated to the right people in the right timeframe. This enables members of your team to do immediate investigations, make informed decisions, and take appropriate and timely action.

 Analysis—When events occur that require investigation, ESM provides an array of investigative tools that enable members of your team to drill down into an event to discover its details and connections, and to perform functions, such as NSlookup, Ping, PortInfo, Traceroute, WebSearch, and Whois.

  Reporting—Briefing others on the status of your network security is vital to all who have a stake in the health of your network, including IT and security managers, executive management, and regulatory auditors. ESM’s reporting and trending tools can be used to create versatile, multi-element reports that can focus on narrow topics or report general system status, either manually or automatically, on a regular schedule.

Individual SmartConnectors and/or a Connector Appliance gather and process event data from network devices and pass it to the Manager. The Manager processes and stores event data in the CORR-Engine. Users monitor events in ArcSight Web, and manage user groups and the CORR-Engine storage using the ArcSight Command Center, and develop content and perform advanced investigation on the ArcSight Console. A comprehensive series of optional products provide forensic-quality log management, network management and instant remediation, regulatory compliance, and advanced event analysis.

SmartConnectors:

SmartConnectors perform the following functions:

1. Collect all the data you need from a source device, so you do not have to go back to the device during an investigation or audit.
2. Save network bandwidth and storage space by filtering out data you know will not be needed for analysis.
3. Parse individual events and normalize them into a common schema (format) for use by ESM.
4. Aggregate events to reduce the quantity of events sent to the Manager.
5. Categorize events using a common, human-readable format. This saves you from having to be an expert in reading the output from a myriad of devices from multiple vendors, and makes it easier to use those event categories to build filters, rules, reports, and data monitors.
6. Pass events to the Manager after they have been processed.

SmartConnectors, hosted individually, or as part of an ArcSight Connector Appliance, are the interface to the objects on your network that generate correlation-relevant data on your network. After collecting event data from network nodes, they normalize the data in two ways: normalizing values (such as severity, priority, and time zone) into a common format, and normalizing the data structure into a common schema. SmartConnectors can then filter and aggregate events to reduce the volume of events sent to the Manager, which increases ESM’s efficiency and accuracy, and reduces event processing time.

ArcSight Connector Appliance:

ArcSight Connector Appliance is a hardware solution that hosts the SmartConnectors you need in a single device with a web-based user interface for centralized management.ArcSight Connector Appliance offers unified control of SmartConnectors on the appliance itself, remote ArcSight Connector Appliances, and software-based SmartConnector installed on remote hosts.

The ArcSight Connector Appliance:

1) Supports bulk operations across all SmartConnector and is ideal in ArcSight deployments with a large number of SmartConnectors

2) Provides a SmartConnector management facility in Logger-only environments

3) Provides a single interface through which to configure, monitor, tune, and update SmartConnectors

4) ArcSight Connector Appliance does not affect working SmartConnectors unless it is used to change their configuration.

Supported Data Sources:
ESM collects output from data sources with network nodes, such as intrusion detection and prevention systems, vulnerability assessment tools, firewalls, anti-virus and anti-spam tools, encryption tools, application audit logs, and physical security logs. The graphic below shows the Common network security data sources that ESM supports and ways you can analyze their output in ESM.

FlexConnector

FlexConnector framework is a software development kit (SDK) that enables you to create your own SmartConnector tailored to the nodes on your network and their specific event data.
FlexConnector types include file reader, regular expression file reader, time-based database reader, syslog, and Simple Network Management Protocol (SNMP) readers. For more information about FlexConnectors and how to use them, contact your ArcSight customer service representative.

ArcSight Manager
The ArcSight Manager is the heart of the solution. It is a Java-based server that drives analysis, workflow, and services. It also correlates output from a wide variety of security systems.The Manager writes events to the CORR-Engine as they stream into the system. It simultaneously processes them through the correlation engine, which evaluates each event with network model and vulnerability information to develop real-time threat summaries.

CORR-Engine Storage
The Correlation Optimized Retention and Retrieval (CORR) Engine is a proprietary data storage and retrieval framework that receives and processes events at high rates, and performs high-speed searches.

User Interfaces

The ArcSight Command Center
The ArcSight Command Center provides a streamlined interface for managing users, storage, and event data; monitoring events and running reports; and configuring storage, updating licenses, managing component authentication, and setting up storage notifications. With content management, you can establish peer relationships with other ESM installations, search, and synchronize ESM content across peers. Searches ranging from simple to complex are easy to configure and saved for regular use.

The ArcSight Console
The ArcSight Console is a workstation-based interface intended for use by your full-time security staff in a Security Operations Center or similar security-monitoring environment. It is the authoring tool for building filters, rules, reports, Pattern Discovery, dashboards and data monitors. It is also the interface for administering users and workflow.

ArcSight Web
ArcSight Web provides a secure web-based interface to the Manager. ArcSight Web provides event monitoring and drill-down capabilities in dashboards and active channels, reporting, case management, and notifications for Security Analysts. As a security feature, ArcSight Web does not allow authoring or administration functions.

ArcSight Risk Insight
ArcSight Risk Insight is an add-on product that enables users to understand the business impact of realtime threats on assets. In ESM, users define asset business layers (including workstations, servers, laptops), use rules to calculate risks factors on these assets, and import the data into Risk InSight. Risk InSight aggregates the scores following the business model, and users assess the impact of a specific threat that could present a risk factor on the business. Users build their own key performance indicators to monitor their organization’s business risks continuously. Once installed, access Risk InSight through the ArcSight Command Center.

Pattern Discovery
Pattern Discovery can automatically detect subtle, specialized, or long-term patterns that might otherwise go undiscovered in the flow of events. You can use Pattern Discovery to

Discover day-zero attacks—Because Pattern Discovery does not rely on encoded domain knowledge (such as predefined rules or filters), it can discover patterns that otherwise go unseen, or are unique to your environment.

Detect low-and-slow attacks—Pattern Discovery can process up to a million events in just a few seconds (excluding read-time from the disk). This makes Pattern Discovery effective to capture even low-and-slow attack patterns.

Profile common patterns on your network—New patterns discovered from current network traffic are like signatures for a particular subset of network traffic. By matching against a repository of historical patterns, you can detect attacks in progress.The patterns discovered in an event flow that either originate from or target a particular asset can be used to categorize those assets. For example, a pattern originating from machines that have a back door (unauthorized program that initiates a connection to the attacker) installed can all be visualized as a cluster. If you see the same pattern originating from a new asset, it is a strong indication that the new asset also has a back door installed.

Automatically create rules—The patterns discovered can be transformed into a complete rule set with a single mouse click. These rules are derived from data patterns unique to your environment, whereas predefined rules must be generic enough to work in many customer environments.

ArcSight Express 

ArcSight Express is a separately licensed Security Information and Event Management (SIEM) appliance that provides the essentials for network perimeter and security monitoring by leveraging the superior correlation capabilities of ESM in combination with the Correlation Optimized Retention and Retrieval (CORR) Engine. ArcSight Express delivers an easy-to-deploy, enterprise-level security monitoring and response system through a series of coordinated resources, such as dashboards, rules, and reports included as part of ArcSight Express content.

Logger
ArcSight Logger is an event data storage appliance that is optimized for extremely high event throughput. Logger stores security events onboard in compressed form, but can always retrieve unmodified events on demand for forensics-quality litigation data. Logger can be deployed stand-alone to receive events from syslog messages or log files, or to receive events in Common Event Format from SmartConnectors. Logger can forward selected events as syslog messages to ESM. Multiple Loggers work together to scale up to support high sustained input rates. Event queries are distributed across a peer network of Loggers.

ArcSight NCM/TRM
ArcSight Network Configuration Manager and Threat Response Manager (NCM/TRM) is an appliance that builds and maintains a detailed understanding of your network’s topology, enabling you to centrally manage your network infrastructure and respond instantly, even automatically, to incidents as they occur.

Network Configuration Manager (NCM)
Network Configuration Manager enables you to centrally manage all configuration-related tasks on a wide array of network devices. You can maintain several versions of configurations for each device, compare configurations before applying them to devices,

Threat Response Manager (TRM)
Threat Response Manager enables you to rapidly and safely respond to network security incidents. You can configure TRM to respond to these incidents automatically, or you can perform those actions manually. Some of the commonly performed actions are quarantine a node on the network, block traffic from a specific IP address range, or block a specific protocol from an IP range.

Life cycle of event through SIEM

This process is detailed in the following topics:

1.Data Collection and Event Processing

2.Priority Evaluation and Network Model Lookup

3.Correlation

4.Monitoring and Investigation‚

5.Workflow‚

6.Reporting and Incident Analysis‚

7.CORR-Engine‚(Storage and Archive).

1.Data Collection and Event Processing

A data source on a network node generates events, which are collected by an ArcSight SmartConnector. The connector normalizes the data into the ESM schema, then tags it with event categories and looks up zone and customer attributes from the ESM network model. You can also configure the SmartConnector to filter and aggregate events to reduce the volume of the event stream.
A) Collect Event Data
 

Event collection is the process of gathering information from network nodes on your network. Network nodes may be primary (such as a firewall or an IDS) or a concentrator (such as a syslog service, Symantec SESA, or SiteProtector) that gathers data from multiple similar primary network nodes. Events are then collected from these sources by ArcSight SmartConnectors. The data collected is log data generated by the different types of sources on your network. Each item of the log is translated into one event. How the data reaches the connector depends on the source that generates the logs.       

B) Normalize Event Data 

Normalize means to conform to an accepted standard or norm. Because networks are heterogeneous environments, each device has a different logging format and reporting mechanism. You may also have logs from remote sites where security policies and procedures may be different, with different types of network devices, security devices, operating systems and application logs. Because the formats are all different, it is difficult to extract information for querying without normalizing the events first.

ArcSight refers to an event that has been processed by a SmartConnector or other ESM component that has gone through this schema normalization as a normalized event. Events that have been processed by the SmartConnector and are ready to be sent to the Manager are also referred to as base events. With the data organized, you can pull all records containing a value that is of interest or sort by any field. Another factor in normalization is converting timestamps to a common format. Since the devices may all use different time zones, ESM normalization converts the timestamps to UTC (GMT).

C) Event Severity 

Device severity captures the language used by the data source to describe its interpretation of the danger posed by a particular event. For example, if a network IDS detects a DHCP packet that does not contain enough data to conform to the DHCP format, the device flags this as a high-priority exploit.

Agent Severity is the translation of the device severity into ESM-normalized values. For example, Snort uses a device severity scale of 1-10, whereas Checkpoint uses a scale of high, medium and low. ESM normalizes these values into a single agent severity scale.

The default ESM scale is Very Low, Low, Medium, High, and Very High. An event can also be classified as AgentSeverity Unknown if the data source did not provide a severity rating

D) Apply Event Categories 

Like the logs themselves, different security devices also include a model for describing the characteristics of the events they process. But as described above, no two devices or vendors use the same event-characteristic model. To solve this problem, ArcSight has also developed a common model for describing events, which enables you to understand the real significance of a particular event as reported from different devices. This common model also enables you to write device-independent content that can correlate events with normalized characteristics. This model is expressed as event categories, and the SmartConnector assigns them using default criteria, which can be configured during connector set-up.

Event categories are a series of six criteria that translate the core meaning of an event from the system that generated it into a common format. These six criteria, taken individually or together, are a central tool in ESM's analysis capability.

                                      

Event Categorization Utility

Events from unsupported or custom devices can generate events that the provided connectors do not know how to categorize. For example, if your organization has developed and deployed ArcSight FlexConnectors to collect and process events specific to customized network nodes, these "custom" events are not categorized by the usual method. From the ArcSight Console, you can manually apply categorization to one or more custom events from a FlexConnector (or other custom or unsupported device). Once you apply categorization to events from a particular device (and its associated connector), the categorization is automatically applied to other events of the same type. 

   

E) Look up Customer and Zone in Network Model

To help the Manager properly identify the endpoints involved in event traffic, the SmartConnector looks up two attributes of the network model: Customer and Zone.

Customers

Customer tagging is a feature developed mainly to support Managed Security Services Provider (MSSP) environments, although it can also be used by private organizations to denote cost centers, internal groups, or subdivisions. The Customer designation keeps event traffic from multiple cost centers and/or business units clearly identified and separate.

In the network model, if you have separate cost centers you need to differentiate, you can assign a Customer designation to the ESM Network those assets reside within. Only then can two Networks that have zones with overlapping IP address ranges be assigned to the same SmartConnector, because the Customer designation is used to differentiate between the overlapping address spaces, so the SmartConnector can look up the correct zone for each endpoint involved in an event.

The Customer attribute is only needed to clarify the zone look-up if the SmartConnector reports over the same address range but for different networks. The SmartConnector then uses the Customer designation to find which network contains the correct zone.

Zones

A zone represents part of the network, and is identified by a contiguous block of IP addresses. Zones usually represent a functional group within the network or a subnet, such as a wireless LAN, the engineering network, the VPN or the DMZ. Zones are also how ESM resolves private networks whose IP ranges may overlap with other existing IP ranges. Every asset or address range must have a zone associated with it. ESM comes configured with the standard global IP address ranges already grouped into zones, so if your network uses only these public IP addresses, ESM can resolve them without setting up any additional zones. However, if your network uses subnets or contains one or more private networks, you must set up zones so that ESM can resolve the IP addresses of the assets on your network.

The address ranges in zones in the same network cannot overlap. Any given IP address will be contained within the address range of at most one zone in that network. 

Filter and Aggregate Events

SmartConnectors can be configured with filter conditions and aggregation logic that focus and reduce the volume of events sent to the Manager.

Configure SmartConnectors to Filter Out Events

Filters for SmartConnectors are exclusive (filter out). Events that meet the connector filtering criteria are not forwarded to the Manager. During SmartConnector set up, you can configure the connector to use filter conditions that do not pass events to the Manager according to specific criteria. For example, you can use filters to exclude events with certain characteristics or events from specific network devices.

Configure SmartConnector to Aggregate Events

You can configure the SmartConnector to aggregate (summarize and merge) events that have the same values in a specified set of fields, either a specified number of times, OR within a specified time limit.
Connector aggregation merges events with matching values into a single aggregated event. The aggregated event contains only the values the events have in common plus the earliest start time and latest end time.This reduces the number of individual events the Manager has to evaluate.

For example, suppose the connector is configured to aggregate events with a certain source IP and port, destination IP and port, and device action if they occur 10 times in 30 seconds. If the connector receives 10 events with these matching values within that time, they are grouped into a single aggregated event with an aggregated event count of 10.

If the 30-second time frame expires and the connector has received only two matching events, the connector will create a single aggregated event with an aggregated event count of two. If 900 matching events come in during the 30 seconds, the connector would create 90 aggregated events, each with an aggregated event count of 10.

ESM refers to this process as "grouping by" those fields. Group by appears again in other ESM features, such as rules, data monitors, and reports. Aggregation starts when an event arrives with values in the group by fields that match the specified conditions. Aggregation continues until either a set time limit is reached or a set event count is reached.

2. Priority Evaluation and Network Model Lookup 

The SmartConnector sends the aggregated and filtered events to the Manager, where they are evaluated and tagged with network and actor modeling information, and priority levels, then stored in CORR-Engine storage.

Evaluate the Priority Formula 

The priority formula (sometimes referred to as the threat level formula) is a series of five criteria that each event is evaluated against to determine its relative importance, or priority, to your network. Priority evaluation is an automatic feature that is always "on," and is applied to all the events received by the Manager. The point of calculating an event's priority is to signal to security operations personnel whether this is an event that warrants further notice. The priority formula consists of four factors that combine to generate an overall priority rating. Each of the criteria described in the table below contributes a numeric value to the priority formula, which calculates the overall importance, or urgency, of an individual event. All values are between 0 and 10, where 0 is low and 10 is high. A high priority factor indicates an event with a higher risk factor. Not every high priority event is necessarily a threat, however. For example, if a critical e-mail server fails, the priority of the events reporting it may be very high, although it does not represent an attack on your network.

Priority factor 

1) Model Confidence

Model confidence refers to whether the target asset has been modeled in ESM and to what degree. Maximum score = 10.

2) Relevance

Relevance refers to whether or not an event is relevant to an asset based on whether the event contains ports and/or known vulnerabilities, and if so, whether those vulnerabilities and/or ports are exposed on the asset. If an asset does not expose the vulnerabilities or ports contained in the event, the event is not relevant to the asset. Maximum score = 10.

3) Severity

Severity is a history function. Has the system been attacked, has it been compromised, or has the attacker scanned or attacked the network in the past? Scores are assigned based on the attacker and target's presence in one of ESM's threat tracking active lists (/All Active Lists/ArcSight System/Threat Tracking), whose contents are updated automatically by ESM rules. Maximum score = 10.

4) Asset Criticality

Asset criticality measures how important the target asset is in the context of your enterprise as set by you in the network modeling process by using the


Write Event to CORR-Engine Storage 

At this point in the process, the event is written to the Correlation Optimized Retention and Retrieval (CORR) Engine with the addition of its priority level and complete network model data. If there is a problem with the CORREngine, the event flow will stop here. Possible problems could be that storage could be full, or the network connection between the Console and the Manager is down. If there is such a problem with the CORR-Engine, SmartConnector data stops flowing into the Manager and correlation activity stops. However, all event data is saved on the Manager until the CORREngine is back up, so no event data is lost. As a configuration safeguard, the cache on the Manager should be set with ample space to store event traffic should the CORR-Engine develop a problem.

3. Correlation Evaluation

Once events have been normalized, prioritized, and their endpoints identified within the network model, they are processed by the correlation engine, where the magic happens 

Events that have been tagged with event categories, priority evaluations, and network model information are then processed by the correlation engine, where filters, rules, and data monitors connect the dots, find the events of interest, and can initiate immediate response.

Most activities of interest are often represented by more than one event. Correlation is a process that discovers the relationships between events, infers the significance of those relationships, prioritizes them, then provides a framework for taking actions.

For example, various systems on a network may report the following events:

n  UNIX operating system: multiple failed log-ins

n  IDS: Attempted brute force attack

n  Windows operating systems: multiple failed log-ins A correlation rule puts these data points together and detects five or more failed log-ins in a one-minute period targeting the same source.

Based on these facts, this combination of events is considered an attempted brute force attack. The Windows operating system next reports a successful log-in from the same source. The attempted brute force attack followed by a successful login from the same source elevates the risk that the attack may have been successful. To verify whether an attack was successful, you can analyze the volume of traffic going to the Windows target. In this case, a sudden spike in traffic to this target can verify that a brute force attack was successful. ESM's correlation tools use statistical analysis, Boolean logic, and aggregation to find events with particular characteristics you specify. Rules can then take automated action to protect your network.

Filters

Filters are a set of conditions that focus on particular event attributes. This focus also reduces the number of events that are processed by the system. Filters are applied in many places in the event life cycle by SmartConnectors, the Manager, and the correlation engine. Filters are also used for monitoring, analysis, and reporting.

Filters applied at the SmartConnector select only events that match the conditions, and it is these events that are forwarded to the Manager for processing. Non-matching events are not forwarded to the Manager.

Filters applied at the Manager select which events it will process based on the conditions specified. Events that don't meet the conditions are not evaluated further, but they are preserved in the data store.

ESM filters come in two major forms:

1.Named conditions (Filters resource)

2.Unnamed conditions 

Named Conditions (Filters Resource)

A filter resource is a named object that other resources and SmartConnectors can reference. Filter resources are reusable, and you can transport them among Managers using Packages or the Archive utility. If you need to use the same condition in multiple places, create a filter resource, which you can then refer to in rules, reports, data monitors, and active channels.

Unnamed Conditions

Unnamed conditions reside within another resource, and are used to specify conditions that are applied locally by that resource only. You can specify unnamed conditions as part of an active channel, rule, or report. These conditions are saved as part of the resource in which they were created, and are not reusable by other resources.

Rules

A rule is a programmed procedure that evaluates incoming events for specific conditions and patterns, and when a match is found, can initiate actions in response. Rules are the centerpiece of the ESM Correlation Engine, and are what reveals specific meaning out of the steady event stream. Rules are similar to intrusion detection system (IDS) rules, except they operate on an event stream instead of a bit stream. They are constructed with aggregation and Boolean pattern matching to evaluate objects, such as event fields, network models, and active lists.

Rules must be activated in order to run on live data. When a rule is under development, you can test it on historical data on a local system before activating it on a live event stream. When activated, rules evaluate each event for the conditions specified.

There are three types of rules available in ESM: standard, lightweight, and pre-persistence.

Standard Rules

Standard rules are triggered when events match one or more set of conditions, for example, events that target a critical asset and are categorized as hostile. If the rule is configured to aggregate (consolidate) multiple events with matching attributes, then the rule is triggered by more than one matching event. For example, if the rule is configured to aggregate three matching events, the rule is triggered when those three matching events occur in the time limit specified.

Joins

Standard rules can have joins. A join means to connect events from different network nodes in order to understand attributes they may have in common. Join rules recognize patterns that involve more than one type of event.

Join rules are triggered by events that match two or more sets of conditions. For example, a join rule can be triggered if there is an event from your intrusion detection system and a corresponding permit event from the firewall, and both target the same asset on the same port from the same attacker. If the join rule is configured for aggregation, the rule is triggered if the specified number of matching events occur within the specified time frame.

Lightweight and Pre-persistence Rules

Lightweight and pre-persistence rules were designed for simplicity and performance. Each type has only one event condition (no joins), is triggered on every matching event, has aggregation fixed at 1, and doesn’t generate correlation events although rule failures are logged.

Lightweight rules can only act on active and session lists and are processed earlier in the flow than standard rules.

Event-enriching pre-persistence rules are best used for threat level formula analysis. These rules set values for incoming base events before the events themselves are persisted in the database. Pre-persistence rules are processed early in the workflow, however, the values they set are available to standard and lightweight rules, which run during the post-persistence event flow. Pre-persistence rules cannot be scheduled or replayed, since events occurring in the past have already been persisted and can no longer be modified.

Rule Aggregation

Standard rules can aggregate, or summarize and consolidate, events with matching (or not matching) values over a specified time frame.

Aggregation can be performed on the initial event stream at the SmartConnector, as described in “Filter and Aggregate Events” on page 35, and again at the Manager by rules.

Aggregation applied at the SmartConnector consolidates numerous repeat events (events with the same essential data, such as firewall events) to reduce the volume of events sent to the Manager without losing crucial event data. The SmartConnector generates a single event whose event type is aggregated event.

Aggregation applied by rules also groups together events with similar characteristics, but with the added benefit of being able to send a correlation event when matches occur, and trigger actions, such as sending a notification if the number of matches meets a certain threshold. For example, a user may only want to be notified if there are more than five login failures in one minute. Aggregation matches are counted and tracked in working memory, so rules with aggregation conditions can be memory-intensive, depending on what they evaluate.

How Rules are Evaluated

The rules engine first looks for matches to specified event conditions. For lightweight and pre-persistence rules, a match on every event immediately triggers the action which the rule is designed to execute. For standard rules, matches are held in working memory. The working memory passes these matches on to the tracker, where they are evaluated against other incoming events for aggregation and join conditions, if present. If the standard rule’s conditions, join conditions, and aggregation conditions are all met within the specified time thresholds, the rules engine will trigger a correlation event. Partial matches in expired thresholds are sent to the garbage collector.The rules engine evaluates the event stream, holds matches in working memory, and processes join and aggregation conditions in the tracker. If all conditions are met within the time thresholds, a correlation event is triggered.
Rule Actions and Thresholds
Rule thresholds tell the rule how many matching occurrences it should considerover what time frame before taking action. Depending on the event type, the situation, and the action you wish the rule to take, your rule can set the action into motion at one of the following thresholds:
How Rules Use Active Lists
Active lists are configurable tables that collect specified fields of event data to enablecrossreferencing during correlation. Active lists serve as a community bulletin board for tracking specific event data over long periods (days or weeks) so it can be available on demand for correlation.
The illustration below shows how one rule can find an asset that shows hostile activity and write that asset's address and zone to an active list. Another rule can then read from the active list and take additional action, such as aggregate further activity from that asset over 10 minutes.

One rule can find an asset that shows hostile activity and write that asset's ID and activity information to an active list. Another rule can then read from the active list and take additional action, such as aggregate additional activity from that asset.

There are two types of active lists:

n Event based -- Event--based active lists retain specific data from live events, and are populated automatically as the result of a rule action triggered by qualifying events. Event-based lists have an explicit event field tied to every field in the active list.

For example, You can configure a rule to look for three failed login attempts in one minute. When the rule is triggered, it generates a correlation event and an takes the action of populating an active list with the event data for the login attempts.

n Fields based -- Fields-based active lists contain data that is not part of the event data, and are thus populated by the user manually, or by importing a comma-separated value list exported from another application. It is used as a reference table lookup by rules.

For example, you could manually populate an active list with the user login names of all the employees in the IT department. Then you can write a rule that looks for special admin logins on critical assets to which only members of the IT department are authorized access. Check for login attempts by employees who are not on the this fields-based active list.

How Rules Use Session Lists

Similar to how active lists associate events happening in one area of the network with events happening in another area, session lists associate users with the event traffic they are involved with on the network.

Session lists capture and record session-related data in a list, where it can be used by the Correlation Engine to:

1.Resolve event endpoints against DHCP sessions to identify which device was located at the reported IP address at the time of the event

2.Utilize existing maps that link MAC addresses and/or host names to users, if available

3.Attribute actions originating from a specific device to its owner

4. Extract and resolve user information from VPN log-ins, including the VPN user name and session characteristics

5.Track who accesses a given network node at a given time to trace events that originate from this device to users that were logged in at the time

Session correlation is a three-step process that involves three or more ESM resources. 

 On the first event

n  On subsequent events

n  On every event (This is the only threshold available for lightweight and pre-persistence rules)  On first threshold

n  On subsequent thresholds

n  On every threshold

n  On time unit

n  On time window expiration

When the threshold is met, a rule can take action, such as notify other users, execute a script, add an event to an active list, or export the event to a third-party system. For lightweight rules, the action is limited to creating or updating an active or session list. For pre-persistence rules, the action is limited to setting an event field.

When a standard rule’s action is triggered by a threshold, the system generates an action event, which is a type of audit event that is used by ESM to keep track of system status and event processing statistics. All audit events, including action events, are sent back through the correlation engine, where they can be evaluated by other filters, rules, data monitors, and active lists that are looking for specific types of audit events. Audit events can be tracked in active channels, and can be useful to those who need to monitor, administrate, and report on ESM system health and behavior

Data Monitors
Data monitors are how the logic is defined for the graphical summaries that are displayed in dashboards. The datamonitors resource is located in the Dashboards area of the navigation tree (dashboards are part of the monitoring phase of the event life cycle, and are discussed in “Dashboards” on page 79). Some data monitors, however, also perform special analysis.
Data monitors are similar to rules, in that they evaluate the event stream and system health statistics, and consolidate (aggregate) events with common elements. Rules focus on inferring meaning from certain event conditions in order to specify actions, whereas data monitors focus primarily on summarizing event data graphically, and in the case of correlation data monitors, on providing a different type of analysis, such as calculating statistics and moving averages, and reconciling event streams.
There are three types of data monitors:

n  Event-based data monitors: used to create graphical or tabular summaries of event data for display in dashboards.

n  Correlation data monitors: used to evaluate the event stream and discover anomalies by calculating statistics, reconciling event streams, and calculating moving averages. Like rules, correlation data monitors generate correlation events when their conditions are met. Correlation data monitors are used in conjunction with rules, which can trigger actions when the correlation data monitor conditions are met.

n  Non-event based data monitors: used to monitor and display ESM system status in a graphical or tabular summary. 

Variable event fields are user-named extensions to ESM's event schema (for more about the event schema, see Chapter 11‚ The Event Schema‚ on page 105). Variables are virtual event fields whose values are the result of a special function performed on another field.

ESM enables you to create local variables that apply to the rule, filter, data monitor, query, active channel, or field set you are editing, and global variables that can be referenced by multiple resources. Variables boil down the complexity of the data in a way that enables calculations or functions to be performed. Local and global variables support the following types of functions:

confidential ESM 101 69 For example, you can write a rule that is triggered when an after-hours login occurs. To calculate the “after hours” time range, you can define a timestamp variable that extracts the hour of day out of a time stamp, then set the rule trigger on events that occur between x and y time. The time stamp value is made up of multiple data points: dd mmm yyyy hh:mm:ms UTC, as shown below:21 Jun 2013 17:28:02 PDT

Velocity Templates

Examples of Velocity Expressions to Retrieve Values

Velocity expressions usually begin with the $ sign followed by the field name in camel case: $<fieldNameInCamelCase> 

Event field values

To get the value from the event field such as Attacker Address, the expression would be $attackerAddress

Global variable values

The following examples show ways to use a velocity expression on variables, depending on the variable name. If it contains a dot, remove the dot and use camel case. If it contains a space, use an underscore: $<VariableName>$<variable_Name>

Rule actions

The following rule action example uses velocity expressions to retrieve values from an event field, Attacker Address, and a variable, dhcp.Hostname, and then send out a notification with the specified text in the subject:

“Brute force login attempt from IP Address: $attackerAddress Hostname: $dhcpHostname” 

Active Channels

There are three types of active channels that display different types of data:

n   Live Channels display continuously refreshed live event data

n   Rules Channels display replay events for testing rules

n   Resource Channels display the status of certain resources, such as the assets in your network model and open cases

Dashboards

Like the instrument panel of a car, dashboards display indicators that communicate the state of your enterprise as reported by SmartConnectors from data sources on your network. Dashboards are made up of individual data monitors and/or query viewers in a variety of graphical and tabular formats that summarize the event flow and communicate the effect of event traffic on specific systems on the network, or display the status of ESM components.

ESM provides many standard dashboards. You can also create your own. The data monitors and query viewers that make up dashboards.

Reports

A report binds one or more queries with a report template. As shown in the diagram on the next page, a query can collect data from trends, session lists, and active lists. In addition to reporting on event data, reports can also summarize data from Cases, Notifications, and Assets.

Queries

A query is a resource that defines the parameters of data you want to gather from a data source reporting events to ESM. The results of the query then become the basis for one or more report or trend. As a data source, queries can use the ESM CORR-Engine event storage, data stored in an active list, session list, or gathered from a trend. Queries can also summarize internal ESM data from assets, cases, and notifications.

In a query, you select the data fields you want to report on, specify any additional functions you want run on them (such as sum, average, and so on), and any sort or group-by conditions you want to add, such as grouping results by source address, zone, or prioritY

Trends

A trend is an ESM resource that defines how and over what time period data will be aggregated and evaluated for prevailing tendencies or currents. A trend executes a specified query on a defined schedule and time duration. A trend is one or more queries run on a schedule. A trend can be used as the primary data source for a report. Or a trend (based on one query) can be used as the data source for another query that further refines the result of the initial query. A collection of trend queries (queries that use trends as their data source) can provide focused views of a data set, which can then be fed into a single report or multiple reports. Trends gather event data over time, which helps identify, for example, the frequency of worm outbreaks, incident time-to-close, or number of cases closed. They can also be used to gather status and operational data about network objects, such as operating systems, asset activity by business role, or regulatory compliance status. ESM provides a set of standard trend reports that show trends on current data, such as trends by operating system, by role, by compliance requirement, time-to-close on cases, and number of cases closed. Depending on the data gathered by the base query, the trend will either be a snapshot trend or an interval trend.

Snapshot Trend

A snapshot trend uses a query that operates on a fixed moment in time, for example, to gather information about assets on your network. Snapshot trends are built from queries based on assets, cases, or notifications.

Snapshot trends answer questions about the status of objects on the network in fixed moments of time. You would use Snapshot trends to determine metrics such as current number of assets, number of systems with a particular operating system, or number of systems with particular vulnerabilities. For example, you would use a snapshot trend to evaluate statistics on vulnerabilities and incident metrics over time to determine whether your vulnerability posture or incident closing rate is getting better or worse.A snapshot trend operates on data in the current moment in time, and only collects data going forward. Thus, a snapshot trend cannot be used to determine how many assets were in a zone 6 weeks ago. You can use snapshot trends to collect data from this point forward, however, and in six weeks from now, you will have six week's worth of data that will tell you how many assets were in this zone at regular intervals over the last six weeks.

Interval Trend

An interval trend uses a query that operates on events that happen over a specified time window, for example, to gather information about how many events of a particular description occurred daily over a 6week period. The query upon which an interval trend is based can use other trends, queries, and lists as data sources.

interval trends answer questions about event characteristics over a specific time period. Because the Manager supports the late arrival of events, interval trends can be refreshed manually at any time.

How Trends Work

Creating a trend and using the data in a report is a three-step process: 

To develop a trend, first create a query that defines the data you are interested in. Next, define the trend time period and other parameters. Finally, you can use the resulting trend data in a report.

You can build a report directly from a single trend, or to get more flexible results out of the trend, you can create additional queries that refine the results of the first trend.

For example, say you wanted to report on daily VPN login statistics. You can create a base query that returns all VPN login attempts, then create a trend that runs this query once per day. To further refine the results between attempts, successes, and failures, you can build additional queries that use the output of your VPN login trend as its data source to differentiate between these three types of log-ins.

Delta Reports

A delta report compares the result sets from two different queries run from the same report definition using different parameters, such as today's date and yesterday's date. This is useful to compare results from one time period with another, or one business division with another.

The differences may be presented in a file or as a series of events. Each row represents one event that differed between the two queries. Delta reports have limited presentation features.

Focused Reports

Focused reports are a type of report that consist of a master report definition and parameters that focus on a subset of the data captured by the master report. This enables you to generate a separate report for each subdivision of data, such as individual zones, based on a single overall query without having to copy and modify the master report every time.

For example, if you need to report total event count for systems with different business roles, you can create a master report definition called Total Event Count per Business Role and add a parameter that points to the Business Role asset category group and an inGroup condition that points to it. Adding the parameter and the inGroup condition make the report "focusable."