Open topic with navigation

Forwarders

Forwarders send all events, or events that match a particular filter, on to a particular host or destination such as ArcSight Manager.

The ability to define a different filter for each forwarder allows Logger to divide traffic among several destinations. For example, because Logger can handle much higher event rates than ArcSight Manager, Logger might be used to forward events to a number of ArcSight Managers. Forwarder filters make it possible to split the flow between the Managers, using one forwarder for each Manager. Additionally, forwarding enables you to send a subset of events to other destinations for further processing while maintaining all events on Logger for long-term storage.

The forwarding filter is a query that searches for matching events, optionally within a time range. You can create two types of forwarder filters—continuous and time-range bound.

• A continuous filter constantly evaluates the incoming events and forwards the matching ones to the specified destination.
• A time-range bound filter uses a time range in addition to the specified condition to determine whether an event should be forwarded to the destination. If the event falls within the specified time range and matches the specified condition, it is forwarded; otherwise, it is not. The Logger receipt time of an event is used to determine whether an event will be forwarded to a destination when a forwarder filter specifies a time range by which events are evaluated for forwarding. Once a forwarder has forwarded all events within a time range, it does not forward any more events.

A forwarder’s operation can be paused and resumed at any point in time. When a forwarder resumes operation, forwarding resumes from the last checkpoint that was established before the forwarding operation was paused.

You can also disable and re-enable a forwarder. When you re-enable a forwarder, all previously established checkpoints are removed and forwarding starts over again as per the forwarder configuration-forwarders with continuous filters start from the current time, while forwarders with time-range bound filters start from beginning of the configured time range.

Forwarder types include:

• UDP Forwarder: forwards events by using the User Datagram Protocol.
• TCP Forwarder: forwards events by using the Transmission Control Protocol.
• Connector Forwarder: sends events to the Logger Streaming Connector.
• TH Forwarder: sends Common Event Format (CEF) events to a TH destination.

• ArcSight ESM CEF Forwarders: send Common Event Format (CEF) events to an ESM Destination. The built-in connector on Logger is used to forward these events to ESM.

  Note: In order to create an ArcSight ESM forwarder, you must first create an ESM Destination. See ESM Destinations for more information.

As a best practice, do not add more than ten regular expression forwarders. Even though each additional forwarder improves the forwarding rate, the relation is not proportional. In high EPS (events per second) situations or situations where other resource-intensive features are running in parallel (alerts, reports, and several search operations) and the forwarding filter is complex, adding too many forwarders may reduce performance because forwarders have to compete for the same Logger resources besides competing for the same built-in connector for forwarding.

You can specify a regular expression or an indexed search query (Unified Query) for the filter. Doing so enables you to take advantage of the indexing technology to quickly and efficiently search for events to forward.

To create a forwarder:

1. Open the Configuration > Data menu and click Forwarders.

2. Click Add to display the following form.

3. Enter a name for the new forwarder. Provide a name that is unique and not likely to be duplicated elsewhere. For example, if you create an Alert called "MyTest" and a forwarder called "MyTest," you will get an error message asking for a unique name.
4. Choose the forwarder type appropriate for your needs: UDP Forwarder, TCP Forwarder, Connector Forwarder, TH (CEF) Forwarder, or ArcSight ESM (CEF) Forwarder type.
5. Select the type of forwarding filter you want this forwarder to use—Unified or Regular Expression. Select “Unified” if you want to specify an indexed search query or “Regular Expression” to specify a regular expression query.
6. Click Next.

7. Enter additional type-specific information as described in the following table.

   Forwarder Parameters

   Parameter	Forwarder Types	Description
   Name	All	The name that you entered in the previous screen is displayed automatically. If you want to change the name, make the change on this screen.
   Query	All	Enter the query that will be used to filter events that the forwarder will forward, or select a filter from the Filters list. Forwarder queries can be constrained by device groups and storage groups, but not by Peers. If you selected Unified Query in the previous screen, enter an indexed search query that includes full-text and field-based indexed fields. You can click the Advanced Search link to access the Search Builder tool to build an indexed query. (See Classic Search: Using the Advanced Search Builder for more information.) Tip: The unified query you specify must follow the following guidelines, or you will not be able to save the query or the forwarder. Queries in the following format are valid; no other formats are allowed. (full-text terms | field search)* | regex That is, the query must only contain full-text (keyword) and field-based query elements; it cannot contain any aggregation search operators, or operators that process the searched data further to refine the search. For example, chart, sort, eval, top, and so on. Therefore, this is a valid query: failed message CONTAINS “failed device” However, this is an invalid query: failed message CONTAINS “failed device” | sort deviceEventCategory The query can contain the regex operator after a pipeline character (|). Therefore, this is a valid query for a forwarder: failed message CONTAINS “failed device” | regex deviceEventCategory = “fan” Tip: All search terms (except the “regex” portion) in a query must be indexed. If a query contains full-text (keyword) terms, full-text indexing must be enabled. Similarly, if the query contains a field, field-based indexing must be enabled and the specified field must be indexed. If you selected Regular Expression in the previous screen, specify a regular expression in this text box. See Searching for Events.
   Filters	All	Instead of specifying a unified query, you can select a filter from the Filters list. The Filters list contains all saved filters and predefined system filters on your Logger. Select a filter that meets the validity guidelines described in Query. Otherwise, the user interface will display an error when you save the forwarder definition. You can only select one unified query filter per forwarder. However, You can select multiple filters for a regular expression-based forwarder. Similarly, when creating a regular expression-based filter, select a filter from this list.
   Filter by time range	All	If you are creating a continuous filter, which continuously evaluates incoming events and forwards the matching ones, skip this parameter. In this case, the query is run continuously and forwarding continues until you pause it. If you are creating a time range bound filter, check this box to specify a time range of events that the forwarder will forward. If you enter a time range, the forwarder sends events that are within that time range and stops. When you check this box, the Start and End dates and Time fields are displayed. Start must be earlier than End. Specifying a time in the future changes that field to the current time. For example, specifying a Start of the current day at 7 AM and an End of current day at 7 PM will produce events with timestamps from 7 AM to the time the filter is saved (that is, earlier than 7 PM).
   Source Type	Connector	Select from the pull-down list of log file types, including: Apache HTTP Server Access Apache HTTP Server Error IBM DB2 Audit Juniper Steel-Belted Radius Microsoft DHCP Log Others... Note: The Source type must be the same in receiver, forwarder, and SmartConnector. See Forwarding Log File Events to ESM. A receiver can only receive events of a single source type. Set up separate receivers for each type of log file.
   Preserve Syslog Timestamp	UDP, TCP	Set to true to preserve the syslog timestamp. The default is true. In this case, the timestamp is the original receipt time of the event. If set to false, original timestamp is replaced with Logger’s receipt time.
   Preserve Original Syslog Sender	UDP, TCP	Set to true to send the event as-is, without inserting Logger’s IP address in the hostname (or equivalent) field of the syslog event. The default is true. If set to false, Logger’s information is inserted in the hostname (or equivalent) field of the syslog event.
   IP/Host	UDP, TCP, Connector	The IP address or host name of the destination that will the receive forwarded events. Note: You cannot configure a Logger forwarder to send data to the same system on which it is configured.
   Port	UDP, TCP, Connector	The port on the destination that will receive the forwarded events. The default port is 514.
   Connection Retry Timeout	TCP, Connector, ESM	The time, in seconds, to wait before retrying a connection. The default is 5 seconds.
   ESM Destination	ESM	An existing ESM Destination that will receive the forwarded events. (For more information, see ESM Destinations.)
   Transformation Hub Destination	Transformation Hub	An existing Transformation Hub Destination that will receive the forwarded events. (For more information, see Transformation Hub Destinations.)

8. Select the Enable checkbox to have the forwarder immediately enabled. If you choose not to enable the forwarder now, you can enable it later.
9. Click Save.

To edit a forwarder:

1. Open the Configuration > Data menu and click Forwarders.
2. Locate the forwarder you want to edit.
3. If the forwarder is enabled, click the Enabled icon () to disable it.

4. Click the Edit icon ().

   The following screen shows the Edit Forwarder screen for a regular expression based forwarder. The Edit Forwarder screen for a Unified Query forwarder lists the Unified Query based filters and the Query text box only allows you to specify one query.

   Specifying Query Terms, Filters, and other forwarder parameters

5. Edit the information in the form, as described in the table Forwarder Parameters.
6. Select the Enable checkbox to have the forwarder immediately enabled. If you choose not to enable the forwarder now, you can enable it later.
7. Click Save.

To delete a forwarder:

1. Open the Configuration > Data menu and click Forwarders.
2. Locate the forwarder that you want to delete.
3. If the forwarder is enabled, click the Enabled icon () to disable it.
4. Click the Remove icon ().
5. Click OK to confirm the deletion.

To pause a forwarder:

1. Open the Configuration > Data menu and click Forwarders.
2. Locate the forwarder that you want to pause.
3. Click the Running icon () to pause the forwarder.

To resume a forwarder:

1. Open the Configuration > Data menu and click Forwarders.
2. Locate the forwarder whose operation you want to resume.
3. Click the Paused icon () to resume forwarder operation.

To disable a forwarder:

1. Open the Configuration > Data menu and click Forwarders.
2. Click Event Output in the left panel.
3. Locate the forwarder that you want to disable.
4. Click the Enabled icon () to disable it.

To enable or re-enable a forwarder:

1. Open the Configuration > Data menu and click Forwarders.
2. Locate the forwarder that you want to enable or re-enable.
3. Click the Disabled icon ().

See Also