Skip to main content

Installation and Administration Guide

Fluent Bit

Fluent Bit is an open-source log collector that uses input and output plug-ins to:

  • collect data from multiple sources

  • distribute or send data to various destinations

The on-premises DSP Platform is shipped with Fluent Bit. It is configured by default to use HTTP source and the NULL output plug-in that throws away events. This mean you can get up and running easily and forward audit records to your chosen SIEM solution. To do that, you need to update a configmap, created by the installer, with any supported Fluent Bit output plug-in.

Note

You can install any other audit service by disabling the installation of Fluent Bit and providing the installer with any audit service URL.

File System Buffering and Retries

Fluent Bit supports file system buffering to handle slow network or unresponsive target services. With file system buffering, Fluent Bit stores the data in memory but also maps a copy on the disk. Since audit data is sensitive, Fluent Bit uses in-memory buffering by default. You can enable file system buffering, but consider the risk of storing sensitive information on disk.

Fluent Bit has a retry mechanism and provides a simple configuration option called Retry_Limit. You can set it on each output to impose a limit to try n times and then discard the data after reaching the limit. When you set Retry_Limit to no_limits, there is no limit to the number of retries.

Fluent Bit-Splunk Integration

The following is an example of using Fluent Bit to forward audit records to Splunk.

  1. Set up a Splunk Cloud instance:

    1. Go to Splunk.

    2. Create a cloud trial account.

    3. Log in to the Splunk Cloud Platform using the URL in the Splunk welcome email.

    4. Click Settings → Data input → Select HTTP Event Collector.

    5. Click New Token.

    6. Enter a name, such as dsp-token.

    7. Click Next.

    8. Select the main index.

    9. Click Review.

    10. Click Submit.

    11. Copy the token value.

  2. Update the Fluent Bit config map for the control plane instance:

    1. kubectl edit configmap fluent-bit -n NAMESPACE

    2. Add a new output section with the following properties (replace host and splunk_token):

      [OUTPUT]
    name splunk
    match incoming.excludingDataPlaneEvents
    host prd-p-wma5y.splunkcloud.com
    splunk_token aad8350d-24a9-4b8d-a6eb-5d6a729927cf
    port 8088
    tls on
    tls.verify off 

  3. Update the Fluent Bit config map for each data plane instance:

    1. kubectl edit configmap fluent-bit -n NAMESPACE

    2. Add a new output section with the following properties (replace host and splunk_token):

      [OUTPUT]
    name splunk
    host prd-p-wma5y.splunkcloud.com
    splunk_token aad8350d-24a9-4b8d-a6eb-5d6a729927cf
    port 8088
    tls on
    tls.verify off
    match *
In this section: