Skip to main content

Export Teleport Audit Events with Fluentd

Report an Issue
Video guide
This is a fake video link, YouTube API key is not available
This is a fake video link, YouTube API key is not available

Teleport's Event Handler plugin receives audit events from the Teleport Auth Service and forwards them to your log management solution, letting you perform historical analysis, detect unusual behavior, and form a better understanding of how users interact with your Teleport cluster.

Fluentd is an open source data collector for a unified logging layer.

This guide also serves as an explanation for the Teleport Event Handler plugin, using Fluentd as the target service.

How it works​

The Teleport Event Handler authenticates to the Teleport Auth Service to receive audit events over a gRPC stream, then sends those events to Fluentd as JSON payloads over a secure channel established via mutual TLS.

In this demo, we'll create a local Docker container for Fluentd as a destination for the Event Handler:

You can follow the instructions below for a local proof-of-concept demo, or use any of the additional installation instructions to configure the Teleport Event Handler to integrate with your infrastructure.

Prerequisites​

  • Fluentd version v1.12.4 or greater. The Teleport Event Handler will create a new fluent.conf file you can integrate into an existing Fluentd system, or use with a fresh setup.
  • A server, virtual machine, Kubernetes cluster, or Docker environment to run the Teleport Event Handler plugin.

This guide requires you to have completed one of the Event Handler setup guides:

The instructions below demonstrate a local test of the Event Handler plugin on your workstation. You will need to adjust paths, ports, and domains for other environments.

Step 1/2. Start the Fluentd forwarder​

The Fluentd plugin will send events to your Fluentd instance using keys generated on the previous step.

The fluent.conf file generated in the prerequisite setup guide configures your Fluentd instance to accept events using TLS and print them:

<source>
    @type http
    port 8888

    <transport tls>
        client_cert_auth true

        # We are going to run fluentd in Docker. /keys will be mounted from the host file system.
        ca_path /keys/ca.crt
        cert_path /keys/server.crt
        private_key_path /keys/server.key
        private_key_passphrase ********** # Passphrase generated along with the keys
    </transport>

    <parse>
      @type json
      json_parser oj

      # This time format is used by the plugin. This field is required.
      time_type string
      time_format %Y-%m-%dT%H:%M:%S
    </parse>

    # If the number of events is high, fluentd will start failing the ingestion
    # with the following error message: buffer space has too many data errors.
    # The following configuration prevents data loss in case of a restart and
    # overcomes the limitations of the default fluentd buffer configuration.
    # This configuration is optional.
    # See https://docs.fluentd.org/configuration/buffer-section for more details.
    <buffer>
      @type file
      flush_thread_count 8
      flush_interval 1s
      chunk_limit_size 10M
      queue_limit_length 16
      retry_max_interval 30
      retry_forever true
    </buffer>
</source>

# Events sent to test.log will be dumped to STDOUT.
<match test.log>
  @type stdout
</match>

To try out this Fluentd configuration, start your Fluentd instance:

docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge

This will consume your current terminal, so open a new one to continue following along.

Step 2/2. Run the Event Handler plugin​

In this section, you will modify the Event Handler configuration you generated and run the Event Handler to test your configuration.

Configure the Event Handler​

Edit the configuration for the Event Handler, depending on your installation method.

Earlier, we generated a file called teleport-event-handler.toml to configure the Teleport Event Handler. This file includes setting similar to the following:

storage = "./storage"
timeout = "10s"
batch = 20
# concurrency is the number of concurrent sessions to process. By default, this is set to 5.
concurrency = 5
# The window size configures the duration of the time window for the event handler
# to request events from Teleport. By default, this is set to 24 hours.
# Reduce the window size if the events backend cannot manage the event volume
# for the default window size.
# The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
window-size = "24h"
# types is a comma-separated list of event types to search when forwarding audit
# events. For example, to limit forwarded events to user logins
# and new Access Requests, you can assign this field to
# "user.login,access_request.create".
types = ""
# skip-event-types is a comma-separated list of audit log event types to skip.
# For example, to forward all audit events except for new app deletion events,
# you can include the following assignment:
# skip-event-types = ["app.delete"]
skip-event-types = []
# skip-session-types is a comma-separated list of session recording event types to skip.
# For example, to forward all session events except for malformed SQL packet
# events, you can include the following assignment:
# skip-session-types = ["db.session.malformed_packet"]
skip-session-types = []

[forward.fluentd]
ca = /home/bob/event-handler/ca.crt
cert = /home/bob/event-handler/client.crt
key = /home/bob/event-handler/client.key
url = "https://fluentd.example.com:8888/test.log"
# The Event Handler appends `.<session-id>.log` to `session-url` when sending
# session recording events. For example, if `session-url` is
# `https://fluentd.example.com:8888/session`, the actual requests are sent to
# paths like `/session.<session-id>.log`. Ensure that your log collector's
# tag matching or routing rules account for this suffix (e.g., use `session.*`
# as a match pattern in Fluentd or Fluent Bit).
session-url = "https://fluentd.example.com:8888/session"

[teleport]
addr = teleport.example.com:443
identity = "identity"

Update the following fields.

[teleport]

addr: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443

identity: Fill this in with the path to the identity file you exported earlier.

If you are providing credentials to the Event Handler using a tbot binary that runs on a Linux server, make sure the value of identity in the Event Handler configuration is the same as the path of the identity file you configured tbot to generate, /opt/machine-id/identity.

[forward.fluentd]

ca: Include the path to the CA certificate: /home/bob/event-handler/ca.crt

cert: Include the path to the Fluentd client certificate. /home/bob/event-handler/client.crt

key: Include the path to the Fluentd client key. /home/bob/event-handler/client.key

url: Include the Fluentd URL where the audit event logs will be sent.

session-url: Include the Fluentd URL where the session logs will be sent.

Start the Event Handler​

Start the Teleport Event Handler by following the instructions below.

Copy the teleport-event-handler.toml file to /etc on your Linux server. Update the settings within the toml file to match your environment. Make sure to use absolute paths on settings such as identity and storage. Files and directories in use should only be accessible to the system user executing the teleport-event-handler service such as /var/lib/teleport-event-handler.

Next, create a systemd service definition at the path /usr/lib/systemd/system/teleport-event-handler.service with the following content:

[Unit]
Description=Teleport Event Handler
After=network.target

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/teleport-event-handler start --config=/etc/teleport-event-handler.toml --teleport-refresh-enabled=true
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-event-handler.pid

[Install]
WantedBy=multi-user.target

If you are not using Machine & Workload Identity to provide short-lived credentials to the Event Handler, you can remove the --teleport-refresh-enabled true flag.

Enable and start the plugin:

sudo systemctl enable teleport-event-handler
sudo systemctl start teleport-event-handler
Choose when to start exporting events

You can configure when you would like the Teleport Event Handler to begin exporting events when you run the start command. This example will start exporting from May 5th, 2021:

teleport-event-handler start --config /etc/teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

You can only determine the start time once, when first running the Teleport Event Handler. If you want to change the time frame later, remove the plugin state directory that you specified in the storage field of the handler's configuration file.

Once the Teleport Event Handler starts, you will see notifications about scanned and forwarded events:

sudo journalctl -u teleport-event-handler
DEBU   Event sent id:f19cf375-4da6-4338-bfdc-e38334c60fd1 index:0 ts:2022-09-2118:51:04.849 +0000 UTC type:cert.create event-handler/app.go:140...

Troubleshooting connection issues​

If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:

  • The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the --ttl flag in the tctl auth sign command, which is 12 hours by default.
  • In your Teleport Event Handler configuration file, you have provided the correct host and port for the Teleport Proxy Service.
  • Start the Fluentd container prior to starting the Teleport Event Handler. The Event Handler will attempt to connect to Fluentd immediately upon startup.

Next steps​

  • See instructions to configure dedicated integrations in the navigation sidebar under Audit Event Export.
  • To see all of the options you can set in the values file for the teleport-plugin-event-handler Helm chart, consult our reference guide.