Skip to main content

Pulse for Logstash

Pulse for Logstash provides comprehensive monitoring of Logstash instances, collecting detailed metrics about health status, node statistics, pipeline performance, and more.

The following instructions explain how to Install the Pulse Agent for Logstash clusters.

For a smooth and efficient setup, we recommended using the Integration Wizard available in the Pulse platform. It will lead you step by step, making it easy to get up and running quickly.

If you'd like to explore Deployment options or dive deeper into installation details, continue with the detailed guide below.

Authentication​

For optimal security and functionality, it is highly recommended to provide the Pulse Agent with credentials for a dedicated user with only the required permissions.

To set up credentials, set the following settings within logstash.yml on all of the nodes:

api.auth.type: basic
api.auth.basic.username: "LOGSTASH_USERNAME"
api.auth.basic.password: "LOGSTASH_PASSWORD"

Replace the following as needed:

  • LOGSTASH_USERNAME : Username for the pulse agent credential
  • LOGSTASH_PASSWORD : Password for the pulse agent credential

Restart logstash for the changes to take effect.

Deployment Options​

Deploying the Pulse Agent for Logstash clusters is simple and quick. You can choose from various deployment configurations. The configurations covered for each option use a predefined host list. See Kubernetes Autodiscovery if you want to discover it using kubernetes labels or a headless service name.

Click on the deployment environment to learn more:

Kubernetes​

To deploy the Pulse Agent on Kubernetes, create the deployment spec in a file called pulse-deployment.yaml.

apiVersion: v1
kind: Secret
metadata:
  name: pulse-config
type: Opaque
stringData:
  LOGSTASH_USER: <USER>
  LOGSTASH_PASSWORD: <PASSWORD>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: pulse-agent
  name: pulse-agent
  namespace: default
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: pulse-agent
  template:
    metadata:
      labels:
        app: pulse-agent
    spec:
      automountServiceAccountToken: false
      containers:
        - image: r.pulse.support/<PULSE_TOKEN_VALUE>/agent
          imagePullPolicy: Always
          name: pulse-agent
          env:
            - name: PULSE_AGENT_HEAP_SIZE
              value: 4G
            - name: PULSE_TOKEN
              value: <PULSE_TOKEN_VALUE>
            - name: LOGSTASH_HOSTS
              value: <LOGSTASH_HOSTS_VALUE>
            - name: FQID
              value: <FQID_VALUE>
            - name: LOGSTASH_USER
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: LOGSTASH_USER
            - name: LOGSTASH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: LOGSTASH_PASSWORD
          resources:
            limits:
              cpu: "1"
              memory: 16Gi
            requests:
              cpu: 100m
              memory: 4Gi
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
      dnsPolicy: ClusterFirst
      restartPolicy: Always

Next, replace the following values with:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <USERNAME> – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step
  • default – The namespace in Kubernetes for the deployment
  • <LOGSTASH_HOSTS_VALUE> – The addresses of all of your Logstash cluster hosts. The hosts list should have an http/https prefix separated by a comma. For example: http://logstash.mycompany.com:9600, http://logstash-service-2.mycompany.com:9600, http://10.23.11.1:9600
  • <FQID_VALUE> – a randomly generated string of 32 characters, unique per cluster set up

Finally, apply the spec:

kubectl apply -f pulse-deployment.yaml

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Kubernetes via Helm​

To deploy the Pulse Agent on Kubernetes via Helm create a values.yaml file using the command below:

image:
  repository: r.pulse.support/<PULSE_TOKEN_VALUE>/agent
  pullPolicy: Always
  tag: latest

env:
- name: PULSE_AGENT_HEAP_SIZE
  value: 4G
- name: LOGSTASH_HOSTS
  value: <LOGSTASH_HOSTS_VALUE>
- name: FQID
  value: <FQID_VALUE>
secretEnv:
  PULSE_TOKEN: <PULSE_TOKEN_VALUE>
  LOGSTASH_USER: <USER>
  LOGSTASH_PASSWORD: <PASSWORD>

Replace the following as needed:

  • <LOGSTASH_HOSTS_VALUE> – The addresses of all of your Logstash cluster hosts. The hosts list should have an http/https prefix separated by a comma. For example: http://logstash.mycompany.com:9600, http://logstash-service-2.mycompany.com:9600, http://10.23.11.1:9600
  • <FQID_VALUE> – a randomly generated string of 32 characters, unique per cluster set up
  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <USER> – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Finally, apply the spec with the following command:

helm upgrade -i pulse-agent oci://public.ecr.aws/x7k7t7h4/pulse-agent --create-namespace
        --namespace <YOUR_NAMESPACE> -f values.yaml

Replace the following as needed:

  • <YOUR_NAMESPACE> – The namespace in Kubernetes for the deployment.

For a full list of values, run the following command:

helm show values oci://public.ecr.aws/x7k7t7h4/pulse-agent

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Docker​

To deploy the Pulse Agent on Docker, use the following commands:

docker pull r.pulse.support/<PULSE_TOKEN_VALUE>/agent
docker run -d --name pulse-agent \
    --restart=always \
    --net=host \
    --env PULSE_AGENT_HEAP_SIZE='4G' \
    --env PULSE_TOKEN='<PULSE_TOKEN_VALUE>' \
    --env LOGSTASH_HOSTS='http://10.23.11.1:9600' \
    --env FQID='<FQID_VALUE>' \
    --env LOGSTASH_USER='<USER>' \
    --env LOGSTASH_PASSWORD='<PASSWORD>' \
    r.pulse.support/<PULSE_TOKEN_VALUE>/agent

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <LOGSTASH_HOSTS_VALUE> – The addresses of all of your Logstash cluster hosts. The hosts list should have an http/https prefix separated by a comma. For example: http://logstash.mycompany.com:9600, http://logstash-service-2.mycompany.com:9600, http://10.23.11.1:9600
  • <FQID_VALUE> – a randomly generated string of 32 characters, unique per cluster set up
  • <USER> – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage, or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Amazon Elastic Container Service (ECS)​

To deploy the Pulse Agent on ECS, you must first register the task definition and then create a service in your Logstash cluster. Note that this is not specific to a single ECS cluster.

Register the task definition​

To create the task definition, create a file called pulse-agent.json:

{
  "family": "pulse-agent",
  "containerDefinitions": [
    {
      "name": "pulse-agent",
      "image": "r.pulse.support/<PULSE_TOKEN_VALUE>/agent",
      "cpu": 1024,
      "memory": 5120,
      "portMappings": [],
      "essential": true,
      "environment": [
        {
          "name": "PULSE_AGENT_HEAP_SIZE",
          "value": "4G"
        },
        {
          "name": "PULSE_TOKEN",
          "value": "<PULSE_TOKEN_VALUE>"
        },
        {
          "name": "LOGSTASH_HOSTS",
          "value": "<LOGSTASH_HOSTS_VALUE>"
        },
        {
          "name": "FQID",
          "value": "<FQID_VALUE>"
        },
        {
          "name": "LOGSTASH_USER",
          "value": "<USER>"
        },
        {
          "name": "LOGSTASH_PASSWORD",
          "value": "<PASSWORD>"
        }
      ],
      "environmentFiles": []
    }
  ],
  "networkMode": "awsvpc",
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "5120",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <LOGSTASH_HOSTS_VALUE> – The addresses of all of your Logstash cluster hosts. The hosts list should have an http/https prefix separated by a comma. For example: http://logstash.mycompany.com:9600, http://logstash-service-2.mycompany.com:9600, http://10.23.11.1:9600
  • <FQID_VALUE> – a randomly generated string of 32 characters, unique per cluster set up
  • <USER> – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Next, register the task definition using this command:

aws ecs register-task-definition --cli-input-json file://pulse-agent.json

Create a service​

After registering the task definition, create a service in the desired cluster. The Pulse Agent needs to be able to connect to both the monitored services and to https://incoming.pulse.support.

Make sure to select a subnet and security group that meets these requirements.

Use the following command to create a service:

aws ecs create-service --cluster <ECS_CLUSTER_NAME> \
    --service-name pulse-agent \
    --task-definition pulse-agent \
    --desired-count 1 \
    --launch-type "FARGATE" \
    --network-configuration "awsvpcConfiguration={subnets=[<subnet-abcd1234>],securityGroups=[<sg-abcd1234>]}"

Replace the following as needed:

  • <ECS_CLUSTER_NAME> – The name of the ECS cluster to run the task on
  • <subnet-abcd1234> – Subnets to use for the task
  • <sg-abcd1234> – Security groups to use for the task

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Nomad​

To deploy the Pulse Agent on Nomad, use this sample job spec:

job "pulse_agent" {
  datacenters = ["dc1"]
  type = "service"
  priority = 50

  update {
    canary = 1
    auto_promote = true
  }

  group "main" {
    network {
      port "agent_healthz" {
        to = 8080
      }
    }

    task "agent" {
      driver = "docker"
      config {
        image = "r.pulse.support/<PULSE_TOKEN_VALUE>/agent"
        ports = ["agent_healthz"]

        volumes = [
          "secrets/config.yml:/etc/pulse-agent/config.yml",
        ]
      }

      template {
        destination = "secrets/config.yml"
        data = <<-EOF
 
          pulseToken: <PULSE_TOKEN_VALUE>

          clusters:
            - type: logstash
              hosts: <LOGSTASH_HOSTS_VALUE>
              cluster_fqid: <FQID_VALUE>
              username: <USER>
              password: <PASSWORD>
          EOF
      }

      resources {
        cpu = 100
        memory = 16384
      }

      service {
        name = "pulse-agent"
        port = "agent_healthz"
        provider = "nomad"

        check {
          type     = "http"
          path     = "/healthz"
          interval = "30s"
          timeout  = "2s"
        }
      }
    }
  }
}

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <LOGSTASH_HOSTS_VALUE> – The addresses of all of your Logstash cluster hosts. The hosts list should have an http/https prefix separated by a comma. For example: http://logstash.mycompany.com:9600, http://logstash-service-2.mycompany.com:9600, http://10.23.11.1:9600
  • <FQID_VALUE> – a randomly generated string of 32 characters, unique per cluster set up
  • <USER> – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Note: You can also place the task in your existing cluster job.

Assuming you use this job spec and save it as pulse_agent.nomad, upload the job spec to your Nomad cluster using this command:

nomad job run pulse_agent.nomad

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Cloud Run​

Before deploying the Pulse Agent on Cloud Run, please refer to Direct VPC Access Limitations to see if Cloud Run works for you.

Create a file named cloudrun.yaml with the following content and replace the values <YOUR_REGION>, <YOUR_NETWORK>, <YOUR_SUBNET> and <PULSE_TOKEN_VALUE> as explained below.

apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: pulse-agent
      annotations:
        run.googleapis.com/launch-stage: BETA
      labels:
        cloud.googleapis.com/location: <YOUR_REGION>
    spec:
      template:
        metadata:
          annotations:
            run.googleapis.com/network-interfaces: '[{"network":<YOUR_NETWORK>,"subnetwork":<YOUR_SUBNET>}]'
            run.googleapis.com/vpc-access-egress: "private-ranges-only"
        spec:
          serviceAccountName: "cloudrun-pulse-agent"
          containers:
            - image: "r.pulse.support/<PULSE_TOKEN_VALUE>/agent:latest"
              env:
                - name: PULSE_AGENT_HEAP_SIZE
                  value: 4G
                - name: PULSE_TOKEN
                  value: <PULSE_TOKEN_VALUE>
                - name: LOGSTASH_HOSTS
                  value: <LOGSTASH_HOSTS_VALUE>
                - name: FQID
                  value: <FQID_VALUE>
                - name: LOGSTASH_USER
                  value: <USER>
                - name: LOGSTASH_PASSWORD
                  value: <PASSWORD>
              resources:
                limits:
                  cpu: "1"
                  memory: 4Gi
                requests:
                  cpu: 100m
                  memory: 4Gi
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: 8080
                initialDelaySeconds: 30
                periodSeconds: 10
                timeoutSeconds: 1
              readinessProbe:
                httpGet:
                  path: /healthz
                  port: 8080
                initialDelaySeconds: 10
                periodSeconds: 10
                timeoutSeconds: 1

Replace the following as needed:

  • <YOUR_REGION> – Your Cloud Run region
  • <YOUR_NETWORK> – Your Cloud Run network
  • <YOUR_SUBNET> – Your Cloud Run subnet, make sure that it's within YOUR_REGION.
  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <LOGSTASH_HOSTS_VALUE> – The addresses of all of your Logstash cluster hosts. The hosts list should have an http/https prefix separated by a comma. For example: http://logstash.mycompany.com:9600, http://logstash-service-2.mycompany.com:9600, http://10.23.11.1:9600
  • <FQID_VALUE> – a randomly generated string of 32 characters, unique per cluster set up
  • <USER> – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Run this command to deploy:

gcloud run services replace cloudrun.yaml --region <YOUR_REGION>

Replace the following as needed:

  • <YOUR_REGION> – Your Cloud Run region

The clusters section of the configuration file should point to the Logstash API endpoints. Multiple hosts are supported (separated by comma), as well as specifying username and password for protected instances.

Kubernetes Autodiscovery​

For dynamic Kubernetes environments, Pulse can automatically discover Logstash instances using two methods:

Label-Based Discovery​

Discovers Logstash pods using Kubernetes label selectors. Set the following environment variables and values in your deployment configuration:

  • LOGSTASH_AUTODISCOVER_ENABLED - true
  • LOGSTASH_AUTODISCOVER_METHOD - lables
  • LOGSTASH_AUTODISCOVER_NAMESPACE - the namespace to autodiscover by. Example: monitoring
  • LOGSTASH_AUTODISCOVER_LABELS - the labels and values to autodiscover by. Example: app=logstash,tier=monitoring
  • LOGSTASH_AUTODISCOVER_SERVICE_NAME
  • LOGSTASH_AUTODISCOVER_PORT - the expected port for Logstash. Example: 9600
  • LOGSTASH_AUTODISCOVER_SCHEME - http or https to construct HTTPS URLs for discovered instances.
  • LOGSTASH_AUTODISCOVER_REFRESH_INTERVAL_SECONDS - 60

Only pods in "Running" state are included. Custom port annotation is supported, so co.elastic.metrics/logstash.port would match the following example:

apiVersion: v1
kind: Pod
metadata:
  name: logstash-0
  labels:
    app: logstash
    tier: monitoring
  annotations:
    co.elastic.metrics/logstash.port: "9601"

For label-based discovery to work correctly, make sure that the labels and their values indeed mark pods belonging to a single cluster, that is, the same version of Logstash running the same pipelines.

Headless Service Discovery​

Discovers Logstash instances through a Kubernetes headless service. Set the following environment variables and values in your deployment configuration:

  • LOGSTASH_AUTODISCOVER_ENABLED - true
  • LOGSTASH_AUTODISCOVER_METHOD - headless_service
  • LOGSTASH_AUTODISCOVER_NAMESPACE - the namespace to autodiscover by. Example: monitoring
  • LOGSTASH_AUTODISCOVER_SERVICE_NAME - the name of the headless service. Example: logstash-headless
  • LOGSTASH_AUTODISCOVER_PORT - the expected port for Logstash. Example: 9600
  • LOGSTASH_AUTODISCOVER_SCHEME - http or https to construct HTTPS URLs for discovered instances.
  • LOGSTASH_AUTODISCOVER_REFRESH_INTERVAL_SECONDS - 60

Example for a headless service matched by the above configuration

apiVersion: v1
kind: Service
metadata:
  name: logstash-headless
  namespace: monitoring
spec:
  clusterIP: None
  selector:
    app: logstash
  ports:
  - port: 9600
    name: http

Kubernetes RBAC Requirements​

The agent needs permissions to query the Kubernetes API. Create a ServiceAccount with appropriate RBAC permissions:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pulse-agent
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pulse-agent
  namespace: monitoring
rules:
- apiGroups: [""]
  resources: ["pods", "endpoints"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pulse-agent
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pulse-agent
subjects:
- kind: ServiceAccount
  name: pulse-agent
  namespace: monitoring

For cluster-wide discovery across namespaces, use ClusterRole and ClusterRoleBinding instead.

Autodiscovery Fallback​

If autodiscovery is enabled but fails (e.g., Kubernetes API is unavailable):

  1. An error is logged
  2. The agent falls back to the statically configured hosts list
  3. Discovery is retried on the next refresh interval

This ensures the agent continues working even if autodiscovery temporarily fails.

Using a configuration file​

The recommended method to deploy the Pulse agent is using environment variables. We also recommend running one agent instance per cluster.

However, if this is not possible for any reason, you can use a configuration file rather than using environment variables in order to define the clusters to be monitored.

See Advanced Adjustments for basic instructions on deploying using a configuration file. Some examples for the "clusters" section of the configuration files for monitoring Logstash follow.

Static Hosts with Authentication​

clusters:
  - type: logstash
    cluster_fqid: <generate_random_32_character_string>
    hosts: https://logstash1:9600,https://logstash2:9600,https://logstash3:9600
    username: logstash
    password: changeme

Label-Based Autodiscovery​

clusters:
  - type: logstash
    cluster_fqid: <generate_random_32_character_string>
    autodiscover.enabled: true
    autodiscover.method: labels
    autodiscover.namespace: monitoring
    autodiscover.labels: app=logstash
    autodiscover.port: 9600
    autodiscover.refresh_interval_seconds: 60
    username: logstash
    password: changeme

Headless Service Autodiscovery​

clusters:
  - type: logstash
    cluster_fqid: <generate_random_32_character_string>
    autodiscover.enabled: true
    autodiscover.method: headless_service
    autodiscover.namespace: monitoring
    autodiscover.service_name: logstash-headless
    autodiscover.port: 9600
    autodiscover.refresh_interval_seconds: 60
    username: logstash
    password: changeme