Skip to main content

OpenSearch

The following instructions explain how to Install the Pulse Agent for OpenSearch clusters.

For a smooth and efficient setup, we recommended using the Integration Wizard available in the Pulse platform. It will lead you step by step, making it easy to get up and running quickly.

If you'd like to explore Deployment options or dive deeper into installation details, continue with the detailed guide below.

Authentication Methods​

For optimal security and functionality, it is highly recommended to provide the Pulse Agent with credentials for a dedicated user with only the required permissions.

You can create a dedicated user via the API. The syntax provided here is for use in the Kibana Dev Tools.

You may also not use any authentication method and skip to deployment.

Note: Operating clusters without authentication is strongly discouraged. Pulse is not held responsible for the possible consequences of doing so.

Username and Password​

Here’s a step-by-step setup via the API.

1. Create a new action group​

Use the following command to create a new action group for your OpenSearch cluster:

PUT _plugins/_security/api/actiongroups/pulse_additional_actions
{
  "allowed_actions": [
    "cluster:admin/snapshot/status*",
    "cluster:admin/snapshot/get",
    "cluster:admin/repository/get",
    "cluster:admin/knn_stats_action",
    "cluster:admin/opendistro/ism/managedindex/explain",
    "cluster:admin/indices/dangling/list"
  ]
}

2. Create a role​

Next, create a role for the Pulse Agent: 

PUT _plugins/_security/api/roles/remote_monitoring_collector
{
  "cluster_permissions" : [
    "cluster_monitor",
    "pulse_additional_actions"
  ],
  "index_permissions" : [
    {
      "index_patterns" : [
        "*"
      ],
      "allowed_actions" : [
        "indices_monitor"
      ]
    }
  ]
}

3. Create the user​

Now you’re ready to create the user for the Pulse Agent with its unique credentials. Use the following command:

PUT  _plugins/_security/api/internalusers/pulse-monitoring-collector
{
  "password": "<PASSWORD>",
  "opendistro_security_roles": ["remote_monitoring_collector"],
  "backend_roles": [],
  "attributes": {
    "full_name": "Cluster metrics collector for pulse",
    "email": "support@pulse.support"
  }
}

Replace the following as needed:

  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Deployment Options​

Deploying the Pulse Agent for OpenSearch clusters is simple and quick. You can choose from various deployment configurations.

Click on the deployment environment to learn more:

Kubernetes​

To deploy the Pulse Agent on Kubernetes, create the deployment spec in a file called pulse-deployment.yaml. 

apiVersion: v1
kind: Secret
metadata:
  name: pulse-config
type: Opaque
stringData:
  PULSE_TOKEN: <PULSE_TOKEN_VALUE>
  OPENSEARCH_USER: pulse-monitoring-collector
  OPENSEARCH_PASSWORD: <PASSWORD>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: pulse-agent
  name: pulse-agent
  namespace: default
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: pulse-agent
  template:
    metadata:
      labels:
        app: pulse-agent
    spec:
      automountServiceAccountToken: false
      containers:
        - image: r.bigdataboutique.com/pulse-agent
          imagePullPolicy: Always
          name: pulse-agent
          env:
            - name: PULSE_AGENT_HEAP_SIZE
              value: 4G
            - name: PULSE_TOKEN
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: PULSE_TOKEN
            - name: OPENSEARCH_HOSTS
              value: <OPENSEARCH_HOSTS_VALUE>
            - name: OPENSEARCH_USER
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: OPENSEARCH_USER
            - name: OPENSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: OPENSEARCH_PASSWORD
          resources:
            limits:
              cpu: "1"
              memory: 16Gi
            requests:
              cpu: 100m
              memory: 4Gi
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
      dnsPolicy: ClusterFirst
      restartPolicy: Always

Next, replace the following values with:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step
  • default – The namespace in Kubernetes for the deployment
  • <OPENSEARCH_HOSTS_VALUE> – The address of your OpenSearch cluster hosts. The hosts list should have an http/https prefix separated by a comma for OpenSearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200

Finally, apply the spec:

kubectl apply -f pulse-deployment.yaml

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Kubernetes via Helm​

To deploy the Pulse Agent on Kubernetes via Helm create a values.yaml file using the command below:

image:
  repository: r.bigdataboutique.com/pulse-agent
  pullPolicy: Always
  tag: latest

env:
- name: PULSE_AGENT_HEAP_SIZE
  value: 4G
- name: OPENSEARCH_HOSTS
  value: <OPENSEARCH_HOSTS_VALUE>
secretEnv:
  PULSE_TOKEN: <PULSE_TOKEN_VALUE>
  OPENSEARCH_USER: pulse-monitoring-collector
  OPENSEARCH_PASSWORD: <PASSWORD>

Replace the following as needed:

  • <OPENSEARCH_HOSTS_VALUE> – The address of your OpenSearch cluster host. The hosts list should have an http/https prefix separated by a comma for OpenSearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Finally, apply the spec with the following command:

helm upgrade -i pulse-agent oci://europe-west1-docker.pkg.dev/bdbq-pulse/pulse-helm/pulse-agent --create-namespace
        --namespace <YOUR_NAMESPACE> -f values.yaml

Replace the following as needed:

  • <YOUR_NAMESPACE> – The namespace in Kubernetes for the deployment.

For a full list of values, run the following command:

helm show values oci://europe-west1-docker.pkg.dev/bdbq-pulse/pulse-helm/pulse-agent

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Docker​

To deploy the Pulse Agent on Docker, use the following command:

docker run -d --name pulse-agent \
    --restart=always \
    --net=host \
    --env PULSE_AGENT_HEAP_SIZE='4G' \
    --env PULSE_TOKEN='<PULSE_TOKEN_VALUE>' \
    --env OPENSEARCH_HOSTS='http://10.23.11.1:9200' \
    --env OPENSEARCH_USER='pulse-monitoring-collector' \
    --env OPENSEARCH_PASSWORD='<PASSWORD>' \
    r.bigdataboutique.com/pulse-agent

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <OPENSEARCH_HOSTS_VALUE> – The address of your OpenSearch cluster host. The hosts list should have an http/https prefix separated by a comma, for OpenSearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage, or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Amazon Elastic Container Service (ECS)​

To deploy the Pulse Agent on ECS, you must first register the task definition and then create a service in your OpenSearch cluster. Note that this is not specific to a single ECS cluster.

Register the task definition​

To create the task definition, create a file called pulse-agent.json:

{
  "family": "pulse-agent",
  "containerDefinitions": [
    {
      "name": "pulse-agent",
      "image": "r.bigdataboutique.com/pulse-agent",
      "cpu": 1024,
      "memory": 5120,
      "portMappings": [],
      "essential": true,
      "environment": [
        {
          "name": "PULSE_AGENT_HEAP_SIZE",
          "value": "4G"
        },
        {
          "name": "PULSE_TOKEN",
          "value": "<PULSE_TOKEN_VALUE>"
        },
        {
          "name": "OPENSEARCH_HOSTS",
          "value": "<OPENSEARCH_HOSTS_VALUE>"
        },
        {
          "name": "OPENSEARCH_USER",
          "value": "pulse-monitoring-collector"
        },
        {
          "name": "OPENSEARCH_PASSWORD",
          "value": "<PASSWORD>"
        }
      ],
      "environmentFiles": []
    }
  ],
  "networkMode": "awsvpc",
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "5120",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <OPENSEARCH_HOSTS_VALUE> – The address of your OpenSearch cluster host. The hosts list should have an http/https prefix separated by a comma, for OpenSearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Next, register the task definition using this command:

aws ecs register-task-definition --cli-input-json file://pulse-agent.json

Create a service​

After registering the task definition, create a service in the desired cluster. The Pulse Agent needs to be able to connect to both the monitored services and to https://incoming.pulse.support.

Make sure to select a subnet and security group that meets these requirements.

Use the following command to create a service:

aws ecs create-service --cluster <ECS_CLUSTER_NAME> \
    --service-name pulse-agent \
    --task-definition pulse-agent \
    --desired-count 1 \
    --launch-type "FARGATE" \
    --network-configuration "awsvpcConfiguration={subnets=[<subnet-abcd1234>],securityGroups=[<sg-abcd1234>]}"

Replace the following as needed:

  • <ECS_CLUSTER_NAME> – The name of the ECS cluster to run the task on
  • <subnet-abcd1234> – Subnets to use for the task
  • <sg-abcd1234> – Security groups to use for the task

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Nomad​

To deploy the Pulse Agent on Nomad, use this sample job spec:

job "pulse_agent" {
  datacenters = ["dc1"]
  type = "service"
  priority = 50

  update {
    canary = 1
    auto_promote = true
  }

  group "main" {
    network {
      port "agent_healthz" {
        to = 8080
      }
    }

    task "agent" {
      driver = "docker"
      config {
        image = "r.bigdataboutique.com/pulse-agent"
        ports = ["agent_healthz"]

        volumes = [
          "secrets/config.yml:/etc/pulse-agent/config.yml",
        ]
      }

      template {
        destination = "secrets/config.yml"
        data = <<-EOF
 
          pulseToken: <PULSE_TOKEN_VALUE>

          clusters:
            - type: opensearch
              hosts: <OPENSEARCH_HOSTS_VALUE>
              username: pulse-monitoring-collector
              password: <PASSWORD>
          EOF
      }

      resources {
        cpu = 100
        memory = 16384
      }

      service {
        name = "pulse-agent"
        port = "agent_healthz"
        provider = "nomad"

        check {
          type     = "http"
          path     = "/healthz"
          interval = "30s"
          timeout  = "2s"
        }
      }
    }
  }
}

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <OPENSEARCH_HOSTS_VALUE> – The address of your OpenSearch cluster host. The hosts list should have an http/https prefix separated by a comma, for OpenSearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

Note: You can also place the task in your existing cluster job.

Assuming you use this job spec and save it as pulse_agent.nomad, upload the job spec to your Nomad cluster using this command:

nomad job run pulse_agent.nomad

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Cloud Run​

Before deploying the Pulse Agent on Cloud Run, please refer to Direct VPC Access Limitations to see if Cloud Run works for you.

Create a file named cloudrun.yaml with the following content and replace the values <YOUR_REGION>, <YOUR_NETWORK>, <YOUR_SUBNET> and <PULSE_TOKEN> as explained below.

apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: pulse-agent
      annotations:
        run.googleapis.com/launch-stage: BETA
      labels:
        cloud.googleapis.com/location: <YOUR_REGION>
    spec:
      template:
        metadata:
          annotations:
            run.googleapis.com/network-interfaces: '[{"network":<YOUR_NETWORK>,"subnetwork":<YOUR_SUBNET>}]'
            run.googleapis.com/vpc-access-egress: "private-ranges-only"
        spec:
          serviceAccountName: "cloudrun-pulse-agent"
          containers:
            - image: "r.bigdataboutique.com/pulse-agent:latest"
              env:
                - name: PULSE_AGENT_HEAP_SIZE
                  value: 4G
                - name: PULSE_TOKEN
                  value: <PULSE_TOKEN>
              resources:
                limits:
                  cpu: "1"
                  memory: 4Gi
                requests:
                  cpu: 100m
                  memory: 4Gi
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: 8080
                initialDelaySeconds: 30
                periodSeconds: 10
                timeoutSeconds: 1
              readinessProbe:
                httpGet:
                  path: /healthz
                  port: 8080
                initialDelaySeconds: 10
                periodSeconds: 10
                timeoutSeconds: 1

Replace the following as needed:

  • <YOUR_REGION> – Your Cloud Run region
  • <YOUR_NETWORK> – Your Cloud Run network
  • <YOUR_SUBNET> – Your Cloud Run subnet, make sure that it’s within YOUR_REGION.
  • <PULSE_TOKEN> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)

Run this command to deploy:

gcloud run services replace cloudrun.yaml --region <YOUR_REGION>

Replace the following as needed:

  • <YOUR_REGION> – Your Cloud Run region