Skip to main content

Elasticsearch on Amazon OpenSearch Service

The following instructions explain how to Install the Pulse Agent for Elasticsearch clusters on Amazon OpenSearch Service.

For a smooth and efficient setup, we recommended using the Integration Wizard available in the Pulse platform. It will lead you step by step, making it easy to get up and running quickly.

If you'd like to explore Deployment options or dive deeper into installation details, continue with the detailed guide below.

Authentication methods​

For optimal security and functionality, it is highly recommended to provide the Pulse Agent with credentials for a dedicated user with only the required permissions. You can create the user via the Kibana Dev Tools or the API. The syntax provided here is for the Kibana Dev Tools.

You may also not use any authentication method and skip to deployment.

Note: Operating clusters without authentication is strongly discouraged. Pulse is not held responsible for the possible consequences of doing so.

You can select from the following authentication methods:

Amazon Identity and Access Management (IAM)​

You can authenticate the Pulse Agent using either an IAM user (with an access key and secret key) or an IAM role.

The following permissions must be added to the domain’s access policy under the Elasticsearch domain configuration (Security settings). These permissions apply to both IAM users and roles:

{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "es:ESHttpGet",
          "Principal": {
            "AWS": "<IAM_ARN>"
          },
          "Resource": [
            "<DOMAIN_ARN>/",
            "<DOMAIN_ARN>/_cluster/state*",
            "<DOMAIN_ARN>/_cluster/settings/",
            "<DOMAIN_ARN>/_cluster/stats*",
            "<DOMAIN_ARN>/_cluster/health/",
            "<DOMAIN_ARN>/_cluster/allocation/explain/",
            "<DOMAIN_ARN>/_cat/recovery/",
            "<DOMAIN_ARN>/_stats/",
            "<DOMAIN_ARN>/_nodes/",
            "<DOMAIN_ARN>/_nodes*/stats*",
            "<DOMAIN_ARN>/_cat/nodes/",
            "<DOMAIN_ARN>/_snapshot/",
            "<DOMAIN_ARN>/_cat/snapshots*",
            "<DOMAIN_ARN>/_snapshot/_status/",
            "<DOMAIN_ARN>/_cat/indices/",
            "<DOMAIN_ARN>/_cat/templates/",
            "<DOMAIN_ARN>/_template*",
            "<DOMAIN_ARN>/*/_settings/",
            "<DOMAIN_ARN>/_tasks*",
            "<DOMAIN_ARN>/_cluster/pending_tasks/",
            "<DOMAIN_ARN>/_opendistro/_knn/stats/",
            "<DOMAIN_ARN>/_opendistro/_ism/explain/*",
            "<DOMAIN_ARN>/_dangling/"
          ]
        }
      ]
    }

Replace the following as needed:

  • <IAM_ARN> – The ARN for the user or role with which Pulse will access the domain
  • <DOMAIN_ARN> – The domain's ARN shown in the domain General information section in AWS.

Additional Step for an IAM Role

If you’re using an IAM role for authentication, you will also need to configure a trust policy to allow your services.

The trust policy for EC2 or ECS:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": ["<SERVICE>"]
      },
      "Effect": "Allow"
    }
  ]
}

Replace the following as needed:

  • <SERVICE> – use either ecs-tasks.amazonaws.com or ecs-tasks.amazonaws.com

For Kubernetes-based clusters on EKS, refer to the AWS documentation on assigning IAM roles to EKS.

Username and Password (cluster version 7.x)​

The guide below takes you through the step-by-step setup via the API.

1. Create a new action group​

Use the following command to create a new action group for your cluster:

PUT _opendistro/_security/api/actiongroups/pulse_additional_actions
{
  "allowed_actions": [
    "cluster:admin/snapshot/status*",
    "cluster:admin/snapshot/get",
    "cluster:admin/repository/get",
    "cluster:admin/knn_stats_action",
    "cluster:admin/opendistro/ism/managedindex/explain",
    "cluster:admin/indices/dangling/list"
  ]
}

2. Create a role​

Next, create a role for the Pulse Agent: 

PUT _opendistro/_security/api/roles/remote_monitoring_collector
{
  "cluster_permissions" : [
    "cluster_monitor","pulse_additional_actions"
  ],
  "index_permissions" : [
    {
      "index_patterns" : [
        "*"
      ],
      "allowed_actions" : [
        "indices_monitor"
      ]
    }
  ]
}

3. Create the user​

Now you’re ready to create the user for the Pulse Agent with its unique credentials. Use the following command:

PUT _opendistro/_security/api/internalusers/pulse-monitoring-collector
{
  "password": "<PASSWORD>",
  "attributes": {
    "full_name": "Cluster metrics collector for pulse",
    "email": "support@pulse.support"
  }
}

Replace the following as needed:

  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

4. Create a role mapping​

Finally, create a role mapping:

PUT _opendistro/_security/api/rolesmapping/remote_monitoring_collector
{
  "users":["pulse-monitoring-collector"]
}

Replace the following as needed:

  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step

Username and Password (Cluster version 6.x or below)​

The guide below takes you through the step-by-step setup via the API.

1. Create a new action group​

Use the following command to create a new action group for your cluster:

PUT _opendistro/_security/api/actiongroups/snapshot_monitor
{
  "permissions": [
    "cluster:admin/snapshot/status*",
    "cluster:admin/snapshot/get",
    "cluster:admin/repository/get"
  ]
}

2. Create a role​

Next, create a role for the Pulse Agent: 

PUT _opendistro/_security/api/roles/remote_monitoring_collector
{
  "cluster": [
    "CLUSTER_MONITOR",
    "snapshot_monitor"
  ],
  "indices": {
    "*": {
      "*": [
        "indices_monitor"
      ]
    }
  }
}

3. Create the user​

Now you’re ready to create the user for the Pulse Agent with its unique credentials. Use the following command:

PUT _opendistro/_security/api/internalusers/pulse-monitoring-collector
{
  "password": "<PASSWORD>",
  "attributes": {
    "full_name": "Cluster metrics collector for pulse",
    "email": "support@pulse.support"
  }
}

Replace the following as needed:

  • pulse-monitoring-collector – The name of the dedicated monitoring user.
  • <PASSWORD> – The password for the dedicated monitoring user.

4. Create a role mapping​

Finally, create a role mapping:

PUT _opendistro/_security/api/rolesmapping/remote_monitoring_collector
{
  "users":["pulse-monitoring-collector"]
}

Replace the following as needed:

  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step

Deployment Options​

Deploying the Pulse Agent for Elasticsearch on Amazon OpenSearch Service clusters is simple and quick. You can choose from various deployment configurations.

Click on the deployment environment option to learn more:

Kubernetes​

To deploy the Pulse Agent on Kubernetes, create the deployment spec in a file called pulse-deployment.yaml. 

apiVersion: v1
kind: Secret
metadata:
  name: pulse-config
type: Opaque
stringData:
  PULSE_TOKEN: <PULSE_TOKEN_VALUE>
  ELASTICSEARCH_USER: pulse-monitoring-collector
  ELASTICSEARCH_PASSWORD: <PASSWORD>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: pulse-agent
  name: pulse-agent
  namespace: default
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: pulse-agent
  template:
    metadata:
      labels:
        app: pulse-agent
    spec:
      automountServiceAccountToken: false
      containers:
        - image: r.bigdataboutique.com/pulse-agent
          imagePullPolicy: Always
          name: pulse-agent
          env:
            - name: PULSE_AGENT_HEAP_SIZE
              value: 4G
            - name: PULSE_TOKEN
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: PULSE_TOKEN
            - name: AWS_ELASTICSEARCH_HOST
              value: <AWS_ELASTICSEARCH_HOSTS_VALUE>
            - name: ELASTICSEARCH_USER
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: ELASTICSEARCH_USER
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pulse-config
                  key: ELASTICSEARCH_PASSWORD
          resources:
            limits:
              cpu: "1"
              memory: 16Gi
            requests:
              cpu: 100m
              memory: 4Gi
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
      dnsPolicy: ClusterFirst
      restartPolicy: Always

Next, replace the following values with:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step
  • default – The namespace in Kubernetes for the deployment
  • <AWS_ELASTICSEARCH_HOSTS_VALUE> – The address of your Elasticsearch cluster host. The hosts list should have an http/https prefix separated by a comma for Elasticsearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200

If you chose to use IAM-based authentication and are not assuming a role: replace the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY created for the user you’re authenticating with. Also add `AWS_REGION with the cluster’s region.

If you are assuming a role: remove the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections.

Finally, apply the spec:

kubectl apply -f pulse-deployment.yaml

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Kubernetes via Helm​

Create a values.yaml file using the command below:

image:
  repository: r.bigdataboutique.com/pulse-agent
  pullPolicy: Always
  tag: latest

env:
- name: PULSE_AGENT_HEAP_SIZE
  value: 4G
- name: AWS_ELASTICSEARCH_HOST
  value: <AWS_ELASTICSEARCH_HOSTS_VALUE>

secretEnv:
  PULSE_TOKEN: <PULSE_TOKEN_VALUE>
  ELASTICSEARCH_USER: pulse-monitoring-collector
  ELASTICSEARCH_PASSWORD: <PASSWORD>

Replace the following as needed:

  • <AWS_ELASTICSEARCH_HOSTS_VALUE> – The address of your Elasticsearch cluster host. The hosts list should have an http/https prefix separated by a comma for Elasticsearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

If you chose to use IAM-based authentication and are not assuming a role: replace the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY created for the user you’re authenticating with. Also add `AWS_REGION with the cluster’s region.

If you are assuming a role: remove the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections.

Finally, apply the spec:

helm upgrade -i pulse-agent oci://europe-west1-docker.pkg.dev/bdbq-pulse/pulse-helm/pulse-agent --create-namespace
        --namespace YOUR_NAMESPACE -f values.yaml

For a full list of values, run the following command:

helm show values oci://europe-west1-docker.pkg.dev/bdbq-pulse/pulse-helm/pulse-agent

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Docker​

Use the following command:

docker run -d --name pulse-agent \
    --restart=always \
    --net=host \
    --env PULSE_AGENT_HEAP_SIZE='4G' \
    --env PULSE_TOKEN='<PULSE_TOKEN_VALUE>' \
    --env AWS_ELASTICSEARCH_HOST='<AWS_ELASTICSEARCH_HOSTS_VALUE>' \
    --env ELASTICSEARCH_USER='pulse-monitoring-collector' \
    --env ELASTICSEARCH_PASSWORD='<PASSWORD>' \
    r.bigdataboutique.com/pulse-agent

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <AWS_ELASTICSEARCH_HOSTS_VALUE> – The address of your Elasticsearch cluster host. The hosts list should have an http/https prefix separated by a comma for Elasticsearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

If you chose to use IAM-based authentication and are not assuming a role: replace the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY created for the user you’re authenticating with. Also add `AWS_REGION with the cluster’s region.

If you are assuming a role: remove the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections.

Confirm that the Agent is running and no errors are logged in the Agent.

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Amazon Elastic Container Service (ECS)​

To deploy the Pulse Agent on ECS, you must first register the task definition and then create a service in your OpenSearch cluster.

Register the task definition​

Before deploying the Pulse Agent on ECS, make sure that the task specification is registered with your AWS account. Note that this is not specific to a single ECS cluster.

To create the task definition, create a file called pulse-agent.json:

{
  "family": "pulse-agent",
  "containerDefinitions": [
    {
      "name": "pulse-agent",
      "image": "r.bigdataboutique.com/pulse-agent",
      "cpu": 1024,
      "memory": 5120,
      "portMappings": [],
      "essential": true,
      "environment": [
        {
          "name": "PULSE_AGENT_HEAP_SIZE",
          "value": "4G"
        },
        {
          "name": "PULSE_TOKEN",
          "value": "<PULSE_TOKEN_VALUE>"
        },
        {
          "name": "AWS_ELASTICSEARCH_HOST",
          "value": "<AWS_ELASTICSEARCH_HOSTS_VALUE>"
        },
        {
          "name": "ELASTICSEARCH_USER",
          "value": "pulse-monitoring-collector"
        },
        {
          "name": "ELASTICSEARCH_PASSWORD",
          "value": "<PASSWORD>"
        }
      ],
      "environmentFiles": []
    }
  ],
  "networkMode": "awsvpc",
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "5120",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <AWS_ELASTICSEARCH_HOSTS_VALUE> – The address of your Elasticsearch cluster host. The hosts list should have an http/https prefix separated by a comma for Elasticsearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step

If you chose to use IAM-based authentication and are not assuming a role: replace the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY created for the user you’re authenticating with. Also add `AWS_REGION with the cluster’s region.

If you are assuming a role: remove the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections.

Next, register the task definition using this command:

aws ecs register-task-definition --cli-input-json file://pulse-agent.json

Create a service​

After registering the task definition, create a service in the desired cluster. The Pulse Agent needs to be able to connect to both the monitored services and to https://incoming.pulse.support.

Make sure to select a subnet and security group that meets these requirements.

Use the following command to create a service:

aws ecs create-service --cluster <ECS_CLUSTER_NAME> \
    --service-name pulse-agent \
    --task-definition pulse-agent \
    --desired-count 1 \
    --launch-type "FARGATE" \
    --network-configuration "awsvpcConfiguration={subnets=[<subnet-abcd1234>],securityGroups=[<sg-abcd1234>]}"

Replace the following as needed:

  • <ECS_CLUSTER_NAME> – The name of the ECS cluster to run the task on
  • <subnet-abcd1234> – Subnets to use for the task
  • <sg-abcd1234> – Security groups to use for the task

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.

Nomad​

Deploying Pulse Agent on a Nomad cluster is simple. Here’s a sample job spec that will deploy just the Pulse Agent:

job "pulse_agent" {
  datacenters = ["dc1"]
  type = "service"
  priority = 50

  update {
    canary = 1
    auto_promote = true
  }

  group "main" {
    network {
      port "agent_healthz" {
        to = 8080
      }
    }

    task "agent" {
      driver = "docker"
      config {
        image = "r.bigdataboutique.com/pulse-agent"
        ports = ["agent_healthz"]

        volumes = [
          "secrets/config.yml:/etc/pulse-agent/config.yml",
        ]
      }

      template {
        destination = "secrets/config.yml"
        data = <<-EOF

          pulseToken: <PULSE_TOKEN_VALUE>

          clusters:
            - type: aws-elasticsearch
              hosts: <AWS_ELASTICSEARCH_HOSTS_VALUE>
              username: pulse-monitoring-collector
              password: <PASSWORD>
          EOF
      }

      resources {
        cpu = 100
        memory = 16384
      }

      service {
        name = "pulse-agent"
        port = "agent_healthz"
        provider = "nomad"

        check {
          type     = "http"
          path     = "/healthz"
          interval = "30s"
          timeout  = "2s"
        }
      }
    }
  }
}

Replace the following as needed:

  • <PULSE_TOKEN_VALUE> – The Pulse token for your organization (you can find it in the Account Settings of the Pulse Platform)
  • <AWS_ELASTICSEARCH_HOSTS_VALUE> – The address of your Elasticsearch cluster host. The hosts list should have an http/https prefix separated by a comma for Elasticsearch clusters. For redundancy purposes, it’s recommended to enter a list of the cluster's hosts and/or a load balancer. For example: http://elastic-service.mycompany.com:9200, http://elastic-service-2.mycompany.com:9200, http://10.23.11.1:9200
  • pulse-monitoring-collector – The name of the dedicated monitoring user set up in the previous step
  • <PASSWORD> – The password for the dedicated monitoring user set up in the previous step Note: You can also place the task in your existing cluster job.

If you chose to use IAM-based authentication and are not assuming a role: replace the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY created for the user you’re authenticating with. Also add `AWS_REGION with the cluster’s region.

If you are assuming a role: remove the OPENSEARCH_USER and OPENSEARCH_PASSWORD sections.

Assuming you use this job spec and save it as pulse_agent.nomad, upload the job spec to your Nomad cluster using this command:

nomad job run pulse_agent.nomad

After successfully running the deployment command, the new cluster should appear on your Pulse homepage within a few minutes. You will receive an email notification once the cluster is available.

If the cluster doesn’t appear on the Pulse homepage or you encounter an error, please contact our support team via the Pulse system. Make sure to include your logs (including the error message) and deployment configuration.