Skip to main content

Integration

Pulse for OpenSearch and OpenDistro for Elasticsearch​

Both the platform and the agent offer full support for all Elasticsearch flavors and deployment locations.

Self-managed OpenSearch and OpenDistro (that aren't on Amazon OpenSearch Service) are fully supported on our platform, as well as AWS OpenSearch (the managed service by AWS) and AWS Elasticsearch (which is based on OpenDistro).

This page explains how to set up pulse for clusters using fine-grained security. If you want to authenticate for AWS OS/ES using IAM, see AWS OpenSearch/Elasticsearch IAM

The recommended integration method is using Pulse Agent.

Using Metricbeat OSS 7.12.X (but not above) is also supported but not recommended, you can follow the same instructions as those for Elasticsearch. This page has the relevant compatibility matrix as well as guidance on a cluster setting required for OpenSearch support. Metricbeat is not compatible with AWS OpenSearch/Elasticsearch.

Using the Pulse Agent​

If you haven't done so yet, follow the instructions under the Pulse Agent integration section for installation and basic configuration instructions.

The clusters section of the configuration file should point to the cluster hosts or a load balancer pointing to them. Multiple hosts are supported (separated by comma), as well as specifying username and password for protected clusters. See below for best practices for creating that user.

For non-managed OpenSearch, use:

clusters:
  - type: opensearch
    hosts: http://10.50.2.1:9200,http://10.50.2.2:9200,http://10.50.2.3:9200
    username: pulse-monitoring-collector
    password: pwd

For monitoring an AWS OpenSearch cluster, use:

  - type: aws-opensearch
    hosts: https://my-aws-es-cluster-fjaow83nfdjw7dh2o98.us-east-1.es.amazonaws.com/
    username: pulse-monitoring-collector
    password: pwd

For monitoring an AWS Elasticsearch cluster, use:

  - type: aws-elasticsearch
    hosts: https://my-aws-es-cluster-fjaow83nfdjw7dh2o98.us-east-1.es.amazonaws.com/
    username: pulse-monitoring-collector
    password: pwd

For monitoring an OpenDistro cluster, use:

  - type: elasticsearch
    hosts: http://10.50.2.1:9200,http://10.50.2.2:9200,http://10.50.2.3:9200
    username: pulse-monitoring-collector
    password: pwd

Setting Up A Dedicated Monitoring User - OpenSearch​

The following is for OpenSearch clusters - see below for OpenDistro/legacy AWS Elasticsearch clusters.

To create a user with the relevant permissions:

  1. Create a new action group:
PUT _plugins/_security/api/actiongroups/pulse_additional_actions
{
  "allowed_actions": [
    "cluster:admin/snapshot/status*",
    "cluster:admin/snapshot/get",
    "cluster:admin/repository/get",
    "cluster:admin/knn_stats_action",
    "cluster:admin/opendistro/ism/managedindex/explain",
    "cluster:admin/indices/dangling/list"
  ]
}
  1. Create a role:
PUT _plugins/_security/api/roles/remote_monitoring_collector
{
    "cluster_permissions" : [
      "cluster_monitor","pulse_additional_actions"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "*"
        ],
        "allowed_actions" : [
          "indices_monitor"
        ]
      }
    ]
  }
  1. Create the user:
PUT  _plugins/_security/api/internalusers/pulse-monitoring-collector
{
"password": "L0ng-r4nd0m-p@ssw0rd",
"opendistro_security_roles": ["remote_monitoring_collector"],
"backend_roles": [],
"attributes": {
  "full_name": "Cluster metrics collector for pulse",
  "email": "support@bigdataboutique.com"
}
}

Setting Up A Dedicated Monitoring User - AWS Elasticsearch 7.x/OpenDistro 1.x -​

The following is for OpenDistro/legacy AWS Elasticsearch clusters - see above for OpenSearch clusters. To create a user with the relevant permissions:

  1. Create a new action group:
PUT _opendistro/_security/api/actiongroups/snapshot_monitor
{
  "allowed_actions": [
    "cluster:admin/snapshot/status*",
    "cluster:admin/snapshot/get",
    "cluster:admin/repository/get"
  ]
}
  1. Create a role:
PUT _opendistro/_security/api/roles/remote_monitoring_collector
{
  "cluster_permissions": [
    "cluster_monitor",
    "snapshot_monitor"
  ],
  "index_permissions": [
    {
      "index_patterns": [
        "*"
      ],
      "allowed_actions": [
        "indices_monitor"
      ]
    }
]
}
  1. Create the user:
PUT  _opendistro/_security/api/internalusers/pulse-monitoring-collector
{
  "password": "L0ng-r4nd0m-p@ssw0rd",
  "attributes": {
    "full_name": "Cluster metrics collector for pulse",
    "email": "support@bigdataboutique.com"
  }
}
  1. Create a role mapping:
PUT _opendistro/_security/api/rolesmapping/remote_monitoring_collector
{
  "users":["pulse-monitoring-collector"]
}

Setting Up A Dedicated Monitoring User - AWS Elasticsearch before 7.x/OpenDistro before 1.x -​

The following is for old OpenDistro/legacy AWS Elasticsearch clusters - see above for OpenSearch clusters or newer OD/AWS ES.

To create a user with the relevant permissions:

  1. Create a new action group:
PUT _opendistro/_security/api/actiongroups/snapshot_monitor
{
  "permissions": [
    "cluster:admin/snapshot/status*",
    "cluster:admin/snapshot/get",
    "cluster:admin/repository/get"
  ]
}
  1. Create a role:
PUT _opendistro/_security/api/roles/remote_monitoring_collector
{
  "cluster": [
    "CLUSTER_MONITOR",
    "snapshot_monitor"
  ],
  "indices": {
    "*": {
      "*": [
        "INDICES_MONITOR"
      ]
    }
  }
}
  1. Create the user:
PUT  _opendistro/_security/api/internalusers/pulse-monitoring-collector
{
  "password": "L0ng-r4nd0m-p@ssw0rd",
  "attributes": {
    "full_name": "Cluster metrics collector for pulse",
    "email": "support@bigdataboutique.com"
  }
}
  1. Create a role mapping:
PUT _opendistro/_security/api/rolesmapping/remote_monitoring_collector
{
  "users":["pulse-monitoring-collector"]
}