Advanced Adjustments

The following are various additional configurations for the Pulse Agent.

Setting the Heap Size

Running the Agent may result in java.lang.OutOfMemoryError for clusters with many indices and shards or when the Agent monitors several clusters. To address that, set the environment variable PULSE_AGENT_HEAP_SIZE when running the Agent.

The default value is 2G. Considering the amount of memory available on the machine hosting the agent, you'll need to use a larger value. The following one-line Docker command for running the Agent sets it to 8G:

docker run -d --name pulse-agent \
    --restart=always \
    --net=host \
    --env PULSE_AGENT_HEAP_SIZE='8G' \
    --env PULSE_TOKEN='PULSE_TOKEN' \
    --env AWS_ELASTICSEARCH_HOST='ELASTICSEARCH_HOSTS_VALUE' \
    --env ELASTICSEARCH_USER='pulse-monitoring-collector' \
    --env ELASTICSEARCH_PASSWORD='PASSWORD' \
    r.bigdataboutique.com/pulse-agent

Controlling Parallelism

By default, the Pulse Agent will use a single thread for efficiency. It will automatically adjust as needed, but you can control the number of threads manually by setting the threads parameter:

docker run -d --name pulse-agent \
    --restart=always \
    --net=host \
    --env PULSE_AGENT_HEAP_SIZE='8G' \
    --env PULSE_TOKEN='PULSE_TOKEN' \
    --env AWS_ELASTICSEARCH_HOST='ELASTICSEARCH_HOSTS_VALUE' \
    --env ELASTICSEARCH_USER='pulse-monitoring-collector' \
    --env ELASTICSEARCH_PASSWORD='PASSWORD' \
    --env PULSE_THREADS=2 \
    r.bigdataboutique.com/pulse-agent:latest

Proxy Configurations

To let the Agent use a proxy server while sending metrics to the Pulse platform, set the environment variables PROXY_HOST and PROXY_PORT. If you have a username and password for the proxy, also use PROXY_USERNAME and PROXY_PASSWORD:

docker run -d --name pulse-agent \
    --restart=always \
    --net=host \
    --env PULSE_AGENT_HEAP_SIZE='8G' \
    --env PULSE_TOKEN='PULSE_TOKEN' \
    --env ELASTICSEARCH2_HOSTS='ELASTICSEARCH_HOSTS_VALUE' \
    --env PROXY_HOST=’172.0.0.1' \
    --env PROXY_PORT=’80' \
    r.bigdataboutique.com/pulse-agent:latest

Make sure to use the right proxy host and port. The Agent will default to the system proxy settings, if any are available.

To explicitly and completely disable any proxy configurations, use the following command:

proxyHost: disable

Kibana Monitoring

If you wish to monitor your Kibana instance as well, and you’re using version 7.14 and above, add the deployment parameter KIBANA_HOST with the value being the endpoint for Kibana. Opensearch Dashboards is not currently supported.

docker run -d --name pulse-agent \
    --restart=always \
    --net=host \
    --env PULSE_AGENT_HEAP_SIZE='8G' \
    --env PULSE_TOKEN='PULSE_TOKEN' \
    --env ELASTICSEARCH2_HOSTS='ELASTICSEARCH_HOSTS_VALUE' \
    --env KIBANA_HOST=’172.0.0.1' \
    r.bigdataboutique.com/pulse-agent:latest

Using a configuration file

The recommended method to deploy the Pulse agent is using environment variables. We also recommend running one agent instance per cluster.

However, if this is not possible for any reason, you can use a configuration file rather than using environment variables in order to define the clusters to be monitored.

Following are explanations on how to set the configuration file and examples for each deployment method that uses the configuration file.

Configuration file

The configuration file needs to contain the following configurations:

• pulseToken (required) - the Pulse Auth Token provided to you when on-boarding to the Pulse platform.
• clusters (required) - the list of clusters to monitor. Follow the guides for the individual deployment methods for type-specific configurations. An example config.yml file:

pulseToken: ${PULSE_TOKEN}

clusters:
 - type: elasticsearch
   hosts: http://hostname:9200
   username: pulse-monitoring-collector
   password: pwd

It is possible to provide the required configurations through environment variables, as the configuration file supports environment variable substitution as demonstrated in the example above.

Remember to substitute the hostname with the address of the cluster that's accessible from the Pulse docker container (using "localhost" might not work).

Deploying on Kubernetes with a configuration file

Follow the instructions below to deploy the agent on your Kubernetes cluster. Create a config file as described in the previous section, and name it pulse-agent-config.yml. Then create a secret from it:

# Create a new secret named pulse-agent-config with local file pulse-agent-config.yml as the config.yml
kubectl create secret generic pulse-agent-config --from-file=config.yml=pulse-agent-config.yml

Now create the deployment spec in a separate file called pulse-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
   app: pulse-agent
 name: pulse-agent
spec:
 replicas: 1
 revisionHistoryLimit: 2
 selector:
   matchLabels:
     app: pulse-agent
 template:
   metadata:
     labels:
       app: pulse-agent
   spec:
     automountServiceAccountToken: false
     containers:
       - image: r.bigdataboutique.com/pulse-agent
         imagePullPolicy: Always
         name: pulse-agent
         env:
           - name: PULSE_ENVIRONMENT_NAME
             value: my-environment
           - name: PULSE_AGENT_HEAP_SIZE
             value: 2G
         resources:
           limits:
             cpu: "1"
             memory: 16Gi
           requests:
             cpu: 100m
             memory: 4Gi
         livenessProbe:
           httpGet:
             path: /healthz
             port: 8080
           initialDelaySeconds: 30
           periodSeconds: 10
           timeoutSeconds: 1
         readinessProbe:
           httpGet:
             path: /healthz
             port: 8080
           initialDelaySeconds: 10
           periodSeconds: 10
           timeoutSeconds: 1
         volumeMounts:
           - mountPath: /etc/pulse-agent
             name: config
             readOnly: true
     dnsPolicy: ClusterFirst
     restartPolicy: Always
     volumes:
       - name: config
         secret:
           defaultMode: 420
           secretName: pulse-agent-config

Finally, apply the spec:

kubectl apply -f pulse-deployment.yaml

Deploying on Docker with a configuration file

The Pulse Agent is available as a container image through r.bigdataboutique.com/pulse-agent:latest.

The Pulse Agent expects a configuration file. Within the directory from which you plan to run Pulse, create a file called config.yml. Once you've finished editing the configuration, run the following command:

docker run -v `pwd`/config.yml:/etc/pulse-agent/config.yml --restart=always r.bigdataboutique.com/pulse-agent

This starts up the pulse agent and maps the file config.yml from the current running directory to the path /etc/pulse-agent/config.yml in the agent container.

Deploying on ECS with a configuration file

Follow the instructions below to deploy the agent on your AWS Elastic Container Service (ECS) cluster. The basic procedure:

1. Create a configuration file to be used by Pulse Agent.
2. Register a task definition in your AWS account.
3. Register the service in your ECS cluster.

Configuration file

Create a config file as described in the previous section, and name it pulse-agent-config.yml. Then create a secret from it:

aws secretsmanager delete-secret --secret-id pulse-agent-config.yml
aws secretsmanager create-secret --name pulse-agent-config.yml --secret-string file://pulse-agent-config.yml

Note the ARN of the stored secret, and update the task execution role to allow access. Assuming that your cluster uses the default task execution role (ecsTaskExecutionRole), follow these directions:

1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. In the navigation pane, choose Roles.
3. Search the list of roles for ecsTaskExecutionRole (or whatever task execution role you will use) and select it.
4. Choose Permissions, Add inline policy.
5. Choose the JSON tab and specify the following JSON text, ensuring that you specify the full ARN of the Secrets Manager secret you created in.
6. Choose Review policy. For Name specify PulseAgentConfig, then choose Create policy.

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": ["secretsmanager:GetSecretValue"],
     "Resource": [
       "arn:aws:secretsmanager:REGION:AWS_ACCOUNT_ID:secret:pulse-agent-config.yml-HASH"
     ]
   }
 ]
}

Task definition

The task specification must be registered with your AWS account. It is not specific to a single ECS cluster. To create the task definition, create a file named in a separate file called pulse-agent.json. Make sure to specify the correct executionRoleArn, and valueFrom for the secret. Note that the sidecar is necessary to work around a deficiency in ECS.

{
 "executionRoleArn": "arn:aws:iam::AWS_ACCOUNT_ID:role/ecsTaskExecutionRole",
 "family": "pulse-agent",
 "containerDefinitions": [
   {
     "command": [
       "sh",
       "-c",
       "echo $PULSE_CONFIG > /etc/pulse-agent/pulse-agent-config.yml"
     ],
     "secrets": [
       {
         "valueFrom": "arn:aws:secretsmanager:REGION:AWS_ACCOUNT_ID:secret:pulse-agent-config.yml-HASH",
         "name": "PULSE_CONFIG"
       }
     ],
     "mountPoints": [
       {
         "containerPath": "/etc/pulse-agent",
         "sourceVolume": "conf"
       }
     ],
     "image": "alpine",
     "essential": false,
     "name": "fetch-config-at-startup"
   },
   {
     "name": "pulse-agent",
     "image": "r.bigdataboutique.com/pulse-agent",
     "cpu": 1024,
     "memory": 5120,
     "portMappings": [],
     "essential": true,
     "environment": [
       {
         "name": "PULSE_AGENT_HEAP_SIZE",
         "value": "4G"
       }
     ],
     "dependsOn": [
       {
         "containerName": "fetch-config-at-startup",
         "condition": "SUCCESS"
       }
     ],
     "environmentFiles": [],
     "mountPoints": [
       {
         "readOnly": true,
         "containerPath": "/etc/pulse-agent",
         "sourceVolume": "conf"
       }
     ],
     "healthCheck": {
       "command": ["CMD-SHELL", "curl -fsSL http://localhost:8080/healthz"],
       "interval": 30,
       "timeout": 5,
       "retries": 3,
       "startPeriod": 30
     }
   }
 ],
 "volumes": [{ "name": "conf" }],
 "networkMode": "awsvpc",
 "requiresCompatibilities": ["FARGATE"],
 "cpu": "1024",
 "memory": "5120",
 "runtimePlatform": {
   "cpuArchitecture": "X86_64",
   "operatingSystemFamily": "LINUX"
 }
}

Then register the task definition:

aws ecs register-task-definition --cli-input-json file://pulse-agent.json

Service

After registering the task definition, create a service in the desired cluster. The Pulse Agent needs to be able to connect to both the monitored services and to https://incoming.pulse.support. Make sure to select a subnet and security group that meets these requirements.

aws ecs create-service --cluster my-cluster-name \
 --service-name pulse-agent \
 --task-definition pulse-agent \
 --desired-count 1 \
 --launch-type "FARGATE" \
 --network-configuration "awsvpcConfiguration={subnets=[subnet-abcd1234],securityGroups=[sg-abcd1234]}"

Troubleshooting ECS

If you encounter difficulties with your deployment, the first step to enable logging to CloudWatch Logs for the Pulse agent. The steps to do this are:

1. Grant permission to write logs to the ECS task execution role.
2. Modify the task definition to enable logging.
3. Deploy the new version of the task definition.
4. Inspect the logs in the ECS console. Grant permission to the task execution role Attach an inline policy to the task execution role in use (if you followed our recommended setup, the task execution role is ecsTaskExecutionRole):

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "logs:CreateLogStream",
       "logs:PutLogEvents"
     ],
     "Resource": "*"
   }
 ]
}

Modify the task definition

Add a logConfiguration section to the task defintion and upload a revision. The relevant section looks like this:

{
 "containerDefinitions": [
   {
     "name": "pulse-agent",
     // ...
     "logConfiguration": {
       "logDriver": "awslogs",
       "options": {
         "awslogs-create-group": "true",
         "awslogs-group": "pulse-agent",
         "awslogs-region": "MY_AWS_REGION",
         "awslogs-stream-prefix": "pulse-agent"
       }
     }
   }
 ]
}

Deploy the new version Upload the modified task definition using aws ecs register-task-definition. Then use aws ecs update-service to deploy the new revision.

aws ecs register-task-definition --cli-input-json file://pulse-agent.json
aws ecs update-service --cluster my-cluster-name --service pulse-agent --task-definition pulse-agent

Inspect the logs From the AWS ECS console, select your cluster, then the pulse-agent service, and then click the logs tab. This provides easy access to the logs, and you can addiionally click the button in the console to go to the full AWS CloudWatch Logs page for the logs. From the AWS CloudWatch Logs console, select "Actions" -> "Download Search results (CSV)" to create a CSV file that you can send to Pulse support. Any messages with "level":"ERROR" or "level":"WARN" are worth looking into.

Prometheus Endpoint

The agent exposes a Prometheus endpoint for monitoring the Pulse Agent’s activity. You can enable it by adding the below snippet to the configurations file:

prometheus:
enabled: true
port: 9090