Deploying on ECS

Follow the instructions below to deploy the agent on your AWS Elastic Container Service (ECS) cluster. The basic procedure:

1. Create a configuration file to be used by Pulse Agent.
2. Register a task definition in your AWS account.
3. Register the service in your ECS cluster.

Configuration file

Create a config file as described in the previous section, and name it pulse-agent-config.yml. Then create a secret from it:

aws secretsmanager delete-secret --secret-id pulse-agent-config.yml
aws secretsmanager create-secret --name pulse-agent-config.yml --secret-string file://pulse-agent-config.yml

Note the ARN of the stored secret, and update the task execution role to allow access. Assuming that your cluster uses the default task execution role (ecsTaskExecutionRole), follow these directions:

1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. In the navigation pane, choose Roles.
3. Search the list of roles for ecsTaskExecutionRole (or whatever task execution role you will use) and select it.
4. Choose Permissions, Add inline policy.
5. Choose the JSON tab and specify the following JSON text, ensuring that you specify the full ARN of the Secrets Manager secret you created in.
6. Choose Review policy. For Name specify PulseAgentConfig, then choose Create policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": [
        "arn:aws:secretsmanager:REGION:AWS_ACCOUNT_ID:secret:pulse-agent-config.yml-HASH"
      ]
    }
  ]
}

Task definition

The task specification must be registered with your AWS account. It is not specific to a single ECS cluster. To create the task definition, create a file named in a separate file called pulse-agent.json. Make sure to specify the correct executionRoleArn, and valueFrom for the secret. Note that the sidecar is necessary to work around a deficiency in ECS.

{
  "executionRoleArn": "arn:aws:iam::AWS_ACCOUNT_ID:role/ecsTaskExecutionRole",
  "family": "pulse-agent",
  "containerDefinitions": [
    {
      "command": [
        "sh",
        "-c",
        "echo $PULSE_CONFIG > /etc/pulse-agent/pulse-agent-config.yml"
      ],
      "secrets": [
        {
          "valueFrom": "arn:aws:secretsmanager:REGION:AWS_ACCOUNT_ID:secret:pulse-agent-config.yml-HASH",
          "name": "PULSE_CONFIG"
        }
      ],
      "mountPoints": [
        {
          "containerPath": "/etc/pulse-agent",
          "sourceVolume": "conf"
        }
      ],
      "image": "alpine",
      "essential": false,
      "name": "fetch-config-at-startup"
    },
    {
      "name": "pulse-agent",
      "image": "r.bigdataboutique.com/pulse-agent",
      "cpu": 1024,
      "memory": 16384,
      "portMappings": [],
      "essential": true,
      "environment": [
        {
          "name": "PULSE_AGENT_HEAP_SIZE",
          "value": "2G"
        },
        {
          "name": "PULSE_ENVIRONMENT_NAME",
          "value": "my-environment"
        }
      ],
      "dependsOn": [
        {
          "containerName": "fetch-config-at-startup",
          "condition": "SUCCESS"
        }
      ],
      "environmentFiles": [],
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/etc/pulse-agent",
          "sourceVolume": "conf"
        }
      ],
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -fsSL http://localhost:8080/healthz"],
        "interval": 30,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 30
      }
    }
  ],
  "volumes": [{ "name": "conf" }],
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "2048",
  "memory": "16384",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

Then register the task definition:

aws ecs register-task-definition --cli-input-json file://pulse-agent.json

Service

After registering the task definition, create a service in the desired cluster. The Pulse Agent needs to be able to connect to both the monitored services and to https://pulse.support/. Ensure that you select a subnet and security group that meets these requirements.

aws ecs create-service --cluster my-cluster-name \
  --service-name pulse-agent \
  --task-definition pulse-agent \
  --desired-count 1 \
  --launch-type "FARGATE" \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-abcd1234],securityGroups=[sg-abcd1234]}"

Troubleshooting

If you encounter difficulties with your deployment, the first step to enable logging to CloudWatch Logs for the Pulse agent. The steps to do this are:

1. Grant permission to write logs to the ECS task execution role.
2. Modify the task definition to enable logging.
3. Deploy the new version of the task definition.
4. Inspect the logs in the ECS console.

Grant permission to the task execution role

Attach an inline policy to the task execution role in use (if you followed our recommended setup, the task execution role is ecsTaskExecutionRole):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Modify the task definition

Add a logConfiguration section to the task defintion and upload a revision. The relevant section looks like this:

{
  "containerDefinitions": [
    {
      "name": "pulse-agent",
      // ...
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "pulse-agent",
          "awslogs-region": "MY_AWS_REGION",
          "awslogs-stream-prefix": "pulse-agent"
        }
      }
    }
  ]
}

Deploy the new version

Upload the modified task definition using aws ecs register-task-definition. Then use aws ecs update-service to deploy the new revision.

aws ecs register-task-definition --cli-input-json file://pulse-agent.json
aws ecs update-service --cluster my-cluster-name --service pulse-agent --task-definition pulse-agent

Inspect the logs

From the AWS ECS console, select your cluster, then the pulse-agent service, and then click the logs tab. This provides easy access to the logs, and you can addiionally click the button in the console to go to the full AWS CloudWatch Logs page for the logs. From the AWS CloudWatch Logs console, select "Actions" -> "Download Search results (CSV)" to create a CSV file that you can send to Pulse support.

Any messages with "level":"ERROR" or "level":"WARN" are worth looking into.