Monitoring Cucumber Test Results on AWS CloudWatch

Monitoring test results is critical for maintaining the health of your automated testing suite. AWS CloudWatch provides a reliable and scalable solution for logging and monitoring your Cucumber test results. This guide will walk you through the process of setting up monitoring for Cucumber test results using AWS CloudWatch, including necessary configurations, considerations, and applicable code.

Prerequisites

AWS Account: Ensure you have an AWS account with necessary permissions.
IAM Role: Create an IAM role with permissions for CloudWatch.
Cucumber Project: Have a working Cucumber project set up.

Step 1: Setting Up AWS CloudWatch Logs

Create Log Group:
- Navigate to the CloudWatch console.
- Go to “Logs” and create a new log group (e.g., /cucumber/test-results).
Create IAM Role and Attach Policy:
- Create an IAM role with permissions to write logs to CloudWatch.
- Attach the following policy to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:<region>:<account-id>:log-group:/cucumber/test-results:*"
        }
    ]
}

Step 2: Add AWS SDK Dependencies

Add the AWS SDK for Java to your pom.xml in your Cucumber project to enable interaction with CloudWatch.

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-cloudwatchlogs</artifactId>
    <version>1.12.200</version>
</dependency>

Step 3: Implement CloudWatch Logging Utility

Create a utility class to log test results to CloudWatch.

package com.example.utils;

import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.logs.AWSLogs;
import com.amazonaws.services.logs.AWSLogsClientBuilder;
import com.amazonaws.services.logs.model.CreateLogStreamRequest;
import com.amazonaws.services.logs.model.PutLogEventsRequest;
import com.amazonaws.services.logs.model.PutLogEventsResult;
import com.amazonaws.services.logs.model.InputLogEvent;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;

public class CloudWatchLogger {

    private static final String LOG_GROUP_NAME = "/cucumber/test-results";
    private static final String LOG_STREAM_NAME = "cucumber-log-stream";
    private static AWSLogs awsLogsClient = AWSLogsClientBuilder.standard()
            .withRegion(Regions.US_WEST_2)
            .withCredentials(new DefaultAWSCredentialsProviderChain())
            .build();
    private static String sequenceToken;

    static {
        awsLogsClient.createLogStream(new CreateLogStreamRequest(LOG_GROUP_NAME, LOG_STREAM_NAME));
    }

    public static void log(String message) {
        List<InputLogEvent> logEvents = new ArrayList<>();
        logEvents.add(new InputLogEvent()
                .withMessage(message)
                .withTimestamp(new Date().getTime()));

        PutLogEventsRequest putLogEventsRequest = new PutLogEventsRequest()
                .withLogGroupName(LOG_GROUP_NAME)
                .withLogStreamName(LOG_STREAM_NAME)
                .withLogEvents(logEvents);

        if (sequenceToken != null) {
            putLogEventsRequest.withSequenceToken(sequenceToken);
        }

        PutLogEventsResult putLogEventsResult = awsLogsClient.putLogEvents(putLogEventsRequest);
        sequenceToken = putLogEventsResult.getNextSequenceToken();
    }
}

Step 4: Integrate Logging in Cucumber Steps

Use the CloudWatch logger in your Cucumber step definitions to log test results.

package com.example.steps;

import com.example.utils.CloudWatchLogger;
import io.cucumber.java.After;
import io.cucumber.java.Before;
import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import static org.junit.Assert.assertEquals;

public class TestSteps {
    
    @Before
    public void setUp() {
        CloudWatchLogger.log("Starting a new test scenario");
    }

    @Given("test data with ID {string} and data {string} is prepared")
    public void prepareTestData(String testId, String data) {
        CloudWatchLogger.log("Preparing test data with ID: " + testId + " and data: " + data);
        // Simulate preparing test data
    }

    @When("I execute the test case with ID {string}")
    public void executeTestCase(String testId) {
        CloudWatchLogger.log("Executing test case with ID: " + testId);
        // Simulate test execution
    }

    @Then("the result should be {string}")
    public void verifyResult(String expectedResult) {
        String actualResult = "success"; // Simulate getting actual result
        CloudWatchLogger.log("Expected result: " + expectedResult + ", Actual result: " + actualResult);
        assertEquals(expectedResult, actualResult);
    }

    @After
    public void tearDown() {
        CloudWatchLogger.log("Test scenario completed");
    }
}

Step 5: Run Your Tests

1. Compile and Run Tests:

mvn compile
mvn test

2. Monitor Logs in CloudWatch:

Go to the CloudWatch console.
Navigate to Logs, and then to the log group /cucumber/test-results.
Select the log stream cucumber-log-stream to view the log events.

Considerations and Best Practices

Log Levels: Implement different log levels (INFO, DEBUG, ERROR) to control the verbosity of logs.
Error Handling: Ensure that the logging utility handles errors gracefully and does not interrupt the test flow.
Log Rotation: Manage log retention policies in CloudWatch to control storage costs and maintain log hygiene.
Performance: Minimize the performance impact of logging by batching log events and using asynchronous logging if necessary.
Monitoring and Alerts: Set up CloudWatch Alarms based on log patterns to alert you of critical issues in your test runs.
Environment Configuration: Use environment variables to manage different configurations for logging in development, staging, and production environments.

Conclusion

By following this guide, you will be able to efficiently monitor your Cucumber test results using AWS CloudWatch. This setup ensures that you have real-time visibility into your test executions, helping you quickly identify and address issues in your testing suite.

If you found this article helpful and are interested in integrating Cucumber Automation Framework with AWS CloudWatch, I suggest you check out some of the other articles I’ve written for this series:

If you’re looking for a deeper dive into some of the concepts and specifics discussed in my article, feel free to reach out to me directly or as always you can checkout the official AWS CloudWatch Developer Documentation for more information.