Welcome to this tutorial on how to check AWS CloudWatch logs using Java. This tutorial is intended for developers who are familiar with Java programming and AWS concepts and are using macOS or Linux as their operating system.
In this tutorial, we will be focusing on how to use Java to check CloudWatch Logs. We will assume that you have an AWS account and that you have set up your AWS environment behind a JumpCloud identity provider. We will also assume that you have basic knowledge of AWS services, including CloudWatch, and have configured your AWS credentials for programmatic access.
To test our code, we will be using AWS CodeBuild, which is a fully managed build service that compiles your source code, runs tests, and produces software packages that are ready to deploy.
By the end of this tutorial, you will have a good understanding of how to use the AWS SDK for Java to interact with CloudWatch Logs, including how to retrieve and filter logs, create and delete log groups, and set up CloudWatch Logs subscriptions. We will guide you through step-by-step instructions to get started, and assume you have the necessary tools installed, including SAML2AWS and JDK.
So let's get started and learn how to check AWS CloudWatch logs using Java on macOS or Linux behind a JumpCloud identity provider, and test our code using AWS CodeBuild.
Notes:
- The tests can be run either locally or directly on AWS CodeBuild.
- If you choose to run the tests locally, you will need to have SAML2AWS and JDK installed on your machine, as mentioned in the previous step.
- Alternatively, you can choose to run the tests directly on AWS CodeBuild, which does not require you to have SAML2AWS or JDK installed on your local machine. This can be useful if you want to test your code on a clean environment, without the need to install additional tools or dependencies.
Step 1: Install SAML2AWS and JDK
Before we begin, you will need to have the following tools installed on your machine:
SAML2AWS: This is a command-line tool that enables you to obtain temporary AWS credentials using your organization's SAML identity provider. SAML2AWS makes it easy to authenticate and access AWS resources from the command line. You can download and install SAML2AWS from the official website: https://github.com/Versent/saml2aws/releases
JDK: The Java Development Kit (JDK) is required to compile and run Java code. Make sure you have JDK version 8 or higher installed on your machine. You can download and install the latest version of the JDK from the official Oracle website: https://www.oracle.com/java/technologies/downloads/
SAML2AWS and JDK are essential tools for interacting with AWS services using Java. SAML2AWS allows you to authenticate with AWS using your organization's SAML identity provider, which is required to access CloudWatch logs. The JDK, on the other hand, is required to compile and run Java code, including the code we will be writing in this tutorial.
Once you have installed SAML2AWS and JDK on your machine, you are ready to proceed to the next step.
Step 2: Configurations
Open Terminal, then configure saml2aws by typing saml2aws configure. This will prompt you to enter the required information, including your username, password, security provider, and URL. You will need to have this information ready before you can proceed with the configuration.
$ saml2aws configure
? Please choose a provider: JumpCloud
? AWS Profile myaccount
? URL https://example.com
? Username me@example.com
? Password
...
account {
URL: https://example.com
Username: me@example.com
Provider: JumpCloud
MFA: Auto
SkipVerify: false
AmazonWebservicesURN: urn:amazon:webservices
SessionDuration: 3600
Profile: myaccount
Region: us-east-1
}
Configuration saved for IDP account: default[default]
name = default
app_id =
url = https://sso.jumpcloud.com/saml2/aws-dev
username = abc@gmail.com
provider = JumpCloud
mfa = Auto
skip_verify = false
timeout = 0
aws_urn = urn:amazon:webservices
aws_session_duration = 3600
aws_profile = default
resource_id =
subdomain =
role_arn = arn:aws:iam::123456789:role/Developer
region =
http_attempts_count =
http_retry_delay =
credentials_file =
saml_cache = false
saml_cache_file =
target_url =
disable_remember_device = false
disable_sessions = false
prompter = Before we start writing Java code to access CloudWatch logs, we need to make sure that the AWS CLI is properly configured with the necessary credentials. To do this, we will first check if the .aws/credentials file has been created and contains a default profile.
The AWS CLI stores its credentials in the .aws/credentials file, which is located in your home directory. You can check if this file exists and contains a default profile by entering the following command in your terminal: cat ~/.aws/credentials
If the file exists and contains a default profile, you should see something similar to the following:
[default]
aws_access_key_id = <your_access_key_id>
aws_secret_access_key = <your_secret_access_key>
If the file does not exist or does not contain a default profile, you can create it by running the following command in your terminal: aws configure
This will prompt you to enter your AWS Access Key ID, AWS Secret Access Key, Default region name, and Default output format. After you enter this information, AWS CLI will create the .aws/credentials file with a default profile containing your credentials.
Make sure that the .aws/credentials file exists and contains a default profile before proceeding to the next step.
Step 3: Writing the code
private static Credentials switchRole(String roleArn, String roleSessionName) {
StsClient stsClient = StsClient.builder()
.region(Region.AP_SOUTHEAST_1)
.credentialsProvider(ProfileCredentialsProvider.create())
.build();
AssumeRoleRequest roleRequest = AssumeRoleRequest.builder()
.roleArn(roleArn)
.roleSessionName(roleSessionName)
.build();
AssumeRoleResponse roleResponse = stsClient.assumeRole(roleRequest);
Credentials myCreds = roleResponse.credentials();
return myCreds;
}
private static BasicSessionCredentials setUpLocalCredentials() {
Credentials myCreds = AwsUtil.switchRole("arn:aws:iam::987654321:role/Developer", "mysession");
String awsAccessKey = myCreds.accessKeyId();
String awsSecretKey = myCreds.secretAccessKey();
String sessionToken = myCreds.sessionToken();
BasicSessionCredentials credentials = new BasicSessionCredentials(awsAccessKey, awsSecretKey, sessionToken);
return credentials;
}
private static AWSLogs setUpCloudWatchClient() {
BasicSessionCredentials credentials = setUpLocalCredentials();
AWSLogs logsClient = AWSLogsClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withRegion(Regions.AP_SOUTHEAST_1)
.build();
return logsClient;
}In this step, we will write Java code to access CloudWatch logs using the AWS SDK for Java. The code will include three methods:
- switchRole(String roleArn, String roleSessionName) - This method switches to an AWS Identity and Access Management (IAM) role and returns the temporary security credentials needed to access AWS resources. It takes in the roleArn and roleSessionName as parameters, which specify the Amazon Resource Name (ARN) of the role to assume and the name to identify the session.
- setUpLocalCredentials() - This method sets up the local credentials to be used for accessing AWS resources. It calls the switchRole() method to obtain temporary security credentials and returns the BasicSessionCredentials object, which contains the access key, secret key, and session token needed to authenticate with AWS services.
- setUpCloudWatchClient() - This method sets up the CloudWatch client to access CloudWatch logs. It calls the setUpLocalCredentials() method to obtain the necessary credentials, creates an AWSLogs client using the AWSLogsClientBuilder class, and returns the AWSLogs object, which is used to access CloudWatch logs.
By using these three methods, we can easily switch to an IAM role, obtain temporary security credentials, and use those credentials to authenticate with AWS services. We can then use the AWSLogs object to access CloudWatch logs and perform various operations, such as retrieving log events, creating log groups, and deleting log streams.
public static ArrayList<String> retrieveCloudWatchLogs(
String logGroupName,
int within,
String pattern) {
boolean runOnCodeBuild = !(System.getenv("CODEBUILD_BUILD_ID") == null);
String msg = String.format("RunningOnCodeBuild=%b;logGroup=%s,within=%dmins,pattern=%s", runOnCodeBuild, logGroupName, within, pattern);
System.out.println(msg);
AWSLogs logsClient = runOnCodeBuild ? AWSLogsClientBuilder.defaultClient() : setUpCloudWatchClient();
// Set the log group name and create a FilterLogEventsRequest
FilterLogEventsRequest filterLogEventsRequest = new FilterLogEventsRequest();
if (logGroupName != null) filterLogEventsRequest.withLogGroupName(logGroupName);
if (pattern != null) filterLogEventsRequest.withFilterPattern(pattern);
long startTime = System.currentTimeMillis() - within * 60 * 1000;
long endTime = System.currentTimeMillis();
filterLogEventsRequest.withStartTime(startTime);
filterLogEventsRequest.withEndTime(endTime);
// Execute the FilterLogEventsRequest and process the log events returned by it
ArrayList<String> messages = new ArrayList<>();
do {
FilterLogEventsResult filterLogEventsResult = logsClient.filterLogEvents(filterLogEventsRequest);
for (FilteredLogEvent event : filterLogEventsResult.getEvents()) {
String message = event.getMessage().trim();
messages.add(message);
}
filterLogEventsRequest.setNextToken(filterLogEventsResult.getNextToken());
} while (filterLogEventsRequest.getNextToken() != null);
return messages;
}The retrieveCloudWatchLogs method is the main method responsible for retrieving logs from the AWS CloudWatch service. It takes in three arguments: logGroupName for specifying the log group name to be searched, within for specifying the number of minutes of logs to retrieve, and pattern for filtering the log events based on a specified pattern.
The method first checks if the code is running on AWS CodeBuild or not. If it is running on CodeBuild, it uses the default AWSLogs client, otherwise it sets up the client by calling the setUpCloudWatchClient() method, which we discussed earlier.
After setting up the client, the method creates a FilterLogEventsRequest object and sets the log group name, time range, and filter pattern. The method then executes the request and processes the log events returned by it. It uses a do-while loop to retrieve all the log events, since AWS CloudWatch returns log events in batches.
For each log event retrieved, the method extracts the log message and adds it to an ArrayList of log messages. Finally, the method returns the list of log messages.
This method is the backbone of the tutorial, as it demonstrates how to retrieve logs from AWS CloudWatch using Java.
Step 4: Writing the test
Finally, let’s write a test:
@Test
public void testRetrieveAwsCloudWatchLogs() {
String logGroupName = "/aws/lambda/hello";
ArrayList<String> messages = AwsUtil.retrieveCloudWatchLogs(logGroupName, 60, "abc");
assertTrue(messages.size() > 0);
System.out.println("There are " + messages.size() + " messages fetched from CloudWatch");
int count = 1;
for (String message : messages) {
System.out.println("* [" + count + "]: " + message);
count++;
}
}This test method is used to validate the retrieveCloudWatchLogs() method by fetching and asserting that there are logs returned from the specified logGroupName within the last 60 minutes and containing the specified pattern.
The assertTrue() assertion is used to check if there are any logs returned from the CloudWatch logs. If there are, the test prints out the number of logs retrieved and each log message with a count.
This test can be run locally or on AWS CodeBuild to ensure that the retrieveCloudWatchLogs() method is working correctly.
Conclusion
In conclusion, checking AWS CloudWatch logs using Java can be a valuable tool for troubleshooting and analyzing application behavior. In this tutorial, we covered the steps required to retrieve logs using Java, including configuring SAML2AWS, setting up local credentials, and creating a CloudWatch client. We also discussed how to write a simple test to verify the log retrieval process. With these tools and techniques, you can easily monitor your application's logs and detect issues before they become critical problems. By taking advantage of AWS's powerful logging and analysis capabilities, you can ensure the smooth operation of your applications and provide a better experience for your users.
Loading...