Tracing Lambda with AWS X-Ray (using Powertools)

 


Working in cloud and in general in a distributed system, means that code is executed among different services which have different loggers and maybe different location where this log are placed.

To understand the code flow (because if you have a problem, maybe you can't read all the code), you need an instrument to build who is calling what and when in order to check which is the bottleneck or where your error is located.

If you have a legacy application, the way is to create such a trace information spread all over different calls (which means using a Logger Context for internal and header for external call) and then use a log collector to visualize all the information, grouped by this trace.

If you are working with a Spring application there are some libraries that do it for you: you add them in maven dependencies and collect logs with specific tools (kibana or similar) and then you have the flow.

You can also use some agent (like datadog agent) that collect information directly from JVM and send information to a collector: this is of course the most powerful tracer but it is so verbose because it trace really everything.

All of this methods have one thing in common: they suppose to have an infrastructure to collect the logs and to visualize them, which is not always possible, especially if you haven't Devops for you (maybe you are a startup). So which is the solution? X-Ray is here for you


X-RAY - INTRODUCTION

X-Ray is just a common trace utility that AWS offers (not for free) but at very cheap costs. It collect trace information that comes out from your code and then give you a nice way to build all the flow of your different calls among different services.

X-Ray can be turned on in your Lambda (or your application running in other services) but you need to fill it, it is not just about writing Logs in a certain ways


CASE STUDY

Suppose we have 2 different Lambda applications, both triggered from ApiGateway. The first Api has 2 different method:

  • POST which accepts a user and insert it into DynamoDB table
  • GET which scan the DynamoDB table and then - for each element - call the second Api
The second Api receive a User and do nothing, just create a response message and return it back.

We want to trace all the connected system through a call to see the interaction.

For this purpose we will use X-Ray but not directly but using a wrapper offered by Powertools.

Let's integrate in our application

POM.XML

To integrate it in the first API we will import the powertools-tracing dependency

<dependency>
    <groupId>software.amazon.lambda</groupId>
    <artifactId>powertools-tracing</artifactId>
    <version>1.17.0</version>
</dependency>

and the aspectj configuration. Why aspectj? because the code need to be packaged at compile time, you haven't proxy like Spring

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>aspectj-maven-plugin</artifactId>
    <version>1.14.0</version>
    <configuration>
        <source>16</source>
        <target>16</target>
        <complianceLevel>${maven.compiler.target}</complianceLevel>
        <aspectLibraries>
            <aspectLibrary>
                <groupId>software.amazon.lambda</groupId>
                <artifactId>powertools-tracing</artifactId>
            </aspectLibrary>
        </aspectLibraries>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
            </goals>
        </execution>
    </executions>

This allows us to use the powertools. By the way we need to pass tracing values also to external call, so we need a dedicated apache-http client that does it for us. For this we need to also use this dependency

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-xray-recorder-sdk-apache-http</artifactId>
    <version>2.14.0</version>
</dependency>

In the Lambda we will also use the DynamoDb sdk and the apache-client library from aws-sdk

For testing purpose we also need to add a variable that is used from the library

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.1.2</version>
    <configuration>
        <environmentVariables>
            <LAMBDA_TASK_ROOT>handler</LAMBDA_TASK_ROOT>
        </environmentVariables>
    </configuration>
</plugin>

THE CODE

The usage is very simple: @Tracing annotation can be used. It just trace it to X-Ray. There is the captureMode attribute which can be used to change the behavior


public enum CaptureMode {
    RESPONSE,
    ERROR,
    RESPONSE_AND_ERROR,
    DISABLED,
    ENVIRONMENT_VAR;

    private CaptureMode() {
    }
}

Easy to understand. The environment_var means you will pass the values to environment for lambda.

X-Ray give you the ability to add annotation and Metadata: the annotation is a dedicate section, the metadata is the one used directly by Tracing. To add value you can use

TracingUtils.putAnnotation("annotation", "JustAnAnnotation");
TracingUtils.putMetadata("content", "JustAmetadataContent");

The second one add the value to the metadata filled by the next trace

In our case:

@Logging(logEvent = true, correlationIdPath = "/headers/trace-header")
@Tracing(captureMode = CaptureMode.RESPONSE)
public APIGatewayProxyResponseEvent handleRequest(final APIGatewayProxyRequestEvent input

The App.java trace the handleRequest method. Then it calls a serveRequest method, traced again

@Tracing(captureMode = CaptureMode.RESPONSE_AND_ERROR)
private String serveRequest(String requestType, String body) throws IOException{

Then a FindService is called, and all the method are annotated with Tracing. There is another service - DownstreamService which purpose is to perform the call. It is annotated and it will use the special HttpClientBuilder from x-ray library

package crud.downstream;

import com.amazonaws.xray.proxies.apache.http.HttpClientBuilder;
import com.fasterxml.jackson.databind.ObjectMapper;
import crud.User;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import software.amazon.lambda.powertools.tracing.CaptureMode;
import software.amazon.lambda.powertools.tracing.Tracing;

import java.io.IOException;
import java.io.InputStream;

public class DownstreamService {

    private static final Logger log = LogManager.getLogger(DownstreamService.class);
    private static ObjectMapper objectMapper = new ObjectMapper();
    private DownstreamService(){}

    public static DownstreamService of(){
        return new DownstreamService();
    }

    @Tracing(captureMode = CaptureMode.RESPONSE_AND_ERROR)
    public SimpleResponse callExternalService(User user) throws IOException {
        CloseableHttpClient httpclient = HttpClientBuilder.create().build();
        HttpPost httpPost = new HttpPost("xxxxxxxxxxx");
        final StringEntity stringEntity = new StringEntity(objectMapper.writeValueAsString(user));
        httpPost.setEntity(stringEntity);
        CloseableHttpResponse response = httpclient.execute(httpPost);
        try {
            log.debug("Try to call");
            HttpEntity entity = response.getEntity();
            InputStream inputStream = entity.getContent();
            SimpleResponse simpleResponse = objectMapper.readValue(inputStream, SimpleResponse.class);
            return simpleResponse;
        } finally {
            response.close();
        }

    }
}

TEMPLATE

In cloudFormation template we need to add the Action Policy for Lambda to let it write on xray:


- PolicyName: AWSXrayWriteOnlyAccess
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - 'xray:PutTraceSegments'
          - 'xray:PutTelemetryRecords'
          - 'xray:GetSamplingRules'
          - 'xray:GetSamplingTargets'
          - 'xray:GetSamplingStatisticSummaries'
        Resource: '*'


For the Function we need to pass the service name for powertools service. In the example also how to use the environment variable to trace response and error


Environment:
  Variables:
    JAVA_TOOL_OPTIONS: -XX:+TieredCompilation -XX:TieredStopAtLevel=1
    POWERTOOLS_SERVICE_NAME: CrudDatabaseApi
    POWERTOOLS_TRACER_CAPTURE_RESPONSE: true
    POWERTOOLS_TRACER_CAPTURE_ERROR: true

The other API has pretty the same configuration but of course has different classes.

TESTING X-RAY

If we perform the call we will see the interaction between each service.

Here the visualization


Down you can find each single Trace


Also grouped by single service



CONCLUSION

It is a very easy way to trace information and flow execution using Powertools. Using just annotation you can trace whatever you need in a json format and have the flow of involved services.

There is a good start point if you haven't any other infrastructure then... AWS




Commenti

Posta un commento

Post popolari in questo blog

HTML and AI - A simple request, and a beautiful response for ElasticSearch without Kibana

Immagine
THE PROBEM I recently installed an ElasticSearch in a Docker environment and I want to be sure that it is working, before waiting the consumers start to connect to it. The Container is started using a volume which was cloned from another Docker installation, so it was not empty: we already have some indexes and some documents. I want to be sure that everything was available but I don't have any Kibana installation (and I can't install it at all). So only API was available and I need to check something  ElasticSearch REST APIs THE FIRST SOLUTION If you ask to Microsoft Copilot how to interact with ElasticSearch api, it will reply by suggesting: curl scripting postman But the interaction is not so human readable and it is quite noisy to write the commands. I want to have something to be easily used but I don't want to spend to much time on that. I need an accelerator THE SECOND SOLUTION I know that there is such an api to ask to ElasticSearch the list of indexes. And I also k...
Continua a leggere

A simple CD using AWS Pipeline

Immagine
  CD is a friend in software development: there is no other way to speed up feedback from your application without having a solid Continuous Deployment. I am skipping continuous integration because all of this examples are related to single repo.. where the only developer allowed to commit code it's me: the integration is with my self :) In this example we will create an automation related to a simple situation where we have a LambdaFunction exposed through an ApiGateway, where the lambda function is running a docker image. There are infinite ways of creating and managing Lambda and pipelines of course, but in this example we will use: CodeCommit to store the code ECR to store the code CloudFormation to create and update all the infrastructure. CodeBuild to create the artifacts we need Let's see it CODE COMMIT     To easily have a repository to store the code and to have a natural integration with AWS echosystem, I decided to use Code commit. Code is structured as other e...
Continua a leggere

Websocket Chat with Lambda (but all in Java)

Immagine
  I don't know if I am not able to perform search on Internet (I doubt it) but every examples of websocket chat created with Lambda was all created using Node.js. And I can cheat on Java, that's my love. So I decided to transform the usual chat into a Java project, by of course performing some changes in order to be easy to check if it works.  THE IDEA The idea behind the project is simple: a simple WebSocket API exposed with api gateway with 4 different route and a unique Lambda who is in charge to serve each request. The change I made is about the name registration: instead of just propagate the message to other connected users, I want to have the way to register a name for chat users so every time a message is published, the receiver can see who sent the message. As I said in other project, this is to be intended for testing purpose, so there are really infinite ways to do have the same behavior maybe in a better way. THE PROJECT The entire project is based on cloud formati...
Continua a leggere