Imagine the following scenario: You are a developer on a team of serverless developers who use AWS API Gateway for their APIs. While CloudFormation is commonly used to deploy stacks, CloudFormation macros are less popular. You receive a request from your project manager to monitor the duration of each API call individually, so that the team can identify poor performance. Currently, you have CloudWatch monitoring your APIs, but they only give you a look at the overall health of the API, not for individual calls.

After doing some research, you realize that there is in fact a built-in setting for enabling detailed metrics in API Gateway:

Red circle around enabled detailed cloudwatch metrics
Red circle around enabled detailed cloudwatch metrics

Red circle around enabled detailed cloudwatch metrics

Since your organization is very large and there are over fifty APIs, a programmatic solution is warranted. This is beneficial not only for the sake of speed, but also for repeatability and other robust benefits of writing infrastructure-as-code.

There is a property which you can add to an existing AWS::ApiGateway::Stage resource that will effectively select this box for you. This will enable detailed CloudWatch metrics on each endpoint. Since each of the services in your organization has something like this resource in their
CloudFormation template:

RestApiStage:
 Type: "AWS::ApiGateway::Stage"
 Properties:
   DeploymentId: !Ref RestApiDeployment
   RestApiId: !Ref RestApi
   TracingEnabled: true
   StageName: "api"

You will be able to add the new property that you discovered to each of the templates, changing it to:

RestApiStage:
 Type: "AWS::ApiGateway::Stage"
 Properties:
   DeploymentId: !Ref RestApiDeployment
   RestApiId: !Ref RestApi
   TracingEnabled: true
   StageName: "api"
   MethodSettings:
     - ResourcePath: '/*'
       HttpMethod: '*'
       MetricsEnabled: 'true'

Clearly, in an organization with many services, you want to avoid doing this manually. A CloudFormation macro is the perfect solution.

CloudFormation Macros

Architecture of Cloudformation Macro
Architecture of Cloudformation Macro

Architecture of Cloudformation Macro

CloudFormation Macros are an excellent way to cut down on boilerplate infrastructure-as-code. They allow developers to reuse and transform CloudFormation templates by editing existing resources or injecting new ones into them. The architecture is as follows:

  1. A developer uploads a template to CloudFormation which contains the Transform property so that CloudFormation knows the template must be transformed by a macro before deployment.

  2. CloudFormation sends the template to the macro lambda specified in the template.

  3. The lambda returns a transformed template to CloudFormation.

  4. CloudFormation deploys the transformed template.

At Nuvalence, we use this technology to simplify the creation of CloudWatch alarms across our services. In this article, we’re going to look at the relatively simple example of editing existing resources in order to enhance their observability.

Macro Implementation

Consumers are going to interface with your macro by adding the Transform property to their template and providing it with the name of the macro.

AWSTemplateFormatVersion: "2010-09-09"
Description: Sample
Transform: NuvalenceExampleMacro
Resources:
 # list of resources, etc.

Now that we know how the macro will be consumed, we can implement the actual lambda. I will use TypeScript in this example. First, I created types to match the CloudFormation macro interface, which is the following:

export interface CloudFormationTransformRequest {
   requestId: number
   fragment: Template
}

export interface CloudFormationTransformResponse {
   requestId: number
   status: string
   fragment: Template
}

Note the fragment field models a CloudFormation template. Also, note that the status field is what tells CloudFormation whether the transform was a success or not.

For completeness, I define Template and related types here:

export interface Template {
   Transform?: string
   Resources: Array<Resource<any>>
}

export interface Resource<T> {
   name: string,
   attributes: ResourceAttributes<T>
}

export interface ResourceAttributes<T> {
   Condition?: string
   Type: string
   Properties: T
}

Using the above-defined CloudFormation types, you will create a handler for your lambda that looks something like this:

import {CloudFormationTransformRequest, CloudFormationTransformResponse} from "./models/cloudformation/CloudFormationTransforms";
import {Template} from "./models/cloudformation/CloudFormationTemplateTypes";
import ApiGatewayStageTransformer from "./services/ApiGatewayStageTransformer";

export default class Handler {
   private apiGatewayStageTransformer: ApiGatewayStageTransformer;

   constructor() {
       this.apiGatewayStageTransformer = new ApiGatewayStageTransformer();
   }

   public handle = async (event: CloudFormationTransformRequest): Promise<CloudFormationTransformResponse> => {
       try {
           let templateClone: Template = event.fragment;
           const transformedTemplate = this.apiGatewayStageTransformer.transform(templateClone);
           return {
               "requestId": event.requestId,
               "status": "success",
               "fragment": transformedTemplate
           };
       } catch (e) {
           console.log(e);
           return {
               "requestId": event.requestId,
               "status": "failure",
               "fragment": event.fragment
           }
       }
   };
}

export const handler = new Handler().handle;

Here we take in a CloudFormationTransformRequest, pull the template fragment off of the event, and then transform it before sending it back.

Implementation of the transform itself:

export default class ApiGatewayStageTransformer {
   public transform(template: Template): Template {
   // Get all AWS::ApiGateway::Stage and append new property.
   Object.values(template.Resources)
       .filter(resource => resource.attributes.Type === 'AWS::ApiGateway::Stage')
       .forEach(stage => {
           stage.attributes.Properties.MethodSettings = {
               "ResourcePath": '/*',
               "HttpMethod": '*',
               "MetricsEnabled": "true"
           };
       });
   return template;
  }
}

This method will parse all resources of type AWS::ApiGateway::Stage from the template and modify them so that they have the MethodSettings property and all necessary settings so that detailed CloudWatch metrics will be enabled on each endpoints individually.

Macro Deployment

When you first deploy a macro, you need to make CloudFormation aware of the macro name in addition to deploying the macro lambda itself. (The template below assumes you have uploaded your macro code to an S3 bucket called macro-bucket and named your distribution dist.zip in the root of that directory.)

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  MacroVersion:
   Type: "AWS::SSM::Parameter::Value<String>"
   Default: "macro-version"
Resources:
 Macro:
   Type: AWS::CloudFormation::Macro
   Properties:
     Description: "CloudFormation Macro example"
     FunctionName: !GetAtt MacroLambda.Arn
     Name: "NuvalenceExampleMacro"
 MacroLambda:
   Type: AWS::Lambda::Function
   Properties:
     Code:
       S3Bucket: "macro-bucket"
       S3Key: "/dist.zip"
     FunctionName: "NuvalenceExampleMacro"
     Handler: "dist/index.handler"
     Runtime: nodejs12.x
     MemorySize: 1024
     Role: !Sub "arn:aws:iam::${AWS::AccountId}:role/lambda_basic_execution"
     Timeout: 20
     Tags:
       - Key: "version"
         Value: !Ref MacroVersion

The AWS::CloudFormation::Macro resource is responsible for making the macro callable via the Transform property, and the lambda does the actual transform. Deploying this will enable your consumers to find and use your CloudFormation macro.

Macro Consumption

Now that you’ve deployed your macro, you can inform other teams that simply by adding the right Transform property at the top of their CloudFormation template, they can enable detailed monitoring of their API Gateway endpoints.

One can imagine the wealth of possibilities that can be achieved using CloudFormation macros: automatic CloudWatch alarm creation, registration of a service to a database based only on values provided in the template, or doing a find-and-replace on the subscribers of an SNS topic so that stakeholders can ensure all team members are subscribed to the topic to name a few.

We hope we have sparked some new ideas to help better manage your AWS Services.