14th September 2020 | Written by Zachary Sedefian

Using CloudFormation Macros to create repeatable solutions


Imagine the following scenario: You are a developer on a team of serverless developers who use AWS API Gateway for their APIs. While CloudFormation is commonly used to deploy stacks, CloudFormation macros are less popular. You receive a request from your project manager to monitor the duration of each API call individually, so that the team can identify poor performance. Currently, you have CloudWatch monitoring your APIs, but they only give you a look at the overall health of the API, not for individual calls.

After doing some research, you realize that there is in fact a built-in setting for enabling detailed metrics in API Gateway:


Red circle around enabled detailed cloudwatch metrics


Since your organization is very large and there are over fifty APIs, a programmatic solution is warranted. This is beneficial not only for the sake of speed, but also for repeatability and other robust benefits of writing infrastructure-as-code.

There is a property which you can add to an existing AWS::ApiGateway::Stage resource that will effectively select this box for you. This will enable detailed CloudWatch metrics on each endpoint. Since each of the services in your organization has something like this resource in their CloudFormation template:

RestApiStage:
Type: "AWS::ApiGateway::Stage"
Properties:
  DeploymentId: !Ref RestApiDeployment
  RestApiId: !Ref RestApi
  TracingEnabled: true
  StageName: "api"

you will be able to add the new property that you discovered to each of the templates, changing it to:

RestApiStage:
Type: "AWS::ApiGateway::Stage"
Properties:
  DeploymentId: !Ref RestApiDeployment
  RestApiId: !Ref RestApi
  TracingEnabled: true
  StageName: "api"
  MethodSettings:
    - ResourcePath: '/*'
      HttpMethod: '*'
      MetricsEnabled: 'true'

Clearly, in an organization with many services, you want to avoid doing this manually. A CloudFormation macro is the perfect solution.

CloudFormation Macros

CloudFormation Macros are an excellent way to cut down on boilerplate infrastructure-as-code. They allow developers to reuse and transform CloudFormation templates by editing existing resources or injecting new ones into them. The architecture is as follows:

architecture of cloudformation macro

  1. A developer uploads a template to CloudFormation which contains the Transform property so that CloudFormation knows the template must be transformed by a macro before deployment.
  2. CloudFormation sends the template to the macro lambda specified in the template.
  3. The lambda returns a transformed template to CloudFormation.
  4. CloudFormation deploys the transformed template.


At Nuvalence, we use this technology to simplify the creation of CloudWatch alarms across our services. In this article, we’re going to look at the relatively simple example of editing existing resources in order to enhance their observability.

Macro implementation

Consumers are going to interface with your macro by adding the Transform property to their template and providing it with the name of the macro.

AWSTemplateFormatVersion: "2010-09-09"
Description: Sample
Transform: NuvalenceExampleMacro
Resources:
# list of resources, etc.

Now that we know how the macro will be consumed, we can implement the actual lambda. I will use TypeScript in this example. First, I created types to match the CloudFormation macro interface, which is the following: 

export interface CloudFormationTransformRequest {
  requestId: number
  fragment: Template
}

export interface CloudFormationTransformResponse {
  requestId: number
  status: string
  fragment: Template
}

Note the fragment field models a CloudFormation template. Also note that the status field is what tells CloudFormation whether the transform was a success or not.

For completeness, I define Template and related types here:

export interface Template {
  Transform?: string
  Resources: Array<Resource<any>>
}

export interface Resource<T> {
  name: string,
  attributes: ResourceAttributes<T>
}

export interface ResourceAttributes<T> {
  Condition?: string
  Type: string
  Properties: T
}

Using the above-defined CloudFormation types, you will create a handler for your lambda that looks something like this:

import {CloudFormationTransformRequest, CloudFormationTransformResponse} from "./models/cloudformation/CloudFormationTransforms";
import {Template} from "./models/cloudformation/CloudFormationTemplateTypes";
import ApiGatewayStageTransformer from "./services/ApiGatewayStageTransformer";

export default class Handler {
  private apiGatewayStageTransformer: ApiGatewayStageTransformer;

  constructor() {
      this.apiGatewayStageTransformer = new ApiGatewayStageTransformer();
  }

  public handle = async (event: CloudFormationTransformRequest): Promise<CloudFormationTransformResponse> => {
      try {
          let templateClone: Template = event.fragment;
          const transformedTemplate = this.apiGatewayStageTransformer.transform(templateClone);
          return {
              "requestId": event.requestId,
              "status": "success",
              "fragment": transformedTemplate
          };
      } catch (e) {
          console.log(e);
          return {
              "requestId": event.requestId,
              "status": "failure",
              "fragment": event.fragment
          }
      }
  };
}

export const handler = new Handler().handle;

Here we take in a CloudFormationTransformRequest, pull the template fragment off of the event, and then transform it before sending it back.

Implementation of the transform itself:

export default class ApiGatewayStageTransformer {
  public transform(template: Template): Template {
  // Get all AWS::ApiGateway::Stage and append new property.
  Object.values(template.Resources)
      .filter(resource => resource.attributes.Type === 'AWS::ApiGateway::Stage')
      .forEach(stage => {
          stage.attributes.Properties.MethodSettings = {
              "ResourcePath": '/*',
              "HttpMethod": '*',
              "MetricsEnabled": "true"
          };
      });
  return template;
  }
}


This method will parse all resources of type AWS::ApiGateway::Stage from the template and modify them so that they have the MethodSettings property and all necessary settings so that detailed CloudWatch metrics will be enabled on each endpoints individually.


Macro deployment

When you first deploy a macro, you need to make CloudFormation aware of the macro name in addition to deploying the macro lambda itself. (The template below assumes you have uploaded your macro code to an S3 bucket called macro-bucket and named your distribution dist.zip in the root of that directory.)

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  MacroVersion:
  Type: "AWS::SSM::Parameter::Value<String>"
  Default: "macro-version"
Resources:
Macro:
  Type: AWS::CloudFormation::Macro
  Properties:
    Description: "CloudFormation Macro example"
    FunctionName: !GetAtt MacroLambda.Arn
    Name: "NuvalenceExampleMacro"
MacroLambda:
  Type: AWS::Lambda::Function
  Properties:
    Code:
      S3Bucket: "macro-bucket"
      S3Key: "/dist.zip"
    FunctionName: "NuvalenceExampleMacro"
    Handler: "dist/index.handler"
    Runtime: nodejs12.x
    MemorySize: 1024
    Role: !Sub "arn:aws:iam::${AWS::AccountId}:role/lambda_basic_execution"
    Timeout: 20
    Tags:
      - Key: "version"
        Value: !Ref MacroVersion

 

Macro consumption

The AWS::CloudFormation::Macro resource is responsible for making the macro callable via the Transform property, and the lambda does the actual transform. Deploying this will enable your consumers to find and use your CloudFormation macro.

Now that you’ve deployed your macro, you can inform other teams that simply by adding the right Transform property at the top of their CloudFormation template, they can enable detailed monitoring of their API Gateway endpoints.

One can imagine the wealth of possibilities that can be achieved using CloudFormation macros: automatic CloudWatch alarm creation, registration of a service to a database based only on values provided in the template, or doing a find-and-replace on the subscribers of an SNS topic so that stakeholders can ensure all team members are subscribed to the topic to name a few.

We hope we have sparked some new ideas to help better manage your AWS Services.


Leave a Reply

Your e-mail address will not be published. Required fields are marked *

Related Articles