Today we are starting an infrequent blog series called AWS Tips and Tricks. This series will focus on useful lessons learned throughout our AWS journey ranging from the simplest tricks to more sophisticated solutions that we feel might be useful to the everyday Cloud enthusiast. To kick off this series, we take a look at how to avoid getting the ThrottlingException when fetching Systems Manager parameters in your Lambda functions. We explore a more cost effective and efficient way of using secure string parameters without sacrificing security.
The ThrottlingException problem!
AWS Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management. We usually use it to store our passwords and secrets. We then fetch those secrets from our Lambda functions using an API call (GetParameter). This is fine for most simple workloads. But when your function is getting invoked more than hundreds or thousands of times a second, you will quickly run into the dreaded ThrottlingException error.
You get this error because of a transactions per second (TPS) limit that AWS enforces. This limit is shared across the AWS account and region. The limit stands at 1000 TPS now but it used to be 40 TPS.
So how do we work around this problem? You could enable higher throughput limits but lots of API calls comes with additional cost. The below table shows the pricing.
We decided to go a different route. Since we use CloudFormation templates to create Lambda functions, we thought of simply passing the parameter value to the function as an environment variable. Unfortunately, CloudFormation does not support Secure String as parameter type. You can not use it as input parameter like you can with String and String List.
The Solution
CloudFormation Custom Resource.
Custom resource enables you to create your own provisioning logic and have CloudFormation run it during a stack operation, such as when you create, update or delete a stack. This is especially useful for resource types that are not supported by CloudFormation. In this case, we use Custom Resource to fetch a SecureString parameter from Systems Manager Parameter Store. We then pass that parameter to our Lambda function.
Essentially, by doing this we invoke the GetParameter API once instead of potentially thousand times. This avoids the ThrottlingException error. It also reduces the number of API calls to almost zero!
How it works
We will show it all in code so it’s easier to understand.
Create a Lambda function. We name this function secure-string-function. This lambda function receives and responds to requests from CloudFormation when you use the Custom Resource type in your template. In the CloudFormation template, we will point to this function when defining our Custom resource.
Here is the code. Note that this is the simplified version. Explaining how the code works is not part of this post but we might do a deep dive about it. Stay tuned!
import os
import urllib3
import json
import boto3
def lambda_handler(event, context):
'''
Main Lambda hander - evaluates control and makes decisions
'''
get_secure_string(event,context)
def get_secure_string(event,context):
try:
param_name = event['ResourceProperties']['ParameterName']
param_val = get_ssm_param(param_name)
response_data = {
"ParameterValue" : param_val
}
send(event, context, "SUCCESS", response_data,"secureStringLambda",True)
except Exception as e:
response_data = {"Param": "GetParamError"}
print(e)
send(event, context, "FAILED", response_data,"secureStringLambda",True)
def get_ssm_param(param_name):
'''
Returns value from AWS SSM parameter store.
'''
client = boto3.client('ssm')
response = client.get_parameter(Name=param_name, WithDecryption=True)
print("param: {}".format(response["Parameter"]["Value"]))
if not 'Parameter' in response or not len(response['Parameter']):
print(f'ssm: {param_name} was not found')
return "ParameterNotFoundError"
return response["Parameter"]["Value"]
def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=True):
responseUrl = event['ResponseURL']
print(responseUrl)
responseBody = {}
responseBody['Status'] = responseStatus
responseBody['Reason'] = ""
responseBody['PhysicalResourceId'] = physicalResourceId
responseBody['StackId'] = event['StackId']
responseBody['RequestId'] = event['RequestId']
responseBody['LogicalResourceId'] = event['LogicalResourceId']
responseBody['NoEcho'] = noEcho
responseBody['Data'] = responseData
json_responseBody = json.dumps(responseBody)
print("Response body:n" + json_responseBody)
headers = {
'content-type' : '',
'content-length' : str(len(json_responseBody))
}
try:
http = urllib3.PoolManager()
encoded_data = json_responseBody.encode('utf-8')
response = http.request('PUT',responseUrl,body=encoded_data,headers=headers)
print("Status code: " + response.reason)
except Exception as e:
raise(e)
Define your CloudFormation template with a custom resource.
- To define a Custom Resource we use the type AWS::CloudFormation::CustomResource or Custom::MyCustomResourceTypeName;
- Add the following Parameter and Resource definition in your template to Get the secure string;
- MyParameterName is the parameter name for which we are storing a secure value;
- SecureStringLambdaName is the name of the Lambda function that we created in the previous step.
Parameters:
MyParameterName:
Description: Parameter name to get the value of.
Type: String
SecureStringLambdaName:
Description: Name of the Secure String Lambda function
Type: String
Default: secure-string-function
Resources:
GetSecureStringResource:
Type: Custom::SecureStringFunction
Properties:
ServiceToken: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${SecureStringLambdaName}"
ParameterName: !Ref MyParameterName
You can then use the ParameterValue in your template as you desire. In this case, we pass this ParameterValue as an environment variable to a Lambda function.
Resources:
MyFunction:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables: MY_SECURE_STRING: !GetAtt GetSecureStringResource.ParameterValue
That’s it! You now have a simple solution to fetch a Secure String from Systems Manager parameter store and use that parameter in your CloudFormation templates.
We hope you find this post informative. If you require any further assistance, please do not hesitate to contact us at RedBear.
