For some time now it is possible to add Layers to a Lambda function which allows you to use additional libraries and dependencies in your functions without the need to include these in your deployment pacakge. This is for instance very convenient when using Cloudformation to manage your Lambda functions. In most cases this means you can simply point to a single file instead of creating a deployment package each time.
By using Lambda Layers we get:
- all functions are using the same library versions
- easy to use by attaching it to the function
- simplify the deployment with Cloudformation (AWS SAM)
A specific use-case we had was the boto3 library that we use very often to automate all kinds of tasks. The default boto3 library provided when creating a Python based Lambda function is quite old and doesn’t support all the latest features and services released by AWS.
You can also add multiple Lambda Layers to a single Lambda Function which is great to split up functionality (ex. a layer for boto3 and another one for your own libraries etc..). Lambda Layers can also be versioned which makes it easy to transition from one version to the other. You can also add permissions to a Lambda Layer. More info on Lambda Layers can be found here
The Cloudformation template and Makefile used in the example below can be found in this Github repository
Adding CI
Because building and pushing these layers from our local machine is quite cumbersome, time consuming and generally not a best practice in days where CI/CD is (or should be) the default, we added CI to automate this using AWS CodeCommit and CodeBuild.
Setup
We have choosen to use CodeBuild which will be triggered by a CodeCommit
repo where we keep the source. All build and publish actions are managed by a
Makefile which will be used in the CodeBuild build process.
make deploy-ci
This will deploy a Cloudformation stack which creates the followng resources:
- CodeCommit git repository
- Codebuild Project
- IAM Role with inline Policy
When the Cloudformation stack finishes successfully the CodeCommit http url will
be shown. You can then commit the Makefile and buildspec.yml to this new
repository and start a build of the CodeBuild project.
CodeBuild: Build Lambda Layer
Creating a Lambda Layer is easy to do. Create a folder based on your runtime (ex: ‘python’) and add your libraries to that folder using your language’s package manager (ex pip for Python).
Example from the Makefile:
build: ## PIPELINE COMMAND: build lambda layer
mkdir -p build/python
pip install boto3 -t ./build/python
cd ./build && zip -r ../$(LAYER_NAME).zip .
CodeBuild: Publish Lambda Layer
Once we have have created our Lambda Layer we can publish it so it becomes available for use with our Lambda functions. Each publish action will increment the layer’s version number.
Example from the Makefile:
publish:
@aws lambda publish-layer-version \
--layer-name test-boto3 \
--description "boto3 1.9.125" \
--zip-file fileb://$(LAYER_NAME).zip \
--compatible-runtimes python3.6
Updating boto3 version
When a new version of boto3 is released you can update the BOTO_VERSION
variable in the Makefile, commit your changes and run a new build.
After the build a new Lambda layer version is created with the boto version in
the description.
This is a rather basic example using only the boto3 sdk but you can use this as a base to other sdk’s and improve the build process.