Consolidate your NAT Gateways with Transit Gateway and save some money

At ReInvent of 2018 AWS announced Transit Gateway. Transit Gateway is a managed service that allows you to connect multiple VPC’s from different accounts and on-prem connections through a single endpoint.

People who manage more then just a few accounts and need on-prem access during their transition to the cloud often use VPN tunnels, Direct Connect or other solutions provided by 3rd party providers like Cisco. Add VPC peering to that mix and you end up with quite a complex setup you need to manage. Transit Gateway can simplify this.

These images from the reInvent slides illustrate this in the case of VPN tunnels.

Optimise cost

At the customer where I’m currently working we have around 80 accounts and this number will probably increase further. So when Transit Gateway became available we were keen to test it to find out how it could reduce complexity. But reducing complexity was not the only factor, another one was cost.

From the current 80 accounts there are around 50 that have one or more VPC’s each with their own NAT gateway and VPN connection to on-prem. If you do the calculation of the monthly cost you end up with a rather high number.

Example

  • Price per NAT gateway / hour = $0.048 ($1.15/day -> $34.56/month)
  • $34.56 x 6 (2 VPC’s with subnets in 3AZs) * 50 accounts = $10368/month

This could be reduced to $34.56 x 3 = $103.68. Nice savings..

Transit Gateway acts as a central place to connect networks (basically route tables) so we came up with the idea of using NAT Gateways in the Transit Gateway account for all the attached accounts instead of using NAT Gateways in all the individual accounts.

I tried to describe the necessary steps to setup central NAT Gateways using Transit Gateway as clear as I could. The setup is actually quite simple but it might take some time to get your head around it.

Setup

Inspired by the Well Architected framework we decided to put all the Transit Gateway(s) in a separate AWS account (the Transit Gateway Account). This keeps things clean and we can limit access to this account which improves security which is imporatant since this will be a crucial part of our infrastructure.

This diagram might help to understand the setup.

Setup a Transit Gateway

NOTE: The following steps are done in the Transit Gateway account!

Creating a Transit Gateway is rather straightforward. The following example creates a Transit Gateway with the default settings using the CLI.

aws ec2 create-transit-gateway --description tgw1

Once created you can get the details by running

aws ec2 describe-transit-gateways

To make the Transit Gateway visible in your other account(s) we need to share it using Resource Access Manager. Specify the AWS Account ID with the --principals parameter

aws ram create-resource-share \
    --name FirstTGW \
    --resource-arns arn:aws:ec2:eu-west-1:120987654321:transit-gateway/tgw-095ebf9f67a91882a \
    --principals "123456789012"

NOTE: You can also share resources on an Organizational level which makes the resource available without the Pending Accept phase.

Configure a VPC to use Transit Gateway

NOTE: The following steps are done in a regular account!

Now that the Transit Gateway is created and shared we can attach a VPC from the account we shared the Transit Gateway with.

Check resource share invitations

Run the following command to get a list of resource share invitations.

 aws ram get-resource-share-invitations

Accept the resource share

Use the resourceSahreInvitationArn property from the output in the previous step.

aws ram accept-resource-share-invitation \
    --resource-share-invitation-arn <resourceSahreInvitationArn>

Get the Transit Gateway info

Once we accepted the resource share we should see a Transit Gateway resource appearing in our account.

aws ec2 describe-transit-gateways

Create the attachment

Now we can attach the private subnets from our VPC to the Transit Gateway. The transit-gateway-id property can be found in the output of the previous command.

aws ec2 create-transit-gateway-vpc-attachment \
    --transit-gateway-id tgw-095ebf9f67a91292a \
    --vpc-id vpc-bac5c8dc \
    --subnet-ids subnet-03184365 subnet-0761d95d

Accept the attachment from the Transit Gateway account.

NOTE: The following steps are done in the Transit Gateway account!

First list the pending attachments

aws ec2 describe-transit-gateway-attachments --filters Name=state,Values=pendingAcceptance

Accept the attachment. The transit-gateway-attachment-id can be found in the output of the previous step. This can take some time.

aws ec2 accept-transit-gateway-vpc-attachment  \
    --transit-gateway-attachment-id tgw-attach-01c82551a66514b6c

Ok. If you were able to follow this without too much trouble you did a good job already. Almost there..

Transit Gateway VPC

To provide outgoing internet connectivity (NAT) we need to create a VPC with public and private subnets in the same AZ’s as we have in our other accounts. This is important because an instance in a given AZ will use the NAT Gateway in the Transit Gateway account that is deployed in the same AZ.

example: instance in AZ ID ‘euw1-az3’ will use a NAT Gateway from the Transit Gateway account which is also deployed in euw1-az3.

I leave the deployment of the VPC as an exercise for the reader, but you should have public subnets with a default route to an Internet Gateway and private subnets each with a default route to a NAT Gateway. This is pretty standard and should not be too hard to do.

Once you have this VPC deployed you also need to attach this VPC’s private subnets to the Transit Gateway like we did for the account in the previous steps.

Transit Gateway VPC route tables

To allow traffic from established connections back in we need to configure the necessary routing so that traffic from the internet finds it’s way back to the subnets of the account were it originated from.

In this example the VPC subnet of the attached account is 192.168.76.0/22. So we add a route to the route tables of the public and private subnets that points to the Transit Gateway as a destination for this network.

Route table private subnet A

Route table private subnet B

Route table private subnet C

Route table public subnets

NOTE: If you have multiple accounts and use a larger CIDR block to carve smaller subnets from, you can use the larger CIDR block instead.

Transit Gateway Route Table

Each VPC attachment will be added associated to a Transit Gateway Route Table and the available networks of that VPC will be propageted into this route table.

When creating a Transit Gateway with the default settings, a default Transit Gateway Route table is created and VPC attachments are automatically associated and propagated to that route table. So each attachment has a route to all the other attachments

NOTE: If you create a Transit Gateway with no default route table association you need to do this manually.

Because we want to use the NAT Gateways in the VPC of the Transit Gateway to connect the private subnets of all our other accounts to the internet we need to add a static route to the Transit Gateway Route Table. The target for the destination 0.0.0.0/0 will be the VPC in the Transit Gateway Account. We basically send all traffic that is not destined for one of the attached subnets to the VPC of the Transit Gateway. From that point on the route tables of the Transit Gateway VPC are used.

Finishing

Ok, that were a lot of (not always obvious) steps but we are now set to remove the NAT gateways from all the VPC’s that are attached to the Transit Gateway and start using the NAT Gateways in the Transit Gateway VPC.

Take into account that NAT Gateways have a limit of 5Gbit/sec. We are currently not even close to that limit but might be different in your case. Keep an eye on those NAT Gateway metrics while migrating or adding accounts.

Need help? Feel free to contact us