Addressing and Routing
Physical addresses are separate from VPC addresses, to enable independent scaling of the virtual network and the physical network.
A proprietary encapsulation format is used to carry VPC information to the destination. It includes 3 layers of headers: IP packet from the payload, VPC encapsulation, and IP on physical network. On VM hosts, encapsulation is handed by the AWS Nitro System.
Multicast and broadcast routing are not supported in a regular VPC.
Mapping Service
The mapping service is a distributed web service that handles mappings between (VPC ID, IP address) and physical destinations like target physical host IP.
There are two kinds of mappings, /32 host mappings and routes (CIDR mappings) for Blackfoot edge devices.
Mappings are cached, pushed out and pre-loaded to memory, and proactively invalidated when they change.
Mappings from the mapping service are 100% cached on the physical host, and the cache miss path is not implemented (i.e. drop them) to ensure predictable performance (especially latency) and mitigate a bunch of availability concerns for the mapping service. This is the Preprogrammed Model described in Andromeda’s paper.
For EC2 instances communicating within the same subnet, the fake ARP response from VM hosts will return the actual but virtual MAC address of the other instance. Other packets are sent to the subnet gateway, for which a fake MAC address is assigned and VM host parses the IP header to find the destination for those packets.
On receipt of an encapsulated packet, VM hosts check if the source is valid, and an alert is triggered if invalid packets are found, which could be caused by a program error or an attack.
Flow Tracking
Not all flows are tracked on VM hosts. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-connection-tracking.html.
Edge Gateway
Blackfoot Edge devices handles VPC ingress and egress, including Internet traffic, Direct Connect, VPN and S3 & DynamoDB Endpoints. It’s horizontally scaled, redundant, high available and stateless.
Blackfoot encapsulates ingress traffic for VPC and decapsulates egress traffic. it also operates NAT in a stateless manner, mapping private and public IPs (EIPs) one-to-one.
VPC Services
AWS exposes a set of supporting services within customer VPCs at well-known or reserved addresses. These services are traditionally exposed from the IPv4 link-local address range (169.254.0.0/16). For AWS Nitro System instances, AWS also provides these services using IPv6 ULAs.
Services include
- Instance Metadata Service (IMDS)
- Route 53 DNS resolver
- Network Time Protocol server
VPC Quotas
Each VPC can have up to 256,000 NAU units (think of it as IP addresses in use), and the maximum number of NAU units for a VPC and all of its intra-Region peered VPCs is up to 512,000.
Previously, only around 50k NAU units are supported. It was likely because of the Preprogrammed Model in local mapping service cache.