Two designs, and the firewall that only sees half a conversation
East-west traffic between VPCs is forced through a stateful firewall in a shared inspection VPC. There is an appliance in every zone for availability, the routing is symmetric on paper, and the whole thing passed its security review.
In production it works for some calls and not others, with no pattern anyone can name. The firewall logs show requests arriving and no matching responses. The failing calls are the ones where the caller and the callee happen to be in different zones.
Boundaries, outermost first: Inspection VPC: Appliance (zone 1), Appliance (zone 2) VPC A — zone 1: Caller (zone 1) VPC B — zone 2: Callee (zone 2) Outside every boundary: Transit gateway (default) Connections: Caller calls Transit gateway — request Transit gateway calls Appliance — inspected Callee publishes to Transit gateway — response Transit gateway publishes to Appliance — wrong appliance
Boundaries, outermost first: Inspection VPC: Appliance (zone 1), Appliance (zone 2) VPC A — zone 1: Caller (zone 1) VPC B — zone 2: Callee (zone 2) Outside every boundary: Transit gateway (mode on) Connections: Caller calls Transit gateway — request Transit gateway calls Appliance — inspected Callee publishes to Transit gateway — response Transit gateway publishes to Appliance — same appliance