CloudFront VPC Origins Tied Unrelated Apps to One Fault

AWS CloudFront spent 3 hours 33 minutes on July 16, 2026, returning 5xx errors to every customer on VPC Origins. Canvas, Blackboard, Hugging Face, and the UK National Lottery went dark together. Distributions that still used public origins kept serving.

The rest of CloudFront was not down. A connection-management fleet in Frankfurt’s euc1-az2 zone stopped loading routing updates, and every edge node that needed those updates failed in the same window. That is a feature-level pile-up, and it is the part campus IT still cannot see on a vendor spreadsheet.

Canvas, PayPay, and a Lottery Shared One Path

IncidentHub founder Hrishikesh Barua’s monitors caught the cascade in real time. Instructure, which runs Canvas, put CloudFront on its status page. Anthology’s Blackboard page did the same. Hugging Face told users the model hub was unreachable from most regions. None of those products share a parent company. They shared a CloudFront feature that launched in late 2024.

Rutgers University’s Office of Information Technology posted at 6:08 a.m. ET that Canvas was unreachable and called it “a vendor-driven, nationwide event affecting multiple institutions.” Students at Rutgers do not hold an AWS account. They still sat in the blast radius of a private-origin routing table in Germany.

THE JULY 16 BLAST RADIUS

Japanese payment, live-stream, and blogging sites failed in the same hours, including PayPay and Niconico Live. Downdetector logged about 350 AWS reports, against more than 6.5 million during the October 20, 2025, US-EAST-1 DNS failure. The July event was smaller on that meter and wider in what it revealed: products that look diverse still concentrated on one origin type.

What CloudFront VPC Origins Locks Down

AWS CloudFront as the single front door when it announced VPC Origins on November 20, 2024. Before that, an Application Load Balancer, Network Load Balancer, or EC2 origin needed a public IP so CloudFront could reach it. Anyone who found that IP could skip the CDN, the WAF, and the geo rules.

The new path puts a service-managed elastic network interface in the customer’s private subnet. Edge servers connect to that ENI. The origin never needs a public address or an internet-gateway route for return traffic. AWS also attaches a managed security group named CloudFront-VPCOrigins-Service-SG, which the origin can trust and nothing else has to.

The developer guide tells teams to keep load balancers in private subnets and then strip public access after a staging distribution is promoted. That last step is the lock. Once the subnet is private, CloudFront is the only ingress. If the VPC Origins routing layer stops, there is no leftover public hostname to fail over to unless someone built one in advance.

AWS’s live workaround on July 16 was to change the origin type. That is a security step backward, and it only works if the public origin, the DNS, and the firewall rules still exist. Teams that adopted the feature to close an attack surface had also closed the hatch they needed at 12:45 a.m. PDT.

The Health Dashboard Logged 3 Hours 33 Minutes

AWS dated impact from 12:45 a.m. to 4:18 a.m. PDT, which is 07:45 to 11:18 UTC. The first public note on the Health Dashboard landed at 1:44 a.m. PDT, about an hour after errors started. AWS Support posted on X at 1:57 a.m. PDT.

JULY 16 ON THE AWS CLOCK (PDT)

1. 12:45 a.m.: VPC Origins customers start seeing elevated 5xx errors.
2. 1:44 a.m.: Health Dashboard opens an investigation for VPC Origins connectivity.
3. 1:57 a.m.: AWS Support repeats the 5xx alert and points at the dashboard.
4. 2:57 a.m.: Engineers identify an internal constraint on the fleet that manages connections to private VPC origins.
5. 3:18 a.m.: Update blames a packet-processing subsystem that routes edge requests into customer VPCs, and repeats the origin-type workaround.
6. 3:52 a.m.: Multiple mitigation actions go out.
7. 4:18 a.m.: Retrospective summary calls full recovery; live notes at 4:27 a.m. and 4:57 a.m. PDT still said recovery was in progress.

Blackboard’s status page opened an investigation at 4:33 a.m. ET, confirmed CloudFront at 5:40 a.m. ET, and only logged recovery signs at 8:57 a.m. ET, after AWS had already marked the CDN healthy at 7:18 a.m. ET. Downstream status pages lagged the control plane that broke them.

AWS Support posted the resolution the same morning. The company has not named the resource behind the “internal constraint,” and it has not said whether that ceiling moved.

https://x.com/AWSSupport/status/2077733500732522882

Two Campus Platforms Sat on the Same Feature

Canvas and Blackboard had already gone down together during the October 20, 2025, US-EAST-1 DNS incident. IncidentHub timed that one at more than 17 hours for both learning platforms. July 16 was shorter and narrower, and it still took both of them, because both still sit on CloudFront without a second CDN in the default setup.

Axiom Flow’s August 15, 2026, pass over 3,400 US colleges and universities, starting from an Education Department College Scorecard file of 5,291 schools, is the clearest public headcount since the outage. It classified 64 percent of that starting list with enough public evidence to name an LMS.

US HIGHER ED LMS SHARE IN 3,400 SCHOOLS

Brian Watkins, Instructure’s communications director, said during the October 2025 outage that half of US college and university students use Canvas, and 38 percent of K-12 students do too. Institution count and student count are different meters. Both still put Canvas, and Canvas plus Blackboard, in the path of a single CDN feature.

Mayur Upadhyaya, CEO of API observability firm APIContext, put the wider pattern in one line: the industry is “increasingly consolidating around a small number of providers because they’re the most convenient and economically attractive choice,” and “a fault that might once have affected a handful of organizations can now impact thousands simultaneously because so many businesses depend on the same infrastructure.” Campus IT cannot multi-home Canvas. The correlation sits one layer below the LMS contract.

AWS Kept Extending VPC Origins After July

The feature did not freeze after Frankfurt. AWS added cross-account sharing in November 2025 through Resource Access Manager, so a distribution in one account can point at a private origin in another. On May 1, 2026, it opened WebSockets traffic through VPC origins for chat, collaborative editors, live dashboards, and IoT, with no extra charge for that traffic.

HOW AWS GREW THE PRIVATE ORIGIN PATH

• November 20, 2024: VPC Origins launches for private ALB, NLB, and EC2 origins in commercial Regions.
• November 2025: Cross-account VPC Origins lets teams split the distribution and the origin across accounts.
• May 1, 2026: WebSockets join HTTP on the same private front door.
• July 15, 2026: An AWS networking post by Salman Ahmed and Sandeep Panda walks through a private AI agent streaming over WebSocket VPC Origins.
• July 16, 2026: The connection-management fleet in euc1-az2 hits its constraint and VPC Origins 5xx goes global.
• August 26, 2026: AWS publishes a guide on multi-region active-active designs that still use CloudFront VPC Origins.

Each expansion puts more application types, and more account layouts, onto the same control plane that failed to load a routing table. Chat and live dashboards that moved off public subnets in May inherited the July failure mode. The July 15 how-to landed the day before the outage. The August 26 architecture post landed six weeks after it.

Engineering teams that lived through that night have been writing the boring version of a fix: keep the private origin, and keep a tested public-origin switch ready. Origin failover by itself did not save them. Classmethod engineer Ryo Suzuki documented a bypass that used a public ALB in US-EAST-1 plus inter-region peering; that build took about 39 minutes, which is only useful if someone is awake and already knows the pattern.

No Help From a Second AWS Region

Standard zonal advice says fail away from euc1-az2. That advice assumes the broken piece is in the data plane of one zone. CloudFront’s connection-management fleet distributes routing configuration to network processors at the edge, worldwide. When that load failed, Tokyo, São Paulo, and Chicago edges still accepted requests and still could not route them to a VPC origin.

AWS was explicit that other origin types were unaffected. S3 origins, public ALBs, and custom HTTP origins were on a different path. Region failover, the usual answer to a Frankfurt hardware problem, does not move a global configuration loader. That is the same control-plane pattern Azure already hit when incompatible metadata went out across an edge network in October 2025.

IncidentHub lined the July 16 mode up with three 2025 events: Google Cloud’s June 12 authorization-config push, Azure’s October 29 edge-metadata change, and Cloudflare’s November 18 bot-management file. Those incidents shipped a bad config everywhere. AWS’s variant never loaded the good config. Users saw the same thing, a global edge running without valid instructions.

IncidentHub also logged a separate AWS event on July 15, 2026, from 11 p.m. UTC, limited to euc1-az2 in eu-central-1, the same availability zone named in the CloudFront write-up. AWS has not tied the two nights together. The overlap is on the record; the causal link is not.

WHAT WE KNOW

• The fleet: AWS said an internal constraint on the fleet that manages connections to private VPC origins stopped routing configuration from loading.
• The zone: The constraint was associated with euc1-az2 in EU-CENTRAL-1 (Frankfurt).
• The blast: Only VPC Origins connectivity returned 5xx; other origin types stayed up.
• The clock: Impact ran 12:45 a.m. to 4:18 a.m. PDT, 3 hours 33 minutes.

WHAT IS UNCONFIRMED

• The meter: AWS has not said whether the ceiling tracked connections, threads, memory, or another resource.
• The fix: There is no public note that the limit was raised or that a circuit breaker was added.
• The night before: A same-zone event on July 15 is logged; AWS has not called it a precursor.

Without those details, a customer cannot tell whether its own traffic could trip the fleet again, or whether the ceiling sits entirely outside any one account. Barua’s advice stays operational: pre-stage the origin-type change in Terraform, map identity providers that sit on the same CDN path, and watch CloudFront’s Health Dashboard instead of waiting for students to mail the help desk.

A Public Origin Switch Only Helps If It Is Ready

Identity is the second-order cut. Frontegg’s outage could lock users out of applications that were themselves healthy. A university that “only” uses Canvas still depends on whatever CDN Canvas chose, and on whatever identity broker sits in front of it. Multi-cloud on first-party workloads does not follow the student into that SaaS.

The docs still describe VPC Origins as a way to make CloudFront the single point of entry. That sentence is accurate, and on July 16 it was also the failure report. Until AWS publishes the constraint it hit in euc1-az2, the teams that already staged a public origin are the ones who have a move at 12:45 a.m.

Frequently Asked Questions

How long does AWS take to deploy a VPC origin?

The CloudFront console asks you to wait until the VPC origin status is Deployed, and AWS says that wait can take up to 15 minutes. You also need at least one free private IPv4 address in the subnet so the managed ENI can be created. That address carries no extra IPv4 charge.

Can a Network Load Balancer with a TLS listener be a VPC origin?

No. AWS blocks Gateway Load Balancers entirely, and it also blocks Network Load Balancers that use TLS listeners. A Network Load Balancer that you do use must have a security group attached, which is not the default NLB setup.

Does CloudFront VPC Origins support gRPC or IPv6?

VPC Origins does not support gRPC traffic, and it does not support IPv6. AWS also withholds Lambda@Edge origin request and origin response triggers on this origin type. Inbound NACL rules are not evaluated on the CloudFront-to-origin path; outbound NACLs still must allow ephemeral TCP ports 1024 to 65535.

Do you still need an internet gateway in a VPC that only uses private origins?

Yes. AWS requires an internet gateway on the VPC so the VPC can be treated as internet-reachable, even though that gateway is not used to route CloudFront traffic to the private origin and you do not add it to the origin subnet’s route table. The actual hop is the managed ENI.

Which CloudFront origin types stayed up on July 16, 2026?

AWS said customers using origin types other than VPC Origins were not affected and could ignore the workaround. The live guidance was only for teams that could temporarily change a VPC origin to a public origin type, then revert after 4:18 a.m. PDT.