Uncaught Rust Exception Disrupts Cloudflare Operations

On November 18, 2025, a significant disruption occurred within Cloudflare’s infrastructure, resembling a major Internet outage. This incident stemmed from a failure in the Rust-based FL2 proxy, resulting in an HTTP 5xx error that interrupted customer traffic. Following extensive debugging efforts, engineers managed to restore operations.

Cause of the Disruption

The malfunction was traced to a corrupted input file. This file, known as the features file, is dynamically generated based on customer settings, such as those related to bot traffic. A recent update caused this file to include duplicate entries, expanding the typical features list from approximately 60 to over 200.

Memory Pre-allocation Issues

The FL2 proxy had pre-allocated memory designed to accommodate a specific range of features. While the older FL proxy managed this without incident, the FL2 code mishandled the overflow, leading to a critical exception. The chained processing functions within the updated proxy ingested an erroneous value, which cascaded to trigger a panic.

Identified Errors

• Error Handling: The absence of an exception handler for this common error drastically impacted system stability.
• Input Validation: The oversized features file did not cause immediate problems until it was passed into the pre-allocated memory, highlighting a flaw in input checks.

Common Causes of Vulnerabilities

This incident underscores a recurring theme in software vulnerabilities, particularly in the realm of input validation and error handling. Despite the advantages of modern programming languages like Rust, developers must prioritize rigorous input checks and thoroughly manage return values to avoid such failures.

Future Considerations for Cloudflare

In light of these findings, it is imperative that Cloudflare reassess its reliance on the FL2 proxy. Rolling back to the older FL proxy, which demonstrated greater stability, may be a critical step in restoring customer confidence. Moreover, this incident serves as a reminder of the importance of adhering to established best practices in software development.