Cloudflare announced the general availability of MySQL support in Hyperdrive, a service aimed at Workers users for pooling and reusing database connections across the company’s network. The transition from beta to general availability followed the resolution of approximately 20 bugs, including a rare issue in query-result caching that produced empty responses even when data was expected.
The issue occurred at a rate of approximately 0.001%, making it difficult to detect through tracing logs. Cloudflare discovered that the bug appeared only when caching was enabled, after it was reported by the customer with the highest MySQL service usage during the beta phase.
How does Hyperdrive work?
Hyperdrive keeps SQL connections warm and open across Cloudflare’s network, then shares them among Workers requests instead of creating a new connection for every request. It can also cache the results of common queries in the local CDN memory, reducing latency and pressure on the origin database.
The service’s architecture consists of a Client node operating near the Worker and an Endpoint node that manages the connection pool and is located in a data center near the origin database. The two nodes may share the same cache when they operate in the same data center.
The bug involved a difference in the MySQL protocol format
Hyperdrive handles MySQL from two different perspectives: it operates as a server in front of the Worker and as a client in front of the origin database. Because the protocol capabilities negotiated by the Worker may differ from those Hyperdrive uses with the database, the service must reformat result messages to suit the final client.
The investigation focused on how result sets are terminated. Some MySQL libraries, including the popular mysql2 library in Node.js, do not support the CLIENT_DEPRECATE_EOF capability. When this capability is absent, the server uses EOF packets, while newer protocol behavior replaces them with an OK packet at the end of a result set.
The Client node sometimes stored bytes after translating them to match the Worker’s capabilities, while the Endpoint node expected a canonical version using Hyperdrive’s capabilities. When one node read a result cached by the other, it could process the messages in an unsuitable format, resulting in an early EOF, a hang, or an empty result.
What changed in practice?
Cloudflare fixed the issue by making all nodes write a canonical version of the query result to the cache, always encoded according to Hyperdrive’s capabilities, regardless of which node cached it. After the fix was deployed, the empty-row issue disappeared and the Client connection success rate reached 99.99%, according to Cloudflare.
The company also addressed other issues that emerged during testing, including an error in emoji encoding. Using utf8 actually referred to utf8mb3, which does not support all four-byte Unicode code points. The service switched to utf8mb4. It also reset the encoding after executing the COM_RESET_CONNECTION command, because this command resets the connection to the server’s default settings, which may be latin1.
Availability and limitations
MySQL support is now generally available through the Cloudflare dashboard or the wrangler tool, and can be used with compatible MySQL databases, including connecting to a new PlanetScale database. The case shows that connection pooling and caching in an intermediary layer depend not only on SQL correctness, but also on an exact match between protocol states and the format expected by each layer.