Programming and Software Development

GitHub Reveals Details of Reliability Improvements After Two September Outages

GitHub announced that it reduced database pressure, added load-control and fault-containment tools, and moved more production workloads to Azure after two incidents affected its services during September 2026. However, the company failed to meet its goal of deploying all production services in the Central US region and recorded an outage affecting Copilot code reviews.

2026-10-14
4 min read
3 views
certi.news Editorial Team
GitHub Reveals Details of Reliability Improvements After Two September Outages

In its September 2026 availability report, GitHub disclosed two incidents that affected the performance of its services, alongside progress in redistributing infrastructure and improving database resilience under load. The company said remediation work was not yet complete, despite implementing changes intended to reduce the risk of failures when load increases or a shared component fails.

As of October 7, 120 of 160 production services had been deployed in the Central US region, while 13 of 51 primary MySQL databases had been moved. Azure handled 65.4% of read traffic for the core GitHub application, 48.3% of Git read traffic, and 80.2% of stream-processing traffic. The company emphasized that these percentages apply to different workloads and should not be added together.

Infrastructure Migration Progress, but Behind Schedule

GitHub halted deployments and traffic increases in Central US on September 11 because of capacity disparities among availability zones. Permanent increases resumed on September 18 after temporary capacity was provided, but long-term capacity work is still underway, with a goal of making all available capacity accessible by the end of October. The company did not meet its ambitious goal of deploying all production services there by September 30.

During a planned database switchover, an automated emergency procedure separated database replicas in Central US, causing disruption for customers in the region. GitHub disabled the procedure during planned handoffs until recovery mechanisms could be improved.

Reducing Database Pressure and Containing Failures

GitHub moved 24 tables from its core authentication system away from its oldest shared database, removing approximately one million queries per second from the replicas. It also deleted approximately 980 million unnecessary records from the modification history in the cluster supporting Issues and Pull Requests, freeing approximately 1.8 terabytes of reclaimable storage. Weekly transaction-conflict events declined by approximately 77% compared with the June peak.

Half of the sampled Pull Request traffic can now continue with limited functionality when a shared dependency fails. Database traffic routing can also add capacity automatically during spikes; one cluster expanded from 14 to 19 replicas during an increase in Pull Request activity on September 23. GitHub added a load-control mechanism that can reject lower-priority requests before services reach broader saturation.

The Two September Incidents

On September 13, a background data-cleanup operation increased load on a shared database. The protection mechanism monitored replica lag, but not the accumulated load on the primary server, which reached its available connection limit. The absence of a short query timeout left request handlers waiting, while a retry loop continued sending failed write operations. The incident lasted one hour and 28 minutes and ended after internal load was reduced and the task was stopped.

On September 28, a change to the service that runs Copilot code reviews caused a limited version of review results to be sent to some customers instead of the complete version containing comments. As a result, tasks appeared complete even though the reviews did not appear in pull requests. The incident lasted 52 minutes and was resolved by GitHub reverting the change, with customers asked to request the affected reviews again.

What Is Changing in Practice?

The two incidents show that capacity improvements alone are not enough; reliability also depends on detecting failures early, stopping unproductive retries, and ensuring that a task is not declared successful before its output is actually delivered. GitHub is working to add controls for background processing, improve service startup behavior, and strengthen the monitoring and delivery of Copilot code review results.

GitHub also recorded improvements in the infrastructure supporting GitHub Actions: measured Redis operations declined by approximately 28% and relative CPU usage by approximately 30%. A separate caching layer reduced bandwidth usage from Redis by 84%, average response time by 66%, median response time for the primary inventory request by 96%, and waiting requests by 95%. Open questions remain about the company's ability to complete the capacity work and prevent recurring failures of shared dependencies under actual production conditions.

News source
c
Author

certi.news Editorial Team

What you need to know

GitHub announced reliability improvements following two incidents in September 2026, including database load reduction, load control, fault containment, and shifting more workloads to Azure. However, the company has not yet completed deploying all production services in the Central US region and is still working to prevent recurring failures involving shared dependencies and Copilot code reviews.

  • GitHub services were affected by two incidents on September 13 and 28; the first resulted from increased load on a shared database, while the second caused limited results to be displayed for Copilot code reviews.
  • GitHub moved 24 tables from the authentication system away from the oldest shared database, removing about one million queries per second and reducing weekly transaction conflicts by about 77% compared with the June peak.
  • The company added automatic database-traffic routing and a load-control mechanism that rejects lower-priority requests before services become saturated.
  • As of October 7, 120 of 160 production services and 13 of 51 primary MySQL databases had been deployed or moved to Central US under the redistribution plan.
  • Azure handled different proportions of read and processing traffic for GitHub workloads, and these percentages should not be added together because they relate to separate workloads.
  • GitHub Actions’ architecture improved, with measured Redis operations decreasing by about 28% and relative CPU utilization decreasing by about 30%.

FAQ

What caused the first incident in September?

A background data-cleanup operation on September 13 increased load on a shared database, while the protection mechanism did not monitor the accumulated load on the primary server.

What happened in the Copilot code-review incident?

A service change on September 28 caused a limited version of review results to be sent to some customers without the full comments, making the tasks appear complete even though the reviews did not appear in pull requests.

Has GitHub completed deploying all production services in Central US?

No. As of October 7, 120 of 160 production services had been deployed there, and GitHub had not met its goal of deploying all services by September 30.

What measures did GitHub take to improve reliability?

It reduced database load, disabled an automated replica-detachment procedure during planned delivery operations, added load control, and improved background processing plus the monitoring and delivery of Copilot review results.

In the same category

You may also like

View all news