The problem with autoscaling GPU workloads may not be Kubernetes’ failure to make a decision, but the decision arriving too late. Ramkumar Nagaraj and Bingi Narasimha Karthik of Adobe describe an incident in which a critical production service experienced a demand wave that raised user error rates to 15–20%, even though the Horizontal Pod Autoscaler had begun scaling. The reason was that hundreds of containers remained pending, while new GPU nodes took a long time to become ready.
In the sequence documented by the authors, the demand wave arrived at 06:00, HPA metrics crossed the threshold at 06:05, and container scheduling began at 06:15. However, the first GPU nodes did not complete provisioning until 06:45, after the spike had ended. Provisioning GPU nodes typically takes three to five times longer than provisioning CPU-based services because of firmware loading, driver initialization, and CUDA setup.
From Reacting to Demand to Preparing for It
The team proposed running a controller inside Kubernetes every 60 seconds. It reads an hour of historical metrics and predicts demand ten minutes ahead. The goal is not perfect prediction, but to begin preparing capacity early enough for the nodes and containers to be ready when the wave arrives.
The design used data collected by Prometheus, including CPU and memory utilization, response time, request rate, and GPU utilization. The team tested ARIMA, exponential smoothing, the Prophet library, and LSTM before selecting a two-layer Bi-LSTM model with 64 and then 32 units. According to the article, the choice was a response to data patterns that included short spikes, recovery periods, and anomalous flat values, rather than because it was theoretically the best option in every case.
The model is retrained weekly, while the deployed model runs only in inference mode inside a binary for a controller written in Go, using TensorFlow Lite. As a result, the design does not require an external machine-learning platform or a model-serving layer.
Three Layers for Controlling Scaling
The design consists of three interconnected functions: prediction, provisioning, and absorption. The model predicts demand, the controller then gradually increases the replica count, and the pre-provisioned capacity provides room to absorb the actual wave.
Because prediction cannot handle every surprise, the team added a spike detector that operates in parallel. This component compares actual demand with forecasts using an adaptive threshold based on a moving standard deviation. If demand exceeds the forecast by a margin that meets the specified confidence level, the detector increases the scaling speed. The authors describe this component as an inferential safety net, not a second predictive model.
The gradual scaler limits the increase to 20 containers per minute. This is intended to prevent a massive scheduling wave from putting pressure on the scheduler and etcd, and from causing contention among image pulls, container starts, init-container initialization, and sidecar injection. Target utilization was also set at 70% rather than 100%, leaving room to absorb spikes and allowing the model to be occasionally inaccurate without turning the error into a chain of failures.
What Did the Tests Demonstrate?
The team first ran the system in shadow mode, recording predictions without carrying out actual scaling, and collected more than 500 hours of data. The results showed 85% accuracy when the ten-minute-ahead demand forecast was within ±10% of actual demand. The spike detector also caught nine out of ten waves, with two false alarms, while gradual-scaler tests showed no cascading failures or scaling oscillation.
The system also operated alongside HPA v2 without conflicts. During one week of validation in a controlled development environment, it passed 23 out of 23 tests. In a simulation of the spike patterns that caused the original incident, the system detected the wave approximately 11 minutes in advance.
When Is This Approach Appropriate?
Predictive scaling is valuable when node provisioning takes more than two or three minutes and demand is partially predictable, such as daily or weekly patterns or known events. It also requires good monitoring data, with at least one week of Prometheus metrics. Conversely, the idea becomes less worthwhile if nodes can be provisioned within 30 seconds, demand is entirely random, or the team’s top priority is reducing cost; keeping warm nodes means paying for spare capacity.
Editorial reading: The practical change here is not replacing HPA, but adding a prediction horizon to a system that typically responds to demand after it appears. This is particularly important for GPU workloads, where a fast scaling decision is not enough if the infrastructure needs tens of minutes before containers can run. However, the evidence presented still comes from controlled validation, not from broad, independent measurement across multiple production environments.
The team’s own experience also points to important limitations: a well-tuned ARIMA model may achieve a similar result with a simpler architecture, and the model may become outdated within days as demand patterns change. It also remained difficult to explain why a particular number of replicas was predicted, and questions about the optimal data volume, retraining frequency, and the ability of the inferential spike detector to catch unprecedented events have not yet been resolved.
Therefore, the article recommends starting by collecting one week of metrics, training a simple model, running it in shadow mode, and then measuring accuracy before enabling scaling. When moving to production, teams should impose a maximum replica limit, provide a clear disable procedure, and preferably pass through a constrained-scaling phase before opening the system fully. The conclusion repeated by the authors is practical: model complexity is not an advantage in itself, and if the simpler solution works, it is better to run it.