Why ML Ended Up on Kubernetes
The CNCF's 2024 survey puts Kubernetes production adoption at 80%, and a lot of ML came along for the ride. What that number actually measures, what it doesn't, and where the platform still hurts for ML workloads.
A number has been doing the rounds for a year: Kubernetes hit 80% production adoption for ML workloads in 2024. It is close to a real number with the interesting part sanded off. The CNCF 2024 annual survey actually reports that 80% of respondents run Kubernetes in production, up from 66% in 2023. That is all workloads, not ML. The same survey reports that 48% of respondents are not running AI/ML workloads on Kubernetes yet. Both facts are true, and together they describe the real situation better than the headline does: Kubernetes became the default place to run software, and ML is moving into a house the company already owns.
The default beat the specialist
Nobody sits down and chooses Kubernetes for machine learning in the abstract. It wins because the cluster already exists, along with the CI, the identity setup, the cost allocation and the on-call rotation. Adding an ML workload is a namespace and a device plugin. It is not a platform decision.
The quiet hero is the device plugin model. Once a GPU is expressible as a schedulable resource (nvidia.com/gpu), the same scheduler that bin-packs web services places training jobs:
apiVersion: v1
kind: Pod
metadata:
name: gpu-training-pod
spec:
containers:
- name: training
image: nvidia/cuda:12.0-runtime
resources:
limits:
nvidia.com/gpu: 4
command: ["python", "train_model.py"]None of that is ML-specific, and that is the point. A generic mechanism beat the purpose-built ML schedulers because organisations would rather run one platform than two.
What the ecosystem actually gives you
The ML layer on top of Kubernetes is real and, by 2024, had names you can actually evaluate: Kubeflow Pipelines for workflow DAGs, KServe for model serving with scale-to-zero, Katib for hyperparameter search, Argo Workflows if you want pipelines without the ML branding, and Seldon Core as an alternative serving stack. None of it is free to run. The price is operational complexity, and the honest question for a small team is whether that tax is worth paying yet.
Where it still hurts
That 48% figure is not a lag indicator. It is friction. GPU bin-packing gets awkward the moment you need fractional devices or MIG slices, and the same spot preemption that makes training cheap also makes it flaky, so checkpointing discipline ends up mattering more than the orchestrator. Observability is the other gap: the platform answers "is the pod healthy" but not "is the model still good", which is why ML teams keep building a second monitoring stack on top of the first. None of this will surprise anyone who has run a model in production.
The honest takeaway
Kubernetes won ML by default, not by design. So the question for 2025 is not "Kubernetes or not". It is how much of the ML stack you hand it. Serving on the existing cluster is nearly free. Owning training on it is a real commitment. Start with the first, and make the second earn its place.
Source: CNCF Annual Survey 2024. The 80% production-adoption figure covers all workloads among survey respondents; the 48% AI/ML figure is from the same survey.