Troubleshooting and Resolving a Pod Stuck in 'CreateContainerConfigError' in Kubernetes
The other day I was making changes to my helm charts and, after deploying my application, I noticed that one of my pods was stuck in a CreateContainerConfigError state. This is a pretty tricky error because it doesn’t give you any details on what the underlying issue could be.
What is the CreateContainerConfigError?
To understand this, let’s look at what happens at deployment time to give you an idea of the flow and what could go wrong at each step.
When you deploy a pod, the first step is to pull the image from the registry and then create the container. If the image is not found, then Kubernetes will return an ErrImagePull error. If the image is found, then it will proceed to create the container.
If the container creation fails, then it will return a CreateContainerError error. If the container creation succeeds, then Kubernetes will then start the container.
If the container start fails, then it will return a CreateContainerConfigError error.
In other words, the error happens when the container is transitioning from a Pending state to a Running state. It is at this point that the deployment configuration will be validated to make sure that the container can be started. If the configuration is invalid, then it will return a CreateContainerConfigError error.
How to Troubleshoot the CreateContainerConfigError
Disclaimer: There can be many reasons why the container configuration is invalid and it will depende on your specific configuration. I will only be covering the one that I have encountered. If you have encountered a different cause, please leave a comment below.
Because the error happens during the validation of the configuration, a good starting point is to double-check the following:
- is the ConfigMap missing? Is it properly configured?
- is a Secret missing? Is it properly configured?
- is the PersistentVolume missing? Is it properly configured?
- is the Pod being created correctly? Are there any empty or invalid fields?
Now that we understand what the error is, and what we should be looking at, let’s look at how to troubleshoot it and narrow down the problem.
Check the Pod Status
The first thing I did was get the pod that I had the error and that I wanted to drill into.
You can do this by running kubectl get pods -n <namespace>.
~ kubectl get pods -n my-service
NAME READY STATUS RESTARTS AGE
my-service-00000000078c9fff-dssbk 0/2 CreateContainerConfigError 1 (10s ago) 28s
my-service-00000000bcddf7d-xfsmk 2/2 Running 25 (42h ago) 16d
Check the Events
Next, we are interested to see all the events on the pod.
You can do this by running kubectl describe pod <pod-name> -n <namespace> and look at the bottom at the Events. This will give you a lot of information about the pod, including the events that have happened to it similar to the following, which has been redacted to remove sensitive information.
~ kubectl describe pod my-service-00000000078c9fff-dssbk -n my-service
Name: my-service-00000000078c9fff-dssbk
Namespace: my-service
Priority: 0
Service Account: default
Node: <node-details>
Start Time: Wed, 25 Jan 2023 15:36:14 +0100
Labels: app.kubernetes.io/instance=my-service
app.kubernetes.io/name=my-service
pod-template-hash=00000000
Annotations: <annotations>
Status: Pending
IP:
IPs:
IP:
Controlled By: ReplicaSet/
Containers:
my-service:
Container ID:
Image: <image-name>
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CreateContainerConfigError
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Liveness: http-get http://:http/ delay=15s timeout=60s period=60s #success=1 #failure=3
Readiness: http-get http://:http/ delay=15s timeout=60s period=60s #success=1 #failure=3
Environment:
(...)
AzureWebJobsStorage: <set to the key 'AzureWebJobsStorage' in secret 'my-service'> Optional: false
AzureAccessKey: <set to the key 'AzureAccessKey' in secret 'my-service'> Optional: false
AzureTopicEndpoint: <set to the key 'AzureTopicEndpoint' in secret 'my-service'> Optional: false
ClientId: <set to the key 'ClientId' in secret 'my-service'> Optional: false
(...)
State: Waiting
(...)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 94s default-scheduler Successfully assigned my-service/my-service-00000000078c9fff-dssbk to <node-name>
Normal Pulled 94s kubelet Successfully pulled image "image" in 165.014261ms
Warning Failed 77s (x4 over 94s) kubelet Error: couldn't find key ClientId in Secret my-service/my-service
(...)
The Events section shows a list of all the events that have occurred in the process of creating the pod.
And here we find the issue. The pod is actually missing a secret, the ClientId in my case, that it needs to start. And that is why the pod is in:
State: Waiting
Reason: CreateContainerConfigError
If you want to double-check that the secret is missing, you can run kubectl get secrets -n <namespace> and check if the secret is not there.
Or you can output it in a JSON format and check that the key is missing by running the following command:
kubectl get secret my-service -n my-service -o json | jq '.data | map_values(@base64d)'
How to Resolve the CreateContainerConfigError
In my case, because I store my infrastructure and configuration (including the kubernetes secrets) in Terraform, I just needed to add the secret to the Terraform configuration, apply it and because the deployment had already timed out, re-run the deployment. But it would’ve picked it up automatically if I had applied it a bit sooner.
Now that the pod has its necessary configuration and is valid if we run kubectl get pods -n <namespace> again, we can see that the pod is now in a Running state.
kubectl get pods -n my-service
NAME READY STATUS RESTARTS AGE
my-service-00000000078c9fff-dssbk 2/2 Running 1 (10s ago) 28s
my-service-00000000bcddf7d-xfsmk 2/2 Terminating 25 (42h ago) 16d
And there you have it. You have successfully resolved the CreateContainerConfigError.
This was an easy one, let me know what you encountered in the comments below and how you fixed it.
Happy Coding and I hope this helps someone!