Networking
Security Groups for Pods
- If you use the
default
CNIaws-node
then you are limited to hosting a number of pods based on the instance type:
https://docs.amazonaws.cn/en_us/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
- If you wish to use
security groups for pods
you have to use aec2
type on the list below:
https://docs.amazonaws.cn/en_us/eks/latest/userguide/security-groups-for-pods.html#supported-instance-types
- If you have ran
kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true
and still seevpc.amazonaws.com/has-trunk-attached=false
for all nodes in the cluster. Try rotating your nodes (ie auto-scaling instance refresh) OR Again checking if you nodes are on thesupported instance types
list above (This was our problem! and wasted half of my day :()
Troubleshooting
- You can safely ignore the below the logs which can be seen in
k describe pod
Normal SecurityGroupRequested 8m18s vpc-resource-controller Pod will get the following Security Groups [sg-01abfab8503347254]
Normal ResourceAllocated 8m17s vpc-resource-controller Allocated [{"eniId":"eni-0bf8102e8bf0fa369","ifAddress":"02:78:59:8f:ee:b2","privateIp":"10.243.50.203","vlanId":1,"subnetCidr":"10.243.48.0/20"}] to the pod
Warning FailedCreatePodSandBox 8m17s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bdacc9416438c30c46cdd620a382a048cb5ad5902aec9bf7766488604eef6a60" network for pod "pgadmin": networkPlugin cni failed to set up pod "pgadmin_pgadmin" network: add cmd: failed to assign an IP address to container
Normal SandboxChanged 8m16s kubelet Pod sandbox changed, it will be killed and re-created.
- You can see if your pod has connected to the
sg
andeni
via running ak describe pod..
as you should get an output like:
Annotations: kubernetes.io/psp: eks.privileged
vpc.amazonaws.com/pod-eni:
[{"eniId":"eni-0bf8102e8bf0fa369","ifAddress":"02:78:59:8f:ee:b2","privateIp":"10.243.50.203","vlanId":1,"subnetCidr":"10.243.48.0/20"}]
Limits:
vpc.amazonaws.com/pod-eni: 1
Requests:
vpc.amazonaws.com/pod-eni: 1
As well as the logs from describe showing:
Pod will get the following Security Groups [sg-01abfab8503347254]