terraform.aws-baseline-infra/examples/eks-lab/eks/README.md

eks-lab

This module creates the following resources

• VPC
• Public and private subnets
• NAT gateway
• EKS cluster
• EKS nodegroup
• EKS bastion
• Install eksctl and kubectl on EKS bastion

How to use eksctl and kubectl

By default, AWS EKS are installed with an aws-auth configmap which allows only the cluster creator to work with the cluster. Therefore, one must first assume to the creator IAM role before running eksctl or kubectl. For example, to create kube config, run these commands:

export AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY="yyyy" AWS_DEFAULT_REGION=ap-northeast-1
aws eks update-kubeconfig --name lab-apne1-xpk-iac-cluster01

Configure VPC CNI to use custom networking

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=failure-domain.beta.kubernetes.io/zone

cat <<EOF  | kubectl apply -f -
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
 name: ap-northeast-1a
spec:
  subnet: subnet-0d015cc72715685ca
EOF

cat <<EOF | kubectl apply -f -
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
 name: ap-northeast-1c
spec:
  subnet: subnet-030ee2c3e2b730fcc
EOF

Then redeploy the nodegroup

terraform apply -replace="aws_eks_node_group.eks-nodegroup"

If successfully done, you will start to see 100.64.0.0 addresses being used on the EKS worker nodes. You can also see it with kubectl:

root@ip-192-168-123-48:~# kubectl  get pods --all-namespaces -o wide
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE     IP                NODE                                                 NOMINATED NODE   READINESS GATES
kube-system   aws-node-5892k                      1/1     Running   0          4m9s    192.168.123.245   ip-192-168-123-245.ap-northeast-1.compute.internal   <none>           <none>
kube-system   coredns-5fc8d4cdcf-c75z6            1/1     Running   0          13m     100.64.9.249      ip-192-168-123-245.ap-northeast-1.compute.internal   <none>           <none>
kube-system   coredns-5fc8d4cdcf-h5lnl            1/1     Running   0          13m     100.64.13.41      ip-192-168-123-245.ap-northeast-1.compute.internal   <none>           <none>
kube-system   ebs-csi-controller-d6bff959-8459z   6/6     Running   0          13m     100.64.8.74       ip-192-168-123-245.ap-northeast-1.compute.internal   <none>           <none>
kube-system   ebs-csi-controller-d6bff959-vnwlf   6/6     Running   0          5m28s   100.64.11.124     ip-192-168-123-245.ap-northeast-1.compute.internal   <none>           <none>
kube-system   ebs-csi-node-h7w8r                  3/3     Running   0          4m9s    100.64.11.188     ip-192-168-123-245.ap-northeast-1.compute.internal   <none>           <none>
kube-system   kube-proxy-vgmdf                    1/1     Running   0          4m9s    192.168.123.245   ip-192-168-123-245.ap-northeast-1.compute.internal   <none>           <none>

Edit configmap/aws-auth

kubectl edit -n kube-system configmap/aws-auth

Add a group with system:master role

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::040216112220:role/clusterCreator
      username: system:node:Template:EC2PrivateDNSName
    - groups:
      - system:masters
      rolearn: arn:aws:iam::040216112220:role/lab-apne1-xpk-iac-bast-role
      username: lab-apne1-xpk-iac-bast-role    
kind: ConfigMap
metadata:
  creationTimestamp: "2022-12-29T11:02:15Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "59670"
  uid: 7cf9d889-8ed2-4c8d-ac0f-092184cede8a

Addon updates

When updating addons, please select advanced options and choose preserve settings.