terraform.aws-baseline-infra/examples/eks-lab-ip6/eks

eks-lab/eks

This layer creates the following resources

• EKS cluster using ipv6 for service network
• EKS nodegroup
• EKS bastion
• Install eksctl, kubectl, awscliv2, helm on EKS bastion with user_data script

Be patient. EKS cluster takes 12min to provision. Node group will take another 5 min. And the cluster addon takes another ?? min.

Worker node instance size

Choose t3.large at the minimum. This is due to AWS's limitation on number of IPs. Smaller instanecs are limited with 6 IP which is not enough. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

How to use eksctl and kubectl

By default, AWS EKS are installed with an aws-auth configmap which allows only the cluster creator to work with the cluster. Therefore, one must first assume to the creator IAM role before running eksctl or kubectl. For example, to create kube config, run these commands:

export AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY="yyyy" AWS_DEFAULT_REGION=ap-northeast-1
aws eks update-kubeconfig --name lab-apne1-xpk-iac-cluster01

Edit configmap/aws-auth

kubectl edit -n kube-system configmap/aws-auth

Add a group with system:master role

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::040216112220:role/clusterCreator
      username: system:node:Template:EC2PrivateDNSName
    - groups:
      - system:masters
      rolearn: arn:aws:iam::040216112220:role/lab-apne1-xpk-iac-bast-role
      username: lab-apne1-xpk-iac-bast-role    
kind: ConfigMap
metadata:
  creationTimestamp: "2022-12-29T11:02:15Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "59670"
  uid: 7cf9d889-8ed2-4c8d-ac0f-092184cede8a

Addon updates

When updating addons, please select advanced options and choose preserve settings.

Install ALB ingress controller

AWS provides documentation on how to deploy a sample application with ingress (ALB) https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

That depends on the load balancer container, which can be deployed by

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.4/docs/install/iam_policy.json

aws iam create-policy \
--policy-name AWSLoadBalancerControllerIAMPolicy \
--policy-document file://iam_policy.json

Create an openid provider on iam https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

eksctl create iamserviceaccount \
  --cluster=lab-apne1-xpk-iac-cluster01 \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::040216112220:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve
  
helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=lab-apne1-xpk-iac-cluster01 \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller   

Tag subnets

Reference: https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

The following tags are set in the network layer:

On private subnets: Key – kubernetes.io/role/internal-elb Value – 1

On public subnets: Key – kubernetes.io/role/elb Value – 1

Install sample app the 2048 game

See https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.4/docs/examples/2048/2048_full.yaml
edit the file
kubectl apply -f 2048_full.yaml
kubectl get ingress/ingress-2048 -n game-2048

In a moment, the lb address should be displayed

root@ip-192-168-123-187:~# kubectl get ingress/ingress-2048 -n game-2048
NAME           CLASS   HOSTS   ADDRESS                                                                               PORTS   AGE
ingress-2048   alb     *       internal-k8s-game2048-ingress2-5f196824a1-20502803.ap-northeast-1.elb.amazonaws.com   80      7s