AWS: CDK and Python — installing EBS CSI driver Add-On on EKS

Installing EBS CSI Driver Add-On on an AWS Elastic Kubernetes Service cluster with AWS CDK, Python and CfnAddon

Let’s continue with AWS CDK and Python. I’m not writing because I like CDK, but because there are quite a few examples on the Internet for AWS CDK with Python, so let them at least be here.

So, we have a cluster — AWS: CDK and Python — building an EKS cluster, and general impressions of CDK, we have a couple of controllers — AWS: CDK and Python — configure an IAM OIDC Provider, and install Kubernetes Controllers. As if everything is ready — I started installing a VictoriaMetrics chart, and everything was working except for the pod with VMSingle, which hung in the Pending status.

“VolumeBinding”: binding volumes: timed out waiting for the condition

Let’s check the Events of this Pod:

…
Events:
Type Reason Age From Message
 — — — — — — — — — — — — -
Warning FailedScheduling 10m default-scheduler running PreBind plugin “VolumeBinding”: binding volumes: timed out waiting for the condition

Quick googling led me to a question on StackOverflow, where I recalled about EKS Add-ons, in particular, about the EBS CSI diver, which should create EBS when a PersistentVolumeClaim appears.

So today we’ll look at how to add add-ons to a cluster with the AWS CDK.

Actually, it’s quite simple, the only thing I had to google was how to use the CfnAddon, but this time the documentation was found quickly, and even with examples in Python, not TypeScript.

IAM Role for EBS CSI driver

We already have OIDC Provider, see AWS: EKS, OpenID Connect, and ServiceAccounts

For the driver, we also will use IRSA. So we need to describe a ServiceAccount, and attach an AWS Managed Policy with the iam.ManagedPolicy.from_aws_managed_policy_name():

...
        # Create an IAM Role to be assumed by ExternalDNS
        ebs_csi_addon_role = iam.Role(
            self,
            'EbsCsiAddonRole',
            # for Role's Trust relationships
            assumed_by=iam.FederatedPrincipal(
                federated=oidc_provider_arn,
                conditions={
                    'StringEquals': {
                        f'{oidc_provider_url.replace("https://", "")}:sub': 'system:serviceaccount:kube-system:ebs-csi-controller-sa'
                    }
                },
                assume_role_action='sts:AssumeRoleWithWebIdentity'
            )
        )
        ebs_csi_addon_role.add_managed_policy(iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AmazonEBSCSIDriverPolicy"))
...

In the from_aws_managed_policy_name specify the name as "service-role/ManagedPolicyName".

CfnAddon for EBS CSI driver

Find a current version of the diver by specifying the version of the cluster — we have EKS version 1.26, because the CDK still does not support 1.27:

$ aws eks describe-addon-versions — addon-name aws-ebs-csi-driver — kubernetes-version 1.26 — query “addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]” — output text
v1.20.0-eksbuild.1
True
…

And describe the connection of the add-on itself with the CfnAddon – specify the cluster name, version, and ServiceAccount's IAM Role ARN taken from the ebs_csi_addon_role created above::

...
        # Add EBS CSI add-on
        ebs_csi_addon = eks.CfnAddon(
            self,
            "EbsCsiAddonSa",
            addon_name="aws-ebs-csi-driver",
            cluster_name=cluster_name,
            resolve_conflicts="OVERWRITE",
            addon_version="v1.20.0-eksbuild.1",
            service_account_role_arn=ebs_csi_addon_role.role_arn
        )
...

Deploy, and check:

Check Pods:

$ bkk -n kube-system get pod | grep csi
ebs-csi-controller-896d87c6b-7rv9z 6/6 Running 0 9m59s
ebs-csi-controller-896d87c6b-v7xg7 6/6 Running 0 9m59s
ebs-csi-node-2zwnr 3/3 Running 0 9m59s
ebs-csi-node-pt5zs 3/3 Running 0 9m59s

And now we have a PVC for VictoriaMetrcis in the Bound status:

$ kk -n dev-monitoring-ns get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
vmsingle-victoria-metrics-k8s-stack Bound pvc-151a631b-f6de-4567–8baa-97adb4e04a87 20Gi RWO gp2 91m

And the VMSingle Pod now in the Running status:

$ kk -n dev-monitoring-ns get po | grep vmsingle
vmsingle-victoria-metrics-k8s-stack-f7794d779-n6sc7 1/1 Running 0 28m

Done.

Originally published at RTFM: Linux, DevOps, and system administration.