AKS Azure Kubernetes

Using Key Vault with AKS – System Assigned – CSI Driver – with Pod Identity

Hey, so rather than re-write the same article, before reading and applying this blog, do and read this one – https://realworldit.net/archives/630 This will get you to the point where you can then install and use Pod Identity.

Let’s do this!!

6, After you have gone through the link above, now it’s time to install the Azure Active Directory (Azure AD) Pod Identity

The Pod Identity allows your cluster to perform Azure related operations such as assigning and un-assigning the identity on the underlying VMSS’s. AKS and aks-engine clusters require an identity to communicate with Azure. This identity can be either a managed ID or a service principal.

Part of the Azure AD Pod Identity is a Controller. This is called the MIC or Managed Identity Controller. This Kubernetes controller watches for changes to the pods, AZureIdentity and AzureIdentityBindings through the Kubernetes API Server. When it detects a change, the MIC adds or deletes AzureAssignedIdentity as needed.

Specifically, when a pod is scheduled, the MIC assigns the identity on Azure to the underlying VMSS during the creation phase. When all pods using the identity are deleted, it removes the identity from the underlying VMSS on Azure. The MIC takes similar actions when AzureIdentity or AzureIdentityBinding are created or deleted.

The NMI (Node Managed Identity) is another controller in Kubernetes that makes an Azure AD authentication library (ADAL) request to get a token on behalf of pods by intercepting IMDS traffic on each node and redirect them to itself.It queries Kubernetes through AzureAssignedIdentity for a matching Azure identity. The NMU then makes a request to get the token for the client ID and returns it as a response.

The least amount of roles required to perform these actions are the “Managed Identity Operator” role which needs to be assigned to the managed identity of your cluster and the scope set to the Key vault Resource group and the Node Resource group.

The Node resource group is the MC_xxxxx resource group for your k8s resources. If you have set the resource group’s name for the noderesourcegroup in your template then you will know the name of what you have set it to.

  1. You need to get the VMMS User assigned ID (the one ending in agentpool) and set the permissions on the Node resource group and Key Vault resource group. Then it will set the permissions for the Key Vault. Use the following commands.
$KvRSG = myKeyvaultVResourcegroup
$NodeRSG = myNodeResourcegroup
$subid = yoursubscriptionID

$clusterinfo = Get-AzVmss -Name 
$userai = $clusterinfo.identity.userassignedidentities
$agentpoolid =  $userai.keys -like '*agentpool'
$userai.Getenumerator() | foreach-object {
if ($_.key -eq $agentpoolid){

$clientid = $_.value.clientid
}
} 


az role assignment create --role "Managed Identity Operator" --assignee $clientId --scope /subscriptions/$subid/resourcegroups/$KvRSG

az role assignment create --role "Managed Identity Operator" --assignee $clientId --scope /subscriptions/$subid/resourcegroups/$NodeRSG

az role assignment create --role "Virtual Machine Contributor" --assignee $clientId --scope /subscriptions/$subid/resourcegroups/$NodeRSG

az keyvault set-policy -n myKeyVault --secret-permissions get --spn $clientId

2. After assigning the roles to the managed ID, we need to install Azure AD identity into AKS. This can be done with a couple of helm commands.

helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install pod-identity aad-pod-identity/aad-pod-identity

to verify (you will see this command after you install pod-identity)
To verify that AAD Pod Identity has started in standard mode, run:
  kubectl --namespace=default get pods -l "app.kubernetes.io/component=mic"
  kubectl --namespace=default get pods -l "app.kubernetes.io/component=nmi"


3. We need to create an AzureIdentity in your cluster that will reference the identity you have just created.

  • put in the clientID from the ID for the VMSS ID (it’s up above $clientid).
  • Fill in the resourceID, it needs to point to the identity you created which is located in the AKSCluster resource group.
  • Fill in the metadata – name – this should be in lowercase. I have called it azure-pod-id
  • Save it as aadpodidentity.yaml

apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
    name: azure-pod-id                 # The name of your Azure identity
spec:
    type: 0                                 # Set type: 0 for managed identity
    resourceID: /subscriptions/<SUBID>/resourcegroups/<RESOURCEGROUP>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<AZUREIDENTITYNAME>
    clientID: "<managed identity clientID>"   # The clientId of the User-assigned managed identity (VMMS agentpool)

4. Then, Create an AzureIdentityBinding which will refer to the AzureIdentity. In the template below,

  • I have called it similar names to keep it easier to read and connect.
  • The name is azure-pod-id-binding
  • the spec - azureIdentity, needs to be set to the name of the AzureIdentity in the template above. In this case azure-pod-id
  • Save it as aadpodIdentityBinding.yaml
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
    name: azure-pod-id-binding
spec:
    azureIdentity: "azure-pod-id"      # The name of your Azure identity
    selector: azure-pod-identity-binding-selector

5. In the article – https://realworldit.net/archives/630 we created a SecretProviderClass object. This needs to be amended for Pod Identity. If you have already deployed the SecretProviderClass, it’s time to “undeploy it”

kubectl delete -f SecretProviderClass.yaml

Now we can edit the file and redeploy it.

# This is a SecretProviderClass example using Pod identity to access Keyvault
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  parameters:
    usePodIdentity: "true"
 ' Set to true for using aad-pod-identity to access keyvault


    keyvaultName: "FILL THIS IN"
    cloudName: ""                   # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: secret1 FILL THIS IN
          objectType: secret        # object types: secret, key or cert
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: key1
          objectType: key
          objectVersion: ""
    tenantId: "FILL THIS IN"                 # the tenant ID of the KeyVault

Ok so if all the files have been created we can deploy them.

kubectl apply -f SecretProviderClass.yaml
kubectl apply -f aadpodIdentity.yaml
kubectl apply -f aadpodIdentityBinding.yaml

6. Next it’s time to deploy a pod and see if it works! Use the template below. Save it as podbinding.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-secrets-store-inline2
  labels:
    aadpodidbinding: azure-pod-identity-binding-selector # The selector defined in AzureIdentityBinding in the previous step
spec:
  containers:
    - name: nginx
      image: nginx
      volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: azure-kvname

Run the following command to deploy it. Note a couple of changes I’ve called the pod nginx-secrets-store-inline2 this is incase you have deployed an nginx-secrets-store-inline pod, as you will get an error if the names are the same. also check that the secretProviderClass is set correctly.

kubectl apply -f podbinding.yaml

To check it’s deployed

kubectl describe pod/nginx-secrets-store-inline2 

you should see at the bottom, events. If the container is created and started all is good.

7. To check the pod can access the secrets

kubectl exec nginx-secrets-store-inline2 -- ls /mnt/secrets-store/

This will display the list of secrets

To see the value of the secret – change the adminpassword to whatever you named your secret.

kubectl exec nginx-secrets-store-inline2 --cat /mnt/secrets-store/adminpassword

Useful Link https://azure.github.io/aad-pod-identity/docs/getting-started/role-assignment/

Hope this helped! Please leave a comment.