Troubleshooting EKS Resource Healthcheck Errors

Cause: DNS resolution for the provided Endpoint hostname failed. 

Resolution: Ensure the Endpoint matches the cluster’s API server endpoint.

Cause: The Server CA provided is incorrect.

Resolution: Ensure the Server CA matches the cluster’s Certificate Authority

Cause: The Server CA in the resource settings is a Secret Store path to a  Base64-encoded value.

Resolution: Append &encoding=base64 to the path. Example: sdm/secrets?key=eksca&encoding=base64

Cause: The resource’s Cluster Name does not match the name of the EKS cluster in AWS. 

Resolution: Ensure the resource Cluster Name matches the cluster’s cluster name.

EKS Cluster Authentication Log:

error="sts getCallerIdentity failed: error from AWS (expected 200, got 403). Body: {\"Error\":{\"Code\":\"SignatureDoesNotMatch\",\"Message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.

Consult the service documentation for details.\",\"Type\":\"Sender\"},\"RequestId\":\"6059df54-f0b6-4cf1-98ad-5a27bcec1db9\"}" method=POST path=/authenticate

Cause: The Instance Profile Role ARN (attached to your StrongDM node) specified in the resource is not 

Resolution: Add the Role ARN to the cluster’s mapRoles in configmap.

EKS Cluster Authentication Log:

level=warning msg="access denied" arn="arn:aws:iam::xxxxxxxxxxxx:role/sdm-node-role" client="127.0.0.1:41666" error="ARN is not mapped" method=POST path=/authenticate

Cause: The Assume Role ARN specified in the resource is not mapped in the Clusters RBAC.

Resolution: Add the Role ARN to the cluster’s mapRoles in configmap.

EKS Cluster Authentication Log:

level=warning msg="access denied" arn="arn:aws:iam::xxxxxxxxxxxx:role/k8sDev" client="127.0.0.1:41666" error="ARN is not mapped" method=POST path=/authenticate

Cause: Group name specified in the configmap doesn’t match the group name in RoleBindings.

Resolution: Correct the RBAC RoleBindings.

EKS Cluster Authentication Log:

level=info msg="access granted" arn="arn:aws:iam::671496566364:role/mary-k8sDev" client="127.0.0.1:38078" groups="[dev-access-group]" method=POST path=/authenticate uid="aws-iam-authenticator:xxxxxxxxxxxxAROAZYWCQOZOHGFV7OAZP" username=k8sDev

EKS Cluster Audit Log:

{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Request", "auditID": "adf1482c-a237-4d46-a9fb-a8f5f8fa1788", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/dev", "verb": "get", "user": { "username": "k8sDev", "uid": "aws-iam-authenticator:xxxxxxxxxxxx:AROAZYWCQOZOHGFV7OAZP", "groups": [ "dev-access-group", "system:authenticated" ], "extra": { "accessKeyId": [ "ASIAZYWCQOZOBZ2DF4G7" ], "arn": [ "arn:aws:sts::xxxxxxxxxxxx:assumed-role/k8sDev/1697836792291792707" ], "canonicalArn": [ "arn:aws:iam::xxxxxxxxxxxx:role/k8sDev" ], "principalId": [ "AROAZYWCQOZOHGFV7OAZP" ], "sessionName": [ "1697836792291792707" ] } }, "sourceIPs": [ "xxx.xxx.xxx.xxx ], "userAgent": "Go-http-client/1.1", "objectRef": { "resource": "namespaces", "namespace": "dev", "name": "dev", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "status": "Failure", "message": "namespaces \"dev\" is forbidden: User \"k8sDev\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"dev\"", "reason": "Forbidden", "details": { "name": "dev", "kind": "namespaces" }, "code": 403 }, "requestReceivedTimestamp": "2023-10-20T21:19:52.410702Z", "stageTimestamp": "2023-10-20T21:19:52.739234Z", "annotations": { "authorization.k8s.io/decision": "forbid", "authorization.k8s.io/reason": "" } }

RBAC detail:

$ eksctl get iamidentitymapping --cluster cluster-name

----------------------------------------------------

ARN   USERNAME   GROUPS    ACCOUNT

arn:aws:iam::xxxxxxxxxxxx:role/eksctl-cluster-name-nodegroup-worker-n-NodeInstanceRole-kE6cjaE3ob2v    system:node:{{EC2PrivateDNSName}}    system:bootstrappers,system:nodes    

arn:aws:iam::671496566364:role/k8sDev   k8sDev   dev-access-group     

arn:aws:iam::671496566364:role/mary-sdm-secrets-role   admin   system:masters  

$ kubectl describe rolebinding -n dev

----------------------------------

Name:     devOps-role-binding

Labels:   <none>

Annotations:  <none>

Role:

  Kind:  Role

  Name:  devOps-role

Subjects:

  Kind   Name         Namespace

  ----   ----         ---------

  Group  dev-access-goup  

Cause: The RBAC Role specified in the RoleBinding doesn’t exist in the specified namespace.

Resolution: Create the role in the specified namespace.

EKS Cluster Audit Log:

"userAgent": "Go-http-client/1.1", "objectRef": { "resource": "pods", "namespace": "dev", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "status": "Failure", "message": "pods is forbidden: User \"k8sdev\" cannot list resource \"pods\" in API group \"\" in the namespace \"dev\": RBAC: role.rbac.authorization.k8s.io \"devOps-role\" not found", "reason": "Forbidden", "details": { "kind": "pods" }, "code": 403

kubectl describe rolebinding -n dev

Name:     devOps-role-binding

Labels:   <none>

Annotations:  <none>

Role:

  Kind:  Role

  Name:  devOps-role

Subjects:

  Kind   Name          Namespace

  ----   ----          ---------

  Group  dev-access-group  

ubuntu@ip-172-31-120-147:~/instanceProfileAssumeRole$ kubectl describe role -n dev

No resources found in dev namespace.