Troubleshooting EKS Resource Healthcheck Errors

Cause: DNS resolution for the provided Endpoint hostname failed. 

Resolution: Ensure the Endpoint matches the cluster’s API server endpoint.

Cause: There is no Ingress rule in the EKS’s security groups allow access to port 443 from any SDM Gateways or Relays

Resolution: Add an Ingress rule to the add-on Security Group associated with the EKS cluster to allow Ingress TCP port 443 from SDM node source(s).

Cause: The Server CA provided is incorrect.

Resolution: Ensure the Server CA matches the cluster’s Certificate Authority

Cause: The Server CA in the resource settings is a Secret Store path to a  Base64-encoded value.

Resolution: Append &encoding=base64 to the path. Example: sdm/secrets?key=eksca&encoding=base64

Cause: The resource’s Cluster Name does not match the name of the EKS cluster in AWS. 

Resolution: Ensure the resource Cluster Name matches the cluster’s cluster name.

EKS Cluster Authentication Log:

error="sts getCallerIdentity failed: error from AWS (expected 200, got 403). Body: {\"Error\":{\"Code\":\"SignatureDoesNotMatch\",\"Message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.

Consult the service documentation for details.\",\"Type\":\"Sender\"},\"RequestId\":\"6059df54-f0b6-4cf1-98ad-5a27bcec1db9\"}" method=POST path=/authenticate

Cause: The Instance Profile Role ARN (attached to your StrongDM node) specified in the resource is not 

Resolution: Add the Role ARN to the cluster’s mapRoles in configmap.

EKS Cluster Authentication Log:

level=warning msg="access denied" arn="arn:aws:iam::xxxxxxxxxxxx:role/sdm-node-role" client="127.0.0.1:41666" error="ARN is not mapped" method=POST path=/authenticate

Cause: The Assume Role ARN specified in the resource is not mapped in the Clusters RBAC.

Resolution: Add the Role ARN to the cluster’s mapRoles in configmap.

EKS Cluster Authentication Log:

level=warning msg="access denied" arn="arn:aws:iam::xxxxxxxxxxxx:role/k8sDev" client="127.0.0.1:41666" error="ARN is not mapped" method=POST path=/authenticate

Cause: Group name specified in the configmap doesn’t match the group name in RoleBindings.

Resolution: Correct the RBAC RoleBindings.

EKS Cluster Authentication Log:

level=info msg="access granted" arn="arn:aws:iam::671496566364:role/mary-k8sDev" client="127.0.0.1:38078" groups="[dev-access-group]" method=POST path=/authenticate uid="aws-iam-authenticator:xxxxxxxxxxxxAROAZYWCQOZOHGFV7OAZP" username=k8sDev

EKS Cluster Audit Log:

{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Request", "auditID":

"adf1482c-a237-4d46-a9fb-a8f5f8fa1788", "stage": "ResponseComplete", "requestURI":"/api/v1/namespaces/dev",

"verb": "get", "user": { "username": "k8sDev", "uid": "aws-iam-authenticator:xxxxxxxxxxxx:AROAZYWCQOZOHGFV7OAZP", "groups": [ "dev-access-group", "system:authenticated" ], "extra": { "accessKeyId": [ "ASIAZYWCQOZOBZ2DF4G7" ], "arn": [ "arn:aws:sts::xxxxxxxxxxxx:assumed-role/k8sDev/1697836792291792707" ], "canonicalArn": [ "arn:aws:iam::xxxxxxxxxxxx:role/k8sDev" ], "principalId": [ "AROAZYWCQOZOHGFV7OAZP" ], "sessionName": [ "1697836792291792707" ] } }, "sourceIPs": [ "xxx.xxx.xxx.xxx ], "userAgent": "Go-http-client/1.1", "objectRef": { "resource": "namespaces", "namespace": "dev", "name": "dev", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "status": "Failure", "message": "namespaces \"dev\" is forbidden: User \"k8sDev\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"dev\"", "reason": "Forbidden", "details": { "name":"dev",

"kind": "namespaces" }, "code": 403 }, "requestReceivedTimestamp": "2023-10-20T21:19:52.410702Z", "stageTimestamp": "2023-10-20T21:19:52.739234Z", "annotations": { "authorization.k8s.io/decision": "forbid", "authorization.k8s.io/reason": "" } }

RBAC detail:

$ eksctl get iamidentitymapping --cluster cluster-name

----------------------------------------------------
ARN                                                                       USERNAME   
       GROUPS  ACCOUNT
arn:aws:iam::xxxxxxxxxxxx:role/eksctl-cluster-name-nodegroup-worker-n-NodeInstanceRole-kE6cjaE
3ob2v  system:node:{{EC2PrivateDNSName}}    system:bootstrappers,system:nodes    
arn:aws:iam::671496566364:role/k8sDev                                       k8sDev   
              dev-access-group     
arn:aws:iam::671496566364:role/mary-sdm-secrets-role                                   admin   
                           system:masters  




$ kubectl describe rolebinding -n dev

----------------------------------
Name:     devOps-role-binding
Labels:   <none>
Annotations:  <none>
Role:
  Kind:  Role
  Name:  devOps-role
Subjects:
  Kind   Name         Namespace
  ----   ----         ---------
  Group  dev-access-goup  

Cause: The RBAC Role specified in the RoleBinding doesn’t exist in the specified namespace.

Resolution: Create the role in the specified namespace.

EKS Cluster Audit Log:

"userAgent": "Go-http-client/1.1", "objectRef": { "resource": "pods", "namespace": "dev", "apiVersion": "v1" },

"responseStatus": { "metadata": {}, "status": "Failure", "message": "pods is forbidden: User \"k8sdev\"

cannot list resource \"pods\" in API group \"\" in the namespace \"dev\": RBAC:

role.rbac.authorization.k8s.io \"devOps-role\" not found", "reason": "Forbidden", "details": { "kind": "pods"

}, "code": 403

kubectl describe rolebinding -n dev
Name:     devOps-role-binding
Labels:   <none>
Annotations:  <none>
Role:
  Kind:  Role
  Name:  devOps-role
Subjects:
  Kind   Name          Namespace
  ----   ----          ---------
  Group  dev-access-group  

ubuntu@ip-172-31-120-147:~/instanceProfileAssumeRole$ kubectl describe role -n dev
No resources found in dev namespace.