| Healthcheck Error | Cause & Resolution | Cluster Log |
| cannot dial resource: cannot probe: Head "https://redacted.gr7.us-west-2.eks.amazonaws.com:443": dial tcp: lookup redacted.gr7.us-west-2.eks.amazonaws.com on 127.0.0.53:53: no such host |
Cause: DNS resolution for the provided Endpoint hostname failed. Resolution: Ensure the Endpoint matches the cluster’s API server endpoint. |
N/A |
| cannot dial resource: cannot probe: Head "https://redacted.gr7.us-west-2.eks.amazonaws.com:443": context deadline exceeded (Client.Timeout exceeded while awaiting headers) |
Cause: There is no Ingress rule in the EKS’s security groups allow access to port 443 from any SDM Gateways or Relays
|
N/A |
| cannot dial resource: cannot probe: Head "https://redacted.gr7.us-west-2.eks.amazonaws.com:443": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes") |
Cause: The Server CA provided is incorrect. Resolution: Ensure the Server CA matches the cluster’s Certificate Authority |
N/A |
| failed to initialize plugin: cannot create TLS transport: cannot parse server CA |
Cause: The Server CA in the resource settings is a Secret Store path to a Base64-encoded value. Resolution: Append &encoding=base64 to the path. Example: sdm/secrets?key=eksca&encoding=base64 |
N/A |
| Invalid Credentials, lacking read permissions, or server not found. |
Cause: The resource’s Cluster Name does not match the name of the EKS cluster in AWS. Resolution: Ensure the resource Cluster Name matches the cluster’s cluster name. |
EKS Cluster Authentication Log: error="sts getCallerIdentity failed: error from AWS (expected 200, got 403). Body: {\"Error\":{\"Code\":\"SignatureDoesNotMatch\",\"Message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\",\"Type\":\"Sender\"},\"RequestId\":\"6059df54-f0b6-4cf1-98ad-5a27bcec1db9\"}" method=POST path=/authenticate |
| Invalid Credentials, lacking read permissions, or server not found. |
Cause: The Instance Profile Role ARN (attached to your StrongDM node) specified in the resource is not Resolution: Add the Role ARN to the cluster’s mapRoles in configmap. |
EKS Cluster Authentication Log: level=warning msg="access denied" arn="arn:aws:iam::xxxxxxxxxxxx:role/sdm-node-role" client="127.0.0.1:41666" error="ARN is not mapped" method=POST path=/authenticate |
| Invalid Credentials, lacking read permissions, or server not found. |
Cause: The Assume Role ARN specified in the resource is not mapped in the Clusters RBAC. Resolution: Add the Role ARN to the cluster’s mapRoles in configmap. |
EKS Cluster Authentication Log: level=warning msg="access denied" arn="arn:aws:iam::xxxxxxxxxxxx:role/k8sDev" client="127.0.0.1:41666" error="ARN is not mapped" method=POST path=/authenticate |
| Invalid Credentials, lacking read permissions, or server not found. |
Cause: Group name specified in the configmap doesn’t match the group name in RoleBindings. Resolution: Correct the RBAC RoleBindings. |
EKS Cluster Authentication Log: level=info msg="access granted" arn="arn:aws:iam::671496566364:role/mary-k8sDev" client="127.0.0.1:38078" groups="[dev-access-group]" method=POST path=/authenticate uid="aws-iam-authenticator:xxxxxxxxxxxxAROAZYWCQOZOHGFV7OAZP" username=k8sDev EKS Cluster Audit Log: { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Request", "auditID": "adf1482c-a237-4d46-a9fb-a8f5f8fa1788", "stage": "ResponseComplete", "requestURI":"/api/v1/namespaces/dev", "verb": "get", "user": { "username": "k8sDev", "uid": "aws-iam-authenticator:xxxxxxxxxxxx:AROAZYWCQOZOHGFV7OAZP", "groups": [ "dev-access-group", "system:authenticated" ], "extra": { "accessKeyId": [ "ASIAZYWCQOZOBZ2DF4G7" ], "arn": [ "arn:aws:sts::xxxxxxxxxxxx:assumed-role/k8sDev/1697836792291792707" ], "canonicalArn": [ "arn:aws:iam::xxxxxxxxxxxx:role/k8sDev" ], "principalId": [ "AROAZYWCQOZOHGFV7OAZP" ], "sessionName": [ "1697836792291792707" ] } }, "sourceIPs": [ "xxx.xxx.xxx.xxx ], "userAgent": "Go-http-client/1.1", "objectRef": { "resource": "namespaces", "namespace": "dev", "name": "dev", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "status": "Failure", "message": "namespaces \"dev\" is forbidden: User \"k8sDev\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"dev\"", "reason": "Forbidden", "details": { "name":"dev", "kind": "namespaces" }, "code": 403 }, "requestReceivedTimestamp": "2023-10-20T21:19:52.410702Z", "stageTimestamp": "2023-10-20T21:19:52.739234Z", "annotations": { "authorization.k8s.io/decision": "forbid", "authorization.k8s.io/reason": "" } }
RBAC detail: $ eksctl get iamidentitymapping --cluster cluster-name |
| Invalid Credentials, lacking read permissions, or server not found. |
Cause: The RBAC Role specified in the RoleBinding doesn’t exist in the specified namespace. Resolution: Create the role in the specified namespace. |
EKS Cluster Audit Log: "userAgent": "Go-http-client/1.1", "objectRef": { "resource": "pods", "namespace": "dev", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "status": "Failure", "message": "pods is forbidden: User \"k8sdev\" cannot list resource \"pods\" in API group \"\" in the namespace \"dev\": RBAC: role.rbac.authorization.k8s.io \"devOps-role\" not found", "reason": "Forbidden", "details": { "kind": "pods" }, "code": 403 kubectl describe rolebinding -n dev |
|
Invalid Credentials, lacking read permissions, or server not found. Resource Type = Elastic Kubernetes Cluster (Instance Profile) |
Cause: No Assume Role Specified, so the Instance Profile (attached to the StrongDM node) is used to authenticate and is not mapped in aws-auth configmap. –or– The Assume Role ARN in resource settings is the Instance Profile Role ARN Resolution: If you’re using the Instance Profile to grant permissions to users connecting to the resource, add it to the aws-auth configmap and do not specify an Assume Role ARN in the resource settings. If you prefer to use an Assume Role, create another IAM role and grant the Instance Profile role STS:Assume Role to this new role. |
EKS Cluster Audit Log: {
|
|
Invalid Credentials, lacking read permissions, or server not found. Resource Type = Elastic Kubernetes Cluster (Instance Profile) |
Cause: The Instance Profile Role ARN (attached to your StrongDM node) used to authenticate with the EKS Cluster is not mapped to a Kubernetes Group in the aws-auth configmap. Resolution: Add the Role ARN to the cluster’s mapRoles in configmap. |
EKS Cluster Audit Log: {
|