Tonight I want to connect to an EC2 instance that resides in a private subnet. I currently don’t have a bastion host I can use as an intermediary to jump to this instance. Well, as you might have guessed from this post’s title, the AWS Session Manager is something that might be helpful in this case. So, join me in this journey to configure and set up the Session Manager, using AWS CLI only.

First, let’s start with some assumptions:

  • The IAM user used to run all the CLI command has full administration access.
  • There exist 3 SSM VPC endpoints (ssm, ec2messages, ssmmessages) configured for the VPC, in which the EC2 instance resides.
  • There exists an instance profile (IAM role) with correct permission for SSM Session Manager.
  • SSM agent has been installed, and the SSM agent service is running on the instance.
  • The instance is in running state, resides in a private subnet.
---------------------------------------------------------------------------
|                            DescribeInstances                            |
+---------------------+--------------------+-------------------+----------+
|     InstanceId      | PrivateIpAddress   |  PublicIpAddress  |  State   |
+---------------------+--------------------+-------------------+----------+
|  i-086ce7c8741234567|  10.0.15.100       |  None             |  running |
+---------------------+--------------------+-------------------+----------+

We know that for Session Manager to work, the EC2 instance needs to be associated with an IAM role (instance profile) with proper permission. To check if the instance has an instance profile associated, run the command below:

❯ aws ec2 describe-iam-instance-profile-associations \
--filters Name=instance-id,Values=i-086ce7c8741234567
{
    "IamInstanceProfileAssociations": []
}

Ok, currently there is no IAM role attached to it. No, worry. I have plenty of IAM roles, just need to figure out which one to use. I normally named my IAM roles with -kenno suffix. The following command list custom IAM roles that have names end with -kenno:

❯ aws iam list-roles | jq '.Roles | map(select(.RoleName | endswith("-kenno")).RoleName)'
[
  "lambda-access-kenno",
  "drs-role-2-kenno",
  "drs-role-kenno",
  "ec2-full-access-s3-kenno",
  "SSMInstanceProfile-kenno"
]

The SSMInstanceProfile-kenno looks like a good one to use in my case. But how do we know for sure that this role has the correct permission required by SSM? Let’s find out:

❯ aws iam list-attached-role-policies --role-name SSMInstanceProfile-kenno
{
    "AttachedPolicies": [
        {
            "PolicyName": "AmazonSSMManagedInstanceCore",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
        },
        {
            "PolicyName": "AmazonSSMPatchAssociation",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonSSMPatchAssociation"
        }
    ]
}

The output above shows that AmazonSSMManagedInstanceCore policy is attached to the role. This is what we need. The next step is to associate this role to the instance:

❯ aws ec2 associate-iam-instance-profile --iam-instance-profile Name=SSMInstanceProfile-kenno \
--instance-id i-086ce7c8741234567
{
    "IamInstanceProfileAssociation": {
        "AssociationId": "iip-assoc-0c2fb0c742bdc0a06",
        "InstanceId": "i-086ce7c8741234567",
        "IamInstanceProfile": {
            "Arn": "arn:aws:iam::144696666666:instance-profile/SSMInstanceProfile-kenno",
            "Id": "AIPASDMEX2OOQ12345678"
        },
        "State": "associating"
    }
}

Wait for about 30 seconds, then verify it again:

❯ aws ec2 describe-iam-instance-profile-associations \
--filters Name=instance-id,Values=i-086ce7c8741234567
{
    "IamInstanceProfileAssociations": [
        {
            "AssociationId": "iip-assoc-0c2fb0c7421234567",
            "InstanceId": "i-086ce7c8741234567",
            "IamInstanceProfile": {
                "Arn": "arn:aws:iam::144696666666:instance-profile/SSMInstanceProfile-kenno",
                "Id": "AIPASDMEX2OOQ12345678"
            },
            "State": "associated"
        }
    ]
}

Important note: even the IAM role is attached now, this does mean that we can connect to this instance.

❯ aws ssm start-session --target i-086ce7c8741234567

An error occurred (TargetNotConnected) when calling the StartSession operation: i-086ce7c8741234567 is not connected.

❯ aws ssm get-connection-status --target i-086ce7c8741234567
{
    "Target": "i-086ce7c8741234567",
    "Status": "notconnected"
}

Assuming that my instance can reach out to the SSM endpoints, I will have to wait for 5 to 10 minutes for the connection to SSM reported as “connected”. There is a workaround to get the SSM agent trying to reconnect to the endpoints soon - restart the EC2 instance, which in turn will trigger the SSM agent to start checking the endoints on boot.

❯ aws ssm get-connection-status --target i-086ce7c8741234567
{
    "Target": "i-086ce7c8741234567",
    "Status": "connected"
}

Finally, let’s connect to the instance using AWS CLI:

❯ aws ssm start-session --target i-086ce7c8741234567

SessionManagerPlugin is not found. Please refer to SessionManager Documentation here: http://docs.aws.amazon.com/console/systems-manager/sessio
n-manager-plugin-not-found

Ahhh… I’m missing the session-manager-plugin on my linux host which I run the above command. This local host runs OpenSUSE Leap 15.6, and there is no official package from AWS [z].

Fast forward to 10 minutes later, I found out that the binary session-manager-plugin from RPM package provided for RHEL 9 also runs on SUSE box.

❯ aws ssm start-session --target i-086ce7c8741234567

Starting session with SessionId: kenno-t4ivvvjnhti5kkkkkdn5mmmmmm
sh-5.2$ whoami
ssm-user
sh-5.2$

That’s it. There are lots of commands to run, so how are we supposed to remember most of them? Well, this is exactly why I wrote this note in the first place. Next time when I wanted to perform similar task, I can just come back to visit this page!

References: