Session is closed after 30 seconds when plugin built from source

[ilons]

Summary

When I'm using the session-manager-plugin built from source (instead of the pre-packaged binary), the plugin would exit after 30 seconds of inactivity when using the AWS-StartPortForwardingSession document.

Pre-built binary does not exit after 30 seconds of inactivity.

Version for plugin built using official instructions are shown incorrectly (displaying only major and minor version), which leads me to believe that third number is the internal AWS build id (?).

Steps to reproduce

1. Build plugin

git clone https://github.com/aws/session-manager-plugin.git
cd session-manager-plugin
git checkout 1.2.553.0
docker build -t session-manager-plugin-image .
docker run -it --rm --name session-manager-plugin \
    -v `pwd`:/session-manager-plugin \
    --entrypoint /bin/sh \
    session-manager-plugin-image \
        -c "head -n1 /etc/issue && go version && make release"
# Debian GNU/Linux 11 \n \l
# go version go1.17.13 linux/amd64

2. Verify version

PATH=${PWD}/bin/linux_amd64_plugin/:${PATH} session-manager-plugin --version
# 1.2.0.0

3. Start session

PATH=${PWD}/bin/linux_amd64_plugin/:${PATH} \
&& time timeout 60 aws ssm start-session \
    --document-name AWS-StartPortForwardingSession \
    --target ecs:<cluster_name>_<task_id>_<container_runtime_id> \
    --parameters '{"portNumber": ["8080"], "localPortNumber": ["8080"]}'
# Starting session with SessionId: XXX
# Port 8080 opened for sessionId XXX
# Waiting for connections...
# Exiting session with sessionId: XXX
# real	0m31.375s

I'm seeing this consistently across different versions of the built plugin, in multiple (Linux) execution environments (Pop OS 22.04, Alpine 3.19) using both the AWS CLI to start a session as well as using python with boto3 to get the session token and calling the session-manager-plugin binary.

Using pre-built binary

Wehn using the pre-built binary everything works as expected:

1. Fetch binary

mkdir pre-built
cd pre-built
wget https://s3.amazonaws.com/session-manager-downloads/plugin/1.2.553.0/ubuntu_64bit/session-manager-plugin.deb -O session-manager-plugin.deb
ar x session-manager-plugin.deb
tar -zxvf data.tar.gz

2. Verify version

PATH=${PWD}/usr/local/sessionmanagerplugin/bin/:${PATH} session-manager-plugin --version
# 1.2.553.0

4. Start session

PATH=${PWD}/usr/local/sessionmanagerplugin/bin/:${PATH} \
&& time timeout 60 aws ssm start-session \
    --document-name AWS-StartPortForwardingSession \
    --target ecs:<cluster_name>_<task_id>_<container_runtime_id> \
    --parameters '{"portNumber": ["8080"], "localPortNumber": ["8080"]}'
# Starting session with SessionId: XXX
# Port 8080 opened for sessionId XXX
# Waiting for connections...
# Exiting session with sessionId: XXX
# real	1m0.009s

Questions

1. Is there any way to tweak this timeout I'm seeing (seems to be purely on the client side)?
2. Why does the binary built using the official instructions differs from the pre-built binary?

Similar issues:

• Session Manager Plugin built from source code exits unexpectedly #57

[clementlecorre]

I have a same issue !

[chrisdd3]

From what i understand, the issue is the version inside the VERSION file.
Set it to the latest release tag (currently 1.2.553.0) and the built binary should work fine ( make release )

Explanation:

• version.Version ends up being sent to the server
• The ssm-agent checks that version and has different handling for some versions
• The value of version.Version is generated from the VERSION file
  • makefile command
  • version-generator

I am guessing the release pipeline from aws is replacing the version properly.

[yuting-fan]

Hi @ilons ,

What's the version of SSM agent you're using on the managed instance (your ECS container)? There were a couple of fixes and improvements done in later versions of both Session Manager plugin (SSM CLI) and SSM agent to handle the premature session termination issue, so I'd recommend to always upgrade to the latest version for both.

As for building the CLI and agent from source, @chrisdd3 is correct that the VERSION file defaults to an older version. During our official release, this version is replaced at runtime. So if you build your own binary from source, you would need to update the VERSION to match the release tag of the source code you're building from.

Please let us know if you continue to experience issue after upgrading to the latest versions, and feel free to open a support case to share your logs for us to investigate more deeply.

Regards,
Yuting

[rajyan]

@yuting-fan @Yangtao-Hua

As for building the CLI and agent from source, chrisdd3 is correct that the VERSION file defaults to an older version. During our official release, this version is replaced at runtime. So if you build your own binary from source, you would need to update the VERSION to match the release tag of the source code you're building from.

This behavior is confusing and causing problems for many people.

• Session is closed after 30 seconds when plugin built from source #93
• Port forwarding session times out after 1min in Golang #83
• Build Instructions wrt VERSION #76
• Session Manager Plugin built from source code exits unexpectedly #57

Additionally, some distributions mistakenly build binaries that send wrong version information (v1.2.0.0) which leads to timeout of 30 seconds.

Could you please a take a look at #104 , or update the instructions in README?