/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found

[slondr]

I made sure the issue is in bootsnap

Yep (also, btw, there is a typo in the GitHub issue template which appears right after this header)

Steps to reproduce

Run any Rails command after installing bootsnap via bundler

Expected behavior

In this case: I ran rails server and expected my server to start

Actual behavior

Application crashed almost immediately, with a big error message starting with LoadError originating in bootsnap.so. I posted the full error message at the bottom of this issue.

System configuration

Bootsnap version: 1.15.0

Ruby version: 2.7.5

Rails version: 6.0.5

Full error message

Puma starting in single mode...
* Puma version: 5.6.5 (ruby 2.7.5-p203) ("Birdie's Version")
*  Min threads: 0
*  Max threads: 5
*  Environment: development
*          PID: 192898
! Unable to load application: LoadError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so) - /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so
Traceback (most recent call last):
	31: from /home/user/.rvm/gems/ruby-2.7.5/bin/puma:23:in `<main>'
	30: from /home/user/.rvm/gems/ruby-2.7.5/bin/puma:23:in `load'
	29: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/bin/puma:10:in `<top (required)>'
	28: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/cli.rb:81:in `run'
	27: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/launcher.rb:193:in `run'
	26: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/single.rb:44:in `run'
	25: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/runner.rb:150:in `load_and_bind'
	24: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/configuration.rb:270:in `app'
	23: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/configuration.rb:348:in `load_rackup'
	22: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:66:in `parse_file'
	21: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:105:in `load_file'
	20: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:116:in `new_from_string'
	19: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:116:in `eval'
	18: from config.ru:3:in `block in <main>'
	17: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	16: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	15: from /home/user/dev/core/config/environment.rb:2:in `<top (required)>'
	14: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	13: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	12: from /home/user/dev/core/config/application.rb:1:in `<top (required)>'
	11: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	10: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	 9: from /home/user/dev/core/config/boot.rb:4:in `<top (required)>'
	 8: from /home/user/dev/core/config/boot.rb:4:in `require'
	 7: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/setup.rb:5:in `<top (required)>'
	 6: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap.rb:100:in `default_setup'
	 5: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap.rb:57:in `setup'
	 4: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:in `setup'
	 3: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:in `require_relative'
	 2: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache/iseq.rb:3:in `<top (required)>'
	 1: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require': /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so) - /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so (LoadError)

[casperisfine]

Interesting. Likely caused by 0de36ee, but I wonder if that's even needed.

[gabrielrinaldi]

I am also having the same issue, on my Mac it runs fine, but when dockerizing it it fails with the same message. For some reason building the docker image on my Mac does not cause the issue, but it does on the CI machine.

[gabrielrinaldi]

After more digging I think this is related to the new release of the images, which only has GLIBC_2.35, would that make sense?

[casperisfine]

Yeah that would male sense. if the base image is public would be nice to share it, that would help confirm the bug on my side.

That said I really don't see why we're put the whole glibc in the cache key. I'm pretty sure we can just remove that code, but I need to dig in the history to confirm.

[casperisfine]

Hum, wait, something super weird here. We remove the GLIBC version check in 1.12.0 https://github.com/Shopify/bootsnap/blob/main/CHANGELOG.md#1120

[casperisfine]

Do you re-use your bundler cache across docker image builds? I'm thinking you might need to recompile bootsnap.

[gabrielrinaldi]

This is my docker file:

# Base image
ARG RUBY_VERSION=3.1.3
ARG VARIANT=slim-bullseye

FROM ruby:${RUBY_VERSION}-${VARIANT} as base

ARG BUNDLE_PATH=vendor/bundle
ARG BUNDLE_WITHOUT=development:test
ARG RAILS_ENV=production

ENV BUNDLE_PATH ${BUNDLE_PATH}
ENV BUNDLE_WITHOUT ${BUNDLE_WITHOUT}
ENV RAILS_ENV=${RAILS_ENV}

RUN mkdir /app

WORKDIR /app

RUN mkdir -p tmp/pids

# Build dependencies
FROM base as build_deps

ARG DEV_PACKAGES="git build-essential libpq-dev wget vim curl gzip xz-utils"

ENV DEV_PACKAGES ${DEV_PACKAGES}

RUN --mount=type=cache,id=dev-apt-cache,sharing=locked,target=/var/cache/apt \
  --mount=type=cache,id=dev-apt-lib,sharing=locked,target=/var/lib/apt \
  apt-get update -qq && \
  apt-get install --no-install-recommends -y ${DEV_PACKAGES} \
  && rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Build gems
FROM build_deps as gems

ARG BUNDLER_VERSION=2.3.26

ENV BUNDLER_VERSION ${BUNDLER_VERSION}

RUN gem install -N bundler -v ${BUNDLER_VERSION}

COPY Gemfile* ./
RUN bundle install && rm -rf vendor/bundle/ruby/*/cache

# Production image
FROM base

ARG NODE_VERSION=18.12.1
ARG PROD_PACKAGES="postgresql-client file vim curl gzip gnupg ca-certificates wget xz-utils"
ARG SERVER_COMMAND="bin/rails fly:server"

ENV PORT 8080
ENV PROD_PACKAGES=${PROD_PACKAGES}
ENV SERVER_COMMAND ${SERVER_COMMAND}

RUN groupadd --gid 1000 node \
  && useradd --uid 1000 --gid node --shell /bin/bash --create-home node

RUN --mount=type=cache,id=prod-apt-cache,sharing=locked,target=/var/cache/apt \
  --mount=type=cache,id=prod-apt-lib,sharing=locked,target=/var/lib/apt \
  apt-get update -qq && \
  apt-get install --no-install-recommends -y \
  ${PROD_PACKAGES} \
  && rm -rf /var/lib/apt/lists /var/cache/apt/archives

RUN set -ex \
  && for key in \
  4ED778F539E3634C779C87C6D7062848A1AB005C \
  141F07595B7B3FFE74309A937405533BE57C7D57 \
  74F12602B6F1C4E913FAA37AD3A89613643B6201 \
  61FC681DFB92A079F1685E77973F295594EC4689 \
  8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600 \
  C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 \
  890C08DB8579162FEE0DF9DB8BEAB4DFCF555EF4 \
  C82FA3AE1CBEDC6BE46B9360C43CEC45C17AB93C \
  108F52B48DB57BB0CC439B2997B01419BD92F80A \
  ; do \
  gpg --keyserver hkps://keys.openpgp.org --recv-keys "$key"; \
  done

RUN ARCH= && dpkgArch="$(dpkg --print-architecture)" \
  && case "${dpkgArch##*-}" in \
  amd64) ARCH='x64';; \
  ppc64el) ARCH='ppc64le';; \
  s390x) ARCH='s390x';; \
  arm64) ARCH='arm64';; \
  armhf) ARCH='armv7l';; \
  i386) ARCH='x86';; \
  *) echo "unsupported architecture"; exit 1 ;; \
  esac \
  && set -x \
  && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-$ARCH.tar.xz" \
  && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
  && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
  && grep " node-v$NODE_VERSION-linux-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
  && tar -xJf "node-v$NODE_VERSION-linux-$ARCH.tar.xz" -C /usr/local --strip-components=1 --no-same-owner \
  && rm "node-v$NODE_VERSION-linux-$ARCH.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt \
  && apt-get purge -y --auto-remove xz-utils \
  && ln -s /usr/local/bin/node /usr/local/bin/nodejs

RUN npm install -g mjml

COPY --from=gems /app /app

ENV SECRET_KEY_BASE 1

COPY . .

RUN bin/rails fly:build

EXPOSE ${PORT}

CMD ${SERVER_COMMAND}

fly:build is just an alias to assets:precompile

[casperisfine]

Ok, so I suspect the issue might come from the fact that you compile the gems in a container, then import in your image.

I assume build-essential upgrade the libc or something.

Just for debugging purposes can you try running that same apt-get install in the main image? If that fix the issue it will confirm that it's the problem. And then the proper fix might be to upgrade only the libc package to make sure it match the one used for compilation.

[gabrielrinaldi]

Removing the caching does solve the problems (everything in one image), but adding build-essential to the final image does not make a difference.

I think it is fine to close this as a bootsnap related issue, the issue seems to be ubuntu-latest becoming 22 vs 20. It just coincided for me that both happened at the same time (bootsnap upgrade and the ubuntu-latest).

[slondr]

Can this issue be re-opened? gabrielrinaldi's problem seems unrelated to mine — I am not using containers or macOS.

[casperisfine]

The issue is likely to be similar nonetheless. Please try to flush your bundler cache (or just uninstall bootsnap and re-bundle).

If the issue persists I'll re-open, but I'll need more info to investigate.

[tcarac]

I'm observing a similar issue after having upgraded to ruby 3.2

/home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/setup.rb:5:in <top (required)>'
/home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap.rb:100:in default_setup' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap.rb:57:in setup'
/home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:in setup' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:in require_relative'
/home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache/iseq.rb:3:in <top (required)>' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require'
/home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' LoadError: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so) - /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so`

[slondr]

@casperisfine Uninstalling bootsnap, re-bundling, then reinstalling bootsnap did not fix the issue.

What other info do you need from me?

[casperisfine]

What other info do you need from me?

At this point, I'm at a loss as to what it might be. If you could repro this in isolation via a Dockerfile that would help a lot.

In the meantime I'll ask some colleagues if that rings a bell.