[Node] Install aws-parallelcluster-node from S3 in all regions by himani2411 · Pull Request #3211 · aws/aws-parallelcluster-cookbook

himani2411 · 2026-06-24T16:10:12Z

Description of changes

The node package was installed from S3 only in us-iso regions; everywhere else it install from PyPI ('pip install aws-parallelcluster-node'), breaking air-gapped/proxied build-image in commercial and govcloud.

Set custom_node_package to the official S3 node URL in all regions by default, routing through the existing custom_parallelcluster_node S3 install path. Reuse the existing install_python_from_internet attribute as the opt-out: set it to install from PyPI instead. A user-provided custom_node_package still takes precedence.

node_spec: default path now asserts S3 install; added a context covering the install_python_from_internet PyPI fallback.

Note: requires the node package to be present at
<s3_url>/parallelcluster//node/ in all partitions

Tests

Build Image with DevSettings

References

Link to impacted open issues.
Link to related PRs in other packages (i.e. cookbook, node).
Link to documentation useful to understand the changes.

Checklist

Make sure you are pointing to the right branch.
If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
Check all commits' messages are clear, describing what and why vs how.
Make sure to have added unit tests or integration tests to cover the new/modified code.
Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The node package was installed from S3 only in us-iso regions; everywhere else it fell back to 'pip install aws-parallelcluster-node' from PyPI, breaking air-gapped/proxied build-image in commercial and govcloud. Set custom_node_package to the official S3 node URL in all regions by default, routing through the existing custom_parallelcluster_node S3 install path. Reuse the existing install_python_from_internet attribute as the opt-out: set it to install from PyPI instead. A user-provided custom_node_package still takes precedence. node_spec: default path now asserts S3 install; added a context covering the install_python_from_internet PyPI fallback. Note: requires the node package to be present at <s3_url>/parallelcluster/<version>/node/ in all partitions

himani2411 · 2026-06-26T15:32:57Z

+default['cluster']['install_node_from_internet'] = false
+
+# Official ParallelCluster Node Package
+default['cluster']['custom_node_package'] = "#{node['cluster']['s3_url']}/parallelcluster/#{node['cluster']['parallelcluster-node-version']}/node/aws-parallelcluster-node-#{node['cluster']['parallelcluster-node-version']}.tgz"


NOTE FOR REVIEWER: Addition of a default attribute will always make is_custom_node? to be always true; which leads to node package being installed in cluster finalize recipe

aws-parallelcluster-cookbook/cookbooks/aws-parallelcluster-entrypoints/recipes/finalize.rb

Lines 21 to 23 in 4019f02

if is_custom_node?

include_recipe 'aws-parallelcluster-computefleet::custom_parallelcluster_node'

end

Setting a default value for an attribute is now making the user-intent unclear.

So every pcluster create and update command will lead to node package being installed on eevry node; the only solution is to add a new attribute or go to the original node.override if-else condition we have set; do you agree?

If the cost of simplifying code and making the user intent clear is to add a new chef attribute, then we should do that

As discussed will change how we identify the node is custom (the URL should differ from the default we set) rather than adding a new attribute.

gmarciani · 2026-06-24T16:36:11Z

 end

-bash "install custom aws-parallelcluster-node" do
+bash "install aws-parallelcluster-node" do


Why removing "custom" since the resource it is actually installing the custom node?

Modified to add the source from which we are installing node package

gmarciani · 2026-06-24T16:37:34Z

+# Install the official node package from S3 in all regions (air-gapped/proxied
+# support), unless the user opts into installing from PyPI via
+# install_python_from_internet. A user-provided custom_node_package always wins.
+if !is_custom_node? && !node['cluster']['install_python_from_internet']


When do we expect to use install_python_from_internet?

The ['cluster']['install_python_from_internet'] is a generalized attribute which was to install latest python and any python related dependencies from internet.

gmarciani · 2026-06-30T13:45:38Z

+default['cluster']['install_node_from_internet'] = false
+
+# Official ParallelCluster Node Package
+default['cluster']['custom_node_package'] = "#{node['cluster']['s3_url']}/parallelcluster/#{node['cluster']['parallelcluster-node-version']}/node/aws-parallelcluster-node-#{node['cluster']['parallelcluster-node-version']}.tgz"


If the cost of simplifying code and making the user intent clear is to add a new chef attribute, then we should do that

We now ship a default custom_node_package (the official S3 package) which made is_custom_node? permanently true and caused finalize/update to reinstall the node package on every cluster create/update. Compare the effective custom_node_package against the full default-precedence value (node.default[...]); only a value the customer changed counts as custom. Log the decision via Chef::Log.info. Add helper unit tests covering the default, nil, empty, customer-supplied, and same-base/different-path cases.

gmarciani · 2026-06-30T17:04:24Z

+default['cluster']['install_node_from_internet'] = false
+
+# Official ParallelCluster Node Package
+default['cluster']['custom_node_package'] = "#{node['cluster']['s3_url']}/parallelcluster/#{node['cluster']['parallelcluster-node-version']}/node/aws-parallelcluster-node-#{node['cluster']['parallelcluster-node-version']}.tgz"


[Minor] Not necessary to handle in this PR, but it would be good to rename the attribute in a follow up PR.

The attribute custom_node_package used to make sense because we were using it only to specify a custom package. Now we are instead using it also to specify the official package, so it should be renamed to node_package.

gmarciani · 2026-06-30T17:06:57Z

 def is_custom_node?
  custom_node_package = node['cluster']['custom_node_package']
-  !custom_node_package.nil? && !custom_node_package.empty?
+  return false if custom_node_package.nil? || custom_node_package.empty?


what is the scenario where it is nil?/empty? My understanding is that it must always have a value (either the official package or the custom package). If nil/empty the cookbook would break even before this check.

I think the logic should be: return True unless node['cluster']['custom_node_package'] is equal to the default value.

himani2411 added the 3.x label Jun 24, 2026

himani2411 force-pushed the enroot branch from b007b6a to 4645700 Compare June 24, 2026 16:18

himani2411 force-pushed the enroot branch from 4645700 to c5c2417 Compare June 24, 2026 17:08

himani2411 commented Jun 26, 2026

View reviewed changes

himani2411 mentioned this pull request Jun 29, 2026

[Release] Add upload-node.py to push aws-parallelcluster-node to S3 aws/aws-parallelcluster#7460

Open

gmarciani reviewed Jun 30, 2026

View reviewed changes

himani2411 force-pushed the enroot branch from 2761054 to 358b552 Compare June 30, 2026 15:59

himani2411 marked this pull request as ready for review June 30, 2026 16:37

himani2411 requested review from a team as code owners June 30, 2026 16:37

gmarciani reviewed Jun 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Node] Install aws-parallelcluster-node from S3 in all regions#3211

[Node] Install aws-parallelcluster-node from S3 in all regions#3211
himani2411 wants to merge 2 commits into
aws:developfrom
himani2411:enroot

himani2411 commented Jun 24, 2026 •

edited

Loading

Uh oh!

himani2411 Jun 26, 2026

Uh oh!

gmarciani Jun 30, 2026

Uh oh!

himani2411 Jun 30, 2026 •

edited

Loading

Uh oh!

gmarciani Jun 24, 2026

Uh oh!

himani2411 Jun 30, 2026 •

edited

Loading

Uh oh!

gmarciani Jun 24, 2026

Uh oh!

himani2411 Jun 30, 2026 •

edited

Loading

Uh oh!

gmarciani Jun 30, 2026

Uh oh!

gmarciani Jun 30, 2026

Uh oh!

gmarciani Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	if is_custom_node?
	include_recipe 'aws-parallelcluster-computefleet::custom_parallelcluster_node'
	end

Uh oh!

Conversation

himani2411 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Tests

References

Checklist

Uh oh!

himani2411 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

gmarciani Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

himani2411 Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmarciani Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

himani2411 Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmarciani Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

himani2411 Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmarciani Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

gmarciani Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

gmarciani Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

himani2411 commented Jun 24, 2026 •

edited

Loading

himani2411 Jun 30, 2026 •

edited

Loading

himani2411 Jun 30, 2026 •

edited

Loading

himani2411 Jun 30, 2026 •

edited

Loading