Skip to content

CLOUDP-295785 - add pipeline_main.py extended help #334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 89 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
6649987
Re-design pipeline
Julien-Ben Jun 12, 2025
675bee4
Remove file
Julien-Ben Jul 29, 2025
833e25f
Put lib back in dependencies
Julien-Ben Jul 29, 2025
15e7f51
add todo
Julien-Ben Jul 29, 2025
120c1af
Fix
Julien-Ben Jul 29, 2025
c9ceabf
Remove multi arch call, fix test image path
Julien-Ben Jul 29, 2025
fb87f4d
Fix agent version for default case
Julien-Ben Jul 30, 2025
c05e180
Lindt
Julien-Ben Jul 30, 2025
747c4ba
isort
Julien-Ben Jul 30, 2025
03fd9b8
Cleanup TODOs
Julien-Ben Jul 30, 2025
1fbb8d5
Rename arch -> platform
Julien-Ben Aug 4, 2025
e9a524f
Don't rely on exception to check for builder existence
Julien-Ben Aug 4, 2025
fa6b899
Remove unused variables
Julien-Ben Aug 4, 2025
426e522
Pre commit
Julien-Ben Aug 4, 2025
6890858
Cleanup
Julien-Ben Aug 4, 2025
aab9592
Correct build envs
Julien-Ben Aug 4, 2025
33173bb
Lindt
Julien-Ben Aug 4, 2025
74e867c
Update Makefile
Julien-Ben Aug 4, 2025
b13b054
Add TODO
Julien-Ben Aug 4, 2025
832ce61
Revert "Pre commit"
Julien-Ben Aug 4, 2025
bc23827
Remove agent unused functions
Julien-Ben Aug 6, 2025
03062c3
Remove debug
Julien-Ben Aug 6, 2025
4d4e4e9
Remove args iteration for multi platform
Julien-Ben Aug 6, 2025
7c9967a
Cleanup unused stuff
Julien-Ben Aug 6, 2025
a7c63c9
Cleanup
Julien-Ben Aug 6, 2025
742e784
Rename file
Julien-Ben Aug 6, 2025
1f0a21b
Remove cli sbom
Julien-Ben Aug 6, 2025
813d539
Renamed image building file
Julien-Ben Aug 6, 2025
c06061b
Freeze python on whales
Julien-Ben Aug 6, 2025
5f9d49a
Lint
Julien-Ben Aug 6, 2025
f390dc9
Remove everything SBOM related
Julien-Ben Aug 6, 2025
a47341d
Lint
Julien-Ben Aug 6, 2025
972b23c
Add TODO
Julien-Ben Aug 6, 2025
4ae4034
Remove --all-agents
Julien-Ben Aug 6, 2025
291f043
Merge branch 'master' into julienben/redesign-pipeline
MaciejKaras Aug 7, 2025
88c76bc
Rename trace
Julien-Ben Aug 7, 2025
0fd4db8
Remove operator build
Julien-Ben Aug 7, 2025
ee86ebf
Doc and logs
Julien-Ben Aug 7, 2025
5f5940f
Use build_image_generic for test images too
Julien-Ben Aug 7, 2025
6dd208f
Remove unused sign images in repositories
Julien-Ben Aug 7, 2025
493d4d6
Remove pipeline_process_image
Julien-Ben Aug 7, 2025
a21b254
Remove process_image
Julien-Ben Aug 7, 2025
a7db180
Rename function
Julien-Ben Aug 7, 2025
52b8662
Lint
Julien-Ben Aug 7, 2025
e465582
wip: Initial changes
MaciejKaras Aug 1, 2025
8a4e4b7
merge redesign-pipeline branch
MaciejKaras Aug 6, 2025
5bfacf6
Fixes after merging remote branch
MaciejKaras Aug 6, 2025
9733b45
Add sign option + change staging registries
MaciejKaras Aug 6, 2025
5ab9c08
Add agent and ops-manager to build_info.json
MaciejKaras Aug 6, 2025
e6d67ca
Fix issue with scenario
MaciejKaras Aug 6, 2025
7a34092
Fix for `build_info` computation
MaciejKaras Aug 7, 2025
a71ccf6
Pipeline fixes
MaciejKaras Aug 7, 2025
3be7731
Remove `all-agents` option
MaciejKaras Aug 7, 2025
1aae28b
Add missing `--sign` option
MaciejKaras Aug 7, 2025
e00b0e3
Fix agent matrix build
MaciejKaras Aug 7, 2025
f894e5b
Disable concurrent builds
MaciejKaras Aug 7, 2025
b140220
Move all env vars to constants.py
MaciejKaras Aug 7, 2025
7b5a064
Remove `operator-quick` image build process
MaciejKaras Aug 7, 2025
8df2ce4
Rebase fix
MaciejKaras Aug 7, 2025
b6b05aa
Add release scenarios for agent and ops-manager
MaciejKaras Aug 7, 2025
c5ad3c5
merge from Julien branch
MaciejKaras Aug 8, 2025
031dd5b
Merge branch 'master' into julienben/redesign-pipeline
Julien-Ben Aug 8, 2025
8183ec5
Fix release_info.py structure
MaciejKaras Aug 8, 2025
e7ab93b
Merge branch 'julienben/redesign-pipeline' into maciejk/use-ar-relate…
MaciejKaras Aug 8, 2025
2ec7587
Explicitly push to ECR with latest tag. Staging as a followup
Julien-Ben Aug 8, 2025
1badff0
Ensure builder in main to fix race conditions
Julien-Ben Aug 8, 2025
9e2815a
Log line
Julien-Ben Aug 8, 2025
e17b323
Remove unused is_running_in_evg_pipeline
Julien-Ben Aug 8, 2025
075fcae
Typo from merge conflict
Julien-Ben Aug 8, 2025
afc9b79
Follow up TODO
Julien-Ben Aug 8, 2025
3ef9e2c
Login for garasign image
Julien-Ben Aug 8, 2025
d2a6153
Handle builder creation race condition with an exception
Julien-Ben Aug 8, 2025
c6fc163
Cleanup ensure
Julien-Ben Aug 8, 2025
347b44b
Revert "Handle builder creation race condition with an exception"
Julien-Ben Aug 8, 2025
e2a4126
Revert "Cleanup ensure"
Julien-Ben Aug 8, 2025
715c4ad
Rename trace
Julien-Ben Aug 8, 2025
e117f79
Merge branch 'julienben/redesign-pipeline' into maciejk/use-ar-relate…
MaciejKaras Aug 8, 2025
2125cb7
Remove comment
Julien-Ben Aug 8, 2025
d047de5
Merge branch 'julienben/redesign-pipeline' into maciejk/use-ar-relate…
MaciejKaras Aug 8, 2025
c8b66fe
Merge branch 'master' into maciejk/use-ar-related-versions
MaciejKaras Aug 8, 2025
fa9c7ee
Review fixes
MaciejKaras Aug 11, 2025
e56591c
Merge branch 'master' into maciejk/use-ar-related-versions
MaciejKaras Aug 11, 2025
597edf2
Added comment to get_version_id() method
MaciejKaras Aug 11, 2025
857f705
Revert parallel `max_workers = 1`
MaciejKaras Aug 11, 2025
59e4420
Agent image fix
MaciejKaras Aug 11, 2025
2eebf0f
Merge branch 'master' into maciejk/use-ar-related-versions
MaciejKaras Aug 11, 2025
bec8626
Merge branch 'master' into maciejk/use-ar-related-versions
MaciejKaras Aug 11, 2025
22f1330
CLOUDP-295785 - add pipeline_main.py extended help
MaciejKaras Aug 11, 2025
cfd66b6
Merge branch 'master' into maciejk/ar-pipeline-help
MaciejKaras Aug 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 52 additions & 11 deletions scripts/release/pipeline_main.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ def build_image(image_name: str, build_configuration: ImageBuildConfiguration):
def image_build_config_from_args(args) -> ImageBuildConfiguration:
image = args.image

build_scenario = get_scenario_from_arg(args.scenario) or BuildScenario.infer_scenario_from_environment()
build_scenario = get_scenario_from_arg(args.build_scenario) or BuildScenario.infer_scenario_from_environment()

build_info = load_build_info(build_scenario)
logger.info(f"image is {image}")
Expand Down Expand Up @@ -177,34 +177,75 @@ def _setup_tracing():

def main():
_setup_tracing()
parser = argparse.ArgumentParser(description="Build container images.")
parser.add_argument("image", help="Image to build.") # Required
parser.add_argument("--parallel", action="store_true", help="Build images in parallel.")
parser.add_argument("--sign", action="store_true", help="Sign images.")
supported_images = list(get_builder_function_for_image_name().keys())
supported_scenarios = list(BuildScenario)

parser = argparse.ArgumentParser(
description="""Builder tool for container images. It allows to push and sign images with multiple architectures using Docker Buildx.
By default build information is read from 'build_info.json' file in the project root directory based on the build scenario.
It can be inferred from the environment variables or overridden with the '--build-scenario' option.""",
)
parser.add_argument(
"image",
metavar="image",
action="store",
type=str,
choices=supported_images,
help=f"Image name to build. Supported images: {", ".join(supported_images)}",
)
parser.add_argument(
"--scenario",
choices=list(BuildScenario),
help=f"Override the build scenario instead of inferring from environment. Options: release, patch, master, development",
"-b",
"--build-scenario",
metavar="",
action="store",
type=str,
choices=supported_scenarios,
help=f"""Override the build scenario instead of inferring from environment. Options: {", ".join(supported_scenarios)}.
Default is to infer from environment variables. For '{BuildScenario.DEVELOPMENT}' the '{BuildScenario.PATCH}' scenario is used to read values from 'build_info.json'""",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think

For '{BuildScenario.DEVELOPMENT}' the '{BuildScenario.PATCH}' scenario is used to read values from 'build_info.json

is confusing.

Depending on if we decide to change this behaviour or keep it, we may want to add a sentence to explain why.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not entirely sure what direction we should go. DEVELOPMENT makes sense, but we rarely build our own images, rather instruct evg to build it for us. We can reuse PATCH configuration to build everything we want by overriding other settings like registry and build scenario

)
# Override arguments for build context and configuration
parser.add_argument(
"-p",
"--platform",
metavar="",
action="store",
type=str,
help="Override the platforms instead of resolving from build scenario. Multi-arch builds are comma-separated. Example: linux/amd64,linux/arm64",
)
parser.add_argument(
"-v",
"--version",
help="Override the version/tag instead of resolving from build scenario",
metavar="",
action="store",
type=str,
help="Override the version/tag instead of resolving from build scenario. Default is to infer from environment variables based on the selected scenario.",
)
parser.add_argument(
"-r",
"--registry",
metavar="",
action="store",
type=str,
help="Override the base registry instead of resolving from build scenario",
)
parser.add_argument(
"-s",
"--sign",
action="store_true",
help="If set force image signing. Default is to infer from build scenario.",
)
# For agent builds
parser.add_argument(
"--parallel",
action="store_true",
help="Build agent images in parallel.",
)
parser.add_argument(
"--parallel-factor",
metavar="",
default=0,
action="store",
type=int,
help="Number of builds to run in parallel, defaults to number of cores",
help="Number of agent builds to run in parallel, defaults to number of cores",
)

args = parser.parse_args()
Expand Down