Copy action

[Silic0nS0ldier]

Proposal to introduce a builtin copy action.

Goals

• Simplify rule development.
• Improve performance of copy-heavy toolchains (e.g. Rules JS, which accommodates NodeJS's relative import logic by copying inputs to the "bin" directory by default).
• Cut down on remote execution and BES protocol chatter caused by having lots of short-lived copy file spawns.
• Discourage optimisation anti-patterns like setting no-remote and no-cache that can ultimately lead to greater overhead in remote builds (e.g. forcing remote files to be downloaded under --remote_download_minimal).

[tjgq]

Does "same type" refer to the artifact type or to the filesystem type? The two don't necessarily match: artifacts of file or directory type can be materialized as symlinks in the filesystem.

What should actions.copy(in, out) do if in and out are both declare_file (or both declare_directory), but in is materialized in the filesystem as a symlink? I think there are three possible answers:

• Copy the symlink as-is
• Indirect through the symlink and make a copy of its target
• Execution error (I like this one the least because it makes builds with the same analysis properties and input digests result in different execution behavior, depending on the state of the filesystem - a sort of non-determinism, if you will).

[Silic0nS0ldier]

Does "same type" refer to the artifact type or to the filesystem type? The two don't necessarily match: artifacts of file or directory type can be materialized as symlinks in the filesystem.

I was thinking artifact type when I wrote this, symlinks are something I've yet to figure out. It's very easy to break them, which to be fair is the case for the existing copy rules. The challenge here is that with some symlinks Bazel will actually dereference them and track the targeted artifact.

Not sure what behaviour to go with here just yet. I feel like "copy-as-is" would be the least surprising option, keeps it simple.

• Indirect through the symlink and make a copy of its target

Would make sense when in the source tree a file has been symlinked specifically so it's content can appear in multiple locations. Such cases would be ideally served by hardlinks (multiple canonical handles, same file) but that's not something version control systems keep track of.

Symlinks to a directory ramps up the complexity. They can be used to present the same content in multiple locations, but they also have more novel uses like in pnpm.

Additionally symlinks can point to anything, including other symlinks (e.g. symlink-a -> symlink-b -> file, symlink-a -> symlink-b -> symlink-a). Solving for that in the copy implementation doesn't sound like much fun.

A case could also be made for reading the symlink target then creating the new "copy" with an adjusted target path.

Opting to not do anything special for symlinks may be the best option, can always do the special niche operation (e.g. copying the file a symlink points to) in a separate action.

[Silic0nS0ldier]

The behaviour here has been clarified in the updated proposal. Some changes vs. my last comment driven by prototyping discoveries. To recap;

• declare_directory -> declare_directory
• declare_file -> declare_file
• declare_symlink -> declare_symlink

If a declare_directory or declare_file input is provided that was fulfilled by ctx.actions.symlink, the symlink is dereferenced to perform the copy. I'm not convinced this is the right decision, it's a tricky spot as symlink fulfilled artifacts are a little special in their own right (materialised symlinks fulfilling directories or files do not have a fixed target path). Whatever the final outcome, the goal is to adhere to the principle of least astonishment.

[Silic0nS0ldier]

Given the behaviour of ctx.actions.symlink(target_file = ...) described in #396 (comment), dereferencing looks to be the most consistent (and maybe the only possible) way to handle this scenario.

[sluongng]

FWIW, this is something that we have discussed internally with BuildBuddy as these copy actions are used heavily among the rules_js, rules_oci ecosystem.

Specifically for rules_oci, the copy contents could range anywhere between a few bytes of json, to multi-gigabytes of compressed tarball container layer. Which make them quite "expensive" to schedule and execute remotely.

An alternative way of achieving this, which we are discussing internally, is for the RBE scheduler to "detect" the known copy actions and skip the executions completely by creating the ActionResult directly instead. Though, it would be much nicer if it's Bazel doing these "short-circuit" executions instead of our server.

[sluongng]

In this example, the desired output here would be a Directory which should be uploaded to remote cache CAS so that subsequent downstream actions could consume it.

If the copy is large (i.e. Dir to Dir recursively), having an ActionResult would also save other Bazel clients from having to do the work.

---

I also think that if File to Dir is supported, then we should consider making it a list of files instead so we can compose the directory contents in one go.

[Silic0nS0ldier]

File to Dir won't be supported, at least not in this proposal. There is too much ambiguity around what the file's path within the created folder will be. For it to be useful, some degree of customisation (i.e. customising subpaths for files placed in the directory) would also be necessary.

Since this comment a new action ctx.actions.map_directory has been implemented, which may be a better fit.

[Silic0nS0ldier]

Work on this proposal had stalled as I needed a prototype to validate the design. AI has made significant strides since late 2024, so I've been able to (now) quickly put one together and give this proposal a much needed update.

It is now ready for another review.

[sluongng]

there is a new submission here #419 with a PR in Bazel as well. I recommend trading notes with @thesayyn first and potentially merging the 2 proposals into one?

[thesayyn]

I'd be happy to work with @Silic0nS0ldier, i do believe that copying is still not a the way to solve this, i do believe leaving the copying to the execution strategy is the way to go since they are extremely optimized for staging contents on specific paths before execution. Anything else we do in the name of copying will be duplicating bytes on disk and same pollution as we have today with Copy actions that use coreutils etc.

[thesayyn]

For copy specifically Bazel already maintains one node for the file, if we design the copy api in a way that duplicates the node, file on disk, we are still not saving anything and frankly making a worse copy action that what we have today.

We really need the new copy api to be as cheap as it could be, i came to conclusion that it is only possible with deferred staging (leaving the materialization to the execution strategy because they are heavily optimized for staging inputs cheaply) and skyframe node aliases (cheap clone of the File handles that carries the digest from the backing file)

This will only be beneficial if its cheap. rulesets such as rules_js, rules_oci, rules_python, rules_py etc need to workaround the fact that the tools that are being run (nodejs, python) want a specific filesystem layout to function, site-packages, node_modules, mtree

[Silic0nS0ldier]

I did notice thesayyn/bazel#3 while updating this PR, but not the proposal till just now.

Reading over #419 (ctx.actions.alias), as written it is orthogonal to this copy action proposal as the on-disk representation is not an explicitly guaranteed part of that proposal.

Essentially;

• ctx.actions.symlink (existing): Materialise the specified target with a symlink.
• ctx.actions.copy (this proposal): Materialise the specified target as a normal file/directory (copy, copy-on-write, hardlink, and potentially junctions on Windows now that I think about it).
• ctx.actions.alias (Proposal: add support for artifact aliasing #419): Materialise the specified target here (symlink, hardlink, copy, copy-on-write, bind-mount).

Optimisations are possible across all three (lazy materialisation, deferral to a separate output service a-la --experimental_remote_output_service, symlink templates a-la --remote_download_symlink_template, etc).

The primary motivation for ctx.actions.copy is that the materialisation won't use symlinks which affect runtimes such as NodeJS (require/import machinery dereferences symlinks by default). ctx.actions.alias does not satisfy this requirement out-of-the-box (symlink-sensitive workloads would require a flag to be passed to Bazel).

---

That all said, I don't believe the proposals are mutually exclusive. Sometimes we want a symlink, sometimes we want a copy, sometimes all we really need is for an artifact to show up in another location. For the latter, ctx.actions.alias can eek out some minor benefits compared to explicit symlinking (works across volume boundaries but has dereferencing overhead at runtime) and copying/hardlinking (no dereferencing). That is, it can pick the best possible strategy when the implementation is less of a concern.

---

As far as the optimisations in this proposal, I have deliberately kept them as follow up items (not part of the proposal itself) so as to not hinder landing of the core API. The 1 must-have optimisation (to graduate the API from experimental status) IMO is deferral of source directory copying under remote execution. It is a weakness vs. what spawn based copies can do, but in prototyping proved to be difficult to implement. Keeping this unspecified allows the disk-requirement gap to be closed via hardlinking and copy-on-write (which would likely be much easier to backport).

[Silic0nS0ldier]

Benchmarks table in the proposal (data from the prototype) is squashed under Githubs markdown renderer, so here is a VSCode rendering that's a little easier to read.

[fmeum]

The current behavior of ctx.actions.symlink(target_file = ...) is particularly bad because how the output is observed on disk or by consuming actions already depends on the context:

1. On Windows, the on-disk representation may be a true copy.
2. With BwoB, the symlink action already creates the symlink in the in-memory file system and could (this is just a missed optimization) create the symlink on disk only when explicitly requested.
3. With remote execution, the artifact is indistinguishable from a true copy to the remote executor on the level of the REAPI protocol. Remote executors already use a bunch of optimizations for efficient staging, which may involve CoW, FUSE filesystems or hardlinks.
4. With local execution on Unix, the artifact is a symlink to the original file.
5. With sandboxed execution on Unix, the artifact is a symlink to a symlink to the original file.

If a tool cares about the particular symlink representation, the above already effectively force it to use ctx.actions.symlink(target_path = ...) instead, which behaves consistently across all platforms and execution strategies.
So whatever new mechanism is introduced, perhaps its time to deprecate ctx.actions.symlink(target_file = ...) in favor of it? This particular parameter just doesn't serve any purpose that isn't better served by something else.

The crucial point of any new mechanism should be that it explicitly doesn't prescribe the materialization strategy for the output as an input to actions, except that it guarantees one that makes the output independent of the input for all practical purposes (in particular, it must not be a symlink to it). Whether hardlinks are allowed is a subtle question since hardlinks change the ctime of the input file. Anything beyond that (CoW etc.) would be an implementation detail of Bazel and the local or remote executors. The more we focus on the semantics of the new primitive, the more optimization opportunities we leave to the implementation, which is a good thing. As performance sensitive users will use BwoB anyway, the materialization on disk also doesn't matter.

I don't think that either proposal needs to focus on how Bazel internally represents these files. Having an action registered for them is already more overhead than anything else Bazel could retain about the files, so that's a natural limit for the scale at which any such action could be used.

[thesayyn]

The crucial point of any new mechanism should be that it explicitly doesn't prescribe the materialization strategy for the output

This is the key point, ctx.actions.copy() can't guarantee on disk presentation because we are dealing with a matrix of 3 kernel x N filesystems + remote execution. Any guarantee we make about semantics of ctx.actions.copy()/alias() is a lie, hence my take on it is that we don't even copy anything its just a pointer to an existing file and gets a different materialization path during execution. This works flawlessly for every execution strategy, even local (without sandbox).

Any other eager materialization strategy, eg CoW/hardlink/reflink by the CopyAction is making a choice that prevents any further optimization down the line, eg by execution strategies. So i agree with Fabians take on this.

I don't think that either proposal needs to focus on how Bazel internally represents these files

I agree, its bazels internal and rulesets don't care how its tracked as long as there are no correctness issues.

[Silic0nS0ldier]

Whether hardlinks are allowed is a subtle question since hardlinks change the ctime of the input file.

That's a point in favour of hardlinks being opt-in, if we care about that side effect. From memory EngFlow's remote execution implementation uses hardlinks to materialise inputs in executors, some other services likely do the same. Biggest concern I can see in Bazel is that a ctime change trips up change detection, leading Bazel to rehash source inputs unnecessarily or possibly rerun the generating action for generated inputs.

The crucial point of any new mechanism should be that it explicitly doesn't prescribe the materialization strategy for the output as an input to actions, except that it guarantees one that makes the output independent of the input for all practical purposes (in particular, it must not be a symlink to it).

Taken in the context of #419, that would mean ctx.actions.alias cannot use symlinks (as that would make the output dependent on the input). Unless the scope were broadened to include REAPI changes necessary to express that a given artifact is safe to materialise with a symlink.

Any other eager materialization strategy, eg CoW/hardlink/reflink by the CopyAction is making a choice that prevents any further optimization down the line, eg by execution strategies. So i agree with Fabians take on this.

I don't quite follow, or maybe I've overlooked something in the proposal. What is the concern that impacts the ability to implement further optimisations?

[Silic0nS0ldier]

Small update. I've been working on a lazy download proposal (allows rules to declare non-configurable downloads that can't be eagerly triggered by bazel cquery, keeps bytes off host under RBE with a remote downloader) which would benefit from making artifact selection more flexible. e.g. picking an executable in a directory (like from an extracted archive) without needing to perform a copy (nor any other functionally equivalent optimisation).

If a artifact selection API were to be introduced, it would make the path argument in ctx.actions.copy redundant. Under the current prototype, copying a file out of a directory would look something like;

foo = ctx.actions.pick_file(bar_dir, "foo")
foo_copy = ctx.actions.declare_file("foo_copy")
ctx.actions.copy(foo, foo_copy)

It's 3 lines vs. 2, but would avoid having 2 ways to achieve the same outcome.

[thesayyn]

I don't quite follow, or maybe I've overlooked something in the proposal. What is the concern that impacts the ability to implement further optimisations

If ctx.actions.copy registers an action that does the copy via reflink,clonefile,hardlink etc, then the optimization of never copying inputs is thrown out the window. If your proposal is not going to do that, that's basically what i have already proposed via lazy materialization ctx.actions.copy api.

Taken in the context of #419, that would mean ctx.actions.alias cannot use symlinks (as that would make the output dependent on the input). Unless the scope were broadened to include REAPI changes necessary to express that a given artifact is safe to materialise with a symlink.

the proposal goes beyond RE, its also taking the local execution strategies into account and has the ability to (open question) express content materialization via execution_requirements so things such as nodejs can say i need something that does not use symlinks.

[Silic0nS0ldier]

Lazy materialisation where possible is included under this proposal (with one caveat).

4. Realisation of the output is deferred where possible.
   When the input's content is remote-backed under Build without the Bytes, the copy completes as a metadata-only operation and the output is materialised on demand, exactly like any other remote-backed output.

   [!NOTE]
   Lazy realisation of source artifacts is out of scope for this proposal as Bazel's rewinding machinery recovers a lost artifact by re-executing its generating action. Source artifacts have no generating action, and ActionRewindStrategy requires all lost artifacts to be derived.

   Related work:

   • Lazily create runfiles symlink trees with BwoB
   • Lazily create local symlinks with BwoB
   • Add --file_write_strategy to Bazel

---

the proposal goes beyond RE, its also taking the local execution strategies into account and has the ability to (open question) express content materialization via execution_requirements so things such as nodejs can say i need something that does not use symlinks.

This is a part of the copy.actions.alias proposal I am concerned about. Local execution materialising via symlinks and remote execution materialising via hardlinks/CoW is essentially the same behaviour as the existing ctx.actions.symlink API (when the destination is not a declare_symlink). Runtimes like NodeJS are known to behave differently in their import resolution when symlinks are involved, but that is not a unique constraint. The same issue can easily come up in other places, and when local and remote diverge it leads to rules that do not behave consistently.

I don't think that execution_requirements is an appropriate answer to this problem, it just pushes the responsibility onto rule authors (in the implementation) and users (there will be cases where the default materialisation under local execution is not the desired behaviour).