fixed-point argument parameter on derivations

[RossSmyth]

Is your feature request related to a problem? Please describe.
Would be nice if the derivations here supported fixed-point arguments, i.e. finalAttrs.

Describe the solution you'd like

This would require reworking the derivations to use lib.extendMkDerivation ideally. I played around with this and there is one main blocker: the stdenv attribute.

Describe alternatives you've considered
Could roll our own helper, but it is nice to be in-sync with any update to extendMkDerivation.

Additional context
Add any other context or screenshots about the feature request here.

[ipetkov]

I've never used lib.extendMkDerivation what are the upsides and where is it useful here?

[emanueljg]

I've never used lib.extendMkDerivation what are the upsides and where is it useful here?

It's the modern way to create derivation builders. It lets attributes of a derivation depend on other attributes with finalAttrs, as if they were created by stdenv.mkDerivation. If you use any random helper, you can't use the finalAttrs fixed-point reference. It becomes important in libs like crane because you as the end user of the library can override attributes in the derivation without clobbering important lib attrs, because the lib derivation builder on its end computes its attrs recursively and recomputed on override correctly. I essentially distrust any overriding on non-mkDerivation-based 3rd-party library derivations for this reason.

A few builders in nixpkgs already implements it (rustPlatform.buildRustPackage comes to mind) but sadly adoption is slow. It should be relatively easy to rewrite it.

The "marketing" on this is very bad; if I was on the team of leading adoption, I would call this "smart derivations" because you're essentially creating an attribute-aware derivation.

let d = stdenv.mkDerivation (finalAttrs: {
  foo = 2;
  bar = finalAttrs.foo * finalAttrs.foo;  # bar = 4
});

d' = d.overrideAttrs { foo = 3; };  # bar = 9
in ...

[foob2483-oss]

Also just got hit by this limitation. Not having access to finalAttrs is a bummer for many usecases.

[musjj]

It's the modern way to create derivation builders.

Yep, very much this. I needed to make a package that overrides another's cargoExtraArgs, but the current design makes this impossible without having to resort to barbaric hacks of abusing package inputs.

[RossSmyth]

The only difficult part is porting mkCargoDerivation to extendMkDerivation.

Once that is done, everything else is trivial to port. Again because of the stdenv parameter which makes this port a bit more tricky than usual.

[musjj]

I'm not really sure what the point of the stdenv parameter is. Users can simply override it if they like by doing .override { stdenv = ...; } right? It will be a breaking change, but this seems far more idiomatic.

[musjj]

I played around with this and there is

Btw, do you still have any leftover of that code? I'm trying to give this a go but all the tests are kicking my butt right now.

I thought of doing a minimal PR with just mkCargoDerivation and mkCrossToolchainEnv, but the tests really reaches deep so I think you would need to do all the migration all at once. Though I'm not 100% sure.

[ipetkov]

Users can simply override it if they like by doing .override { stdenv = ...; } right?

I wish it was but it's not quite that simple when it comes to properly supporting cross compilation

I have some ideas on how to improve things in this regard but I need to work out the exact details and docs

[musjj]

My idea was to do something like this:

{
  pkgsBuildHost,
  pkgsHostTarget,
  stdenv,
  stdenvBuildHost ? pkgsBuildHost.stdenv,
  stdenvHostTarget ? pkgsHostTarget.stdenv,
}:
...

Would this work for cross-compilation?

[ipetkov]

I've given this an earnest exploration, I think landing support for this can make things nicer (esp with minimizing the need to juggle baseArgs/commonArgs just to override say one part for buildDepsOnly). The biggest challenge I'm running into is maintaining the same level of API consistency and lack of surprises we have today.

It's "easy" to change mkCargoDerivation (and its callers) to accept either an attrset (current behavior) or a function which accepts finalAttrs and restructuring the internals (expressing "default values caller can override" and "explicit values we wish to force on the caller [unless they reach for .overrideAttrs]" becomes pretty natural). But this makes it just as "easy" to accidentally trigger infinite recursion errors unless you are super careful (and even then I hit it in a few places I was having an impossible time understanding why an innocuous change would trigger the error while an almost semantically equivalent change wouldn't). And this is before we take into account callers also being able to trigger infinite recursion errors as well.

To make matters worse, there's a number of little behaviors and improvements we've collected over the years (some by design, others more organic) which become difficult or impossible to express: one option is to document "such and such attr is special cased and will not take part in recursive/delayed .overrideAttrs evaluation unless it is present in the original attrset" but I don't want to tolerate more than one or two of these occurrences for the sake of API consistency (and ease of both documenting the behavior and allowing a reader to understand it without being an expert in the internals); other option is to drop support for the existing behavior entirely. Although I'm not opposed to making breaking changes, per se, with enough departures from existing patterns and behaviors it basically amounts to a full on rewrite and maybe warrants being its own thing rather than shifting out the floor underneath all current consumers.

That isn't to say its not possible to design something consistent and coherent with fixed-point derivation attrs in mind, I'm just struggling on how to map a transition from the behaviors we have today to such a design with the least amount of churn and new API "surprises".

[dpc]

Either a long lived branch or another project would work.