Musings on a monorepo versus developer oriented distribution

General
12

I know that we can make the tree, but we need to categorize the tree into strict levels, something that would be really hard to define. And it would get stale.
Wave1.0

glib0.1
llvm0.1
python0.1
numpy 0.1

Wave1.1

glib1.0 ← update to latest
llvm0.1 (test and fix)
python0.1 (test and fix, ex: maybe bump to python 0.2)
numpy 0.1 (test and fix)

Wave1.2

glibc1.0 (dont touch dont test)
llvm1.0. ← update to latest
python0.1 (test and fix)
numpy 0.1 (test and fix)

Wave1.3

glibc1.0 (dont touch dont test)
llvm1.0 (dont touch dont test)
python1.0 ← update to latest
numpy 0.1 (test and fix if needed)

Ok, new top level needed, let’s fork off of wave 1.1. But then we have a regression in llvm and python, ok, lets fork off of 1.3. Wait, python1.0 doesn’t work with glibc2.0, so we need to wait for it to get to that wave. For someone who wants a recent version of numpy, they need to go to an old, completed wave that made it all the way to the end where numpy was finally touched, and they need to wait until the next wave completely finishes. Also, what problem does this solve? If you want to get newer software from a newer wave near the end of the tree, you need to wait longer than if that package had just been updated by itself. If I want a new top level with and old bottom level, this works, but if I want a newer bottom with an older top, I need to wait until the newer bottom level’s wave completely finishes.

The alternative you propose is what is currently in place in Nixpkgs, which works. I like the current system, but I think for rapid iteration we should move to a segmented flakes like was suggested above.

1 Like
13

My one reservation with this is I think its important to have sub-packages (rust crate, npm module, etc), be independently installable. E.g. if auxpkg.rust2018.tokio is the wrong version, I should be able to easily install a different version from flakehub. If each needs an overlay system, then we should try to make it consistent across sub-package managers. For nix profile install, its a little weird since, like if two pythons are installed, and then we install the numpy flakehub flake, we would need a way to specify which python to install the numpy to. Or in other words the CLI profile install currently just can’t handle that case with a distributed system

1 Like
14

This raises another important concern: subpackages. Put another way, nested package sets. Flakes do not currently support these aside from the legacyPackages output.

1 Like
15

I think the best way to go about it is to rely on nix2 tooling for now, and then once we get the core team settled in work on nix3 flakes to enable recursive package sets. I think it’s not ideal, but the start of anything is always going to be rocky.

A little off-topic, but I think that this is a good starting point for Aux to differentiate itself – we could stabilize a lot of nix3 quickly and use it to tie auxpkgs together. Once we get the core team settled in, we could put this on the tracker for high priority, maybe.

3 Likes
16

Something that could potentially be interesting to keep both nix2 compatibility and flake orientation would be to use GitHub - edolstra/flake-compat as a way to bootstrap. This would be a minimal support vector that ensures both nix2 and nix3 can consume the “flakes”. I’ve used it in GitHub - bbjubjub2494/hostapd-mana.docker for example. Downside is we still need nix3 to lock dependencies.

2 Likes
17

Another option could be something like josh to keep a monorepo but make contributing to it easier. This would probably require some restructuring though.

2 Likes
18

I like the idea of using Aux to achieve technical advancements but I would not like to end up building upon sand pilars due to rushing over the implementation.

1 Like
19

josh sounds excellent if we go with a monorepo setup! I think this is what TVL uses for their monorepo too if anyone wants to see it in action:

1 Like
20

I’m still in support of moving away from a monorepo. I haven’t used josh before so I don’t have much of an opinion on it. Either way restructuring would be a huge task. I feel like in the restructuring process we have two main methods of going about it:

  • Fork nixpkgs and break it up into its new structure
  • Create a new repo that depends on nixpkgs as a flake input and work towards migrating everything into the new repo (or set of repos) to the point that we can drop the nixpkgs dependence

Edit: I feel the latter option may be more manageable as it allows us to slowly ramp up to maintaining our set of packages instead of maintaining 80000+ right at the start

3 Likes
21

Create a new repo that depends on nixpkgs as a flake input

I think this is going to be the only way to maintain momentum.

We can do both decentralized and a monorepo. The monorepo is just a collection of pinned versions that work well together.

auxpkgs (curated, not top-level)

  • packages (flat, all values are derivations)
  • packageManagers (exactly two deep, ex: rustCargo.tokio)
  • mark each package derivation, in the meta field, as both an aux package and with the status (properly maintained, ported but up to aux-standards, or patched by aux for functionality but not compliant with standards)
  • every package is given a directory with default.nix
    • and the default.nix is always a function that takes an attrset argument and returns a derivation
    • it always imports a flake from another repo
    • the repo URL is always imported from a serialized file (like json or toml) so that changes can be automated

top level

  • inherit nixpkgs
  • override with auxpkgs

Start with just using nixpkgs, and experiment with overriding packages as aux maintainers maintain them.

Breaking things into core can be a separate project.

4 Likes
22

I think for Stage 1, since we inherit a monorepo anyways, the simplest option is to give each SIG a branch/fork and then have the CI try to perform wave merges. This still gives SIGs the ability to merge at their pace.

1 Like
23

@_deleted I’m really curious about this, and I think its relevent.

Lets say auxpkg wants to overwrite package abc in nixpkgs. What kind of problems could come up?
Like assuming the worst possible senario, that abc is heavily used all over nixpkgs and maybe even inside of bootstrapping.

24

I think overwriting is the wrong way to go about it. Instead, just have flake output for each. Still working it out, but something like this:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs, ... }:
    let
      lib = import ./lib;
      forAllSystems = lib.genAttrs lib.systems.flakeExposed;
    in
    {
      nixPackages = forAllSystems (system:
        (import nixpkgs { inherit system; })
      );

      auxPackages = forAllSystems (system:
        (import ./. { inherit system nixPackages; })
      );
    };
}

So you can depend on nixPackages.<pkg> until there is a suitable replacement

1 Like
25

Well lets say openssl has a vulnerability. The Aux maintainers patch it so auxPackages.openssl has the fix. Well everything in nixPackages doesn’t get the fix, and other stuff in auxPackages might use nixPackages.python, which in turn uses nixPackages.openssl instead of the patched auxPackages.openssl.

I think we will have to do an override on nixPackages. Which is recursive and going to make stuff hard to debug, so we should design carefully. But I think its necessary to be able to maintain without full direct forking.

26

As a temporary bandaid fix yes, but I don’t want to set a precedent of auxPackages just being overrides of nixPackages. A better ideal goal in that situation is bring python into auxPackages

1 Like
27

Patching openssl is going to mean we need to pull everything that uses openssl into auxPackages. Which is on the order of half of all packages including core. I don’t think that’s realistically within our ability.

1 Like
28

Agreed, it’s not. Overriding may become necessary for security reasons. The long-term goal however should be to not depend on nixpkgs at all, so I believe when possible we should package ourselves rather than overlay. You’re right for something like openssl that may not be possible

29

I agree. Aux should absolutely not be advertised as “an overlay for Nix”. Its just a way for us to maintain core components with O(1) sized increments (which is still potentially huge) instead of O(n) sized increments.

What we can do is establish policies like, when a package is pulled into auxPackages, we do a build and trace every nixpkgs attribute that the build touches. (e.g. auxPackages.python would have nixpkgs.gcc and nixpkgs.openssl on that “touched” list). We would mark/record this list in the meta attribute, and once all the things on that list have been overridden with pure auxPackages (e.g. nixpkgs.gcc == auxPackages.gcc, and auxPackages.gcc.meta.pureAux == true) then we can change the meta and say auxPackages.python.meta.pureAux = true and do basically a find-and-replace on the code (swapping the nixpkgs var out with auxPackages var)

Then eventually, there won’t be anything left pulling in nixpkgs.

2 Likes
31

As much as I dislike using fetchers instead of inputs this is probably the best solution here (at least until lazy inputs are a thing). This would pull a minimal amount of code for the user and is reasonably easy to update. In fact I’m pretty sure we could use Drift to do it automatically.

1 Like
32

Re: docs location

I think this is the direction things were leaning in. Is that right @minion & @coded?