SIG Repos: How should they work?

Special Interest Groups
31

I’m not yet convinced it’s possible to avoid this - it would be nice if each individual language was it’s own little silo, kept away from every other programming language, but it never plays out that way. At this point though, I think we approach that when it happens - trying to anticipate it when it’s this theoretical is a cognitive rabbit hole that never ends.

I think for now, @getchoo has the right idea - let’s keep things acyclical and simple, until we find a reason we need to change that. We have the luxury of being able to break stuff while we’re in the early bootstrapping phase - let’s use that to our advantage and cross these hurdles when we get to them.

6 Likes
32

Let me try to summarize the current discussion and extract the main points still in discussion:

  • Goals (from user perspective):
    • SIG repos can be used individually, reducing size and eval time
    • top-level still provides what previously ‘Nixpkgs’ was - a one-stop location for all packages
  • Goals (from SIG/maintainer perspective):
    • reduce maintainer burden
      • smaller scoped repos, with less noise
      • less moving parts in each SIG repo
  • technical problems:
    • circular dependencies between SIG repos
      • proposal: (multi-tiered) top-level repos
        • removes circular dependencies, by moving them one level up
        • possibly increases top-level maintenance burden
        • this might feel like going back to a (quasi) mono-repo though - but one could argue this is inherently the case anyway
      • proposal: using follows specifications to lock all other Auxolot dependencies
        • this looses the independence property of the SIG repos

(Did I miss something? Certainly not intentionally - I’ll be happy to make amendments here)

As far as I can tell, at no part of the discussion so far these goals were up to debate, so I’ll focus on the technical problem(s).

I come to the same conclusion as @srxl: A top-level approach without cyclical dependencies keeps things simpler without compromising on our goals (re: independence of SIG repos).

(And I also agree: let’s put our current possibility to experiment with stuff to good use, when we need it.)

4 Likes
34

First I’m so glad were all onboard with reducing recursive dependencies.

I think experimentation will be needed.

And towards that, I have automated nix code refactoring before (I created a nix bundler similar to a JS bundler). Let me say, there’s some real easy stuff we can do from the begining, even at the formatter level, that will allow us to make massive refactors overnight if we realize we need a completely different structure. Stuff like, always accessing packages through attributes (ex: auxpkg.name instead of using with auxpkg; name (which is also horribly slow at runtime)), being consistent with how top-level attibutes are set rather than sometimes dynamically generating top level attr names (which again is also slow), having a folder name match the top level attribute name, etc.

I can confirm this is not possible. Python’s R module has the entire R language (and R modules) as a dependency. Other languages have python as a dependency.

A Design Draft v1

I have spent 3 years trying to automatedly refactor nixpkgs. So basically I’ve been training for my whole (nix) life for this (not that I have everything figured out).

Here is a structure that I think could get put-to-work this week and later.

  • Note: in the list below dependencies go up
  • E.g. lib depends on nothing, core depends on lib, etc.

Repos:

  • lib - a pure nix lib, no system/stdenv (this already exists as a flake so we can use that)
  • core - a reorganized attrset of minimum-packages needed to build the nix cli (which I know how to create thanks to nixpkgs-skeleton)
  • sig_sources - manually edited/maintained repo. Who is this for? ecosystem/SIG maintainers
  • registry - automated. Who/what is this for? think of this like IR (intermediate representation). It exists to untangle the spaghetti of package dependencies/recursion, which helps cross-ecosystem maintainers, core maintainers, improves runtime performance, and makes package-indexing/security-notifications/testing and other stuff automatable
  • ecosystems - manually edited/designed. Who is it for? end users
  • aux - the aux namespace. Not backwards compatible with nixpkgs. Curated (starts off pretty empty). aux = { auxpkgs = registry; ecosystems = ecosystems; lib = lib; }
  • polypkgs - pollyfilled pkgs, aka nixpkgs with overlay of auxpkgs (e.g. mostly backwards compatible, temporary)

(I don’t care about the names, like polypkgs could definitely have a better name, thats not the point)

Design of each

  • Lib

    • Should probably be split into stdlib (aka just lib, always forward-compatible) and internalSupport (aka stuff the aux-monorepo(s) might need but the rest of the world might not need, and also might break forward compatibility)

  • Core

    • to create core, copy all nixpkgs files into a new repo, disable the cache, nix build -vvvvv --substituters '' '.#nix', do it on Linux x86, Linux Armv7.1, Mac Apple Silicon, and Mac x86
    • if a file path is logged on at least one of those^ builds, then keep it. Delete all other files
    • using the log output and some brute force trial-and-error we should be able to detect which of the top-level attributes that were evaluated
      • note which attributes existed on all systems, vs which were system-specific
    • top-level.nix is going to contain a ton of junk still, with attributes to packages that don’t even have files anymore.
      • While we should eventually clean it, the 1-week fix is to not clean it
      • Make a minimal-legacy-core.nix that imports the top-level attr set
      • Make the attrset of minimal-legacy-core only inherit (from top-level) only attributes that are we know are used
    • create core.nix which imports minimal-legacy-core, but re-structures it:
      • if an attribute is not a derivation, then put it directly on core
      • if an attribute is a derivation, and builds regardless of OS put it under core.pkgs
      • if an attribute is a derivation, but only exists on a certain OS, put it under core.os.${osName}
    • v1 done
      • Later we can work on cleaning this repo.
      • Updates can be semi-automated by looking at the nixpkgs git history and checking for changes to the relevent subset of files

  • Registry (which will become auxpkgs)

    • This is the key to what I call, THE GREAT UNTANGLING.

      • Which I think is the most important change to nixpkgs, and is the cause of inter-ecosystems trouble we are hitting right now.

    • A flat repository

    • The “this week” solution is to have registry start as an empty attrset, but I’ll continue to describe how it fits into the bigger picture.

    • Two kinds of packages

      • base packages (normal nix func that produces 1 derivation as output, ex: python)
      • extension packages (these “modify” an existing derivation. For example, have numpy take python as an argument and return both a new python (that now has numpy), and also just a standalone numpy derivation (for stuff like numpy header files or venv))
      • Both extension-packages and base-packages are stored in the same flat attrset

    • Note: we are forced to have both base and extensions in registry because some base packages (like VS Code) need base+extension packages (like nodePackages) to build themself. So its not possible to fully separate all base packages from all extensions packages.

    • The great untangling/ordering

      • To fix the recursion issues we need the attrs in the top-level of registry to be in a particular order. This can be done, and scaled up without issue if we automate the generation of the top-level.nix file.
      • For an example of the attr order, if a package depends on merely core and/or lib then it is considered to have 0 dependencies. It goes at the top. However, something like npm would need to appear BELOW pythonMinimal because npm depends on pythonMinimal. You might be thinking “But Jeff, some packags–” I know, we’ll get there. Every built-package had a dependency tree (specifically a directed acyclic graph (DAG) of a tree). Conceptually, the order of attrs in the registry is the breath first search (BFS) iteration on the combined dependency tree of all packages. Conceptually. The main reason this post is so frickken long is because nixpkgs pretends the dependency tree has loops, even though, in reality, if packages are to ever be built in finite time, the dependency tree cannot have loops.
      • In practice we can achieve a total ordering of packages, with the following logic:
        • If [pkg] only uses core/lib, put in alphabetical order at the top
        • If all of pkg’s dependencies are already in the registry list; easy, just put the package as high as possible, while still being below all of its dependencies
        • Those two rules alone handle a massive amount of packages, but not everything. Let me introduce the “problem children”
          1. If a package has dynamic/optional dependencies we first try to assume that it uses all of them, even if that is somehow impossible (ex: for a package using gcc on linux and clang on mac, we pretend it uses both gcc and clang at the same time). If, with that assumption, all the pkgs dependencies are on the list, then we’re good. If not, then using tree search and some assumptions we can detect the issue and fallback on the next approach.
          1. We will need to semi-automatedly/manually break up some packages. There are kinda three cases for this. definitelyMinimal+maybeFull, branching groups, and multi-base-case recursive dynamic dependencies.
          • definitelyMinimal+maybeFull+: For dynamic non-recursive dependencies, such as pytorch maybe needing cuda, we can often break them up into a “minimal” package and a “full” package. The reason I say definitelyMinimal is that the minimal case cannot have any optinal arguments. It needs to be the bare-bones and nothing else. On the flip side, some packages like ffmpeg and opencv have tons of options and some options are incompatible. We can’t actually make a ffmpegFull. So instead we have an ffmpegMaybeFull where every option is available, and we ensure ffmpegMaybeFull is below all dependencies for all options. This minimal+full technique also works for trivial recursion. Every trivially recursive package has one base case (by definition). That base case gets put in its own derivation as minimal, then the recursive case becomes the full version.
          • Branching groups: Not all dynamic dependencies work under the minimal+full method. For example, evaluating a package on MacOS might cause it to have a different tree-order – an order that is incompatible with the same package evaluated on Linux. Theoretically this can happen even without OS differences. Solving this is actually pretty straightforward, the package is broken up into branches (different groups of dependencies) such as package_linux and package_macos. Each of those will have their own spot in the ordered list. Then one-below the lowest one (aka the one with the most dependencies), we create a combined package. The combined package depends on all the earlier ones, and the contains the “if … then package_linux else if … then package_macos” logic.
          • Dynamic recursive dependencies: Unfortunately I can confirm there are packages that are deeply, painfully, multi-base-case recursive with dynamic dependencies.
            • Let’s start with easiest example. Let’s say registry.autoconf depends on perl. Well registry.perl (ex: perl 6.x) might depend on perl & autoconf. And now we’ve got a multi-recurisve problem; autoconf needs perl and perl needs autoconf (and perl!), its the dependency tree with loops.
            • Except in reality reality we start with core.perl, then build autoconf::(built with core.perl), then build registry.perl::(built with core.perl and autoconf::(built with core.perl)), and then build autoconf::(built with registry.perl::(built with core.perl and autoconf::(built with core.perl))). It quicky becomes a lot to mentally process … and that’s the simple case!
            • Nixpkgs does stuff exactly like that behind the scenes, at runtime. Thing is, we don’t have to do it at runtime. We can be way more clear about what is going on by adding stages.
              • registry.autoconf_stage1, statically depends on core.perl.
              • registry.perl_stage1, statically depends on registry.autoconf_stage1
              • registry.autoconf_stage2 statically depends on registry.perl_stage1
              • All other registry packages use registry.autoconf_stage2 instead of just “some version of autoconf”.
            • While still complicated, making these stages explicit is, I think, the only way to make this stuff even barely manageable. Just imagine the difference between “Error: autoconf_stage2 failed to build” compared to “Error: autoconf (one of multiple generated at runtime) failed to build”.
            • While this does require skilled manual labor, there’s not too many packages like this.
            • Well … except for one category. Cross compliation.
              • While I think we should have cross compilation in mind from the begining, I don’t think we should immediately (or any time soon) jump into trying to handle cross compiled packages.
              • The normal (not-cross-compiled) version of a package is going to have less dependencies, and be higher up on the dependency tree. We should focus on those first since they’re the foundation.
              • That said, I want to recognize what will eventually need to be done for the true deepest most nasty hairball of spaghetti-code in all of nixpkgs; cross compiling of major tools like VS Code, using QEMU virtualization. Not only is it an explosion of dependencies, its possible to depend on the same version of the same package twice, once for the host architecture and again for the target architecture. If we can eventually tackle that, I don’t think it gets any worse.
              • I know it might feel unclean (give me a chance to talk about SIG sources), but in order to detangle cross compliation, some registry packages will need to have system postfix names like gcc_linux_x86_64, just FYI.

    • Last note on the registry, we can use a _ prefix to indicate when a package attr is “just a helper” rather than a derivation that we want to be user-facing. For example _autoconf_stage1, _autoconf_stage2, and then we would have autoconf (e.g. stage2 renamed and ready for public use)

  • SIG sources

    • While the registry can make detangling the recursion possible, it doesn’t necessarily make things perfectly easy to maintain. At a practical level, we can’t just have one package file for each registry package, because stuff like python (python2, pythonMinimal, CPython, Jython, Cython, pythonFull, etc) are going to have a bunch overlap in terms of nix-code, even if they belong at different levels of the dependency tree.

    • SIG sources can let us have our untangled cake and eat (maintain) it too, but there is a big risk!

    • Each SIG could have a directory inside of the sig_sources repo. For example, let’s say there’s a maintaince group for python. Every sig directory would be designed in a way that a script in the registry-repo could scan the SIG folder, see exported packages, see a static list of dependencies for each exported package, then compute the correct order for all of them, and have each attr import code from the sig directory.

    • The danger is that we accidentally recreate the same nixpkgs mess. For example, a giant python/default.nix file that handles every variant of python, packed to the brim stuff like if isPythonFull then ... if isCython ... if isJython. In that case, we are right back to a recursive mess; because cython needs pythonMinimal, and both pythonMinimal and cython are generated by the same monolithic python/default.nix. We have only added indirection. The registry makes de-tangling possible, it doesn’t guarentee it.

    • How can we solve this without subjective “code-feel” guidelines? Two rules.

        1. Evaluation at different points of the tree (e.g. pythonMinimal vs pythonFull) doesn’t always matter. For example, the aux lib functions wouldn’t care at all since they don’t use derivations. So when does it matter? Well lets say we had a helper like escapePythonString. If that helper is implemented without the registry, then its like lib, it doesn’t really care “where” in dependency tree its evaluated. However, if that same tool, escapePythonString, for some reason, needed registry.sed, then it becomes a risk of being tree-order dependent. Lets say we have another helper, buildWheel, which depends on pythonMinimal but is used inside of pythonFull. While not too common, when helpers depend on registry packages, we can break them up into groups. For example, utils_pre_python.nix could contain escapePythonString, and indicate at the top of the file that there is a dependency on registry.sed. Because buildWheel has different registry dependencies, we would need to make a different utils file, like utils_post_python_minimal.nix to house the buildWheel function. While this handles the tree-ordering issues, it doesn’t necessarily fully stop spaghetti code.
        1. This one is hard to explain, but once it “clicks” its easy to have an intuition for. Going back to escapePythonString, lets say it, and all of the helpers are pure-nix. We use escapePythonString across python2, python3Minimal, python38Full, etc. Everything is great. Then one day someone invents Wython (fictional) and the string-escaping of Wython is just a bit different than python. So we face a choice. Either
        • A. We create an independent escapeWythonString
        • or B. we make escapePythonString a bit more complicated by adding a { isWython ? false, ... } parameter
      • You might think “whatever, those options are merely personal preference” but that’s not entirely accurate. The runtime has slight performance difference in terms of tree-shaking, and we can the detect difference objectively via code coverage. Additionally there’s an argument to be made that option B creates a spaghetti control flow. Quick disclaimer, I’m not a 100% coverage kinda guy – I don’t care if a project has 50% coverage – code-coverage is just a tool.
        • Lets talk about tree-shaking, and look a option B. If we run python2, python3 or any individual build, the code coverage of escapePythonString will be more than 0 but not 100%. All of them miss the if isWython branch inside of escapePythonString. That means the engine is always wasting, at least a bit, of time evaulating code that will never be evaluated while building python3.
        • In contrast, under option A, building any individual package causes each helper function to either be 100% or 0% (e.g.100%=escapePythonString, 0%=escapeWythonString)
        • I’m not saying it needs to always be 100% or 0%, but rather:
          • If a single build calls both escapePythonString { option1 = true; }, and escapePythonString { option1 = false; } then there’s no issue, escapePythonString doesn’t need to be broken up (regardless of how other builds use it).
          • For example escapePythonString { singleQuote = true; }, and escapePythonString { singleQuote = false; }
          • But, if Wython only uses escapePythonString { option1 = true; } and all other builds ONLY use escapePythonString { option1 = false; } then there is a problem.
          • For example escapePythonString { isWython = true; }, and escapePythonString { isWython = false; }
      • For the “this week” implementation, these rule can just be eyeball-enforced.
        • It’ll be good enough to prevent the monolithic recursive dependency spaghetti problem.
        • With a tiny bit of practice it’s not that hard to follow the rules manually
        • If there is a debate it won’t become personal-preference war because there is an objective way of determining the answer
        • If a small case is missed, its not a big deal to find/fix it later
        • Later this can be automated by recording the code coverage of each registry-package in a SIG source. For all nix functions that were evaluated during the build, if the function was defined in a file within the SIG folder, and no individual build got 100% coverage of the function, then its flagged. If there is a different combination of arguments that cause a build to get 100% then it passes the flag, otherwise it needs to be broken up.

    • There’s other technial details of SIG sources to discuss, like having inter-SIG dependencies go through the registry instead of being direct imports, and having all SIG sources provide one file per registry-entry, and each registry dependency be a function argument rather than an import, but I’m trying to not turn this post into my dissertation :sweat_smile: (despite how it might look)

  • Ecosystems

    • Goal: be as ergonomic as possible for users

    • Ecosystems shouldn’t depend on other ecosystems directly: either import derivations from the registry, or import nix-functions from a sig source

    • SIG sources != ecosystem

      • For example, JavaScript might be a SIG group (someone who knows JS has relevent skill for maintaining both bun and nodejs), but in contrast nodejs might be an ecosystem, and bun might be a different ecosystem.
      • SIG sources might need to have messy tooling for bootstrapping like pythonMinimal_stage1. The ecosystem interface should hide all that and just present the final product.
      • If it helps generate packages in the registry, or if a registry needs a tool → then it goes in a SIG source
      • Else → Ecosystem
      • Home manager probably would live in the ecosystem space

    • Enable stuff like the dev-shell mixin experience (ex: ecosystems.aux.tools.mkDev [ ecosystems.nodejs.envTooling.typescript ecosystems.python.envTooling.poetry ecosystems.latex.envTooling.basics ])

    • While registry needs to be rigourously consistent in order to be automated, ecosystems only need to be consistent to help with ergonomics.

      • Like a common interface of
        • ecosystems.${name}.tools for nix functions
        • ecosystems.${name}.variant for minimal/full builds (ex: mruby, jruby, or jython or pythonMinimal)
        • ecosystems.${name}.main for the base tool (e.g. rustc/ruby/python/node)
        • ecosystems.${name}.pkgs. They can deviate on a per-ecosystem basis as needed.
        • ecosystems.${name}."v${version}".main
        • ecosystems.${name}."v${version}".pkgs
        • ecosystems.${name}.envTooling
        • etc
      • But they are allowed to be different when it makes sense, like ecosystems.${name}.tools.mkNodePackage, or ecosystems.${name}.tools.pythonWith

  • Aux

    • Having one layer before getting into packages is important for future expansion, for example aux.pureAuxPkgs or aux.distributedPkgs, etc

  • Polypkgs

    • Its own repo so that tarball-urls are easy drop-in replacements for nixpkg tarball urls
    • If nixpkgs gets a commit, we generate a new flake.lock
    • We have git tags equivlent to nixpkgs git tags
    • Temporary
    • Big special note: I know this goes against what I said at the top (“dependences only go up”), but out of practicality, and because this repo is temporary, sig_sources can use/refer to polypkgs.
      • Yes, this is a recursion issue (sig_sources uses polypkgs, which gets overlayed by auxpkgs, which links back to sig_soruces) but it is necessary. For example, lets say python is NOT in the aux registry yet. Lets also say nixpkgs.openssl is broken from a gcc update.
        • Cowsay can’t use nixpkgs.python (built with nixpkgs.openssl) because nixpkgs.openssl is broken
        • But cowsay can use polypkgs.python (built with the polypkgs.openssl which works because polypkgs is overlayed with registry.openssl)
        • E.g. cowsay doesn’t directly depend on registry.openssl.
        • The registry ordering script pretends cowsay has no dependencies (polypkgs is “invisible”)
        • BUT, as soon as we have a registry.python, (which would end up as polypkgs.python) we need to “collaspe” the difference, mark cowsay as depending on registry.python (instead depending on nothing), in which case the registry generator will put cowsay below python instead of having it at the top level.
11 Likes
35

Oh and last thing I forgot. To wrap things up with a bow: packages outside of SIG’s and ecosystems, like which or ping or even openssl, they could simply be added-to then imported-from flakehub and put directly into the registry. Which would let us leverage individuals maintaining their own things on flakehub in a distributed way.

36

@Jeff - I definitely don’t have the brain to understand all the details here, but my overall impression is very positive! I’m a big fan of “iteratively comb out the complexity” approaches - sounds like you’ve thought through the balance between that and up-front design. Maybe when my meds have worn off I will have another go at understanding the details :slight_smile:

37

First of: that’s a very impressive write-up!

I’ll put my thoughts below quotes that are (hopefully) indicative of the exact place in your text (for lack of a better term) I’m talking about.

I can confirm this is not possible. Python’s R module has the entire R language (and R modules) as a dependency. Other languages have python as a dependency.

Hmm, I’m not yet convinced that this means we have to introduce circular dependencies (which I think this really is about)
For example for the node-gyp thing (Jupyter Notebook (python) ← node-gyp (node) ← python) thing,
could be solved by having a python/js core (the naming is up for bikeshed, e.g. ‘bootstrap’ might better convey the purpose) repo
which only sets up these cross-language concern,
and a pure python repo on top.
After fully reading your proposal: Nice, I belive this is what you already came up with, lol.

Here is a structure that I think could get put-to-work this week and later.

  • core - a reorganized attrset of minimum-packages needed to build the nix cli (which I know how to create thanks to nixpkgs-skeleton)

why focus on nix cli for core? I.e. what is the thing that differentiates nix-cli from other packages (e.g. compilers) (and with lix planning to depend on rust: this would pull in the SIG Rust stuff, depending on exact repo layout. How would we handle that? I.e. what is core and what is SIG <language/ecosystem>)

  • registry - automated.

I’m vary on automating such ‘core’ things, as the registry. I’m scared this makes it much more difficult to fix/understand if stuff breaks (a similar concern was raised in SIG Repos: How should they work? - #30 by getchoo, albeit with a possible more complex scheme behind it (very first paragraph))

  • Lib - Should probably be split into stdlib (aka just lib, always forward-compatible) and internalSupport (aka stuff the aux-monorepo(s) might need but the rest of the world might not need, and also might break forward compatibility)

I have hard time imagining that we can break forward compatability that easy, even if it’s just internally.
I might be wrong, but I think nixpkgs had problems with this in a monorepo - and we’re talking about multiple repos needing to synchronise on this here.

While still complicated, making these stages explicit is, I think, the only way to make this stuff even barely manageable. Just imagine the difference between “Error: autoconf_stage2 failed to build” compared to “Error: autoconf (one of multiple generated at runtime) failed to build”.

Someone who has more experience with how nixpkgs currently bootstraps needs to wheigh in here, but to me this sounds extremely reasonable.

… Cross Compilation …

I don’t quite understand why a cross-compiled package has more dependencies. In my mind it just has different packages (different compiler mainly, since the target arch is different).

How can we solve this without subjective “code-feel” guidelines? Two rules.
2.

I have a hard time following here and don’t quite get what you’re arguing for in the end. It would be really kind if you (or somebody else) could try to reword this, in order to help me understand this. But it’s not that important, in the end the whole think still kinda makes sense to me.

Home Manager probably would live in the ecosystem space

Food for thought: are we conflating things that should be kept apart here? I.e. package building/configuration and system (in the sense of system (host services, networking, boot, filesystems, etc) but also user configuration (dotfiles, user services, i.e. home manager, etc) configuration?
I think we should keep these apart from each other, i.e. the configuration stuff (todays nixos and home manager) should be kept different from packaging stuff (todays bootstrapping (the different stages), pythonPackages, rPackages, luaPackages, vscode, neovim, etc).
And this discussion should focus solely on the packaging aspect (which is then consumend by the configuration stuff).

Enable stuff like the dev-shell mixin experience (ex: ecosystems.aux.tools.mkDev [ ecosystems.nodejs.envTooling.typescript ecosystems.python.envTooling.poetry ecosystems.latex.envTooling.basics ])

And in spirit with the last point: where do we see the dev env story?
Is it something that needs to be kept together with packaing (for technical/maintenance reasons)?
Or is it both plausible and sensible to separate it too?

While registry needs to be rigourously consistent in order to be automated, ecosystems only need to be consistent to help with ergonomics.

To make sure I understand this all correctly:

  • lib: generic pure nix ‘tooling’
  • core: bootstrapping of build minimum amount of required toolchain(s)
  • sig_sources: this is the bootstrapping of SIG/ecosystem toolchains AND ecosystem/SIG specific (pure) nix ‘tooling’?
  • ecosystem: (language (or finer grained)/)SIG specific package sets + nix tooling building upon the bootstrapping in sig_sources
  • registry: combines sig_sources + ecosystem together in a clever way to avoid the circular dependency issues
  • polypkgs: tie everything together, to have a one-stop shop for those who want it

Then it also makes sense why you list these functions under ecosystems, and not sig_sources to me.

Aux

  • Having one layer before getting into packages is important for future expansion

I don’t quite get the example, but I belive what you’re getting at is that we need an abstraction layer between the actual layout of things
and how we present them to users, so that when our layout changes, we can maintain backwards compatability in this abstraction layer.
With that I fully agree. And I like the idea to have specific place for this layer!

Polypkgs

  • Its own repo so that tarball-urls are easy drop-in replacements for nixpkg tarball urls
  • If nixpkgs gets a commit, we generate a new flake.lock

I think we ended up somewhere that it’s best to actually fully separate our public
package sets from nixpkgs,
since providing nixpkgs is increased maintenance complexity for use,
but not much benefit to our users (they can just pull in nixpkgs themselves).
(this was the discussion in On the future of our nixpkgs fork)
(Not fully sure this is the right conclusion, now that I think about it:
I’m not knowledgable enough to know if this is also just as easy when not using flakse,
and us pulling in the nixpkgs has the (implicit) guaranteee that the nixpkgs
we provide is compatible with auxpkgs - which the user pulling it in doesn’t have)

All in all, I feel this really goes in the right direction; while I think there’s some discussion to have on certain details, the overall architecture/broader idea seems very solid to me, and I especially like the idea on making the bootstrapping and library parts of what today is all “nixpkgs” more distinguished in however we will call the different parts of auxolot packaging :heart: (I.e. going even further than just distinguishing nix the build tool, nixos the system configuration and nixpkgs the package set).

P.S.: I’m slowly starting to doubt that the complexity of it all can be appropriately handled in a discourse thread. Yesterday someone shared the CQ2 tool on in the matrix chat. But I really don’t wanna rush introducing yet another tool without proper discussion & consideration & approval of the current interim leadership, so this best stays here in the P.S. for now, and if we come to the point where we really feel productive discussion is simply not possible due to the complexity exceeding what discourse’s format can handle, we can revisit this.

PPS: For some details on the ecosystem/polypkgs package sets I think we should also take the thoughts on how to structure our new packages in On the future of our nixpkgs fork - #13 by getchoo into account (stuff like: do we want to namespace packages similar to e.g. gentoo or aur).

1 Like
38

It is quite the elephant to eat. Here’s a way more simple example of the problem.

I’ve got this std-lib -ish thing for Javascript (my good-js repo, which is only halfway done fixing circular dependencies if you want a look).

  • I first grouped things by like “array_functions.js” and “string_functions.js” etc.
  • Then my array functions needed some of my string functions. So array.js imported string.js. No problemo.
  • But then one of my string functions needed one of the array functions. Big problemo.
  • I can’t have string.js import array.js because that would now be a circular dependency. Which Javascript doesn’t allow (at least with synchronous imports, at least on the web and in Deno)

So I was in quite the pickle. Even worse circular dependencies between groups started happening all over the place (iterator helpers, set helpers, async helpers). I eventually realized I needed a flat structure.

  • Every helper function gets one file. No grouping.
  • Each function imports exactly the helpers it needs from the flat structure. Nothing more nothing less.
  • Then the namespaces (string.js, array.js, etc) just import funcs from the flat structure and organize them into nice little bundles.
  • volia, no more circular dependencies, and all the namespaces still work

Its kind of the same thing for nix, just a lot worse since nix not only allowed circular dependencies/imports but actually the nixpkgs team just went all-in on circular deps for over a decade for thousands of packages.

(Also last note, the flat structure doesn’t work for JS when there are two functions that are mutually recursive. For a dumb example, isEven calling !isOdd and isOdd calling !isEven is mutually recursive. This “limitation” is probably a good thing because if functions are mutually recursive we probably want to define them in the same file anyways).

5 Likes
39

This a good point. The issue is core is arbitrary. And to your point my nixpkgs skeleton was built around “packages needed for cowsay” not the nix CLI.

The nix-cli choice is because, every system using nix at-minimum must’ve had nix-cli built for them. Otherwise they couldn’t be running the code lol. So why not say the packages for nix CLI are the core.

Otherwise I don’t think there’s a good way to differentiate core packages from non-core packages. And we might get this problem where core becomes more and more bloated over time with people arguing over what should and shouldn’t be core.

5 Likes
40

Oh, I think there’s a misunderstanding. The entire post, with all its massive complexity, was the most simple structure I could come up with that PREVENTS circular dependencies (without loosing functionality, and without becoming unmaintainable, and in a way that we can iterate on it). That was the whole goal: destroy circular dependencies. Its just really really really hard, at least for me.

(Also I’ll have to reply to the rest of your post tomorrow! Thanks for the long response!)

(Edit: also also, like srxl said, I don’t think anyone can predict whether the structure is going to work or not until we try it.)

7 Likes
41

I love the idea of multi-repo, and forgive me if I missed something, but why can’t the multiple repos just be an implementation detail? Good for devs and maintainers, but irrelevant to end users.

As far as the user is concerned, they input auxpkgs and have access to whatever they need, like they currently do with nixpkgs. But auxpkgs updates are built from merging all of the multi-repos together. Doesn’t matter what repo a package is in, it ends up in the final package cache. Each repo has a single auxpkgs input, so cross-repo dependencies are automatically resolved when auxpkgs gets built, again as with the current nixpkgs.

2 Likes
42

At some point in this thread I tried to summarize the goals we want to achieve with the multi-repo approach. Regarding goals from the user perspective, this was:

Therefore the answer to

but why can’t the multiple repos just be an implementation detail?

is: because we see (or: saw at some point in the discussion) value in exposing this implementation detail to users.

I do agree though, that for users of the top-level / auxpkgs this multi-repo architecture should just be an invisible implementation detail.

1 Like
43

I agree, but am concerned we are getting stuck in the weeds with all of the talk of resolving circular dependencies, what repos can depend on other repos, etc.

There’s nothing to say a user can’t input a single repo, or a small subset like core and python if that’s all they care about. But there might be value in ending the complexity there. As soon as you want to use a package that crosses repos, you are in auxpkgs territory. Pulling strictly from, say, the python repo wouldn’t make available any packages that depend on other repos, you’d need auxpkgs for that.

44

Okay finally a full response for you @liketechnik (and I think this will help others too)

So imagine two pictures 1. A spider web 2. A tree (a normal/hierarchical tree)

  • Right now nixpkg’s control flow is like a spiderweb, any piece of code is allowed to call/depend anything else at any time, including itself.
  • The proposed repo structure (e.g the registry) allows (but doesn’t enforce) de-tangling the spider web.
  • Those two rules, at least partially, enforce turning the spider web into a hierarchical tree.

Without those rules (or better rules) we would more than likely start making our own custom spiderweb.

I agree we should be cautious about automating things. This automation is for one file, and the only thing that is automated is the order of the attributes. It is effectively a sort function. No advanced logic or “intelligent” dependency detection.

  • sig_sources and flakehub packages will say (or be annotated to say) “this X derivation depends on A,B, and C” (it’ll do this inside of json/toml or a yet-to-be-determined structured way)
  • the automated system says “okay, I’ll make sure X appears below A,B, and C.” (And report an error if that is impossible because of circular dependencies)

That’s the only automated thing.

Doing the sorting by hand would be extremely time consuming and error prone considering multiple people editing upstream and downstream sources.

Thanks for echoing those. I think that’s the best way to identify confusion.

  • core: :white_check_mark: (some room for discussion, but nbd)

  • sig sources: :white_check_mark:

  • registry: No, the registry doesn’t import ecosystem(s). If sig_repos were analogous to fishing, lumber, and mining organizations, then the registry would be an assembly line, and ecosystems would be the big-box retail stores and restaurants (lumber → assembly line → retail store, never the reverse order). I would also go as far as saying it’s not clever at all. It’s the most boring un-clever way possible to organize packages; a flat list in (effectively) chronological build order. Converting the highly-clever recursive categorically-grouped nested nixpkgs structure into the most dumb flat structure ever is what is really hard.

  • ecosystems: doesn’t have derivation definitions. It exposes user-facing nix functions and it points to derivation definitions stored in the registry. If the user-facing nix function is ONLY user facing, it can be defined in ecosystem. If a function is used by maintainers to build packages and also happens to be user facing, then it needs to go in sig_sources and merely be imported into ecosystem.

  • the aux namespace repo: will be like 3 files and ~20 lines of nix code possibly forever. “Update the flake lock and commit” will be its whole workflow. Ideally users use this endpoint and know nothing about the other repos. If they want lib, use aux.lib instead of pulling directly from the lib repo.

  • polypkgs: while “tie everything together” is not necessarily wrong, it is probably misleading. Polypkgs is a drop-in replacement for nixpkgs, but we want to use polypkgs as little as possible. Sadly “as little as possible” is probably still going to mean “all over the place for everything” at first but oh well. However, unlike the aux repo, polypkgs is going to need a notable amount of nix code. At first it will be simple, but as we overlay more and more things it is likely to grow. Sometimes overrides need extra logic to work correctly.

Think of it like this: we deprecate a function in the internal lib, and we do a quick search to see if that deprecated function name exists in any aux repo. If the answer is no, we can safely remove the deprecated function from the internal lib. Nice and clean. With the public lib we can never ever ever ever remove/rename a function because it could break an unknown number of other peoples nix code (which is supposed to not bitrot). We see this already with deprecated junk like toPath in nix. We can’t move very fast when changing lib because every mistake is a permanent mistake. For the internal lib we could move a little faster knowing there is room to fix our mess/mistakes later.

Sorry if my mention of home manager and dev tooling was confusing. They were supposed to be examples to explain the intuition.

Key idea: all packages (derivations) are fully defined before ecosystem even begins. If it has to do with packages then it should be handled in SIG sources, flakehub (e.g. external), or registry (and of course core packages will be handled in core).

Is [dev env] something that needs to be kept together with packaing (for technical/maintenance reasons)?

In practice, I don’t know! We will have to try and see. If it doesn’t get used for packaging, then it can stay in ecosystem. If it does get used for packaging (or if the maintenance team simply finds it helpful) then most of the code will live in sig_sources, then ecosystem will just reference it.

why focus on nix cli for core?

TLDR: it’s a well defined starting point. Its not a must have. We can talk about it more later.

I’m a bit confused on what this part is saying. My understanding is

  1. We need polypkgs internally while we bootstrap
  2. The easiest way for us to test aux will be to start swapping out nixpkgs with polypkgs in our own projects
  3. Other people might like our maintenance. For example, if we fix/maintain an unmaintained package on nixpkgs, then others might use ploypkgs to get that fix and (inadvertently) help us out my testing our code and maybe even contributing.
6 Likes
45

Yes and no. Using packages from different sources can (and does, I’ve tried it) cause conflicting dependencies leading to failed builds. For example: package A from Core and package B from nixpkgs. A and B both depend on C. Package D depends on A and B. The build for package D will fail because there is a disagreement on what version of package C to use. So using it as a source of extra packages is fine, but as a dependency on an aux package may cause issues.

2 Likes
46

Yes, that is very much the idea. At least thats what I had in mind.

Nixpkgs philosophy was basically/accidentally “if it works, it works”. One circular dependency isnt that bad. Two isnt that bad. Maybe even three. But how can you deny the fourth PR for a circular dependency but not the 3rd one? It would feel like mistreatment to the person making the PR (and plus “the PR works and is something important”). That kind of “one more” approach is exactly how we end up with a mountain of tech debt so massive, like nixpkgs, that we effectively we hit “tech debt bankruptcy” – the point where people would rather try starting from scratch (or nearly scratch) than fix the original. People practically need a PhD in nix just to have a “big picture” understanding of the control flow of nixpkgs for building cowsay. And it’s precisely because of circular dependencies and lack of standards.

So I want to be clear, this issue isn’t like a footnote or “clean for the sake of being clean”. This is the meat and potatoes of why some of us are here; Nixpkgs has long failed to address the unmaintainablity crisis and we want to start paying down the mountain of tech debt nixpkgs has created.

7 Likes
47

We are in agreement. I usually have no less than 4 pinned nixpkgs versions in the same project ranging from 2018 to 2024 so I regularly deal with mixture problems. In particular the mismatched dynamic library problems.

That said I don’t really understand why that would affect aux needing/not-needing polypkgs. I don’t think aux can be used to build anything practical anytime soon without depending on nixpkgs and gradually overlaying parts of it. Did you mean to highlight the bullet point about other people pulling in a fix from polypkgs?

1 Like
48

This seems like a really well thought-out design. Is there currently any work towards formalising / implementing it? If not, I think it would be helpful to put this in a github repo or something so that we can have more organised discussion around it (similar to an RFC, but not really).

7 Likes
49

The point I was trying to get at is that I believe using nixpkgs as a crutch while building out our own package set is not a great idea. For end users to use both is fine, but I think we can work to slowly package everything in our own manner independent from nixpkgs at the start rather than overlaying it (also tried some stuff with overlaying and got caught in infinite recursion :upside_down_face:).

I’m working on implementing an independent core set of packages here: add stdenv by vlinkz · Pull Request #2 · auxolotl/core · GitHub. It definitely takes inspiration from nixpkgs-skeleton, but rather than forking and removing, I’m taking bits and pieces necessary for a successful build. @Jeff I would really appreciate if you could take a look and provide any feedback.

2 Likes
50

@aria Thank you :blush:

That is the best part, we can get started on lib and core while continuing the discussion since lib and core don’t need to touch the not-formalized sig_sources/registry/ecosystem.

Awesome!

Successful build of what? nix-cli?

Hey, I mean if you’re capable of “eating the elephant” in one bite and just straight up get a from-scratch core working thats awesome. I don’t think I’m capable of that, at least not with the time available to me.

I’d be happy to! There is just one thing…

Getting Started Blocker

I would be happy to start contributing to the lib and core stuff, but the future of aux licensing is somewhat of a blocker for me. I mean lib and core are independent so as long as they are MIT/similar I can start working on them, but beyond that I don’t think I can contribute code to sig_sources/registry until I know if the aux system is going to end up as copy-left or some gray-area that makes bussinesses avoid it.

2 Likes
51

I’m afraid, but I’m still struggling a little to correctly get a mental image of this.

If you don’t mind, I’d like to try to understand these better with your help again, but I can also totally understand if you’d rather focus on the actual technical aspects of this with people who are already more experienced with how nixpkgs currently works and how your proposal would therefore play out.

Based on your explanations, let me try to echo my updated view on those again (leaving out lib, core since we both have the same understanding of those already):

  • sig sources:
    • imports: core, lib
    • provides:
      • bootstrap of SIG/language toolchains, e.g. the python compiler, nodejs or the rust compiler
      • SIG/language nix functions for creating packages, e.g. today’s buildRustPackage, buildPythonPackage or buildNpmPackage
      • packages of SIGs/languages, e.g. today’s pythonPackages.sphinx, nodePackages.prettier, but not yet actually build-able, since the “connection” with the dependency derivations happens in the registry (?, see also below “registry->provides”)
  • registry:
    • imports: core, lib, sig sources
    • provides:
      • flat structure of package derivations defined in sig sources (+ core?), i.e. it “connects” the derivations (today’s callPackage?)
  • ecosystems:
    • imports: registry, lib
    • provides:
      • exports of the package derivations from the registry, e.g. today’s pythonPackages.sphinx, nodePackages.prettier
      • exports of the SIG/language nix functions for creating packages, e.g. today’s buildRustPackage, buildPythonPackage or buildNpmPackage
      • ecosystem specific nix functionality that is not used for creating package derivations - here I have a hard time coming up with actual examples - is this e.g. the nixos configuration module system?
  • aux namespace repo:
    • imports: ecosystems, lib
    • provides:
      • exports of ecosystem + lib
  • polypkgs: (temporary?) nixpkgs + aux combination, that allows us to lean on nixpkgs for getting things started and then progressively replace nixpkgs content with aux content

Regardless of the details of the individual repos I’m trying to understand above:
From your explanation I gather:

  • we do end up with multiple repos
  • we do not end up with SIG specific repos, that is e.g. both rust and python infrastructure ends up in the same “sig sources” repo
  • we do end up with packaging infrastructure concern specific repos, that is we have cleanly separated library (‘lib’, i.e. derivation-less functions), core (‘core’, bootstrapping compiling anything at all), derivation definitions (‘sig sources’, language specific bootstrapping/derivation helpers, all packages), derivation combination/connection (‘registry’), module defintions? (‘ecosystems’) + the one-thing-for-all repo(s) (aux, polypkgs)

Is that understanding correct? Or are, e.g. sig sources or ecosystems, mean to be split-able into multiple SIG specific repos?

2 Likes