Aux Foundational Packages

jakehamilton · June 5, 2024, 2:11am

I recently announced the creation of Aux Labs for experiments which included a Nix library with a working module system. I have now updated the repository with Aux Foundation, a foundational package set that builds everything starting from a single binary seed file. This is based off of the minimal-bootstrap work done in NixPkgs, but this repository has no dependency on NixPkgs.

Currently, the initial Stage 0 is built and exported. This includes some important components such as:

kaem: a tool for running scripts on systems that don’t have a shell
mescc-tools: compiler tooling
mescc-tools-extra: additional helper commands such as cp and chmod

Everything in the repository is built using the module system. While I do not expect to reuse these modules in a larger package set (rather, we would import just the packages), I am envisioning a somewhat similar structure for a larger package set.

srxl · June 5, 2024, 4:22am

Can I ask that you communicate projects like these before starting? Some SIG Core members have been working on similar efforts (particularly, @Jeff’s work on this branch and this thread), and I would rather we not waste time on duplicated efforts. There needs to be open communication on who’s doing what, especially this early on in bootstrapping.

jakehamilton · June 5, 2024, 7:54am

This is still an early experiment, I wanted to try getting things to work. What stage would you like me to communicate about things like this? I figured that mentioning it now that there is something concrete would be the right time, but perhaps there is a better way?

I’m not sure if the work Jeff has been doing overlaps with this. The foundational stuff is all about bootstrapping from a single binary seed, but the core boot requires things like bash to already exist. This foundational package set contains the primordial pieces that the core stuff would use to bootstrap

srtcd424 · June 5, 2024, 9:00am

The only thing I’d say from my point of view is that because of your status as our ‘founder’ :) I tend to default to seeing your announcements as official “we’re doing this” rather than “I’ve been experimenting” - you didn’t actually say that though, so it’s my fault rather than yours!

I agree that it probably good to prove that something is workable in a basic way before opening it up for comments etc - just in case a particular design proves to be a blind alley, or to avoid bandwidth overload if you’re still trying to hack in the face of lots of excited commentary :)

8bitbuddhist · June 5, 2024, 1:18pm

This does read a bit as an announcement rather than a proof of concept Mostly because it was posted in sig_core. Maybe reframing the post as “hey all, I had this idea I wanted to try out, so I added it to auxlabs as an unofficial experiment. Check it out if you’re interested!”

Jeff · June 6, 2024, 12:05pm

I’m actually fine with independent exploration. (No feelings hurt)

I think we should keep trying different things, criticizing, and then eventually work on merging, evolving, and deciding.

I’ll take a look at it to see if theres any criticism I can give, especially on the detangling side. But so long as we don’t pull the trigger early on one experiment I think experimenting is fine.

And @jakehamilton I think you did a good job making clear this is an experiment

Jeff · June 6, 2024, 2:01pm

@jakehamilton

So, I’m not nearly done, at lot of it looks fantastic and I’m really impressed. In terms of criticism one of the first things I run into is “where is lib defined?”. For example

first I went to src/staging/stage0
then that lead me to builders
not sure what kaem is, so I went to one of the other builders
got to lib.modules.override.default but I don’t know that behavior and I don’t/can’t know where that is defined, because lib is an argument and we (generally) don’t know everywhere this function is called

Which is a bigger design principle I think we should discuss.

_

When we read lib.doSomething I think we pretty much all expect it do the same thing every time. But lib.modules.override.default could return null every time for all we know; it is an argument after all. Instead of importing lib (like most languages), when we have it as an untyped argument we loose all ability to locally/statically reason about the code. As a reader, we can’t even assume lib.modules.override.default will be a function.

Readbility is one thing, but more importantly it makes code really really hard to refactor/maintain because, for example, if there is a null check on lib.modules.override.default then we can’t really ever remove that null check, because maybe – somewhere – an argument does cause it to be null. “Anything is possible (thanks to arguments)” might sound nice, but it does not help with understanding, debugging, and fully-testing code.

_

I know nix likes to be able to overwrite everything, that way (in theory) people don’t need to fork in order to modify. But after years of this approach, I think we can say it hasn’t really worked in practice. Perfect exampke is @ VlinkZ being unable to give an overlay of core stuff on nixpkgs because it breaks so many things. Even though he’s plenty experienced in using Nix.

Its easier to fork and modify, than it is to surgically try to override/hack-in behavior. So I think, at least for foundation, its okay to loose that benefit. If someone wants to change foundational behavior, just fork.

_

The alternative, I think, is having lib as a subrepo or submodule (so that we can have a direct import of lib rather than lib as an argument). I know that goes against my original plan, but hey, that’s good. It means the prototypes are doing their job at changing my opinion/estimates. I now think a subrepo is the best way to make foundation maintainable and easy to reason about for new members, while still keeping lib pure and isolated in its own repo.

I think we could avoid the need for subrepos if the builtin import allowed for importing URLs. But that’ll be a topic for another time.

jakehamilton · June 6, 2024, 10:09pm

Thanks for the feedback Jeff! I’ll try and clarify some things here.

Lib comes from this experiment here, it is a replacement for NixPkgs lib and provides a fully functioning module system in addition to other helpers:

Foundation is built using module and just like the NixOS module system, lib is passed in as an argument:

github.com

auxolotl/labs/blob/5032b8772bf8e6cc40d9be41c0f8cec6d32894b7/lib/src/modules/default.nix#L608


      
                };
              };
            };
          };
          
          merged = let
            collected =
              collect
              (args.path or "")
              (modules ++ [internal])
              ({inherit lib options config;} // args);
          in
            lib.modules.combine prefix (lib.lists.reverse collected);
          
          options = merged.matched;
          
          config = let
            declared =
              lib.attrs.mapRecursiveWhen
              (value: !(lib.types.is "option" value))
              (name: value: value.value)

This is mostly for convenience since otherwise every single module would have to import lib in order to declare options. That can get pretty difficult when working with flakes since you would need to point to the flake input. I think having this argument for the module system is a decent trade off here.

All lib functions are namespaces and (to some extent) documented. For example, lib.modules.overrides.default can be found in the modules namespace implementation here:

github.com

auxolotl/labs/blob/5032b8772bf8e6cc40d9be41c0f8cec6d32894b7/lib/src/modules/default.nix#L294


      
          ##
          ## @notest
          ## @type a -> Attrs
          option = lib.modules.override 1500;
          
          ## Create a default override for a value. This uses a very low priority
          ## so that it can easily be overridden.
          ##
          ## @notest
          ## @type a -> Attrs
          default = lib.modules.override 1000;
          
          ## Create a high priority override for a value. This is not the highest
          ## priority possible, but it will override nearly everything else except
          ## for very explicit cases.
          ##
          ## @notest
          ## @type a -> Attrs
          force = lib.modules.override 50;
          
          ## Create a high priority override intended to be used only for VM targets.

Jeff · June 7, 2024, 1:29pm

Agreed. I’m just talking about foundation at the moment though so I’m going to put the flakes input argument aside for now.

I think we should be very careful when doing things “for convenience”. Most of nix/nixpkgs, including stuff like with lib, were justified with “for convenience” and now we have to clean much of all that stuff up.

Pretty much all other language import libs when defining a module; rust, python, java, C#, typescript, ruby, etc. I don’t think importing a library is inconvenient.

But beyond conveniences, I think we can agree it is imperative that this code is easy for other people to understand. Foundation is going to be complicated, and we shouldnt make it more difficult to newcomers.

In isolation, wouldnt you agree its easier to find the definition of an import compared to finding the definition of an argument?