SIG Core Meeting 2024-05-18 - Meeting Notes

Agenda thread:

SIG Core 2024-05-18



  • Introduction since first meeting, some new folks around

Take stock of current progress

Core direction

  • @dfh: How does core fit into the “big picture?”
    • @srxl: Core is the bare minimum to build critical things
    • @srxl: probably not much more than stdenv, lib, stuff to declare derivations…
    • @srxl: effectively the build machinery, other SIGs will be using/consuming core, probably similar to stdenv.mkDerivation
  • What should it contain?
    • @srxl: currently we’ve duplicated nixpkgs’ standard environment, is there anything else we need?
      • @dfh: some common build dependencies, or will they be handled out of core?
        • @srxl: coreutils should stay in core, it’s needed in stdenv… things like build-essential could be in core, possibly some other build tools like meson, cmake
        • @srxl: I’m not decided on openssl, zlib, etc. What do we think?
          • @dfh: This is why I asked, do we want it minimal or do we want it to be something that can serve every build process?
          • @liketechnik: We could say “everything needed to build lix or nix should go in core”, that’s a clear criteria and won’t snowball
          • @srxl: That makes sense, it’s about bootstrapping something that nix/lix will consume…
        • @srxl: Will there be some other repo that has these other libraries? Who maintains the repo? Core? Another SIG?
          • @dfh: If you have core, which is needed to build lix and nix, and other languages… can you construct the build environment only using core? e.g. languages would have their own build toolchain dependencies
            • @srxl: That feels reasonable at first glance, I worry we’ll duplicate dependencies
            • @srxl: Is “needing a duplicate” a good criteria? Lots of things will be duplicated, so they’ll spiral a bit
  • How should it be used by other SIGs?

Moving lang infrastructure out of Core

  • @srxl: what happens if bits of core need to use other languages to create the stdenv? We don’t want circular dependencies…
    • @liketechnik: @jeff on the forums had some great ideas… have you had time & spoons to check that out?
      • @srxl: Could be worth going back to the thread and rereading it
      • @liketechnik: Gist was that everything is tied together in another place, and you can ensure that there are no circular dependencies in that place… e.g. “you will get something to build rust packages and you do not care what it is”
        • @liketechnik: It requires a lot of coordination and effort to maintain two-way interfaces or test each repository, unfortunately
        • @srxl: This seems like something we need to sort out soon. It’ll be a blocker to getting SIG Language groups up-and-started without Core having decided this. This should be prioritized
        • @srxl: Makes language repos too hard to consume individually. They’re kind of out of luck without more stuff, which is sort of like pulling in all of nixpkgs. We should probably think about an architecture that makes this possible
        • @srxl: This seems the most concrete, maybe not most optimal
    • @minion: I care a lot about not having to eat everything to consume python
      • @srxl: Yea, definitely this is important
      • @aria: aria (she/it) says:if you’re not using flakes and you’re ok with IFD, it would be possible to have that with registry right?
        • @srxl: Maybe? It sounds reasonable but I’d have to figure it out some more
        • @srxl: I know that lix was planning to “solve” IFD, they discussed it a little on matrix. We can maybe accept some minor IFD if we want to preference lix
          • @minion: I don’t know how much we want to make it nix-incompatible, this sort of call hasn’t been made
            • @srxl: Steering should make that call

Moving lib into its own repo

  • @srxl: I wasn’t super sure about this, lib isn’t wholly pure, there are tests, etc. which depend on core
    • @srxl: Can we make lib pure? I understand that making it standalone would be nice, but if it depends on core anyway I don’t know how possible that is
    • @srxl: If lib has a dependency in core, maybe we should keep it in core
  • @aria: Remove flake dependency on nixpkgs/core by tcmal · Pull Request #2 · auxolotl/lib · GitHub removes core dependency…
    • @srxl: Ahh, that’s interesting, if we can get rid of that completely then splitting it out is good! Not pulling in nixpkgs to use lib is awesome!
  • @aria: There is also GitHub - nix-community/nixpkgs.lib: nixpkgs lib for cheap instantiation [maintainer=@github-action] (with initial help from @blaggacao)
    • @srxl: I don’t know if it’s useful to do something similar if we have multi-repo stuff anyway
    • @liketechnik: Indeed, it’s not up for discussion on whether we should split them, it’s just the how
    • @liketechnik: I think lib does many things, it’s some pure nix stuff, some build stuff and some of the module system. I wonder if we should split things into more repos
      • @srxl: Maybe these are getting too small. Perhaps we should split the module system, but by this point some components may be only 1 file
      • @srxl: I don’t think there’s any builders, it’s mostly pure nix code…
      • @dfh: If we intend to support the module system for writing packages, it feels confusing to have modules in a repo other than core/lib.
      • @srxl: Module system being in lib is a little weird in nixpkgs, I think keeping things similar is a good bootstrapping decision though…

Working with Security Committee

  • @srxl: We had a lot of dependabot alerts, they seemed to be in tests (so not particularly relevant)
    • @dfh: 16/25 are in tests, the rest are outside of tests…
  • @dfh: You need two kinds of checks
    • “Am I committing known security vulnerabilities?”
    • A scanner, to check that you are handling known stuff over time…
    • It’s looking at dependencies rather than your code, things that are fixable by providing updates
      • @srxl: Ok, so we’d have some tool, and if we get an alert we’d address it
      • @srxl: When it comes to vulnerabilities that don’t have fixes yet, do we prefer to pull it (and break stuff)
        • @minion: nixpkgs has a way to indicate insecure packages, and everyone then has to agree to them
          • @srxl: then that would break everything for everyone in core…
        • @dfh: Don’t make it impossible to insecure software, but make it hard for people to do accidentally. The insecure flag does really well
          • @dfh: If Aux has old versions that may become even more important to mark things consistently as insecure, but it gives more possibilities to play with secure/insecure bits
        • @srxl: Can we make sure that language repos don’t depend on insecure package versions?
        • @srxl: Has nixpkgs ever had part of stdenv marked insecure?
        • @dfh: in that case we need to coordinate with other SIG repos to ensure other things pull the fix…
        • @dfh: You need an “audit my system right now” feature. It needs a list that doesn’t depend on the package repo. You can’t ask the question about a specific package set, because it might change as security vulnerabilities are discovered…
          • @dfh: I want us to inspect this against a database, or similar… If we start tracking security vulnerabilities it needs to both end up in the database and be marked as insecure in packages. That stops people installing something fresh and insecure, and lets people run a checker on old stuff
          • @srxl: Yes, this needs to not be locked… I don’t know that we can do that in core directly
            • @dfh: Yes, I agree, we need to work on “the audit thingy” in security. It needs to stand vertical to everything else.
  • @srxl: Can we automate some of these things, e.g. pull requests for marking vulnerable?
    • @dfh: There’s not a single central vulnerability database. We need some pipeline where we can ingest from a bunch of sources, review, etc. It’s a huge project. Automation would be very great, but challenging…
    • @srxl: This sounds like a good process, but not particularly easy, so not a short-term goal
  • @srxl: So, the immediate process is to mark insecure as soon as we get a vulnerability notice.
    • @dfh: Ok, can we start with a smaller targer, e.g. look at things similar to the npm audit output and start to apply it to the core repo manually on a regular schedule…

Any other business

Next meeting - when? how often?

  • @srxl: Core can have meetings more ad-hoc… we haven’t been blocked by meetings, and since as @vlinkz and @jeff are not present it’s hard to know how important it is. If anyone in core wants to raise a meeting, please feel free to post something on the forums
  • @srxl: We can revisit this later, if we like

Decisions/Action items


@srxl: We had a lot of dependabot alerts, they seemed to be in tests (so not particularly relevant)

You might be interested in using this script, which allows you to run dependabot updates locally, or in CI with a single PR :slight_smile:


Thanks, will take a look.

Glad we were on the same page! If lib depends on core I also would’ve wanted it to stay in the same repo.

I would be a fan of taking the pure parts of lib out and putting them in a repo. I think all the impure stuff should stay as part of core. Core basically is the “impure lib” at least in my view.


So I was visualizing the dependency tree of stdenv and nix for darwin yesterday and …

This… is just absolutely massive.

I feel like there is no way the nix cli actually depends on python3.11, all of those python packages (I mean sphix is in there and thats just documentation!), and haskell and all those haskell modules. And also look at ALL THESE BASH versions. They’re all 5.2 just slightly different, and there is like +30 of them.


I would really really really like to get haskell and python3.11 out of core. My original idea is that something like a perl language SIG would inherit bootstrap-perl from core and then make the full core, but core already has both python minimal and python full, along with a ton of python modules.


Also good to see you here @infinisil ! Thank you for the tip!

1 Like

I noticed this when trying to build Lix from aux-core on my little i3 nuc. Being actually not that familiar with nix/nixpkgs ‘internals’ I wasn’t sure if I should have been surprised or not.

I’m more familiar with Debian packaging where they have a tendency to split things into pretty small bits to avoid large dependency graphs. I don’t know where all the above dependencies get used in nix/lix, but I wonder if it would split neatly into e.g. daemon, core cli commands, additional / dev-only commands, etc.?

I’m not familiar with the code base so they may all be one executable at the moment, but building multiple instances of the same binary with different dependencies for different uses might not be that wasteful if it reduces the closure size significantly for some cases, e.g. headless builders? One to discuss with the lix folks maybe.

Actually there kind of is, is pretty much the de-facto replace to report CVE’s. I made a scraper and UI for it for security a class at my university.

I think it could be practical if every package’s metadata and version was in a JSON file (ex imported by the default.nix) then I would be practical to scan all those JSON files, match them with their name in cve details, and if there is a vulnerability, just edit the json file and auto make a PR

Their footer says

CVE is a registred trademark of the MITRE Corporation and the authoritative source of CVE content is MITRE’s CVE web site.

The “authoritative” search is at CVE - Search CVE List

I haven’t used CVEdetails before, from their page it looks like they do some “enrichment” and feed curation. Do you know if they have a capability to search for affected OS?
Assuming you had access to their paywalled features.

You describing what vulnix has built, and according to their README

Matching Nix package names to NVD products is currently done via a coarse heuristic. First, a direct match is tried. If no product can be found, variations with lower case and underscore instead of hyphen are tried. It is clear that this mapping is too simplistic and needs to be improved in future versions.

From having used a bunch of different linux distros over time I can second that experience. There are libraries/ programs that are not known by their upstream name in the distros package repository for practical or historical reasons.

E.g. nixpkgs has a protonmail-bridge-gui package because the protonmail-bridge derivation does not ship the UI intentionally.

I’ve been pondering about the automatability of this quite a bit, the best first step I have come up with is to build what you describe and add a manual mapping table to it that allows us to hardcode issues.
Sounds tedious, my pro-argument is that this mapping table might be helpful in the future as a data-source for finding better heuristics - hopefully :crossed_fingers:

My personal stance (this has not yet been discussed by the security committee) is that we should see if we can bring vulnix back to a maintained state, see vulnix issue #95