I think for Stage 1, since we inherit a monorepo anyways, the simplest option is to give each SIG a branch/fork and then have the CI try to perform wave merges. This still gives SIGs the ability to merge at their pace.
1 Like
@_deleted I’m really curious about this, and I think its relevent.
Lets say auxpkg wants to overwrite package abc in nixpkgs. What kind of problems could come up?
Like assuming the worst possible senario, that abc is heavily used all over nixpkgs and maybe even inside of bootstrapping.
I think overwriting is the wrong way to go about it. Instead, just have flake output for each. Still working it out, but something like this:
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
};
outputs = { self, nixpkgs, ... }:
let
lib = import ./lib;
forAllSystems = lib.genAttrs lib.systems.flakeExposed;
in
{
nixPackages = forAllSystems (system:
(import nixpkgs { inherit system; })
);
auxPackages = forAllSystems (system:
(import ./. { inherit system nixPackages; })
);
};
}
So you can depend on nixPackages.<pkg> until there is a suitable replacement
1 Like
Well lets say openssl has a vulnerability. The Aux maintainers patch it so auxPackages.openssl has the fix. Well everything in nixPackages doesn’t get the fix, and other stuff in auxPackages might use nixPackages.python, which in turn uses nixPackages.openssl instead of the patched auxPackages.openssl.
I think we will have to do an override on nixPackages. Which is recursive and going to make stuff hard to debug, so we should design carefully. But I think its necessary to be able to maintain without full direct forking.
As a temporary bandaid fix yes, but I don’t want to set a precedent of auxPackages just being overrides of nixPackages. A better ideal goal in that situation is bring python into auxPackages
1 Like
Patching openssl is going to mean we need to pull everything that uses openssl into auxPackages. Which is on the order of half of all packages including core. I don’t think that’s realistically within our ability.
1 Like
Agreed, it’s not. Overriding may become necessary for security reasons. The long-term goal however should be to not depend on nixpkgs at all, so I believe when possible we should package ourselves rather than overlay. You’re right for something like openssl that may not be possible
I agree. Aux should absolutely not be advertised as “an overlay for Nix”. Its just a way for us to maintain core components with O(1) sized increments (which is still potentially huge) instead of O(n) sized increments.
What we can do is establish policies like, when a package is pulled into auxPackages, we do a build and trace every nixpkgs attribute that the build touches. (e.g. auxPackages.python would have nixpkgs.gcc and nixpkgs.openssl on that “touched” list). We would mark/record this list in the meta attribute, and once all the things on that list have been overridden with pure auxPackages (e.g. nixpkgs.gcc == auxPackages.gcc, and auxPackages.gcc.meta.pureAux == true) then we can change the meta and say auxPackages.python.meta.pureAux = true and do basically a find-and-replace on the code (swapping the nixpkgs var out with auxPackages var)
Then eventually, there won’t be anything left pulling in nixpkgs.
2 Likes
As much as I dislike using fetchers instead of inputs this is probably the best solution here (at least until lazy inputs are a thing). This would pull a minimal amount of code for the user and is reasonably easy to update. In fact I’m pretty sure we could use Drift to do it automatically.
1 Like
Re: docs location
I think this is the direction things were leaning in. Is that right @minion & @coded?
Also I recognize your name, glad to have you around 
1 Like
Correct, I wholeheartedly agree
As this is only talking about the packages itself AFAICS: What’s the stance on all the modules aka. nixos/modules/?
Should/will they be separate from the packages entirely?
Are there already any ideas regarding that? Since that will come into importance rather sooner than later too.
2 Likes
I believe we will be doing the same with NixOS, moving it to a separate repository and creating a group responsible for its work.
5 Likes
(sorry for the wall of text, I tried to make it legible)
Until something like loomio is up, maybe using the wiki as a stopgap could help with a discourse topic for each proposal? I’m a bit lost with the proposals.
So far I see:
- mono repo:
- josh: Monorepo with a git proxy in front? Never used it myself, but I do wonder what pull/merge requests will look like
- branch per SIG: not sure if
masterwill be periodic merges from SIGs?
- multi repo:
- per domain
- per programming language: some projects use multiple languages?
- mixed:
- monorepo = collection of pinned versions: packages in their own repos/domains/languages and mono repo pins them?
Probably missed others… discourse’s lack of a threaded view isn’t helpful 
I’m not entirely convinced multi-repo is a good option. It feels more natural e.g a language has a standard library and new projects using the standard library don’t end up in the same repo.
However how are dependencies tracked? And if a dependency has to be updated that might require pull requests across multiple repositories. At best it’s tedious, at worst, it things can be forgotten and security fixes aren’t propagated.
Maybe I missed it, but my major question is What are we trying to fix? Which difficulties is the monorepo providing?
Each proposal should be investigated for their weighed strengths and weaknesses. Something might be a negative but a minor inconvenience, some things might be positive and a massive boon. Having a scale and a method of counting to reach a score could be quite helpful.
As an example, a scale of:
- 0 : Unimportant
- 1: Convenient
- 2: Important
- 3: Very important
- 1000: Dealbreaker
| Feature/Risk | Weight |
|---|---|
| Looks good | 1 |
| Allows domain ownership | 2 |
| Tooling exists | 2 |
| Supported by forge | 3 |
| Feature/Risk | Proposal Jaba | Proposal Doobie |
|---|---|---|
| Looks good | yes | no |
| Allows domain ownership | yes | yes |
| Tooling exists | yes | no |
| Supported by forge | no | yes |
| Total | 6 | 5 |
The total is calculated by simply converting “yes” to 1 and “no” to 0, multiplying it by the feature weight, and summing it up → sum([ weight * boolean for weight, boolean in column.jaba]).
This would double as a historic record and help during decision making by allowing to see:
- What are the problems
- How important are they to the community
- Which proposals there are
- Which feature/risk every proposal has
- The mathematical “victor”
2 Likes
could help with a discourse topic for each proposal? I’m a bit lost with the proposals.
I’m struggling myself too, but let me try to shed some light onto the current discussion in this area:
I do think we’re starting to debate ‘the same’ topic in three different places now, or rather three so closely connected topics, that they heavily influence each other:
- in this topic, we’re debating mono- vs multi-repo
- in SIG Repos: How should they work? we’re already discussing how our multi-repo approach “will look like” (in the sense that ‘mono-repo’ isn’t an option that’s part of the options in that specific topic)
- in On the future of our nixpkgs fork, we’re discussing how we will (not) integrate nixpkgs into our own package set(s); but with the base assumption also kind of being that we have a multi-repo approach
Nevertheless, you raise some excellent questions.
At some point, (part of) the answer was (or in my impression the answer was):
Let me try to summarize the current discussion and extract the main points still in discussion:
- Goals (from user perspective):
- SIG repos can be used individually, reducing size and eval time
- top-level still provides what previously ‘Nixpkgs’ was - a one-stop location for all packages
- Goals (from SIG/maintainer perspective):
- reduce maintainer burden
- scoped repos, with less noise
- less moving parts in each SIG repo
I’m very unsure how to proceed here, with the discussion being spread across 3 different topics,
with wildly varying underlying assumptions - but also a lot of thoughts and discussion already spent on each of them.
1 Like
Which isn’t to say that we shouldn’t have this discussion - I’m not the one to decide this, I don’t want to decide this and I do think we should have this discussion.
But maybe we should prioritize seeking consensus of the underlying assumption of what we want to achieve and how to achieve that, for now. (Like there are already so many amazing ideas in here - but unfortunately, not all of them are compatible with each other; and what I’d like is that we realize that as soon as possible, so we’re not starting to work in opposite directions of each other.)
1 Like
We are trying to fix that we dont have enough core people to maintain and publish everything ourselves. Distributed allows for outsourcing the job to a large degree. For example usining the flakehub flakes maintained by other people right now.
Agreed. For now I’m focusing on the “Sig Repos: How should they work”
1 Like
Early morning drive-by link posting: https://gittuf.dev/ - if we do end up maintaining a large repo for any length of time, something like this might help with dividing up responsibilities and enforcing ownership boundaries?
There’s a full discussion about this in