Forge software/host

Special Interest Groups SIG Documentation
76

Yeah part of the difficulty here is that our long-term goals are around established, elected governing bodies with delegated authority. Right now we don’t have those (as was mentioned in the roadmap) and decisions have to be made. I want to gather as much input from people on these foundational questions, but at the end of the day someone needs to decide and take action. In the future that will be the role of governing bodies (eg. Steering Committee). For now we may have to bear with the less-than-ideal situation of discussion followed by a possibly less-than-ideal decision.

I’ve mentioned this before, but this is the reason that I posted the values, goals, and roadmap. In doing so I wanted to avoid any unnecessary wheel spinning or arguing about things that didn’t get us to a place where we have solved the problems we experienced in the Nix ecosystem. Getting there is going to take some time and especially at the beginning while things are bootstrapped we are going to have to make do with what we have.

3 Likes
77

I don’t like the executive decisions either, but I do think theres going to need to be some until the steering committee is formed and governance established. Which kind of requires getting basic infrastructure running.

IMO: “Can it be fixed later?”

  • If yes (no extra difficulty) we just need to accept executive decisions, get it working and move on
  • If yes, but with much difficulty (mid level foundation stuff) then we should debate it heavily but not let it block the project.
  • If no (basically set in stone) we should debate fiercely on it, block on it, and push back on executive decisions

Git makes it easy to switch where the data is stored, and even keeping two sources in sync/mirrored. Even with CI, thanks to containers, its decently easy to switch from one CI to another. We just need to avoid depending on github in a way were it snowballs into a big problem later.

Who is on the steering committee, how the packages are done (distributed or centralized), what the project will be named, those things, if done wrong, are nearly impossible to change/fix later. And I think its right to call out executive decisions there.

4 Likes
78

Yeah, for some topics there is a healthy amount of discussion to be had. The name, for example, has some caveats as it currently exists. Overall, though, I think we’re moving in a positive direction.

1 Like
79

Excuse me for being blunt here, but that is just incredibly naive. Gitea has been trying for years to rid themselves of Github and become self hosted, and they are building a forge. Claiming that “Git makes it easy to switch code hosting” is only true if everything a forge is about is just the code. In case of Github (or any proprietary service, really), it’s fighting yourself out of a walled guarden, hoping that you’ll actually be able to if the time comes.

3 Likes
80

I think Codeberg and Forgejo serve to counter that argument, being entirely self-hosted including for development.

5 Likes
81

They do, but only by burning all the bridge, treating Gitea as a hostile fork.

82

From another thread:

I feel like if we take this approach, it may make moving easier either now or later since we wouldn’t need to move an entire fork of nixpkgs, rather just the parts we are working on (although it may grow to nixpkgs size in time). I feel like starting off directly with a fork of nixpkgs will be too much to maintain, especially starting out now. I imagine the core repo importing nixpkgs as a flake input, with other package groups importing the core, and then have a top-level repo (auxpkgs or something else) exporting everything. I think building something from the ground up might be easier than breaking nixpkgs apart to fit our new mold, and could also allow us to more easily move away from GitHub.

4 Likes
83

You can certainly attempt that, but from my own personal experience, sooner or later you’ll run into issues that cannot be addressed through overlays, overrides, etc.
Personally, I’d not attempt this without resolving Support flake references to patches · Issue #3920 · NixOS/nix · GitHub

Your experience will not be great.

1 Like
84

Sorry I don’t think I was clear. I’m not proposing using overlays or overrides, rather creating a new monorepo (or group of repos), but cheat by being able to rely on nixpkgs (flake input) packages in case they aren’t already packaged within the new repo itself. I’m worried that if we just fork and try to start maintaining all of nixpkgs off the bat, we’ll be overwhelmed just maintaining let alone restructuring it

2 Likes
85

I like this plan. I am also happy to assist in making sure Forgejo becomes a suitable forge for Aux to self-host with. It’s not terribly hard to get there, it just isn’t there today.

Forgejo has a built-in CI that is very similar to GitHub Actions (it is using act as a runner, under the hood), so the transition there wouldn’t be very painful. Not a drop in replacement, but it’s close-ish, and the transition is usually not too painful in my experience.

…they have not been trying too hard. If they closed PRs and issues on GitHub for about a week, they could have migrated by now. Trying to migrate while issues and PRs are open is never going to work.

No? Forgejo was self-hosting from day one (perks of forking off), except for CI (Forgejo initially used Woodpecker before switching to Forgejo Actions). No bridges needed to be burned, and Forgejo continued to try and collaborate with Gitea for two years. Some Forgejo PRs even landed in Gitea during that time. Even after the hard fork, we do not treat them as a hostile fork. We cherry pick weekly from Gitea, and they’re free to cherry pick from us too.

3 Likes
86

Forgejo was self-hosting from day one (perks of forking off), except for CI (Forgejo initially used Woodpecker before switching to Forgejo Actions).

I never claimed any different. My point was, forgejo never tried to migrate out of any forge, trying to keep metadata, issues, wiki etc. In tact. They forked the Gitea code base, treated upstream pretty much as hostile, and started fresh.

87

Yes. But there was no reason to migrate. It has nothing to do with hostility.

On the other hand, I helped companies migrate off of GitHub, repos and projects larger than Gitea. It’s not easy, but it is definitely doable and does not take years.

1 Like
88

But there was no reason to migrate.

Basically my point, hence it does not compare.

89

Before NixOS I was using FreeBSD. The scope and size of FreeBSD (including ports) is definitely comparable to Aux, and I believe we can learn a log from the BSD community. The BSDs are structured for independence and being self-sufficient. They’re very conservative and I can’t tell much about governance structure, but I was there as a user during migration from Subversion to Git (which happened in 2020, this is how conservative the BSDs are; I think OpenBSD still uses CVS), and I can talk a bit about their code hosting.

All is self-hosted, no reliance on any third party forge or service.

Canonical central repositories are at https://cgit.freebsd.org/ – no forge. No forks either, there’s one central repo with branches, so the issue with forks taking up space doesn’t apply. I’m not sure how the repositories themselves are hosted – I expect either custom solution with plain ssh access, or something as simple as gitosis/gitolite. On top of that, cgit for read-only web access.

All the contributions I made were simple, so I sent them as patches to Bugzilla. For more complex changes there’s a Phabricator. I’m not sure how does the workflow look like once you’re a comitter, TBH. I can try to look it up if anybody’s interested.

This is simple, but very flexible – since Git itself is distributed, committers can work on branches in the central repo or maintain the fork however they see fit. There’s a read only GitHub mirror that can be forked. Some subystem maintainers organize their work around GitHub – they create a central GH fork for their workstream, accept PRs from other contributors, and then push their integration branch to the central repository to upstream. DRM (graphics subsystem) effort was initially organized this way.

I see that the freebsd GitHub org is used for canonical hosting of some smaller components that aren’t as stable as the core repos. Seems like DRM is now an out-of-tree kernel module (GitHub - freebsd/drm-kmod: drm driver for FreeBSD) – this repository is used as a source for a port (i.e. external package out of the base system tree).

If this is helpful at all, I’ll try to look for descriptions of the development workflows [EDITED TO ADD: Committer's Guide | FreeBSD Documentation Portal documents the development workflow in detail, it’s not as sophisticated as we’d need, but still might be valuable for inspiration]. I feel that BSDs’ focus on self-reliance and self-hosting is something we can learn from.

Does anybody have insight how are other projects of similar scale hosted and structured? (Other BSDs? Debian? Arch? Slackware? Something large that’s not a os/distro and not a de facto corporate project?) Let’s try to learn from other projects and avoid repeating their trials and errors if we can.

3 Likes
90

I’d love to see a gateway to/from Radicle at some point, but I agree for the foreseeable future pragmatism and getting up and running quickly is probably the most important. Also, I hate myself for it, but I do tend to feel mild irritation when I have to deal with a non-Github hosted project - apparently I’m getting old and can no longer cope with learning multiple UIs :slight_smile: So perhaps concerns about about developer friction have some validity.

91

For more complex changes there’s a Phabricator. I’m not sure how does the workflow look like once you’re a comitter, TBH. I can try to look it up if anybody’s interested.

My employer hosts phabricator and uses it for some things, so I’m familiar with it.

I know that phabricator is unmaintained since 2021, see https://phacility.com/

There is a community fork, https://we.phorge.it/, but I don’t know much about it

1 Like
92

No problem, you and I are on the same page. When I said code hosting I meant it very literally: it makes just switching storage locations easy. Not issue tracking, actions, webhooks etc. Sorry that wasn’t clear, I went and edited to be more explicit.

1 Like
93

Update:

I have heard back from Codeberg and I am setting up a call with them to discuss some specifics. I may also ping the Lix folks in the mean time to see how we can collaborate since they have their own Forgejo instance.

19 Likes
94

(reposting something that came up on Matrix, to put it in a place where long-form discussion is easier)

In On the future of our nixpkgs fork, there’s a proposal to abandon forking nixpkgs as-is, and focus on building an alternative directly instead.

@crebwyllion made the connection that if we do go that route, forges other than github become a lot more feasible. Nixpkgs’s massive size and the integration of nixpkgs tooling with github are a technical obstacle to moving elsewhere if we’re trying to fork and maintain nixpkgs as-is.

OTOH, if we’re going straight to the smaller, modular repos and doing our own thing… We’re going to need a bunch of original tooling anyway, and we should end up with sensible repo sizes. Which makes moving off github a lot more tractable.

So… yeah, something to consider in this discussion :slight_smile:

18 Likes
95

I think the biggest consideration is going to be bandwidth since each user clones the repositories at least once (but realistically every update). Not sure if we could reasonably self host that right now, but if we could then I think it would be cool to set up our own forgejo instance. Otherwise Codeberg may still be the best option.

6 Likes