Replace standard utilities with rust

Committees Security Committee
1

What about using coreutils-rs over coreutils, and using systemd-run0 over sudo?

10 Likes
2

+1, but do we have volunteers with Rust skills? We could always list it as an Issue, with Enhancement/ Needs Help flag, in GitHub speak.

Chris

4 Likes
3

IMO one simplest start could be to document the necessary/ recommended/ minimum config + provide a space for feedback to evolve that.
The wiki could serve as such.

Forming a working group around this might be interesting. I see overlap between

6 Likes
4

I’d love to allow replacing coreutils with something like uutils/coreutils-rs. However I’m seeing several possible ways to “replace” coreutils:

  • Putting $replacement in $PATH instead of coreutils. This is pretty easy and shouldn’t break too many packages (I think ? Hopefully ?), because building derivations still uses coreutils. If issues arise they will be annoying to deal with though, since users will need to figure out that their system is not using GNU coreutils and that some software relies on some incorrect or missing implementation of $replacement.

    Note: you can already use uutils this way today, it’s easy enough to put in your $PATH.

  • Using $replacement in stdenv. This is a pretty massive effort imo, but it was already being worked on in nixpkgs (and it seems it went pretty far). I’m not sure what the implications are, I’d guess it would trigger the very massivest of all massive rebuilds, for starters ? If so we can’t practically use it for general use until it gets in a binary cache.

I’m not sure which one is the one you’re thinking of, but I think those are two different kinds of discussion (thus 2 different topics).

5

I can envision them as 2 steps of a longer process, actually in the order you presented.

But yeah the massive rebuild for step 2 is … well … :see_no_evil:

If the multirepo discussion for aux yields a workable structure, maybe there is a way to essentially “shard” the rebuild by repo :thinking:

1 Like
6

I don’t think those really are steps in the same process. Packages like busybox and uutils can already be used instead of coreutils, and it’s more a system config matter than anything else. If we want to do something here, that would probably be just providing a module to help doing it right, but I don’t think there’s much to do packaging-wise.

Basing stdenv off them though, that’s another situation, but it’s already pretty easy to configure your system to use an alternative stdenv that’s e.g. musl-based.

I don’t think this will help much: If something gets updated everything depending the new version will need a rebuild. You just don’t have to update every dependent repo upon updating stdenv, while any nixpkgs package doesn’t have a choice at which stdenv it uses (ignoring shenanigans with pkgsMusl and the likes), and updating stdenv immediately triggers a rebuild for all packages from that point on. But I don’t see why we would de-sync SIG repos like that, it doesn’t solve the massive rebuild problem.

Tbh the massive rebuilds aren’t really an issue for this work, they’re inherent to any core package update and we will have to handle that anyway. It’s just that testing a new stdenv on a lot of packages requires a lot of rebuilds, and I wasn’t really concerned by that, just saying we don’t have the infra right now for this. Though just getting an stdenv with uutils in it would still be an advancement and can be worked on today.

1 Like
7

BTW. I’ve started playing around replacing bash with the backward compatible oils-for-unix osh, which would pave the road to upgrade the stdenv code to ysh and therefore allow an easier build language (don’t tell me bash is easy!).
Given that ysh is still in an extremely early state, it’s probably not something we want to focus on now. But I just wanted to throw it in the pit in case there will be a SIG to make the stdenv better :slight_smile: It will be a very longgoing project…

2 Likes
8

+1 for oilshell or whatever it’s called this week :slight_smile: I’ve used it a little to iteratively comb some complexity out of one my existing “pile of terrifying bash” horror projects and really like it. Some of the decisions look a bit odd from outside but make sense once you start to use it and read around.

My understanding is the “new” ysh language is subject to change, but the bash compatible osh isn’t, so it might already have some utility as a “bash-linter-on-steroids.” It also sounds like the C++ version is close to finished, so performance may not be an issue for long.

1 Like
9

yeah osh is pretty stable and much better than shellcheck already is. C++ translation is finished and the docs say to only package this version now. And the name oils-for-unix & osh/ysh will stay :slight_smile:

But ysh is still very experimental. I recommend to test and give feedback anyway, the author is very humble and gives every feedback a good thought! Makes one feel really valuable :smile:

Anyway enough off-topic I guess :stuck_out_tongue: :

1 Like
10

Part of me recalls “nushell builder” being a discussion that was happening somewhere in the Nixpkgs realm. Don’t remember where I heard that off the top of my head. FWIW, I’d support a similar play here. Not familiar with anything around oil at all, but a very quick glance seems interesting.

I’m always cautious of replacing tried and tested core system components with newer, upcoming replacements, but I do still think it’s worth evaluating. I wouldn’t commit to anything like this unless we were confident that we could replace these tools without introducing any major breakages or security issues, though.

(full support for run0 though - suid binaries should have been a thing of the past many moons ago)

2 Likes
11

Probably an announcement in the nixos forums. It would great if that env didn’t rely on the archaic sed, awk, find, etc. present in the nixpkgs stdenv. ripgrep, rfind, are two tools I can think of from the top of my head that I’d much rather use instead of sticking to the same tools present when nix was conceived 20+ years ago.

12

While I think Bash as a builder is holding nix back (why do we need to invoke a whole bash instance just to copy files to $out or start a child process), and from what I see osh is doing a good job, I would still strongly agree with @srxl: I don’t think we should mess with that low level defaults yet.

I think we can try replacements in parallel, I’d love to see docs giving examples using osh as the builder and leveraging osh features. But in terms of swapping out the default builder, bash is just a nearly universal standard.

As for coreutils. I would be in favor of calling them coreutils-gnu and coreutils-rs as a way of “leveling the playing field” instead of calling gnu just “coreutils”

1 Like
13

I think the trouble with this is often bootstrapping. If we can find alternatives that truly shrink the bootstrapping footprint (like if building ripgrep ONLY needs a c compiler) then I think it might be worth incrementally exploring. But if ripgrep needs autoconf and autoconf needs grep then we simply added bloat.

1 Like