Brainstorming: What should Rust support look like?

Special Interest Groups SIG Rust
1

Hi all!

This aims to basically hit the same note as Brainstorming: What should Javascript support look like?, to get this SIG started.

Although - IMHO - the overall current situation is already pretty solid, there are still certainly things that can be improved upon in terms of Rust support in aux. I’d like to initiate a bit of simply “thinking out loud” here in this thread, on what exactly we can work on within this SIG.

General

  • Dedicated repo where packages and build support will live, aka. auxolotl/rust or auxolotl/rust-support (see also SIG Repos: How should they work? - #32 by liketechnik)
  • Being more “batteries-included” in terms of build support and tooling, usable from outside of our (internal) packaging too
  • Having a project-“blessed” way do to things, but still giving user freedom (of course!)

Different compiler versions

Many Rust projects use oxalica/rust-overlay (or equivalents) to pin their compiler version, either for compatibility versions or to e.g. check building with multiple versions.

This should also include support for other compiler implementations than the reference compiler, e.g. if and when the Rust support in GCC gets ready for broader usage.

I think it would be nice to provide similar tooling directly in aux.

Expand and expose build support

In addition to the above point, many Rust projects also rely on e.g. ipetkov/crane for easily setting up a flake for their Rust project. Again, aux-native tooling would be nice, which should cover at least most usecases, which don’t have specialized requirements.

IOW, our internal build tooling should be improved to a point that can be properly used by Rust projects to set up their flake without having to necessarily rely on third-party flakes. That would also sigificantly lower the entry barrier for using Nix in Rust projects.

Native library linking

As of now in nixpkgs, there is this “great” file default-crate-overrides.nix, which specifies (native) build inputs as needed for building. This is a maintainers nightmare, so getting rid of it in the mid- to long-term needs to a goal to make maintaining Rust support in aux sustainable.

As for -sys crates, there is the package.links field in the Cargo manifest, which has the potential to eliminate at least some of these entries.

Current pain points

  • Non-crates.io dependencies (i.e. mostly git dependencies)
    Though git-dependencies are rather discouraged for various reason, lots of application use them, for varying and valid(!) reasons, e.g. using a patched dependency while waiting for upstream to merge it.

    To give an example, and the reason why I thought about this already quite a bit, is that I’m (among other things) packaging proxmox-backup-client, which uses some company-written libraries that are only available through git.

  • Cargo.lock
    When explicitly needing to specify a Cargo.lock through cargoLock.cargoLock{,Contents}, it can be quite painful to update when the package version get’s bumped.

    Further, having then to copy the same lockfile into the source tree using postPatch = "ln -s ${./Cargo.lock} Cargo.lock"; when missing seems kind of redundant. Maybe this can consolidated in some way.

These are just some of things I thought of, feel free to comment!
I would also regularly keep this top post updated with ideas from this thread, such that they are collected in one place.

Cheers,
Christoph

8 Likes
2

In addition to the above point, many Rust projects also rely on e.g. ipetkov/crane for easily setting up a flake for their Rust project. Again, aux-native tooling would be nice, which should cover at least most usecases, which don’t have specialized requirements.

I just wanted to highlight this specific point because it took me way too long to figure out how to get things set up for rust when I started out on nix.

Having an aux-blessed ‘crane-like’ and a guide to working with it would go a long way in helping devs.

5 Likes
3

This is an amazing write up and encompasses all I would need out of auxpackages. Particularly the below.

2 Likes
4

There’s also fenix, which basically only differs in internal implementation, afaict. It might be worthwhile to investigate which approach seems more appropriate for Aux beforehand.

I’ll also try to publish my personal ‘new rust project quickstart’ flake template today (which is rust-analyzer, clippy, rustfmt support for a toolchain-file provided version (so any non-nix user uses the same rust version too) + cross compilation, based on nixpkgs + fenix only); which I use in a handful of work-related, public or not-yet published projects.

Also, I’ll join in the compliment for the write-up and getting the discussion for rust support started.

2 Likes
5

based on the current proposal, i would actually recommend rust tools remaining in core’s stdenv, especially as we will have some more mission critical packages written in rust

i think this could be a good idea, but we would need to watch for scope creep. i would highly recommend a specific package set for it (let’s call it rustBinaries for now), and only allow regular binaries like rust-overlay and fenix. we should never building packages against each version, as to avoid problems like we may have with python, js, etc. and their multiple versions. we also shouldn’t be dogfooding these versions and prefer our standard, built from source versions; these binaries should be for end users only

i think it may also be interesting to create an offshoot repo named something like rust-binaries to take care of this, that can then be pulled into the primary rust SIG repo

now this will be one of my hotter takes on the forum…but i think we’re already there. rustPlatform is extremely flexible in basically situations, from cross compilation to gut dependencies. i think the only problem here is that people just aren’t aware of its features

a good example is the aforementioned git dependencies — which many will switch to naersk/crane for alone. this is easily missed in the nixpkgs manual, but is totally possible to do without constantly updating hashes (like in a project’s flake for example) through cargoLock.allowBuiltinFetchGit

in the context of a regular repo like ours or nixpkgs, rustPlatform is also much more efficient (and less confusing in all situations, really) with its single derivation output. this is also not to mention how these tools always allow features like builtins.fetchGit that shouldn’t be used here

i really think most could go without either of these third party build tools assuming they’re aware of the features already available to them with rustPlatform and are willing to add a cargoLock.lockFile = ./Cargo.lock line. there isn’t much for aux to improve here besides docs

this won’t be possible without IFD in a vast majority of cases, as we won’t have access to the cargo manifest ahead of time

cargoLock.lockFile doesn’t require any updates when the lock file is changed. the aforementioned allowBuiltinFetchGit also could helps those outside of our repo

for packages inside our repo, i don’t think there’s much we can fix. cargoHash will continue to need to be updated, along with the hashes for git dependencies. i don’t think this is a big deal though, especially with tools like nix-update making this process extremely trivial

i don’t have an idea for a solution here, but i’d love to see at this annoys me so much lol

5 Likes
6

Yeah, that’s definitely a good point that needs to be worked out! My approach was to first see how the discussion in SIG Repos: How should they work? - #32 by liketechnik really works out in the end. Letting at least all the build tooling stay in core would be fine by and, and some critical core packages (thinking of sudo-rs as an example here), to get started.

My idea here was to provide e.g. stable, beta and nightly at first, and as a further step a number N of latest major version, e.g. 1.79.x, 1.78.x and so forth. My suggestion would be to not provide each minor version separately.
The defined number N of the provided versions of the Rust compiler should not be too high obviously - or even make it time-based, for example the major versions released in the last year.
Something along these lines.
And our packages should always be build against one fixed version, definitely. Didn’t mean to imply building against all versions, if it came across like this.

Sure, rustPlatform is definitely solid, never questioned that.
Documentation is a thing for sure that would benefit very much from some improvements, as you mentioned too.
My “goal” with this is more about providing user with a solution that doesn’t require them to re-engineer e.g. to run various cargo subcommands & what not in their flake checks output, much like crane/fenix do the heavy lifting. Internal packaging would not benefit that much from such improvements, but

Maybe not directly this way, but e.g. autogenerating the in-tree overrides file instead of having to manually care for that would be already be a good improvement. But maintaining that file by hand is not a sustainable option in any case.

I think there’s definitely a better solution achievable than the current state, with some good engineering.

Well, I meant when updating a package to a newer version and upstream does not commit a Cargo.lock to their repo, for whatever reason. Currently you’re forced to regenerate the Cargo.lock manually and commit it to the nixpkgs tree, and that is cumbersome with git-dependencies and such.

Yeah, cargoHash was never really a concern of mine, since that works pretty well and is done the same way for other languages’ tooling. But for the (few percent of) cases where you can’t simply use cargoHash, it’s really not much fun IMO.

Thanks for the all feedback so far! I’ll will later update the first post a bit, incorporating some of the thoughts and clarifications. :crab:

4 Likes
7

(We might want to split this topic into two topics: How should Rust packaging support inside Auxolotl look like and how can we support developers using Auxolotl('s Rust support) in their projects.
This post fits into the latter category, while the two post before rather belong in the first, I think. Sorry for interrupting your discussion…)

So, the flake template I talked about is finally published (that was on my Todo list far too long anyway). I think the more helpful part about it is: How can developers using nixpkgs Aux Packages in their projects look like, and what could be better? Therefore let me quickly list what/how the template works and what could be improved.

What / How:

  • builtins.fromTOML to automatically retrieve package name, version etc from the Cargo.toml file
  • fenix to ‘nixify’ a specific version of the rust toolchain (saved in rust-toolchain.toml, so that non nix aux users also use the same version)
  • building a binary with aux packages’ makeRustPlatform
  • providing a dev environment/shell using aux packages’ mkShell

What could be improved? Tbh, this probably has some sharp edges and usability issues - but since it’s a template and I just copy it in each project, I can’t really think of any right now.

1 Like
8

some suggestions i have are

  • drop nix-filter as a dependency
  • use nixpkgs’ rust toolchain where possible and reasonable
    • for a standard build, there isn’t much reason to use fenix in the first place
    • this would make fenix another optional dependency, allowing for only depending on nixpkgs
    • the rust mirrors fenix and similar use are also very slow for users in countries such as china. hearing from people i know, being able to take advantage of local cache mirrors is much faster and more convenient
  • better compose cross compilation toolchains
    • the current use of a function to both instantiate a toolchain and build a package is a bit messy
    • clearly separating supported cross targets, the instantiation of toolchains, and the overriding of a package using the standard toolchain from nixpkgs would allow for easier iteration
    • a good example of what i mean can be found in one of my repos here. it’s mainly used to create multi arch docker images from any arm or x64 machine on macos or linux
    • more callPackage in general would be good here
  • RUST_SRC_PATH = "${pkgs.rustPlatform.rustLibSrc}" should be set in devShells with rust-analyer per Rust - NixOS Wiki
  • examples of checks would be nice
    • from the same project i linked before, i have a rustfmt check here for example
  • the structure needs some reworking
    • this is mainly a nitpick about the requirement of some paths to use string concatenation. this is ugly and does have a (very minimal) performance impact in comparison to atomic expressions (i.e., standard paths)

and as for the aux-specific stuff…i don’t think there’s really anything. frankly at this stage i don’t see any advantage aux’s fork of nixpkgs brings. i do enjoy this template though and would love to help upstream it into the org’s (or rust SIG’s) templates eventually

2 Likes
9

and would love to help upstream it into the org’s (or rust SIG’s) templates eventually

The current templates repo description sounds very system configuration specific, so I would place the template in the Rust SIG (templates?) repository. Not sure if we really want to split templates across all the SIG repos though.

When we have decided on a place where the templates should go, I propose to open a PR in the relevant repo with the current state of the template, and implement the improvements suggested as part of the PR adding the rust template. (Which is not to dismiss your suggestions, they seem pretty on point! On the contrary, but using the current state of the template as the basis for a PR and implementing your suggestions in that PR seems the best course of action to me).

(And if the agreement is, they should go in the Rust SIG (templates) repo, then I wouldn’t mind setting up the barebones Rust repo too (akin to the other SIGs repos).)

P.S.: Thanks for all the suggestions and the kind words! I especially like the examples you linked, they look like much more sensible approaches.

10

First of, thanks for the great work!

I’d also go with the generic templates, as splitting templates up over multiple repos hurts discoverability, to start with.
Also, SIG repos are still under heavy discussion, so having it somewhere centralized makes sense also for the current state.

Maybe we can introduce a e.g. lang-tooling/ (naming is hard!) subfolder in that repo, placing such flake templates inside there?
In any case, if you create a PR, feel free to tag me / request a review from me!

11

I don’t see a lot Aux can do right now, I find fenix pretty solid as it is, and if improvements need to be made, they probably should go in it.

I think I’d still put the template in the generic repo, I agree that splitting up template repos may confuse people and hurt discoverability.

I personally wouldn’t mind placing it directly alongside other templates, and would feel weird about some templates being in the root and others not. That means either “all templates goes in the root” (which might end up being just a big heap of folders ?) or put current templates in a folder under root (and other SIGs would need to agree).

For starters though, I think it’s more important to put the template in an aux repo, and things can be moved around later ?

1 Like
12

I think one part that is missing in this discussion right now is us maintaining the packages in nixpkgs. I feel like theres a ton of them which are in a rather stale state. Not mentioning the countless packages still waiting to get packaged.

1 Like
13

One thing I hope aux embraces is more automation to help with this, eg mergebots

3 Likes
14

Based or your pr, I made a small template for C

1 Like