redesign stage 0 std

[onur-ozkan]

Summary

This PR changes how bootstrap builds the stage 1 compiler by switching to precompiled stage 0 standard library instead of building the in-tree one. The goal was to update bootstrap to use the beta standard library at stage 0 rather than compiling it from source (see the motivation at rust-lang/compiler-team#619).

Previously, to build a stage 1 compiler bootstrap followed this path:

download stage0 compiler -> build in-tree std -> compile stage1 compiler with in-tree std

With this PR, the new path is:

download stage0 compiler -> compile stage1 compiler with precompiled stage0 std

This also means that cfg(bootstrap)/cfg(not(bootstrap)) is no longer needed for library development.

Building "library"

Since stage0 std is no longer in-tree x build/test/check library --stage 0 is now no-op. The minimum supported stage to build std is now 1. For the same reason, default stage values in the library profile is no longer 0.

Because building the in-tree library now requires a stage1 compiler, I highly recommend library developers to enable download-rustc to speed up compilation time.

---

If you encounter a bug or unexpected results please open a topic in the #t-infra/bootstrap Zulip channel or create a bootstrap issue.

(Review thread: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Review.20thread.3A.20stage.200.20redesign.20PR/with/508271433)

Blocked on #122709

try-job: dist-x86_64-linux

[rustbot]

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

[onur-ozkan]

@rustbot ready

r? bootstrap
cc @rust-lang/libs

[onur-ozkan]

@rustbot author (currently stage 2 std is copied from stage 1 and this behaviour should change with the beta std change)

[bors]

☀️ Try build successful - checks-actions
Build commit: 424ee44 (424ee4431faa6f45caca8312cb7e92b94a294898)

[onur-ozkan]

It worked, nice :)

@rustbot ready

[RalfJung]

I just learned about compiletest-use-stage0-libtest. Does this PR make true the default value for that? It seems like there's no good reason to even still support false for that option, or is there?

[jieyouxu]

It seems like there's no good reason to even still support false for that option, or is there?

true might make sense as a default (since libtest programmatic API changes relatively unfrequently), but if someone is working on a libtest (programmatic API) change that can require changing compiletest, in which case the contributor would probably want to use false so that they can test their changes for compiletest too.

[onur-ozkan]

I just learned about compiletest-use-stage0-libtest. Does this PR make true the default value for that? It seems like there's no good reason to even still support false for that option, or is there?

The default value for that option is already true and false is useful to detect breaking changes from libtest to compiletest. Currently we run x check compiletest with compiletest-use-stage0-libtest=false on couple of CI runners to prevent that.

[RalfJung]

The default value for that option is already true

The reason I thought it was "false" is that the change notification says
"Added a new option build.compiletest-use-stage0-libtest to force compiletest to use the stage 0 libtest."
That sure sounds like if I want compiletest to use stage 0 libtest I need to "force" that to happen by manually changing the configuration.

[RalfJung]

The default value for that option is already true

Ah, that part was not clear to me, thanks. :) (Sorry for not replying in-thread, github reply-by-mail does not support that.)

[onur-ozkan]

Any idea what we are waiting for?

[jieyouxu]

A few small things:

• compiletest: Remove the libtest-based executor and its dependency #140392 is in the queue, after which compiletest will no longer depend on in-tree libtest (or libtest at all), after which we won't need the compiletest stage 0 hack (unless that somehow has major issues that requires a revert). It will still depend on an unstable library feature, but that's less likely to change (at worst, #[cfg(bootstrap)]).
• I'll finish the internal blog post update tonight ([WIP] Add an Inside Rust blog post for stage 0 std redesign blog.rust-lang.org#1582), where if we remove the compiletest stage0 hack, I'll drop that mention which will simplify it.
  • I'll get a compiler and library team member respectively who's not super familiar w/ bootstrap to read over the internal blog post.

FYI @rust-lang/bootstrap since this is a major bootstrap change, in case you have any last minute review feedback.

[onur-ozkan]

A few small things:

• compiletest: Remove the libtest-based executor and its dependency #140392 is in the queue, after which compiletest will no longer depend on in-tree libtest (or libtest at all), after which we won't need the compiletest stage 0 hack (unless that somehow has major issues that requires a revert). It will still depend on an unstable library feature, but that's less likely to change (at worst, #[cfg(bootstrap)]).

• I'll finish the internal blog post update tonight ([WIP] Add an Inside Rust blog post for stage 0 std redesign blog.rust-lang.org#1582), where if we remove the compiletest stage0 hack, I'll drop that mention which will simplify it.

  • I'll get a compiler and library team member respectively who's not super familiar w/ bootstrap to read over the internal blog post.

FYI @rust-lang/bootstrap since this is a major bootstrap change, in case you have any last minute review feedback.

Okay.

@rustbot blocked

[krasimirgg]

onur-ozkan, first, I think this work is really cool!

We've got a custom rust bootstrapping setup that doesn't invoke x directly. I've applied the complier and library changes from this PR and adapted it to use beta-rustc and beta-stdlibs to build the first new rustc. I ran into a curious build failure about an interaction between compiler/rustc_expand/src/proc_macro_server.rs and library/proc_macro/src/bridge/server.rs:

error[E0046]: not all trait items implemented, missing: `SourceFile`
   --> compiler/rustc_expand/src/proc_macro_server.rs:457:1
    |
457 | impl server::Types for Rustc<'_, '_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `SourceFile` in implementation
    |
    = help: implement the missing item: `type SourceFile = /* Type */;`

error[E0046]: not all trait items implemented, missing: `source_file`
   --> compiler/rustc_expand/src/proc_macro_server.rs:665:1
    |
665 | impl server::Span for Rustc<'_, '_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `source_file` in implementation
    |
    = help: implement the missing item: `fn source_file(&mut self, _: <Self as Types>::Span) -> <Self as Types>::SourceFile { todo!() }`

error[E0277]: the trait bound `Rustc<'_, '_>: server::SourceFile` is not satisfied
   --> compiler/rustc_expand/src/proc_macro_server.rs:833:25
    |
833 | impl server::Server for Rustc<'_, '_> {
    |                         ^^^^^^^^^^^^^ the trait `server::SourceFile` is not implemented for `Rustc<'_, '_>`
    |
note: required by a bound in `Server`
   --> /rustc/45165c82a4c5315ff52c391ad138f41ff40b52d8/library/proc_macro/src/bridge/server.rs:121:1
    = note: `Server` is a "sealed trait", because to implement it you also need to implement `pm::bridge::server::SourceFile`, which is not accessible; this is usually done to force you to use one of the provided types that already implement it
    = help: the following type implements the trait:
              pm::bridge::server::MarkedTypes<S>
    = note: this error originates in the macro `declare_server_traits` which comes from the expansion of the macro `with_api` (in Nightly builds, run with -Z macro-backtrace for more info)

Naively, this looks related to this commit 3962069#diff-002b2429c67f4267af1485b8b9e2f95cbfe586971ed97f832da4d4d45d0e1e41, where the source file was removed from both the library and compiler sides in lockstep. How does the redesigned bootstrap workflow handle cases like this?

[Kobzol]

Do we still need stage0-sysroot after this PR? (

rust/src/bootstrap/src/core/build_steps/compile.rs

Line 1773 in d6f178a

host_dir.join("stage0-sysroot")

). I guess not, but also it can probably stay and we can remove it later once this is merged?

[bjorn3]

I think rustc_expand will need to be changed to depend on the in-tree libproc_macro directly rather than fetching it from the sysroot. It needs to link against a libproc_macro built from the same sources as the one that will end up in it's own sysroot. It doesn't care about the libproc_macro copy that is in the sysroot with which rustc itself was built. It just happens to be the case that before this PR, both copies are built from the same source and thus are ABI compatible (libproc_macro uses the C ABI internally, so only the source needs to match for ABI compat, not the rustc it was built with).