Why I got error 102 in makefile when running nextest.

[BraveY]

This action works successfully on Wed, 12 Mar 2025 03:14:50 GMT.
But it failed on 13 Mar, in this aciton workflow, I got following error:

TEST_WORKDIR_PREFIX="/tmp" RUST_BACKTRACE=1 /home/runner/.cargo/bin/cargo nextest run --no-fail-fast --filter-expr 'test(test) - test(integration)' --workspace  --features=virtiofs --release --test-threads 8
make: *** [Makefile:[11](https://github.com/dragonflyoss/nydus/actions/runs/13826281977/job/38681749367#step:6:12)1: ut-nextest] Error 102
Error: Process completed with exit code 2.

I am wondering why I got Error 102? Is there any updates on 13 Mar caused the error? Since there is no update in our program between 12 Mar and 13 Mar.

This docs say Error 102 is Running cargo metadata produced an error. But it is ok, when i add command cargo metadata in smoke test.

And when i run the test command TEST_WORKDIR_PREFIX="/tmp" RUST_BACKTRACE=1 /home/runner/.cargo/bin/cargo nextest run --no-fail-fast --filter-expr 'test(test) - test(integration)' --workspace --features=virtiofs --release --test-threads 8 in smoke directly, i got the output as expected.

The makefile is here. And the related part is fllowing:

# you need install cargo nextest first from: https://nexte.st/book/pre-built-binaries.html
ut-nextest: .release_version
	TEST_WORKDIR_PREFIX=$(TEST_WORKDIR_PREFIX) RUST_BACKTRACE=1 ${CARGO} nextest run --no-fail-fast --filter-expr 'test(test) - test(integration)' --workspace $(EXCLUDE_PACKAGES) $(CARGO_COMMON) $(CARGO_BUILD_FLAGS) --test-threads 8

Originally posted by @BraveY in #2245

[sunshowers]

Thanks. That's strange. Do you have any other output (stdout/stderr) available? Generally nextest will produce output to stderr if something goes wrong -- are you squelching it?

Looking at

nextest/cargo-nextest/src/errors.rs

Lines 389 to 392 in d4d3b40

	Self::CargoMetadataExecFailed { .. }
	| Self::CargoMetadataFailed { .. }
	| Self::CargoLocateProjectExecFailed { .. }
	| Self::CargoLocateProjectFailed { .. } => NextestExitCode::CARGO_METADATA_FAILED,

, it looks like we also produce 102 if cargo locate-project failed. But we should print something to stderr no matter what. An example with a nextest patched so cargo metadata fails:

% cargo nextest r --workspace
error: no such command: `metadata2`

	Did you mean `metadata`?

	View all installed commands with `cargo --list`
	Find a package to install `metadata2` with `cargo search cargo-metadata2`

Also -- can you reproduce this locally? If not, can you ssh into the runner and try it out? You can use something like https://github.com/marketplace/actions/debugging-with-ssh.

[sunshowers]

Ahhh so it turns out nextest doesn't produce any output of its own, instead delegating to Cargo. If your Cargo is broken, nextest won't produce anything. That's misleading (nextest should print something). But I think the issue is with your installation of cargo.

[sunshowers]

I've just released cargo-nextest 0.9.93 which improves reporting here, and will tell you which command failed to run. This won't fix your issue but it will hopefully help you diagnose it.

Thanks again for reporting the issue!

[BraveY]

I've just released cargo-nextest 0.9.93 which improves reporting here, and will tell you which command failed to run. This won't fix your issue but it will hopefully help you diagnose it.

Thanks again for reporting the issue!

Thanks a lot !!! Now i got the failed command after runing /home/runner/.cargo/bin/cargo nextest list -v --cargo-verbose

error: command `/home/runner/.cargo/bin/rustup metadata '--format-version=1' --all-features --filter-platform x86_64-unknown-linux-gnu` failed with exit status: 1

I'm still working on it to figure out why this command failed run after running make.

It should be cargo metadata instead of rustup metadata.

[BraveY]

I think the new release of rustup v1.28's PR rust-lang/rustup#4022 where cargo-home/bin/cargo is symlinked to cargo-home/bin/rustup caused this problem. Please see the detail in rust-lang/rustwide#94 and rust-lang/rustup#4224.

[BraveY]

BTW, I fixed the broken CI in our acitons by running ${RUSTUP} run stable cargo nextest run instead of ${CARGO} nextest run.

[ChrisDenton]

Sorry, I've only briefly investigated this. However I don't believe nextest is at fault.

I think the issue is that if you set the CARGO environment variable then cargo may replace it with the canonicalized version in some circumstances (i.e. with the symlinks resolved).

BTW, I fixed the broken CI in our acitons by running ${RUSTUP} run stable cargo nextest run instead of ${CARGO} nextest run.

Another possible workaround is to try setting CARGO to the output of rustup which cargo. This will point to the real cargo's location instead of the rustup shim.

[ChrisDenton]

Ultimately the fix will need to come from cargo and rustup. I don't think there's much nextest can do. Well, I guess one hack would be to check if the CARGO environment variable ends with rustup[.exe] and replacing it with cargo[.exe] if so. But that doesn't exactly fill me with joy.

[BraveY]

Sorry, I've only briefly investigated this. However I don't believe nextest is at fault.

I think the issue is that if you set the CARGO environment variable then cargo may replace it with the canonicalized version in some circumstances (i.e. with the symlinks resolved).

BTW, I fixed the broken CI in our acitons by running ${RUSTUP} run stable cargo nextest run instead of ${CARGO} nextest run.

Another possible workaround is to try setting CARGO to the output of rustup which cargo. This will point to the real cargo's location instead of the rustup shim.

I have tried rustup which cargo before, but i got another error about rustc.

[BraveY]

If there is nothing nextest can do, feel free to close this issue. And let's wait rustup to fix it.

[ChrisDenton]

I've done a bit more investigating today and I'm pretty confident the reason for this I gave above is accurate. If the CARGO environment is set (and only if) then cargo will it with the canonicalized path when calling sub-processes (this is a bug). nextest very reasonably assumes the CARGO environment variable contains a path to cargo. However, if the path CARGO was initially set to is a symlink to rustup then it will be canonicalised to rustup instead.

Other than the workaround you implemented, some possible workarounds I tried:

• Don't use the CARGO variable in your Makefile. Call it CARGO_BIN, CARGO_EXE or anything other than CARGO. Example: https://github.com/ChrisDenton/nydus/commit/1ba09c997aa89c56e4c51b7aca21d17faaaa3539.
• Alternatively unset CARGO when calling cargo.

nextest could treat the CARGO variable as being "cargo or rustup". But that's trickier to handle. And the environment variable is used is a fair few places: https://github.com/search?q=repo%3Anextest-rs%2Fnextest%20%2F(%3F-i)%22CARGO%22%2F&type=code

[sunshowers]

Thanks for the investigation! Is the issue that nextest is canonicalizing the Cargo path and dereferencing symlinks (which wouldn't quite work with a multicall binary like rustup, I think)? I don't think it did but maybe it does.

[ChrisDenton]

No, nextest isn't doing anything wrong here. It's a Cargo bug that happens in specific circumstances.

[sunshowers]

Thanks. Closing this out then.