Run doctests across the workspace on CI

[EliahKagan]

The attempt to run doctests, added to the CI test-fast jobs in 89a0567 (#1556), actually runs zero doctests, because there are none in the top-level project and neither --workspace nor its deprecated --all alias is passed.

   Finished `test` profile [unoptimized + debuginfo] target(s) in 28.55s
   Doc-tests gitoxide

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

This was the case both before and after #1559 (which revised #1556 in other ways). I am actually not sure doctests across all creates had ever been run automatically. The doctest command in the justfile, which can be run manually but is also in the CI test job, is the same as the the command added to test-fast in 89a0567. Furthermore, there are two situations with failing doctests, one of which is long-standing:

• Recently, in #1585, some code fences were added that were not intended as doctests, holding text with conflict markers (present intentionally to document gix-merge), or in one case a suggested shell command. Since the opening ``` was not followed by text to indicate the language, however, they were taken to be Rust doctests.
• Years ago, zip! was vendored from itertools, including the original zip! macro's doctest. This fails because it tries to use it from itertools, and the failure is not--as far as I can tell--straightforwardly repairable by making the doctest use the vendored macro, due to the rules about macro visibility, the way scoping interacts with doctests, and the status of the vendored zip! as something that should not be part of the interface of gix-pack. That doctest could of course be converted into a regular unit test with #[test]. But the goal doesn't seem to be to actually test that, so much as to keep its origin in itertools clear.

This PR adds --workspace to the command to run doctests in all projects/crates, as well as --no-fail-fast so the status of all doctests will be seen in any fail. After observing those failures, I fixed them, though for zip! the "fix" was to ignore the doctest. It also replaces --all with --workspace for cargo subcommands where it is a deprecated alias of --workspace, which is not specifically related to doctests. There is a bit more detailed information in the commit messages.

[Byron]

That's a great catch, thanks a lot!

For a moment I was wondering if it makes sense to spend more than a minute on compiling the entire workspace again just to run the very few doctests that there are, instead of selectively running only the known ones.

But then again, I thought it's probably not worth it to change that at the cost of having to remember to add a new crate to CI if doc-tests would be added to a crate that wasn't previously included.

Also I hope that one day cargo nextest will support doctests out of the box, maybe it could run faster then ever then.

[EliahKagan]

For a moment I was wondering if it makes sense to spend more than a minute on compiling the entire workspace again just to run the very few doctests that there are, instead of selectively running only the known ones.

This is a good point. Although the build times are not new, their impact was likely underestimated since it was not combined with the time to actually run the doctests.

But then again, I thought it's probably not worth it to change that at the cost of having to remember to add a new crate to CI if doc-tests would be added to a crate that wasn't previously included.

I think we already have something like that in the unit-tests rule in the justfile. The overall situation would probably be made no worse if we also listed crates with doctests. I wonder, though, if there is a way to quickly parse which crates have doctests (not just documentation comments with code fences that might be marked in such a way as not to be taken as doctests, but doctests and not ignored).

[Byron]

Thinking again, I am now even more in fear that one day such a selection could backfire for the sake of saving CPU that won't be noticeable (except for cents on the bill of GitHub).

The unit-tests are explicitly run with different features because running all permutations of all features isn't always feasible or possible, so I think it's a different intent.

I certainly wouldn't object to such a change, but I'd be much more interested in a way to either solve the 15 failing tests, or have a safeguard. And even that, the safeguard, is a nice to have and only if it's simple enough, all in all.

[EliahKagan]

Thinking again, I am now even more in fear that one day such a selection could backfire for the sake of saving CPU that won't be noticeable (except for cents on the bill of GitHub).

Another option, if it's not important for the doctests to run on all platforms, could be to change

gitoxide/justfile

Line 155 in c5955fc

cargo test --doc

to pass --workspace, so that the full test job does this on Ubuntu, and to remove it from the test-fast runs. The primary benefit of #1556 would still be intact, because everything that uses cargo nextest since then still would, and because, right now, the test-fast job for windows-latest tends to take longer than the full test job, so removing this from the test-fast jobs and putting it in test would probably not make the workflow as a whole take longer.

The unit-tests are explicitly run with different features because running all permutations of all features isn't always feasible or possible, so I think it's a different intent.

That makes sense, and yes, that does not apply to the doctest runs.

I certainly wouldn't object to such a change, but I'd be much more interested in a way to either solve the 15 failing tests, or have a safeguard. And even that, the safeguard, is a nice to have and only if it's simple enough, all in all.

For now, I've opened #1657 for that.