cmd/go: provide a convenient way to iterate all packages in go.work workspace

[hyangah]

go.work makes it easier to work with multiple modules.

Often users want to run tests or analysis for all packages in their workspace. For a single module case, this can be easily achieved by using ./... pattern. Currently, however, there is no equivalent of ./... in workspace mode. So users need to run go commands from inside of each individual module, or supply the paths for each module directories listed in go.work. (e.g. go test -v $(go list -f '{{.Dir}}/...' -m | xargs)).

We need to figure out how to offer ./...-equivalent behavior when working with `go.work.

My personal preference is to reuse./... (VSCode Go uses ./... pattern to implement workspace-wide test, lint, coverage computation, etc).

p.s. I noticed the issue about clarifying the meaning of ... in module mode is still open #37227

[katexochen]

go test -v $(go list -f '{{.Dir}}/...' -m | xargs)

This is especially not possible in CI, as the go.work file shouldn't be checked in (according to the proposal). To have a consistent solution, it would be great to solve this problem for modules in general, not only for repositories with a go.work file.

[seankhliao]

how would you decide what to include if you don't have a go.work file?

[katexochen]

how would you decide what to include if you don't have a go.work file?

I just would like to iterate over all sub modules, e.g., run all unit test in the active module and all sub modules.

[eighty4]

Having a go.work file for local workflow setup is a use case, but I am setting up a project that I definitely plan on adding my go.work file to my committed sources. There's not much value add, other than enabling a fork to engage in development without having to recreate an identical go.work file, but that's only because the workspace feature doesn't have workflows baked into the tooling. It seems like a huge miss to add workspaces and not have a way to run go test across all the workspace modules or to be able to run go get for a module of a workspace without using cd.

I'm not familiar with the internals of go's community. Are they accepting of proposals and PRs?

[katexochen]

@eighty4 If you have go.work checked in, you can easily list all modules with

go list -f '{{.Dir}}' -m | xargs

You could then do something like

mods=$(go list -f '{{.Dir}}' -m | xargs)
for mod in $mods; do
            (cd "$mod"; go test ./...)
done

Not perfect, but works. It's much more pain if you don't check in go.work (as recommended by the documentation) and want to do such thing...

[eighty4]

Sorry you had to repeat the answer for me. Works pretty well:

go list -f '{{.Dir}}' -m | xargs go test

[eighty4]

I don't see how I could achieve the same one liner for go mod tidy. Is it possible to achieve this with an xargs/pipe:

# workspace dir
go mod tidy
# cd for each module
cd composable
go mod tidy
cd ../git
go mod tidy
cd ../testutil
go mod tidy
cd ../util
go mod tidy

[seankhliao]

Reusing ./... for workspaces would clash if you had a module in the workspace root (use .).

Would ./.... (4 dots) be too subtle?

[katexochen]

Would ./.... (4 dots) be too subtle?

Yes, I think this would be too subtle. :( But I also lack a better idea. Maybe a flag, to make commands workspace aware, in the same way -m makes them module aware?

[eighty4]

I don't know what the ./... and go test [packages] do or are for. I could only get pattern matching to work with go test such as go test my_test.go. Do you have to specify go test github.com/fullyqualified/modulepath to test a package? ./... seemed to have no effect.

[eighty4]

What's an example of a module aware command with -m?

[cyrusv]

4 dots is interesting idea--maybe a little more obvious would be ./.../...?

An issue with these shell scripts is that you don't get an aggregated coverage report.

I am curious about the framing of the issue here: Is this a gap in workspace, or a gap in go test? My understanding is that workspace is an optional, local convenience. The ability to run all tests incl submodules, I think, should work whether or not an individual chooses to use workspaces feature. Is that a fair understanding?

[katexochen]

The ability to run all tests incl submodules, I think, should work whether or not an individual chooses to use workspaces feature

I agree with that. Notice that this is a general issue with go tooling, not only for running tests. I'd also like to go tidy all submodules etc.

[katexochen]

mods=$(go list -f '{{.Dir}}' -m | xargs)
for mod in $mods; do
(cd "$mod"; go test ./...)
done

With Go 1.20, the new -C flag can be used to simplify this to

mods=$(go list -f '{{.Dir}}' -m)
for mod in $mods; do
            go test -C "$mod" ./...
done

This also works for go tidy @eighty4 :

mods=$(go list -f '{{.Dir}}' -m)
for mod in $mods; do
            go mod tidy -C "$mod"
done

[cyrusv]

@seankhliao, is there agreement from the maintainers that

• There is a gap in go tooling any time "./..." is allowed where a user may want to specify additionally to apply the tool to submodules
• The fix should work whether or not someone uses workspaces
  ?

If there is agreement, then what are next steps?
Do we need a formal proposal for what the syntax should be for the fix, then a PR to implement?
I'm happy to take action if the maintainers give some guidance -- I have not yet contributed to golang before!

[cyrusv]

bump @bcmills, any advice on how to advance a solution?

[eighty4]

I think it works great this way.

Workspaces should evolve to simplify the go mod tidy upkeep across all modules of a workspace with a command that performs mod tidy in each workspace module, with each tidy being its own independent operation. The statement proposal's wording feels like a hang up.

[samherrmann]

@katexochen, the following one-liners appear to work:

• test:

  go list -f '{{.Dir}}/...' -m | xargs go test

• tidy:

  go list -f '{{.Dir}}' -m | xargs -L1 go mod tidy -C

[eighty4]

Should it be

go work sync
go list -f '{{.Dir}}' -m | xargs -L1 go mod tidy -C
go work sync
go list -f '{{.Dir}}' -m | xargs -L1 go mod tidy -C

or

go list -f '{{.Dir}}' -m | xargs -L1 go mod tidy -C
go work sync
go list -f '{{.Dir}}' -m | xargs -L1 go mod tidy -C

?

[yuriy-yarosh]

@eighty4 go mod tidy then go work sync so go work could pick up the respective changes from workspace module dependencies.

Here's my do.sh helper script.

[matloob]

I think what we'd want here is not an exactt equivalent to ./... but instead the same as what was suggested in #37227 (comment): a package pattern that matches all the packages in the work (formerly called main) modules.

We use ./... in many of these cases because that pattern does not exist.

It's not clear what to call such a pattern though.

[matloob]

I think we should call the pattern work given that it matches the pattern in the work modules. I'm going to file a separate proposal issue for that.

[matloob]

Filed #71294

[dmitshur]

Given the change in #71294 is now implemented, we can't release Go 1.25 without having it also be documented and mentioned in release notes. Since that issue is already closed but this one's still open, marking it as a release blocker.

[gopherbot]

Change https://go.dev/cl/658495 mentions this issue: doc/next: add release note for work package pattern

[ccoVeille]

Hi @matloob

Thanks for working on this.

I found this issues a few week ago,because I had a need for something like this.

I'm happy this issue is now addressed with the PR you made.

I tried to follow the implementation, and also looked at the changes that was made to the documentation.

The Go 1.25 is not yet release, but I'm curious.

I might have missed something, so please be lenient if I misunderstood something.

So let me sum up:

• the starting was point was "we have no easy way to launch test/build on all modules"
• right now solution is to use things like go list -f '{{.Dir}}/...' -m | xargs go test and other variation

So the idea was to bring a way to test (build, mod clean... whatsoever...) all modules from the go.work file

I hope it's pretty close to the idea that was debated here

What is unclear to me is about the new way to work with the changes that were introduced

By reading some messages, we may understand that launching "go test ./.." will magically launch the tests for all submodule listed in the go.work file.

I think it might be great, but then it would comes with changes that could have consequences. Someone trying to test a root modules might test all submodule, but some submodule may require different environment like a docker instance. So it would be a possible breaking changes.

That's why I looked at the code. From what I saw (especially in the unit tests), I would say that what was added is about having a new argument value for go list

So now, unless I'm wrong go list work was added to help solving the issues people had.

But the whole point was not to list them, so here I assume the code changes are about adding something equivalent to go list -f '{{.Dir}}/...' -m | xargs go test, but now it would be easier and maybe something like this go list work | xargs go test

Could you confirm ?

TL;DR; how to use the changes that were added to test all submodules ?

Thanks

[matloob]

The work package pattern won't test all submodules, only the packages in the modules in the go.work workspace. So go test work will test the packages that belong to any of the modules in the workspace.

There is no one single command to test all submodules of a workspace.

[ccoVeille]

Thank for your reply.

go test work command will test submodules present in go.work file, and it's what I'm looking for.

Now, it's clearer. Thanks