build: could not import runtime (open : no such file or directory)

[xhd2015]

Go version

go version go1.23.6 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/xhd2015/Library/Caches/go-build'
GOENV='/Users/xhd2015/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/xhd2015/Projects/gopath/pkg/mod'
GONOPROXY='git.garena.com'
GONOSUMDB='git.garena.com'
GOOS='darwin'
GOPATH='/Users/xhd2015/Projects/gopath'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/xhd2015/installed/go1.23.6'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/xhd2015/installed/go1.23.6/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.23.6'
GODEBUG=''
GOTELEMETRY='local'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/s_/nd3t_zbx61747w0qdryxh4wm0000gp/T/go-build2905057735=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Hi go masters, I was developing an instrumentation framework(namely xgo) on top of the -overlay flag. I happened to encounter a build failure with the following specific building sequence:

• go test succeeds
• go test -overlay some_overlay.json fails
  • NOTE: the overlay contains a modified the file to include a new import: import "runtime", which causes the build failure

After investigation, I found this bug:

• happens only when
  • two builds use the same GOCACHE
  • an overlay introduces new imports to the original file
  • the replaced file belongs to a third party module
• was introduced in go1.19, last to latest go1.24 and possibly later

Below are the steps to reproduce this bug:

git clone --branch fix-xgo-v1.1.0 https://github.com/xhd2015/xgo
cd xgo/runtime/test/build/overlay_build_cache_error_with_go
go test -run TestGoModNonOverlayFirstShouldError -v ./

The test code that verifies the output:

func TestGoModNonOverlayFirstShouldError(t *testing.T) {
	overlayFile, err := setupReverse("gomod_first")
	if err != nil {
		t.Fatalf("failed to setup reverse: %v", err)
	}

        // ensure we have a clean GOCACHE
	gocache, err := getTmpGocache()
	if err != nil {
		t.Fatalf("failed to get tmp gocache: %v", err)
	}
	defer os.RemoveAll(gocache)

	var normOut bytes.Buffer
	err = cmd.Debug().Stdout(&normOut).Stderr(&normOut).Env([]string{"GOCACHE=" + gocache, "GO_BYPASS_XGO=true"}).Run("go", "test", "-v", "./overlay_test_with_gomod")
	if err != nil {
		t.Log(normOut.String())
		t.Fatalf("failed to run test: %v", err)
	}

	// this should fail
	var errOut bytes.Buffer
	afterErr := cmd.Debug().Stdout(&errOut).Stderr(&errOut).Env([]string{"GOCACHE=" + gocache, "GO_BYPASS_XGO=true"}).Run("go", "test", "-v", "-overlay", overlayFile, "./overlay_test_with_gomod")

	goMinor, err := getGo1MinorVersion()
	if err != nil {
		t.Fatalf("failed to get go minor version: %v", err)
	}
        // this issue was introduced in go1.19
	if goMinor >= 19 {
		if afterErr == nil {
			t.Errorf("expect cache+overlay combination error, actual nil")
		}
		expectContains := "could not import runtime (open : no such file or directory)"
		if runtime.GOOS == "windows" {
			expectContains = "could not import runtime (open : The system cannot find the file specified.)"
		}
		errOutStr := errOut.String()
		if !strings.Contains(errOutStr, expectContains) {
			t.Errorf("expect containing %q, actual none", expectContains)
			t.Logf("DEBUG: %s", errOutStr)
			return
		}
	} else {
		if afterErr != nil {
			t.Errorf("expect cache+overlay success, actual error: %v", afterErr)
		}
	}
}

My workaround(xhd2015/xgo#311 (comment)) is to modify the pkg loading under src/cmd/go/internal/load/pkg.go, to always import runtime for all packages except unsafe, internal/... and runtime/internal/....

But practically if the package is a third party package, this workaround won't work.

The underlying error actually comes from go tool compile, which sees an empty # import config file that resolves to empty packagefile of runtime, thus the error message. It's probably there is some bug when generating the import config file when invoking go tool compile.

More details can refer to the original issue link: xhd2015/xgo#311

What did you see happen?

The build fails with the message:

runtime_link.go:8: could not import runtime (open : no such file or directory)

What did you expect to see?

The build should compile with the same GOCACHE, regardless whether -overlay flag is present or not, and new imports introduced or not.

[gabyhelp]

Related Issues

• cmd/go: test with `-overlay` failed with a go-vet error #50044 (closed)
• cmd/go: Setting up a go workspace breaks building #57738 (closed)
• cmd/go: go test does not parse -overlay files for test functions #44946 (closed)
• x/tools/go/packages: TestNewPackagesInOverlay/Modules failing on Go 1.11 #33491 (closed)
• build: run.bash: make test configuration independent of the user's GOENV file #54084 (closed)
• cmd/go: TestScript/build_overlay failing when gccgo is present #42688 (closed)
• cmd/go: go vet does not support -overlay #44957 (closed)
• cmd/go: invoking go run from go test can corrupt build cache #69566
• x/tools/go/gcexportdata: can't import package from pkg/mod #52269
• cmd/go: build failed #59694 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[xhd2015]

The root reason is that go's module cache(GOMODCACHE) does treat modules readonly and immutable. So it won't be aware of that when you import new packages through -overlay.

That said, to bypass this issue, just don't import any extra packages, i.e. don't change the package's import sets.

If the instrumentation relies on external ability, do that via IR-link which can be done by instrumenting the compiler.

See xhd2015/xgo#323 xhd2015/xgo@5e9b1fb