sum.golang.org: should not respond with "410 Gone"

[mpalmer]

What version of Go are you using (go version)?

$ go version
go version go1.15.2 linux/amd64

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/tmp/.cache/go-build"
GOENV="/tmp/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/backend/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build244541875=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Added an initial release (v0.1.0) to the module github.com/sol1/gormzerolog, ran go run . in a private codebase which uses that module.

What did you expect to see?

Something slightly less final and forever than "410 Gone".

What did you see instead?

An error which indicated that sum.golang.org had responded to a request for the gormzerolog module with "410 Gone".

Commentary

410 is a HTTP response code that means "access to the target resource is no longer available at the origin server and that this condition is likely to be permanent." (RFC7231, s6.5.9). In the case of sum.golang.org, it seems unlikely that the origin server knows, or has reason to believe, that the lack of a third-party module version is likely to be permanent. Indeed, only a few minutes after serving that 410 response, the sum database was able to serve checksums for that module version, indicating that the lack of access to the target resource was, indeed, not permanent.

Rather than a 410, requests for module checksums which do not refer to a known module version should provide a response of 404 Not Found. Again, per RFC7231, s6.5.9:

If the origin server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) ought to be used instead.

[hyangah]

@mpalmer Thanks for reaching out. As briefly hinted in #31995 (comment), this is an implementation detail of our caching. We want to utilize HTTP caching and, unfortunately, some CDNs do not support 404 response caching yet. We are keeping an eye on this.

The primary consumer of sum.golang.org is the go command and that does not care whether it's 404 or 410.
Other than this unfortunate deviation from the HTTP spec, is there any problem caused by the use of 410?

cc @katiehockman @heschik

[mpalmer]

Well, it confused the living heck out of me for a while, because the go command reported the "unexpected" error (despite it apparently being an expected and known error), so it sent me on quite a goose chase.

[bcmills]

the go command reported the "unexpected" error (despite it apparently being an expected and known error)

What was the error it gave you? Perhaps we can improve the error message somehow.

[mpalmer]

Whilst I'm not in a position to reproduce the problem just at the moment, and the original error disappeared off the backscroll buffer due to an unfortunately large find output, to the best of my recollection it matched that given in #31995; that is: "unexpected status (<url>): 410 Gone". If 410 has a particular (non-standard) meaning to the go command, then it can't really be "unexpected" when a 410 arrives. At the very least, saying "such and such was not found", rather than "unexpected error" would be a good start. Something like "Could not find checksums for (description); this could be caused by delayed caching, or an incorrect module name or version." would be even more usefully descriptive.

[weberc2]

What was the error it gave you? Perhaps we can improve the error message somehow.

@bcmills I just struck upon this. The error I encountered was:

go: downloading github.com/weberc2/auth v0.0.9
go get: github.com/weberc2/auth@v0.0.9: verifying module: github.com/weberc2/auth@v0.0.9: reading https://sum.golang.org/lookup/github.com/weberc2/auth@v0.0.9: 410 Gone
        server response: not found: github.com/weberc2/auth@v0.0.9: invalid version: unknown revision v0.0.9

In my case, I tagged my library but forgot to git push --tags before running go get github.com/weberc2/auth@v0.0.9 from my downstream module.

[hyangah]

@suzmue @findleyr @heschi According to the latest google cdn doc, 404 is classified as a cacheable status code. Should we flip to use 404?
https://cloud.google.com/cdn/docs/caching

[hyangah]

This is done.

$ curl -I -L https://sum.golang.org/lookup/example.com/mod@v1.0.0
HTTP/2 404 
...