Skip to content

Geth just keeps rewinding after crash in hash mode #31586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
cosmicnet opened this issue Apr 7, 2025 · 15 comments
Open

Geth just keeps rewinding after crash in hash mode #31586

cosmicnet opened this issue Apr 7, 2025 · 15 comments

Comments

@cosmicnet
Copy link

Hi,

Geth crashed. I don't understand why, I have a lot of output.

When I started it back up it just keeps saying Block state missing, rewinding further and at this rate is heading back to 0 :(

I've stopped geth so it doesn't go any further.

Idk what to do, I don't want to start from scratch. I'm running a full node.

For me Geth had been acting oddly since the last update, Ctrl-C would start a shutdown but after giving it some time it wouldn't actually shut down.

Below is some output, but I've got more from the crash if needed.

INFO [04-07|20:07:04.123] Loaded last snap-sync pivot marker       number=15,888,578
WARN [04-07|20:07:04.124] Head state missing, repairing            number=22,210,289 hash=1771e5..d0abf0 snaproot=16cbc8..55979a
INFO [04-07|20:07:12.125] Block state missing, rewinding further   number=22,198,123 hash=e51222..7a2a5a elapsed=8.000s
INFO [04-07|20:07:20.126] Block state missing, rewinding further   number=22,182,681 hash=599d9a..17880e elapsed=16.001sINFO [04-07|20:07:28.126] Block state missing, rewinding further   number=22,160,535 hash=c29f2b..10809a elapsed=24.001sINFO [04-07|20:07:36.127] Block state missing, rewinding further   number=22,139,428 hash=6093ea..d81bc9 elapsed=32.002sINFO [04-07|20:07:44.128] Block state missing, rewinding further   number=22,117,312 hash=1742ee..23ad9a elapsed=40.003sINFO [04-07|20:07:52.129] Block state missing, rewinding further   number=22,097,476 hash=d23ed4..139f90 elapsed=48.004sINFO [04-07|20:08:00.129] Block state missing, rewinding further   number=22,075,353 hash=ae945e..6c0742 elapsed=56.004sERROR[04-07|20:08:01.546] Number of finalized block is missing
INFO [04-07|20:08:08.130] Block state missing, rewinding further   number=22,052,585 hash=00462a..742eae elapsed=1m4.005s
INFO [04-07|20:08:16.131] Block state missing, rewinding further   number=22,029,830 hash=ccf1e0..12d4fa elapsed=1m12.006s
INFO [04-07|20:08:24.131] Block state missing, rewinding further   number=22,006,178 hash=971bfb..fd7cf8 elapsed=1m20.007s
INFO [04-07|20:08:32.132] Block state missing, rewinding further   number=21,981,645 hash=1986a5..ae378d elapsed=1m28.007s
INFO [04-07|20:08:40.133] Block state missing, rewinding further   number=21,957,158 hash=7ffe30..cfdb8d elapsed=1m36.008s
INFO [04-07|20:08:48.133] Block state missing, rewinding further   number=21,933,020 hash=5172e7..e44add elapsed=1m44.008s
INFO [04-07|20:08:56.134] Block state missing, rewinding further   number=21,908,527 hash=2463b4..63d415 elapsed=1m52.009s
ERROR[04-07|20:09:01.546] Number of finalized block is missing
INFO [04-07|20:09:04.134] Block state missing, rewinding further   number=21,886,258 hash=870dee..458b6b elapsed=2m0.009s
INFO [04-07|20:09:12.135] Block state missing, rewinding further   number=21,861,266 hash=0838cf..3de59f elapsed=2m8.010s
INFO [04-07|20:09:20.136] Block state missing, rewinding further   number=21,837,051 hash=5a2a9a..ad882f elapsed=2m16.011s
INFO [04-07|20:09:28.136] Block state missing, rewinding further   number=21,811,525 hash=d62622..e0be86 elapsed=2m24.011s
INFO [04-07|20:09:36.137] Block state missing, rewinding further   number=21,788,391 hash=cdd512..8c4fb3 elapsed=2m32.012s
INFO [04-07|20:09:44.137] Block state missing, rewinding further   number=21,762,937 hash=bc7c02..7c6d53 elapsed=2m40.012s
INFO [04-07|20:09:52.138] Block state missing, rewinding further   number=21,736,789 hash=4ecc11..8d9326 elapsed=2m48.013s
INFO [04-07|20:10:00.138] Block state missing, rewinding further   number=21,710,797 hash=6faa56..be3d55 elapsed=2m56.013s
ERROR[04-07|20:10:01.546] Number of finalized block is missing
...
INFO [04-07|20:24:08.179] Block state missing, rewinding further   number=18,972,241 hash=319ca5..08b05f elapsed=17m4.054s
INFO [04-07|20:24:16.179] Block state missing, rewinding further   number=18,945,668 hash=15b763..549b18 elapsed=17m12.054s
INFO [04-07|20:24:24.179] Block state missing, rewinding further   number=18,920,880 hash=419f82..f88b1f elapsed=17m20.054s
INFO [04-07|20:24:32.180] Block state missing, rewinding further   number=18,910,070 hash=54a70b..0a5534 elapsed=17m28.055s
INFO [04-07|20:24:40.180] Block state missing, rewinding further   number=18,883,575 hash=a392e9..8ac31a elapsed=17m36.055s
INFO [04-07|20:24:48.180] Block state missing, rewinding further   number=18,856,647 hash=778f94..a94e3e elapsed=17m44.055s

fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x7ff7059ed669]

goroutine 23929 gp=0xc000586fc0 m=17 mp=0xc000501008 [running]:
runtime.throw({0x7ff7072843be?, 0x9fdf83975bcea605?})
        runtime/panic.go:1101 +0x4d fp=0xc0005d3af0 sp=0xc0005d3ac0 pc=0x7ff70591b3ad
runtime.sigpanic()
        runtime/signal_windows.go:414 +0xd0 fp=0xc0005d3b38 sp=0xc0005d3af0 pc=0x7ff7058fccf0
math/big.nat.cmp({0xc0b16c6700?, 0xc067ba1950?, 0xc06965be20?}, {0xc0001b97c0?, 0x20?, 0xc0005d3ba8?})
        math/big/nat.go:164 +0x49 fp=0xc0005d3b58 sp=0xc0005d3b38 pc=0x7ff7059ed669
math/big.(*Int).Cmp(0xc0b16bfc20, 0x1?)
        math/big/int.go:380 +0x5d fp=0xc0005d3b98 sp=0xc0005d3b58 pc=0x7ff7059e601d
github.com/ethereum/go-ethereum/crypto.ValidateSignatureValues(0x1, 0xc0b16bfc00, 0xc0b16bfc20, 0x1)
        github.com/ethereum/go-ethereum/crypto/crypto.go:283 +0x75 fp=0xc0005d3bb8 sp=0xc0005d3b98 pc=0x7ff705b4b735
github.com/ethereum/go-ethereum/core/types.recoverPlain({0xf0, 0x95, 0x18, 0x51, 0x27, 0x67, 0x12, 0xb1, 0x3e, 0xa6, ...}, ...)
        github.com/ethereum/go-ethereum/core/types/transaction_signing.go:466 +0x12b fp=0xc0005d3c60 sp=0xc0005d3bb8 pc=0x7ff705bc6b6b
github.com/ethereum/go-ethereum/core/types.EIP155Signer.Sender({0xc000110aa0?, 0xc030affcc0?}, 0xc0b16c0740)
        github.com/ethereum/go-ethereum/core/types/transaction_signing.go:356 +0x2f4 fp=0xc0005d3d78 sp=0xc0005d3c60 pc=0x7ff705bc5d74
github.com/ethereum/go-ethereum/core/types.(*EIP155Signer).Sender(0x7226dc3ebc1d17aa?, 0x4430497c27264bba?)
        <autogenerated>:1 +0x45 fp=0xc0005d3dd0 sp=0xc0005d3d78 pc=0x7ff705bcfd05
github.com/ethereum/go-ethereum/core/types.(*modernSigner).Sender(0xc030affca0, 0xc0b16c0740)
        github.com/ethereum/go-ethereum/core/types/transaction_signing.go:249 +0x31d fp=0xc0005d3f00 sp=0xc0005d3dd0 pc=0x7ff705bc555d
github.com/ethereum/go-ethereum/core/types.Sender({0x7ff7078100a0, 0xc030affca0}, 0xc0b16c0740)
        github.com/ethereum/go-ethereum/core/types/transaction_signing.go:152 +0x8c fp=0xc0005d3f68 sp=0xc0005d3f00 pc=0x7ff705bc4c2c
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x8f2?)
        github.com/ethereum/go-ethereum/core/sender_cacher.go:74 +0x72 fp=0xc0005d3fc8 sp=0xc0005d3f68 pc=0x7ff705f54a72gh.loli.garden/ethereum/go-ethereum/core.newTxSenderCacher.gowrap1()
        github.com/ethereum/go-ethereum/core/sender_cacher.go:64 +0x25 fp=0xc0005d3fe0 sp=0xc0005d3fc8 pc=0x7ff705f549c5runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0005d3fe8 sp=0xc0005d3fe0 pc=0x7ff705923a01
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 24155
        github.com/ethereum/go-ethereum/core/sender_cacher.go:64 +0x72

...

goroutine 3118090 gp=0xc1113ed6c0 m=nil [runnable]:
github.com/ethereum/go-ethereum/core/state.newSubfetcher.gowrap1()
        github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go:278 fp=0xc031fc7fe0 sp=0xc031fc7fd8 pc=0x7ff705d75b60
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc031fc7fe8 sp=0xc031fc7fe0 pc=0x7ff705923a01
created by github.com/ethereum/go-ethereum/core/state.newSubfetcher in goroutine 24155
        github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go:278 +0x259

goroutine 3118045 gp=0xc1113edc00 m=nil [runnable]:
runtime.gopark(0xc000a6bf78?, 0x2?, 0x67?, 0x0?, 0xc000a6bf0c?)
        runtime/proc.go:435 +0xce fp=0xc000a6bd68 sp=0xc000a6bd48 pc=0x7ff70591b4ce
runtime.selectgo(0xc000a6bf78, 0xc000a6bf08, 0xb01701fbfacbc27?, 0x1, 0xc082bb8f00?, 0x1)
        runtime/select.go:351 +0x837 fp=0xc000a6bea0 sp=0xc000a6bd68 pc=0x7ff7058fa357
github.com/ethereum/go-ethereum/core/rawdb.iterateTransactions.func2()
        github.com/ethereum/go-ethereum/core/rawdb/chain_iterator.go:160 +0x1fd fp=0xc000a6bfe0 sp=0xc000a6bea0 pc=0x7ff705cc917d
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000a6bfe8 sp=0xc000a6bfe0 pc=0x7ff705923a01
created by github.com/ethereum/go-ethereum/core/rawdb.iterateTransactions in goroutine 3115152
        github.com/ethereum/go-ethereum/core/rawdb/chain_iterator.go:169 +0x1e8
@jwasinger
Copy link
Contributor

The trace is very strange because the crash math/big/nat.go:164 corresponds (in Go 1.23.0) to this line, indicating an out-of-bounds array access on the internal limbs of a big.Int.

but clearly, the case of a nil input should be caught above in that library function before it can panic. seems like memory corruption could manifest as a mismatch between the stored vs actual size of a slice?

@cosmicnet
Copy link
Author

I've no idea. Machine details:

OS Name Microsoft Windows 10 Pro
Processor AMD Ryzen Threadripper PRO 3945WX 12-Cores, 4001 Mhz, 12 Core(s), 24 Logical Processor(s)
Installed Physical Memory (RAM) 256 GB

Previous memory tests have all come back clear. I could run some more tonight and see.

Is there anything I can do to recover from here?

@rjl493456442
Copy link
Member

Whats the command line flags you used for Geth, and what's the Geth version?

@cosmicnet
Copy link
Author

cosmicnet commented Apr 8, 2025 •
edited
Loading

Geth 1.15.7
geth --config config.toml
cat config.toml
[Eth]
NetworkId = 1
SyncMode = "full"
EthDiscoveryURLs = ["enrtree://AKA3AM6LPBYEUDMVNU3BSVQJ5AD45Y7YPOHJLEF6W26QOE4VTUDPE@all.mainnet.ethdisco.net"]
SnapDiscoveryURLs = []
NoPruning = false
NoPrefetch = false
LightPeers = 100
#UltraLightFraction = 75
DatabaseCache = 2048
DatabaseFreezer = ""
TrieCleanCache = 614
TrieDirtyCache = 1024
TrieTimeout = 3600000000000
SnapshotCache = 409
Preimages = false
EnablePreimageRecording = false
RPCGasCap = 25000000
RPCTxFeeCap = 1e+00

[Eth.Miner]
GasCeil = 8000000
GasPrice = 1000000000
Recommit = 3000000000

[Eth.TxPool]
Locals = []
NoLocals = false
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 1
PriceBump = 10
AccountSlots = 16
GlobalSlots = 4096
AccountQueue = 64
GlobalQueue = 1024
Lifetime = 10800000000000

[Eth.GPO]
Blocks = 20
Percentile = 60
MaxPrice = 500000000000

[Node]
DataDir = REDACTED
IPCPath = "geth.ipc"
HTTPHost = "127.0.0.1"
HTTPPort = 8547
HTTPVirtualHosts = ["localhost"]
HTTPModules = ["eth", "web3", "debug", "txpool", "net", "shh", "db", "admin", "debug"]
WSHost = "127.0.0.1"
WSPort = 8546
WSOrigins = ["localhost"]
WSModules = ["eth", "web3", "debug", "txpool", "net", "shh", "db", "admin", "debug"]
GraphQLVirtualHosts = ["localhost"]

[Node.P2P]
MaxPeers = 50
NoDiscovery = false
BootstrapNodes = ["enode://d860a01f9722d78051619d1e2351aba3f43f943f6f00718d1b9baa4101932a1f5011f16bb2b1bb35db20d6fe28fa0bf09636d26a87d31de9ec6203eeedb1f666@18.138.108.67:30303", "enode://22a8232c3abc76a16ae9d6c3b164f98775fe226f0917b0ca871128a74a8e9630b458460865bab457221f1d448dd9791d24c4e5d88786180ac185df813a68d4de@3.209.45.79:30303", "enode://ca6de62fce278f96aea6ec5a2daadb877e51651247cb96ee310a318def462913b653963c155a0ef6c7d50048bba6e6cea881130857413d9f50a621546b590758@34.255.23.113:30303", "enode://279944d8dcd428dffaa7436f25ca0ca43ae19e7bcf94a8fb7d1641651f92d121e972ac2e8f381414b80cc8e5555811c2ec6e1a99bb009b3f53c4c69923e11bd8@35.158.244.151:30303", "enode://8499da03c47d637b20eee24eec3c356c9a2e6148d6fe25ca195c7949ab8ec2c03e3556126b0d7ed644675e78c4318b08691b7b57de10e5f0d40d05b09238fa0a@52.187.207.27:30303", "enode://103858bdb88756c71f15e9b5e09b56dc1be52f0a5021d46301dbbfb7e130029cc9d0d6f73f693bc29b665770fff7da4d34f3c6379fe12721b5d7a0bcb5ca1fc1@191.234.162.198:30303", "enode://715171f50508aba88aecd1250af392a45a330af91d7b90701c436b618c86aaa1589c9184561907bebbb56439b8f8787bc01f49a7c77276c58c1b09822d75e8e8@52.231.165.108:30303", "enode://5d6d7cd20d6da4bb83a1d28cadb5d409b64edf314c0335df658c1a54e32c7c4a7ab7823d57c39b6a757556e68ff1df17c748b698544a55cb488b52479a92b60f@104.42.217.25:30303"]
BootstrapNodesV5 = ["enr:-KG4QOtcP9X1FbIMOe17QNMKqDxCpm14jcX5tiOE4_TyMrFqbmhPZHK_ZPG2Gxb1GE2xdtodOfx9-cgvNtxnRyHEmC0ghGV0aDKQ9aX9QgAAAAD__________4JpZIJ2NIJpcIQDE8KdiXNlY3AyNTZrMaEDhpehBDbZjM_L9ek699Y7vhUJ-eAdMyQW_Fil522Y0fODdGNwgiMog3VkcIIjKA", "enr:-KG4QDyytgmE4f7AnvW-ZaUOIi9i79qX4JwjRAiXBZCU65wOfBu-3Nb5I7b_Rmg3KCOcZM_C3y5pg7EBU5XGrcLTduQEhGV0aDKQ9aX9QgAAAAD__________4JpZIJ2NIJpcIQ2_DUbiXNlY3AyNTZrMaEDKnz_-ps3UUOfHWVYaskI5kWYO_vtYMGYCQRAR3gHDouDdGNwgiMog3VkcIIjKA", "enr:-Ku4QImhMc1z8yCiNJ1TyUxdcfNucje3BGwEHzodEZUan8PherEo4sF7pPHPSIB1NNuSg5fZy7qFsjmUKs2ea1Whi0EBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpD1pf1CAAAAAP__________gmlkgnY0gmlwhBLf22SJc2VjcDI1NmsxoQOVphkDqal4QzPMksc5wnpuC3gvSC8AfbFOnZY_On34wIN1ZHCCIyg", "enr:-Ku4QP2xDnEtUXIjzJ_DhlCRN9SN99RYQPJL92TMlSv7U5C1YnYLjwOQHgZIUXw6c-BvRg2Yc2QsZxxoS_pPRVe0yK8Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpD1pf1CAAAAAP__________gmlkgnY0gmlwhBLf22SJc2VjcDI1NmsxoQMeFF5GrS7UZpAH2Ly84aLK-TyvH-dRo0JM1i8yygH50YN1ZHCCJxA", "enr:-Ku4QPp9z1W4tAO8Ber_NQierYaOStqhDqQdOPY3bB3jDgkjcbk6YrEnVYIiCBbTxuar3CzS528d2iE7TdJsrL-dEKoBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpD1pf1CAAAAAP__________gmlkgnY0gmlwhBLf22SJc2VjcDI1NmsxoQMw5fqqkw2hHC4F5HZZDPsNmPdB1Gi8JPQK7pRc9XHh-oN1ZHCCKvg", "enr:-IS4QLkKqDMy_ExrpOEWa59NiClemOnor-krjp4qoeZwIw2QduPC-q7Kz4u1IOWf3DDbdxqQIgC4fejavBOuUPy-HE4BgmlkgnY0gmlwhCLzAHqJc2VjcDI1NmsxoQLQSJfEAHZApkm5edTCZ_4qps_1k_ub2CxHFxi-gr2JMIN1ZHCCIyg", "enr:-IS4QDAyibHCzYZmIYZCjXwU9BqpotWmv2BsFlIq1V31BwDDMJPFEbox1ijT5c2Ou3kvieOKejxuaCqIcjxBjJ_3j_cBgmlkgnY0gmlwhAMaHiCJc2VjcDI1NmsxoQJIdpj_foZ02MXz4It8xKD7yUHTBx7lVFn3oeRP21KRV4N1ZHCCIyg", "enr:-Ku4QHqVeJ8PPICcWk1vSn_XcSkjOkNiTg6Fmii5j6vUQgvzMc9L1goFnLKgXqBJspJjIsB91LTOleFmyWWrFVATGngBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC1MD8qAAAAAP__________gmlkgnY0gmlwhAMRHkWJc2VjcDI1NmsxoQKLVXFOhp2uX6jeT0DvvDpPcU8FWMjQdR4wMuORMhpX24N1ZHCCIyg", "enr:-Ku4QG-2_Md3sZIAUebGYT6g0SMskIml77l6yR-M_JXc-UdNHCmHQeOiMLbylPejyJsdAPsTHJyjJB2sYGDLe0dn8uYBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC1MD8qAAAAAP__________gmlkgnY0gmlwhBLY-NyJc2VjcDI1NmsxoQORcM6e19T1T9gi7jxEZjk_sjVLGFscUNqAY9obgZaxbIN1ZHCCIyg", "enr:-Ku4QPn5eVhcoF1opaFEvg1b6JNFD2rqVkHQ8HApOKK61OIcIXD127bKWgAtbwI7pnxx6cDyk_nI88TrZKQaGMZj0q0Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC1MD8qAAAAAP__________gmlkgnY0gmlwhDayLMaJc2VjcDI1NmsxoQK2sBOLGcUb4AwuYzFuAVCaNHA-dy24UuEKkeFNgCVCsIN1ZHCCIyg", "enr:-Ku4QEWzdnVtXc2Q0ZVigfCGggOVB2Vc1ZCPEc6j21NIFLODSJbvNaef1g4PxhPwl_3kax86YPheFUSLXPRs98vvYsoBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC1MD8qAAAAAP__________gmlkgnY0gmlwhDZBrP2Jc2VjcDI1NmsxoQM6jr8Rb1ktLEsVcKAPa08wCsKUmvoQ8khiOl_SLozf9IN1ZHCCIyg"]
StaticNodes = []
TrustedNodes = []
ListenAddr = ":30303"
EnableMsgEvents = false

[Node.HTTPTimeouts]
ReadTimeout = 30000000000
WriteTimeout = 30000000000
IdleTimeout = 120000000000

[Metrics]
HTTP = "127.0.0.1"
Port = 6060
InfluxDBEndpoint = "http://localhost:8086"
InfluxDBDatabase = "geth"
InfluxDBUsername = "test"
InfluxDBPassword = "test"
InfluxDBTags = "host=localhost"

@cosmicnet
Copy link
Author

I restarted and it seems to be downloading from scratch:

INFO [04-08|09:15:13.593] Block state missing, rewinding further   number=15,964,184 hash=043b95..d533bf elapsed=41m44.230s
ERROR[04-08|09:15:17.918] Number of finalized block is missing
INFO [04-08|09:15:21.593] Block state missing, rewinding further   number=15,937,897 hash=446453..380639 elapsed=41m52.231s
INFO [04-08|09:15:29.594] Block state missing, rewinding further   number=15,911,147 hash=fa597e..7182da elapsed=42m0.231s
INFO [04-08|09:15:36.344] Rewinding limit reached, resetting to genesis number=15,888,577 hash=8f73aa..1c82df limit=15,888,578
INFO [04-08|09:15:36.347] Loaded most recent local header          number=22,210,289 hash=1771e5..d0abf0 age=1d18h23m
INFO [04-08|09:15:36.347] Loaded most recent local block           number=0          hash=d4e567..cb8fa3 age=56y3w5d
INFO [04-08|09:15:36.348] Loaded most recent local snap block      number=22,210,289 hash=1771e5..d0abf0 age=1d18h23m
INFO [04-08|09:15:36.348] Loaded last snap-sync pivot marker       number=15,888,578
WARN [04-08|09:15:36.348] Enabling snapshot recovery               chainhead=0 diskbase=22,212,794
WARN [04-08|09:15:36.364] Loaded snapshot journal                  diskroot=16cbc8..55979a diffs=unmatched
WARN [04-08|09:15:36.364] Snapshot is not continuous with chain    snaproot=16cbc8..55979a chainroot=d7f897..0f0544
INFO [04-08|09:15:36.365] Initialized transaction indexer          range="last 2350000 blocks"
INFO [04-08|09:15:36.838] Initialized log indexer                  "first block"=19,847,731 "last block"=22,206,187 "first map"=215,040 "last map"=273,740 "head indexed"=false
WARN [04-08|09:15:36.974] Switch sync mode from full sync to snap sync reason="snap sync incomplete"
WARN [04-08|09:15:36.976] Sanitizing invalid gasprice oracle ignore price provided=<nil> updated=2
WARN [04-08|09:15:36.976] Sanitizing invalid gasprice oracle max header history provided=0     updated=1
WARN [04-08|09:15:36.976] Sanitizing invalid gasprice oracle max block history provided=0     updated=1
WARN [04-08|09:15:36.980] Old unclean shutdowns found              count=25
WARN [04-08|09:15:36.980] Unclean shutdown detected                booted=2025-03-17T23:19:39+0000 age=3w8h55m
WARN [04-08|09:15:36.981] Unclean shutdown detected                booted=2025-03-26T16:36:16+0000 age=1w5d15h
WARN [04-08|09:15:36.981] Unclean shutdown detected                booted=2025-03-26T19:47:56+0000 age=1w5d12h
WARN [04-08|09:15:36.981] Unclean shutdown detected                booted=2025-03-30T21:58:59+0100 age=1w1d11h
WARN [04-08|09:15:36.982] Unclean shutdown detected                booted=2025-04-02T01:51:36+0100 age=6d7h24m
WARN [04-08|09:15:36.982] Unclean shutdown detected                booted=2025-04-03T01:27:36+0100 age=5d7h48m
WARN [04-08|09:15:36.982] Unclean shutdown detected                booted=2025-04-04T02:01:57+0100 age=4d7h13m
WARN [04-08|09:15:36.982] Unclean shutdown detected                booted=2025-04-05T02:03:23+0100 age=3d7h12m
WARN [04-08|09:15:36.983] Unclean shutdown detected                booted=2025-04-05T22:49:29+0100 age=2d10h26m
WARN [04-08|09:15:36.983] Unclean shutdown detected                booted=2025-04-07T02:05:15+0100 age=1d7h10m
WARN [04-08|09:15:36.983] Unclean shutdown detected                booted=2025-04-07T17:45:46+0100 age=15h29m50s
WARN [04-08|09:15:36.985] Engine API enabled                       protocol=eth
INFO [04-08|09:15:36.987] Starting peer-to-peer node               instance=Geth/v1.15.7-stable-827d3fcc/windows-amd64/go1.24.1
INFO [04-08|09:15:37.189] New local node record                    seq=855 id=37579a0fe8b6dd10 ip=127.0.0.1 udp=30303 tcp=30303
INFO [04-08|09:15:37.279] Started P2P networking                   self=enode://a46f2fa94389b86335101f01a6c652654398018d3aa2a98dbeee7f01ce93865dc953975c52480cbb4c6c036028b67cec84bd20030a32e664e74e55a0a4b8e6e8@127.0.0.1:30303
INFO [04-08|09:15:37.283] IPC endpoint opened                      url=\\.\pipe\geth.ipc
ERROR[04-08|09:15:37.283] Unavailable modules in HTTP API list     unavailable="[shh db]" available="[admin debug web3 eth txpool miner net]"
ERROR[04-08|09:15:37.284] Unavailable modules in HTTP API list     unavailable="[shh db]" available="[admin debug web3 eth txpool miner net]"
INFO [04-08|09:15:37.291] HTTP server started                      endpoint=127.0.0.1:8547 auth=false prefix= cors= vhosts=localhost
INFO [04-08|09:15:37.291] WebSocket enabled                        url=ws://127.0.0.1:8546
INFO [04-08|09:15:37.297] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [04-08|09:15:37.297] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=localhost
INFO [04-08|09:15:37.297] Loaded local transaction journal         transactions=0 dropped=0
INFO [04-08|09:15:37.299] Started log indexer
INFO [04-08|09:15:39.255] New local node record                    seq=856 id=37579a0fe8b6dd10 ip=86.8.142.109 udp=30303 tcp=30303
INFO [04-08|09:15:39.673] NAT mapped port                          proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [04-08|09:15:40.091] NAT mapped port                          proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [04-08|09:15:47.405] Looking for peers                        peercount=3 tried=41 static=0
INFO [04-08|09:15:47.677] Removing old bloom bits database in progress... elapsed=10.378s
INFO [04-08|09:15:57.432] Looking for peers                        peercount=8 tried=79 static=0
INFO [04-08|09:15:58.207] Removing old bloom bits database in progress... elapsed=20.908s
...
INFO [04-08|11:44:39.924] Syncing: state healing in progress       accounts=920,649@46.55MiB slots=831,529@62.66MiB codes=1625@8.03MiB nodes=9,994,285@2.55GiB    pending=135,769
INFO [04-08|11:44:45.835] Syncing: chain download in progress      synced=100.00% chain=885.93GiB headers=22,223,665@10.61GiB bodies=22,223,571@634.23GiB receipts=22,223,571@241.08GiB eta=1m2.213s
WARN [04-08|11:44:46.285] Unexpected trienode heal packet          peer=ea8c9811 reqid=555,172,951,694,506,294
WARN [04-08|11:44:47.434] Unexpected bytecode heal packet          peer=ea8c9811 reqid=7,922,220,155,405,186,855
INFO [04-08|11:44:47.938] Syncing: state healing in progress       accounts=921,973@46.61MiB slots=831,529@62.66MiB codes=1629@8.06MiB nodes=9,999,664@2.55GiB    pending=136,617
INFO [04-08|11:44:50.339] Forkchoice requested sync to new head    number=22,223,666 hash=0cc60d..a6dea2 finalized=22,223,596
INFO [04-08|11:44:53.839] Syncing: chain download in progress      synced=100.00% chain=885.93GiB headers=22,223,666@10.61GiB bodies=22,223,571@634.23GiB receipts=22,223,571@241.08GiB eta=1m2.932s
INFO [04-08|11:44:56.312] Syncing: state healing in progress       accounts=923,750@46.69MiB slots=831,529@62.66MiB codes=1631@8.07MiB nodes=10,006,278@2.55GiB   pending=137,191

At least it hasn't had any errors for a while, but it does have these Unexpected bytecode heal packet warnings.

The pending number goes up and down as does the eta. Synced has said 100% for 3hrs now.

@cosmicnet
Copy link
Author

cosmicnet commented Apr 8, 2025 •
edited
Loading

It ran for a few hours, then started erroring and eventually crashed. Odd to see out of memory when the system has over 200Gb ram free.

ERROR[04-08|13:58:49.922] State snapshotter failed to iterate trie err="missing trie node 23f9ec24c54406b44290a773d05de1cf1b00f7db8e1f99066113e70014190142 (owner 0a54b6838b5747a5f77b009914aeef855b2bf6d38bb65531aee20b0b40db8130) (path 020509030e) <nil>"
INFO [04-08|13:59:04.347] Resuming state snapshot generation       root=f560d5..0b1810 in=0a54b6..db8130 at=0a3bcf..44b874 accounts=11,592,699         slots=51,146,332          storage=4.35GiB    dangling=0 elapsed=1h4m18.424s eta=25h29m13.365s
INFO [04-08|13:59:04.445] Imported new potential chain segment     number=22,224,337 hash=f19855..c25835 blocks=1         txs=261         mgas=25.585  elapsed=2.179s      mgasps=11.739 snapdiffs=369.70KiB triedirty=447.98MiB
INFO [04-08|13:59:04.497] Chain head was updated                   number=22,224,337 hash=f19855..c25835 root=1849bb..476804 elapsed=3.9955ms
ERROR[04-08|13:59:04.527] State snapshotter failed to iterate trie err="missing trie node 23f9ec24c54406b44290a773d05de1cf1b00f7db8e1f99066113e70014190142 (owner 0a54b6838b5747a5f77b009914aeef855b2bf6d38bb65531aee20b0b40db8130) (path 020509030e) <nil>"
runtime: VirtualAlloc of 11491688448 bytes failed with errno=1455
fatal error: out of memory

runtime stack:
runtime.throw({0x7ff7a72efac1?, 0xc71f559000?})
        runtime/panic.go:1101 +0x4d fp=0xc8755ffcb8 sp=0xc8755ffc88 pc=0x7ff7a597b3ad
runtime.sysUsedOS(0xc530410000, 0x2acf54000)
        runtime/mem_windows.go:83 +0x1bb fp=0xc8755ffd18 sp=0xc8755ffcb8 pc=0x7ff7a591ec3b
runtime.sysUsed(...)
        runtime/mem.go:77
runtime.(*mheap).allocSpan(0x7ff7a86f77e0, 0x1567aa, 0x0, 0x1)
        runtime/mheap.go:1353 +0x487 fp=0xc8755ffdc0 sp=0xc8755ffd18 pc=0x7ff7a5931c67
runtime.(*mheap).alloc.func1()
        runtime/mheap.go:970 +0x5c fp=0xc8755ffe08 sp=0xc8755ffdc0 pc=0x7ff7a593143c
runtime.systemstack(0xc001b00700)
        runtime/asm_amd64.s:514 +0x49 fp=0xc8755ffe18 sp=0xc8755ffe08 pc=0x7ff7a5981a29

goroutine 4367 gp=0xc000552a80 m=31 mp=0xc007709808 [running]:
runtime.systemstack_switch()
        runtime/asm_amd64.s:479 +0x8 fp=0xc020999640 sp=0xc020999630 pc=0x7ff7a59819c8
runtime.(*mheap).alloc(0x2acf54000?, 0x1567aa?, 0x1?)
        runtime/mheap.go:964 +0x5b fp=0xc020999688 sp=0xc020999640 pc=0x7ff7a593139b
runtime.(*mcache).allocLarge(0x1eb74754260?, 0x2acf53ff8, 0x1)
        runtime/mcache.go:235 +0x7d fp=0xc0209996d8 sp=0xc020999688 pc=0x7ff7a591d9dd
runtime.mallocgcLarge(0x0?, 0x0, 0x0)
        runtime/malloc.go:1540 +0x79 fp=0xc020999738 sp=0xc0209996d8 pc=0x7ff7a591ab79
runtime.mallocgc(0x2acf53ff8, 0x0, 0x0)
        runtime/malloc.go:1063 +0xc5 fp=0xc020999768 sp=0xc020999738 pc=0x7ff7a5978aa5
runtime.growslice(0x1c000000000, 0x7ff7a8738a20?, 0x58c90acb8?, 0xc020999810?, 0x0?)
        runtime/slice.go:264 +0x5c9 fp=0xc0209997d8 sp=0xc020999768 pc=0x7ff7a597db69
github.com/syndtr/goleveldb/leveldb.(*Batch).appendRec(0xc089fb92c0, 0x0, {0xc065be2b70?, 0xc, 0x7ff7a597d509?}, {0x0, 0x0, 0xc095050e01?})
        github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/batch.go:110 +0x3c8 fp=0xc0209998b0 sp=0xc0209997d8 pc=0x7ff7a60d84a8
github.com/syndtr/goleveldb/leveldb.(*Batch).Delete(...)
        github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/batch.go:125
github.com/ethereum/go-ethereum/ethdb/leveldb.(*batch).Delete(0xc07df931d0, {0xc065be2b70?, 0xc, 0x380?})
        github.com/ethereum/go-ethereum/ethdb/leveldb/leveldb.go:443 +0x3a fp=0xc020999900 sp=0xc0209998b0 pc=0x7ff7a61a373a
github.com/ethereum/go-ethereum/core/rawdb.WriteFilterMapExtRow({0x1eb6e11a2c0, 0xc07df931d0}, 0xa80cb8dc, {0x0?, 0x0?, 0x0?}, 0x18?)
        github.com/ethereum/go-ethereum/core/rawdb/accessors_indexes.go:298 +0x16d fp=0xc020999998 sp=0xc020999900 pc=0x7ff7a5d1bf2d
github.com/ethereum/go-ethereum/core/filtermaps.(*FilterMaps).storeFilterMapRowsOfGroup(0xc09506bb88, {0x7ff7a7873e38, 0xc07df931d0}, {0xc0a3f24300, 0x20, 0x43580?}, 0x32e, {0xc102b01200, 0x20, 0x39f40})
        github.com/ethereum/go-ethereum/core/filtermaps/filtermaps.go:598 +0x1cc fp=0xc020999a98 sp=0xc020999998 pc=0x7ff7a653264c
github.com/ethereum/go-ethereum/core/filtermaps.(*FilterMaps).storeFilterMapRows(0xc09506bb88, {0x7ff7a7873e38, 0xc07df931d0}, {0xc0a3efa000?, 0x42d4d?, 0x44000?}, 0x32e, {0xc102a04000, 0x42d4d, 0x44800})
        github.com/ethereum/go-ethereum/core/filtermaps/filtermaps.go:560 +0x136 fp=0xc020999b30 sp=0xc020999a98 pc=0x7ff7a6532376
github.com/ethereum/go-ethereum/core/filtermaps.(*mapRenderer).writeFinishedMaps(0xc07b92fa10, 0xc00e8ffef8)
        github.com/ethereum/go-ethereum/core/filtermaps/map_renderer.go:428 +0x4e5 fp=0xc020999e48 sp=0xc020999b30 pc=0x7ff7a6539c45
github.com/ethereum/go-ethereum/core/filtermaps.(*mapRenderer).run(0xc07b92fa10, 0xc00e8ffef8, 0xc020999ee8)
        github.com/ethereum/go-ethereum/core/filtermaps/map_renderer.go:289 +0x1b2 fp=0xc020999e98 sp=0xc020999e48 pc=0x7ff7a6538c72
github.com/ethereum/go-ethereum/core/filtermaps.(*FilterMaps).tryIndexHead(0xc09506bb88)
        github.com/ethereum/go-ethereum/core/filtermaps/indexer.go:241 +0xf6 fp=0xc020999f18 sp=0xc020999e98 pc=0x7ff7a6535a76
github.com/ethereum/go-ethereum/core/filtermaps.(*FilterMaps).indexerLoop(0xc09506bb88)
        github.com/ethereum/go-ethereum/core/filtermaps/indexer.go:59 +0x1c7 fp=0xc020999fc8 sp=0xc020999f18 pc=0x7ff7a6534b07
github.com/ethereum/go-ethereum/core/filtermaps.(*FilterMaps).Start.gowrap2()
        github.com/ethereum/go-ethereum/core/filtermaps/filtermaps.go:258 +0x25 fp=0xc020999fe0 sp=0xc020999fc8 pc=0x7ff7a652ff05
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc020999fe8 sp=0xc020999fe0 pc=0x7ff7a5983a01
created by github.com/ethereum/go-ethereum/core/filtermaps.(*FilterMaps).Start in goroutine 1
        github.com/ethereum/go-ethereum/core/filtermaps/filtermaps.go:258 +0x185
...
goroutine 77651339 gp=0xc004204fc0 m=nil [select]:
runtime.gopark(0xc01be53f58?, 0x4?, 0xb8?, 0xac?, 0xc01be53ed8?)
        runtime/proc.go:435 +0xce fp=0xc01be53d30 sp=0xc01be53d10 pc=0x7ff7a597b4ce
runtime.selectgo(0xc01be53f58, 0xc01be53ed0, 0x1e1d5e9aadd5eb10?, 0x0, 0x7ff7a72a9cc0?, 0x1)
        runtime/select.go:351 +0x837 fp=0xc01be53e68 sp=0xc01be53d30 pc=0x7ff7a595a357
github.com/ethereum/go-ethereum/eth/protocols/eth.(*Peer).broadcastTransactions(0xc0067c8400)
        github.com/ethereum/go-ethereum/eth/protocols/eth/broadcast.go:72 +0x16e fp=0xc01be53fc8 sp=0xc01be53e68 pc=0x7ff7a65b9cae
github.com/ethereum/go-ethereum/eth/protocols/eth.NewPeer.gowrap1()
        github.com/ethereum/go-ethereum/eth/protocols/eth/peer.go:81 +0x25 fp=0xc01be53fe0 sp=0xc01be53fc8 pc=0x7ff7a65c23a5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc01be53fe8 sp=0xc01be53fe0 pc=0x7ff7a5983a01
created by github.com/ethereum/go-ethereum/eth/protocols/eth.NewPeer in goroutine 77651338
        github.com/ethereum/go-ethereum/eth/protocols/eth/peer.go:81 +0x387

@s1na
Copy link
Contributor

s1na commented Apr 9, 2025

For me Geth had been acting oddly since the last update, Ctrl-C would start a shutdown but after giving it some time it wouldn't actually shut down.

This part is related to #31508

@s1na
Copy link
Contributor

s1na commented Apr 9, 2025 •
edited
Loading

@cosmicnet did you manage to run that memory test? So far best guess is the RAM is fried as @jwasinger mentioned

@cosmicnet
Copy link
Author

Memory test came back clear. I'm about to go away for a few days, I could run a longer test when I get back, but I don't think it's the memory. I remember another go based program throwing some memory errors at one point, I ran a lot of memory tests and nothing showed up. Then in their next version the problem was fixed. I used this machine for a few things, and none of the other programs are having issues. When I've had a machine with bad ram in the past all sorts of stuff gets affected.

It's looking like I'll have to start from scratch. I'm seeing odd stuff I've never seen before:

ERROR[04-09|11:39:05.549] State snapshotter failed to iterate trie err="missing trie node 23f9ec24c54406b44290a773d05de1cf1b00f7db8e1f99066113e70014190142 (owner 0a54b6838b5747a5f77b009914aeef855b2bf6d38bb65531aee20b0b40db8130) (path 020509030e) <nil>"
INFO [04-09|11:39:05.858] Resuming state snapshot generation       root=bfb1c9..8bfb47 in=0a54b6..db8130 at=245ea5..70a634 accounts=11,592,699 slots=52,596,407 storage=4.46GiB dangling=0 elapsed=21m1.488s
ERROR[04-09|11:39:05.896] State snapshotter failed to iterate trie err="missing trie node 23f9ec24c54406b44290a773d05de1cf1b00f7db8e1f99066113e70014190142 (owner 0a54b6838b5747a5f77b009914aeef855b2bf6d38bb65531aee20b0b40db8130) (path 020509030e) <nil>"
INFO [04-09|11:39:06.165] Resuming state snapshot generation       root=935fd5..365138 in=0a54b6..db8130 at=245ea5..70a634 accounts=11,592,699 slots=52,597,042 storage=4.46GiB dangling=0 elapsed=21m1.795s
ERROR[04-09|11:39:06.203] State snapshotter failed to iterate trie err="missing trie node 23f9ec24c54406b44290a773d05de1cf1b00f7db8e1f99066113e70014190142 (owner 0a54b6838b5747a5f77b009914aeef855b2bf6d38bb65531aee20b0b40db8130) (path 020509030e) <nil>"
ERROR[04-09|11:39:06.585]
########## BAD BLOCK #########
Block: 22225255 (0x6789cc8a13a9d26703a5ebaad5c10b3feec949abd2a4717dc88325c6a3ffa792)
Error: invalid gas used (remote: 19218740 local: 19235840)
Platform: geth v1.15.7 go1.24.1 amd64 windows
VCS: 827d3fcc-20250331
Chain config: &params.ChainConfig{ChainID:1, HomesteadBlock:1150000, DAOForkBlock:1920000, DAOForkSupport:true, EIP150Block:2463000, EIP155Block:2675000, EIP158Block:2675000, ByzantiumBlock:4370000, ConstantinopleBlock:7280000, PetersburgBlock:7280000, IstanbulBlock:9069000, MuirGlacierBlock:9200000, BerlinBlock:12244000, LondonBlock:12965000, ArrowGlacierBlock:13773000, GrayGlacierBlock:15050000, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(0xc0003820f0), CancunTime:(*uint64)(0xc0003820f8), PragueTime:(*uint64)(nil), OsakaTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:58750000000000000000000, DepositContractAddress:0x00000000219ab540356cBB839Cbe05303d7705Fa, EnableVerkleAtGenesis:false, Ethash:(*params.EthashConfig)(0x7ff74fb05660), Clique:(*params.CliqueConfig)(nil), BlobScheduleConfig:(*params.BlobScheduleConfig)(0x7ff74fab0900)}
Receipts:
  0: cumulative: 113897 gas: 113897 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7d3356b5adc108e2f86c9a5359630383a2b2c836bd9192b015ec9cba6a3c2a76 logs: [0xc0212f04d0 0xc0212f0840 0xc0212f0a50 0xc0212f0b00] bloom: 00220000000002000000000080000000000000000000000000000000000000800000000020000000000000000000000002000008080000000002000000000000000002000200000000000008000000200000008000000000000000000000000000000008000000000000000000000000000000000000000001080010000000000000000000000000000000000000000000000000000000081000004000000000000000000000000400000000000000000000000000000000000000000000000004000002000000000000000000000000000000000000001000000000000000000000200000000000000000000000000000000000000000000000000000000000 state:
  1: cumulative: 228464 gas: 114567 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x49aa8ca1ad46ffaea1dcc9fe10d142390c98cdfd7386ffd41be5228d47727448 logs: [0xc0212f0fd0 0xc0212f1290 0xc0212f14a0] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000080020000000000000000000000000000000000c00000008000000000000000000000000000000000000000000000800000000000000000000000000100000000000000000000010000800000000001400000000001000000000000100000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000002400000000000000000001000000000000000000000000001000000000002200000000000000000000000000000000000000000000000000000000000 state:
  2: cumulative: 354771 gas: 126307 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x934d83a0521cdbcede60812095c7205ab96692455a56891cb8d3bf330929eb39 logs: [0xc0212f18c0 0xc0212f1b80 0xc0212f1e40 0xc021a2e000 0xc021a2e210] bloom: 002000000100000000000000800000000000000000000000000100000000000100000000000000000002000000000000020000000800000000000000000000000000000000000000010000080000002000000000004000000000000000000000000000000000000000000000000000000000000000000400000800100000000000000000000000000040001000000000000000000000000a0000004000000000000000000000000000020000000000000000000000000000000000000000000000000002000004000000000000000000000000000000801800000002000020000000200000000000080000000000000000000000000000000000000000000000 state:
  3: cumulative: 439860 gas: 85089 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x598375e8c093ca91ab081cf699558cb10298b6a65bb54d683a8e8df81b68b267 logs: [0xc021a2e420 0xc021a2e6e0 0xc021a2e9a0 0xc021a2ea50] bloom: 002000000100000000000000800000000800000000000000000000000000000100000000000000000002000000000000020000000800000000000000000000000000000000000000010000080000002000000000000000000000000000000000000000000000000000000000000000004000000000000000000800100000000000000000000000000000000000000000000000000000000a0000004000000000000000000000000000000000000000000000000000000000004000000000000000000002000000000000000000000000000000000000801800000000000000000000200000000000080000000000000000000000000000000000000000000000 state:
  4: cumulative: 579500 gas: 139640 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6b7c6e0b81c7737d921957cd795b0bbd1182d1a1394c17903ded037a381c407d logs: [0xc021a2ec60 0xc021a2edc0 0xc021a2f340 0xc021a2f600 0xc021a2f6b0 0xc021a2fb80 0xc021a46000] bloom: 002000000100000000000000800000000000000000000000000000000000000100000000000800000002000000000000020000000800000000000000000000000000000000000000010000080040002000000000000000000000000080000000000000000000002000000000000000000000000000000000000800100000000000000000000000000000000000200000000000010000000a0000004000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000002000000400000000801800000001000000000000200000400000080000000000000000000000000000c00000000000000000 state:
  5: cumulative: 664871 gas: 85371 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x8ef7261ce973e0bd0e380350ecd74efc5bdbfbf6375ab891e0ed50482399aae1 logs: [0xc021a462c0 0xc021a464d0 0xc021a46790 0xc021a46840] bloom: 002000000100000000000000800000000800000000000000000000000000000100000000000000000002000000000000020000000800000000000000000000000000000000000000010000080000002000000000000000000000000000000000000000000000000000000000000000004000000000000000000800100000000000000000000000000000000000000000000000000000000a0000004000000000000000000000000000000000000000000000000000000000004000000000000000000002000000000000000000000000000000000000801800000000000000000000200000000000080000000000000000000000000000000000000000000000 state:
  6: cumulative: 775299 gas: 110428 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x752c794d55d568ee6ccbc84ee1d87c713c9dce49d669186fdddc987315dca6ca logs: [0xc021a46b00 0xc021a46dc0 0xc021a46fd0 0xc021a47080] bloom: 00200000000000000000000080000000000000000000000000000000000000000000000000040000000000000000000002000020080000000000000000000000000001000000000400000008000000200000000000000000000000000000000000000000000000000000000000000000100000000000000000000010002000000000001000000000000000000000000000000000000000080200004000000000000000000000000000000000000000000000000100000000000000000000000000000002400000000000000000001000000000000000001000000000000000000002200000000000000000000000000000000000000000000000000000000000 state:
  7: cumulative: 806704 gas: 31405 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x22aa3daad59ddf2c2f3ddf09ad8ae55d2e776ed85a35bb476e18b88d012d6528 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
...
ERROR[04-09|11:39:06.606] Beacon backfilling failed                err="retrieved hash chain is invalid: invalid gas used (remote: 19218740 local: 19235840)"
INFO [04-09|11:39:10.714] Log index head rendering in progress     "first block"=19,847,731 "last block"=22,225,254 processed=811   remaining=0     elapsed=6m20.266s
INFO [04-09|11:39:10.714] Log index head rendering finished        "first block"=19,847,731 "last block"=22,225,254 processed=811   elapsed=6m20.266s
WARN [04-09|11:39:38.505] Marked new chain head as invalid         hash=3043ba..fab18a badnumber=22,225,255 badhash=6789cc..ffa792
INFO [04-09|11:42:07.992] NAT mapped port                          proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [04-09|11:42:08.428] NAT mapped port                          proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"

I'm hoping there is a short cut to restarting a full node rather than syncing from block 0?

@rjl493456442 rjl493456442 changed the title Geth just keeps rewinding after crash Geth just keeps rewinding after crash in hash mode Apr 9, 2025
@rjl493456442
Copy link
Member

It's on hash mode, reason: Rewinding limit reached, resetting to genesis

@cosmicnet do you ever update to v1.15.6? There is a bug affects the hash mode, causing the state to be corrupted

@cosmicnet
Copy link
Author

I probably was on 1.15.6 for a while.

@cosmicnet
Copy link
Author

Saying that I can't see it in my download history. Looks like I went 1.15.1 -> 1.15.7

@s1na
Copy link
Contributor

s1na commented Apr 10, 2025

For me Geth had been acting oddly since the last update, Ctrl-C would start a shutdown but after giving it some time it wouldn't actually shut down.

You said geth wasn't gracefully shutting down. You probably had to kill it. It should in that case print a stack trace. That would be very useful for debugging. Do you still have it?

@fjl
Copy link
Contributor

fjl commented Apr 10, 2025

So the bad news is, since this node may have been running v1.15.6 at some point, the state database could be broken. I would advise to just upgrade to v1.15.7 and resync. It will take a day or so, since your node is not an archive node.

@cosmicnet
Copy link
Author

I've only got the stack traces from the two crashes I captured.

I ran another memory test and it came back clear again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fjl @cosmicnet @s1na @jwasinger @rjl493456442