Openshift local version 4.18.2 goes to unreachable status if the environment is idle for sometime.

[codersyacht]

General information

Issue only occur after sometime, like 1 hour of idleness.

[admin@ocp1 ~]$ crc status
CRC VM:          Running
OpenShift:       Unreachable (v4.18.2)
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     28.13GB
Cache Directory: /home/admin/.crc/cache
[admin@ocp1 ~]$ 

[admin@ocp1 ~]$ sudo cat /var/log/libvirt/qemu/crc.log
2025-04-15 16:14:19.999+0000: Starting external device: virtiofsd
/usr/libexec/virtiofsd --fd=34 --shared-dir /home/admin
2025-04-15 16:14:20.011+0000: starting up libvirt version: 10.5.0, package: 7.5.el9_5 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2025-01-21-08:08:49, ), qemu version: 9.0.0qemu-kvm-9.0.0-10.el9_5.2, kernel: 5.14.0-427.42.1.el9_4.x86_64, hostname: ocp1.fyre.ibm.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-crc \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-crc/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-crc/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-crc/.config \
/usr/libexec/qemu-kvm \
-name guest=crc,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-crc/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/ovmf/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/crc_VARS.fd","node-name":"libvirt-pflash1-storage","read-only":false}' \
-machine pc-q35-rhel9.4.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-storage,acpi=on \
-accel kvm \
  Booting `Red Hat Enterprise Linux CoreOS 418.94.202502250906-0 (ostree:0)'

[admin@ocp1 ~]$ oc login -u developer https://api.crc.testing:6443
error: dial tcp 127.0.0.1:6443: connect: connection refused - verify you have provided the correct host and port and that the server is currently running.

Operating System

Linux

Hypervisor

KVM

Did you run crc setup before crc start?

yes

Running on

Laptop

Steps to reproduce

A running crc instance abruptly stops after sometime of idleness.

CRC version

4.18.2

CRC status

[admin@ocp1 ~]$ crc status --log-level debug
DEBU CRC version: 2.49.0+e843be                   
DEBU OpenShift version: 4.18.2                    
DEBU MicroShift version: 4.18.2                   
DEBU Running 'crc status'                         
CRC VM:          Running
OpenShift:       Unreachable (v4.18.2)
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     28.13GB
Cache Directory: /home/admin/.crc/cache

CRC config

[admin@ocp1 ~]$ crc config view
- consent-telemetry                     : no
- cpus                                  : 8
- disk-size                             : 100
- enable-cluster-monitoring             : true
- memory                                : 32768
- pull-secret-file                      : /home/admin/apps/ocp/pull-secret.txt

Host Operating System

[admin@ocp1 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.5 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.5"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.5 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.5"

Expected behavior

crc should not stop.

Actual behavior

crc abruptly stops

CRC Logs

restart works. But requires restart every 1 hour.

Additional context

No response

[codersyacht]

Issue noted only when running with network-mode as user. When it is set to system with a HA proxy redirecting request to the openshift IP and port, issue does not occur.

[cfergeau]

When this happens, can you still ssh into the cluster? https://github.com/crc-org/engineering-docs/blob/main/content/Debugging.md#access-the-vm (the linux instructions are outdated…)

$ ssh -i ~/.crc/machines/crc/id_ed25519 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 2222 core@127.0.0.1

[codersyacht]

@cfergeau

Thanks for the quick response. It does not work.

[admin@system1 crc]$ ls -lrt
total 21258324
-rw------- 1 admin admin          81 Apr 22 08:44 id_ed25519.pub
-rw------- 1 admin admin         387 Apr 22 08:44 id_ed25519
-rw------- 1 admin admin          23 Apr 22 08:44 kubeadmin-password
srwxr-xr-x 1 admin admin           0 Apr 22 08:44 docker.sock
-rw------- 1 admin admin         901 Apr 22 08:44 config.json
-rw------- 1 admin admin       15275 Apr 22 08:49 kubeconfig
-rw-r--r-- 1 qemu  qemu  21767782400 Apr 22 21:04 crc.qcow2
[admin@system1 crc]$ 
[admin@system1 crc]$ ssh -i ~/.crc/machines/crc/id_ed25519 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 2222 core@127.0.0.1
ssh: connect to host 127.0.0.1 port 2222: Connection refused

You can find some explanations for typical errors at this link:
            https://red.ht/support_rhel_ssh
[admin@system1 crc]$ 
[admin@system1 crc]$ crc status
CRC VM:          Running
OpenShift:       Unreachable (v4.18.2)
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     28.13GB
Cache Directory: /home/admin/.crc/cache
[admin@system1 crc]$