Skip to content

CI: see if wsl2 works #1826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Conversation

AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda force-pushed the test-wsl2 branch 9 times, most recently from 98678cd to 60e615c Compare September 21, 2023 16:58
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Sep 21, 2023

"failed to run wsl --list --verbose, err: exit status 0xffffffff"

+ ./hack/test-templates.sh ./templates/experimental/wsl2.yaml
TEST| [INFO] Validating "./templates/experimental/wsl2.yaml"
time="2023-0[9](https://github.com/lima-vm/lima/actions/runs/6264735883/job/17012085612?pr=1826#step:7:10)-21T17:03:07Z" level=info msg="\"./templates/experimental/wsl2.yaml\": OK"
time="2023-09-21T17:03:07Z" level=warning msg="No instance found. Run `limactl create` to create an instance."
TEST| [INFO] Setup port forwarding rules for testing in "/c/Users/runneradmin/lima-config-tmp/wsl2.yaml"
time="2023-09-21T17:03:09Z" level=info msg="\"C:/Users/runneradmin/lima-config-tmp/wsl2.yaml\": OK"
TEST| [INFO] Creating "wsl2" from "/c/Users/runneradmin/lima-config-tmp/wsl2.yaml"
+ limactl create --tty=false --cpus=1 --memory=1 /c/Users/runneradmin/lima-config-tmp/wsl2.yaml
time="2023-09-21T17:03:09Z" level=info msg="Terminal is not available, proceeding without opening an editor"
time="2023-09-21T[17](https://github.com/lima-vm/lima/actions/runs/6264735883/job/17012085612?pr=1826#step:7:18):03:09Z" level=fatal msg="errors inspecting instance: [failed to run `wsl --list --verbose`, err: exit status 0xffffffff]"
+ eval rm -rf '"/c/Users/runneradmin/lima-config-tmp";' limactl delete -f '"wsl2"'
++ rm -rf /c/Users/runneradmin/lima-config-tmp
++ limactl delete -f wsl2
time="[20](https://github.com/lima-vm/lima/actions/runs/6264735883/job/17012085612?pr=1826#step:7:21)23-09-[21](https://github.com/lima-vm/lima/actions/runs/6264735883/job/17012085612?pr=1826#step:7:22)T17:03:09Z" level=info msg="The wsl2 driver process seems already stopped"
time="2023-09-21T17:03:09Z" level=info msg="The host agent process seems already stopped"
time="20[23](https://github.com/lima-vm/lima/actions/runs/6264735883/job/17012085612?pr=1826#step:7:24)-09-21T17:03:09Z" level=info msg="Removing C:\\Users\\runneradmin\\.lima\\wsl2 under \"*.pid *.sock *.tmp\""
time="2023-09-21T17:03:09Z" level=fatal msg="failed to delete instance \"wsl2\": failed to unregister \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\": failed to run `wsl --list --verbose`, err: exit status 0xffffffff"
Error: Process completed with exit code 1.

https://github.com/lima-vm/lima/actions/runs/6264735883/job/17012085612?pr=1826

@AkihiroSuda
Copy link
Member Author

  wsl --list --verbose
  shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
Windows Subsystem for Linux has no installed distributions.


Distributions can be installed by visiting the Microsoft Store:


https://aka.ms/wslstore

@AkihiroSuda AkihiroSuda force-pushed the test-wsl2 branch 10 times, most recently from a5e9480 to fc1ace2 Compare September 21, 2023 18:08
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Sep 21, 2023

TEST| [INFO] Starting "wsl2"
+ limactl start wsl2
time="2023-09-21T18:11:10Z" level=info msg="Using the existing instance \"wsl2\""
time="2023-09-21T18:11:10Z" level=warning msg="expected status \"Stopped\", got \"Uninitialized\""
time="2023-09-21T18:11:10Z" level=warning msg="Ignoring: vmType wsl2: [OS CPUs Memory Firmware Audio Video GuestInstallPrefix CACertificates Rosetta]"
time="2023-09-21T18:11:10Z" level=info msg="Attempting to download the nerdctl archive" arch=x86_64 digest="sha[25](https://github.com/lima-vm/lima/actions/runs/6265448839/job/17014322683?pr=1826#step:9:26)6:2c5c43a8b77ed620902410[27](https://github.com/lima-vm/lima/actions/runs/6265448839/job/17014322683?pr=1826#step:9:28)361baa62d8fb70a759bc9e7a82c637135598701f" location="https://github.com/containerd/nerdctl/releases/download/v1.6.0/nerdctl-full-1.6.0-linux-amd64.tar.gz"
time="2023-09-21T18:11:10Z" level=info msg="Using cache \"C:\\\\Users\\\\runneradmin\\\\AppData\\\\Local\\\\lima\\\\download\\\\by-url-sha256\\\\addb72512fd2fef72b08d8648173d7aaec3b4d5ae0566ac0bfd03c5adeeafc78\\\\data\""
time="2023-09-21T18:11:10Z" level=info msg="[hostagent] Replacing \"ftp_proxy\" value \"http://localhost:2121\" with \"http://192.168.5.2:2121\""
time="2023-09-21T18:11:10Z" level=info msg="[hostagent] Replacing \"FTP_PROXY\" value \"http://localhost:2121\" with \"http://192.168.5.2:2121\""
time="2023-09-21T18:11:11Z" level=info msg="[hostagent] hostagent socket created at C:\\Users\\runneradmin\\.lima\\wsl2\\ha.sock"
time="2023-09-21T18:11:11Z" level=info msg="[hostagent] Starting WSL VM"
time="2023-09-21T18:11:11Z" level=info msg="[hostagent] Attempting to download the image"
time="2023-09-21T18:11:18Z" level=info msg="[hostagent] 79.99 MiB / 98.11 MiB (81.52%) ? p/s"
time="2023-09-21T18:11:18Z" level=info msg="[hostagent] 98.11 MiB / 98.11 MiB (100.00%) 127.47 MiB/s{\"level\":\"info\",\"msg\":\"decompressing .zst with [zstd -d]\",\"time\":\"2023-09-21T18:11:18Z\"}"
time="2023-09-21T18:11:18Z" level=info msg="[hostagent] Decompressing data\n"
time="2023-09-21T18:11:18Z" level=info msg="[hostagent] 98.11 MiB / 98.11 MiB (100.00%) ? p/s{\"level\":\"debug\",\"msg\":\"res.ValidatedDigest=true\",\"time\":\"2023-09-21T18:11:18Z\"}"
time="2023-09-21T18:11:18Z" level=info msg="[hostagent] Downloaded the image from \"https://deps.runfinch.com/common/x86-64/finch-rootfs-production-amd64-1690920103.tar.zst\""
time="2023-09-21T18:11:18Z" level=info msg="[hostagent] Download succeeded"
time="2023-09-21T18:11:18Z" level=info msg="[hostagent] Importing distro from \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\\\\basedisk\" to \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\""
2023/09/21 18:11:[28](https://github.com/lima-vm/lima/actions/runs/6265448839/job/17014322683?pr=1826#step:9:29) Error in Watcher Error channel: CancelIo: The handle is invalid.
2023/09/21 18:11:28 Error in Watcher Error channel: CloseHandle: The handle is invalid.
time="2023-09-21T18:11:28Z" level=fatal msg="invalid character 'D' looking for beginning of value"
+ ERROR 'Failed to start "wsl2"'
Error: 'TEST| [ERROR] Failed to start "wsl2"'
Error: ERROR] Failed to start "wsl2"
+ diagnose wsl2
+ NAME=wsl2
+ set -x +e
+ tail /c/Users/runneradmin/.lima/wsl2/ha.stderr.log /c/Users/runneradmin/.lima/wsl2/ha.stdout.log
==> /c/Users/runneradmin/.lima/wsl2/ha.stderr.log <==
98.11 MiB / 98.11 MiB (100.00%) 127.47 MiB/s{"level":"info","msg":"decompressing .zst with [zstd -d]","time":"2023-09-21T18:11:18Z"}
{"level":"info","msg":"Decompressing data\n","time":"2023-09-21T18:11:18Z"}

98.11 MiB / 98.11 MiB (100.00%) ? p/s{"level":"debug","msg":"res.ValidatedDigest=true","time":"2023-09-21T18:11:18Z"}
{"level":"info","msg":"Downloaded the image from \"https://deps.runfinch.com/common/x86-64/finch-rootfs-production-amd64-1690920103.tar.zst\"","time":"2023-09-21T18:11:18Z"}
{"level":"info","msg":"Download succeeded","time":"2023-09-21T18:11:18Z"}
{"level":"info","msg":"Importing distro from \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\\\\basedisk\" to \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\"","time":"2023-09-21T18:11:18Z"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 3: \"ssh\"","time":"2023-09-21T18:11:28Z"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2023-09-21T18:11:28Z"}
{"level":"debug","msg":"executing ssh for script \"ssh\": C:\\Program Files\\Git\\usr\\bin\\ssh.exe [ssh -F /dev/null -o IdentityFile=\"C:/Users/runneradmin/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=^aes128-gcm@openssh.com,aes256-gcm@openssh.com -o User=lima -o ControlMaster=auto -o ControlPath=\"C:/Users/runneradmin/.lima/wsl2/ssh.sock\" -o ControlPersist=yes -p 22 172.19.253.101 -- /bin/bash]","time":"2023-09-21T18:11:28Z"}

==> /c/Users/runneradmin/.lima/wsl2/ha.stdout.log <==
Downloading the image (finch-rootfs-production-amd64-1690920103.tar.zst)
{"time":"2023-09-21T18:11:28.465197Z","status":{"sshLocalPort":22}}
+ limactl shell wsl2 systemctl --no-pager status
command-line line 0: invalid quotes
+ limactl shell wsl2 systemctl --no-pager
command-line line 0: invalid quotes
+ limactl shell wsl2 sudo cat /var/log/cloud-init-output.log
command-line line 0: invalid quotes
+ set +x -e
time="2023-09-21T18:11:30Z" level=info msg="The wsl2 driver process seems already stopped"
time="2023-09-21T18:11:30Z" level=info msg="The host agent process seems already stopped"
time="2023-09-21T18:11:30Z" level=info msg="Removing C:\\Users\\runneradmin\\.lima\\wsl2 under \"*.pid *.sock *.tmp\""
time="2023-09-21T18:11:30Z" level=info msg="Removing \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\\\\ha.sock\""
time="2023-09-21T18:11:30Z" level=info msg="Unregistering WSL2 VM"
time="2023-09-21T18:11:31Z" level=info msg="Deleted \"wsl2\" (\"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\")"
Error: Process completed with exit code 1.

https://github.com/lima-vm/lima/actions/runs/6265448839/job/17014322683?pr=1826
fc1ace2

@pendo324 PTAL

@AkihiroSuda
Copy link
Member Author

time="2023-09-21T18:11:18Z" level=info msg="[hostagent] Importing distro from \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\\\\basedisk\" to \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\""
2023/09/21 18:11:[28](https://github.com/lima-vm/lima/actions/runs/6265448839/job/17014322683?pr=1826#step:9:29) Error in Watcher Error channel: CancelIo: The handle is invalid.
2023/09/21 18:11:28 Error in Watcher Error channel: CloseHandle: The handle is invalid.
time="2023-09-21T18:11:28Z" level=fatal msg="invalid character 'D' looking for beginning of value"

@pendo324 @afbjorklund Do you repro the issue locally? Is it working on your local env?

@AkihiroSuda AkihiroSuda mentioned this pull request Sep 27, 2023
@balajiv113
Copy link
Member

I think the error is related to using bash mode. Y not try a simple limactl start template:// to see if its able to start fully

@AkihiroSuda
Copy link
Member Author

Removed bash, still failing

Run .\_output\bin\limactl.exe start template://experimental/wsl2
time="2023-09-28T15:0[4](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:5):49Z" level=info msg="Terminal is not available, proceeding without opening an editor"
time="2023-09-28T1[5](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:6):04:50Z" level=warning msg="expected status \"Stopped\", got \"Uninitialized\""
time="2023-09-28T15:04:50Z" level=warning msg="Ignoring: vmType wsl2: [OS CPUs Memory Firmware Audio Video GuestInstallPrefix CACertificates Rosetta]"
time="2023-09-28T15:04:50Z" level=info msg="Attempting to download the nerdctl archive" arch=x8[6](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:7)_64 digest="sha256:2c5c43a8b[7](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:8)7ed62090241027361baa62d8fb70a759bc9e7a82c637135598701f" location="https://github.com/containerd/nerdctl/releases/download/v1.6.0/nerdctl-full-1.6.0-linux-amd64.tar.gz"
Downloading the nerdctl archive (nerdctl-full-1.6.0-linux-amd64.tar.gz)

240.55 MiB / 240.55 MiB (100.00%) ? p/stime="2023-09-2[8](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:9)T15:04:51Z" level=info msg="Downloaded the nerdctl archive from \"https://github.com/containerd/nerdctl/releases/download/v1.6.0/nerdctl-full-1.6.0-linux-amd64.tar.gz\""
time="2023-0[9](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:10)-28T15:04:52Z" level=info msg="[hostagent] hostagent socket created at C:\\Users\\runneradmin\\.lima\\wsl2\\ha.sock"
time="2023-09-28T15:04:52Z" level=info msg="[hostagent] Starting WSL VM"
time="2023-09-28T15:04:52Z" level=info msg="[hostagent] Attempting to download the image"
time="2023-09-28T15:04:53Z" level=info msg="[hostagent] 98.11 MiB / 98.11 MiB ([10](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:11)0.00%) ? p/s{\"level\":\"info\",\"msg\":\"decompressing .zst with [zstd -d]\",\"time\":\"2023-09-28T15:04:53Z\"}"
time="2023-09-28T15:04:53Z" level=info msg="[hostagent] Decompressing data\n"
time="2023-09-28T15:04:53Z" level=info msg="[hostagent] 98.[11](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:12) MiB / 98.11 MiB (100.00%) ? p/s{\"level\":\"debug\",\"msg\":\"res.ValidatedDigest=true\",\"time\":\"2023-09-28T[15](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:16):04:53Z\"}"
time="2023-09-28T15:04:53Z" level=info msg="[hostagent] Downloaded the image from \"https://deps.runfinch.com/common/x86-64/finch-rootfs-production-amd64-[16](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:17)90920103.tar.zst\""
time="2023-09-28T15:04:53Z" level=info msg="[hostagent] Download succeeded"
time="2023-09-28T15:04:53Z" level=info msg="[hostagent] Importing distro from \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\\\\basedisk\" to \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\""
time="[20](https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826#step:9:21)23-09-28T15:05:03Z" level=fatal msg="invalid character 'D' looking for beginning of value"

https://github.com/lima-vm/lima/actions/runs/6340681012/job/17222647981?pr=1826

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Sep 28, 2023

Added some debug prints, it tries to unmarshal Downloading the image (finch-rootfs-production-amd64-1690920103.tar.zst) as JSON

time="2023-09-28T15:24:38Z" level=info msg="[hostagent] Importing distro from \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\\\\basedisk\" to \"C:\\\\Users\\\\runneradmin\\\\.lima\\\\wsl2\""
time="[20](https://github.com/lima-vm/lima/actions/runs/6340933579/job/17223439234?pr=1826#step:9:21)23-09-28T15:24:49Z" level=fatal msg="failed to unmarshal \"Downloading the image (finch-rootfs-production-amd64-1690920103.tar.zst)\" as events.Event: invalid character 'D' looking for beginning of value"

https://github.com/lima-vm/lima/actions/runs/6340933579/job/17223439234?pr=1826

@pendo324 Do you see this error locally? Is it working on your machine?

@balajiv113
Copy link
Member

Not using logrus here, will this be a problem here ??

fmt.Printf("Downloading %s\n", description)

@AkihiroSuda
Copy link
Member Author

Not using logrus here, will this be a problem here ??

fmt.Printf("Downloading %s\n", description)

Yes, IIUC this was on purpose to avoid messing up the output, but probably we should use logrus, or at least switch the output to stderr.

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Sep 28, 2023

"bash: line 1: $'\\\\\\\\r': command not found\\\\n\\\" 🤔

time="2023-09-28T17:18:46Z" level=info msg="[hostagent] Driver stopped due to error: \"error running wslCommand that executes boot.sh: exit status 127, check /var/log/lima-init.log for more details (out=\\\"bash: line 1: $'\\\\\\\\r': command not found\\\\n\\\")\""

https://github.com/lima-vm/lima/actions/runs/6342128771/job/17227239927?pr=1826

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@AkihiroSuda AkihiroSuda force-pushed the test-wsl2 branch 8 times, most recently from a57614c to 5a8b908 Compare September 30, 2023 19:46
The script was failing prior to this commit

```
time="2023-09-28T17:18:46Z" level=info msg="[hostagent] Driver stopped due to error: \"error running wslCommand that executes boot.sh: exit status 127, check /var/log/lima-init.log for more details (out=\\\"bash: line 1: $'\\\\\\\\r': command not found\\\\n\\\")\""
```

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@AkihiroSuda
Copy link
Member Author

Temporarily giving up:

Still help wanted to carry this.

@AkihiroSuda AkihiroSuda added the help wanted Extra attention is needed label Sep 30, 2023
@pendo324
Copy link
Contributor

pendo324 commented Oct 2, 2023

Sorry @AkihiroSuda, I just saw this for this first time (apparently I had my notifications off? I'll try to fix that).

The issue seems to be that you don't have the binaries needed to extract the rootfs archive. It works on my machine because I do have those in my path.

Is the zstd binary installed on the GHA runner? Is tar/gzip installed? We can switch to an OS image that uses tar/gzip if zstd is not installed.

To install tar/gzip we can install Git for Windows and add it to the path. Something like this will do that:

$env:Path = 'C:\Program Files\Git\usr\bin;' + $env:Path
Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -Value $env:Path

If you re-open this PR and change the rootfs in wsl2.yaml to this, it should work if tar/gzip are installed:

- location: "https://deps.runfinch.com/common/x86-64/finch-rootfs-production-amd64-1694791577.tar.gz",
  arch: "x86_64"
  digest: "sha256:2d4d2e7386450899c6d0587fd0db21afadb31d974fa744aa9365c883935c5341"

@AkihiroSuda
Copy link
Member Author

Thanks @pendo324 would you mind opening a PR?
The error has to be human-readable too.

@pendo324 pendo324 mentioned this pull request Oct 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/test Tests and CI help wanted Extra attention is needed platform/Windows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants