Understanding SSH in mirai

[wlandau]

I would like to leverage the SSH capabilities in mirai for crew (wlandau/crew#200) but I am not sure I understand the implicit network topology. If I am reading the SSH docs correctly, then if I call:

daemons(
  n = 4,
  url = "tcp://127.0.0.1:5100",
  remote = ssh_config("ssh://10.0.0.4:5200", tunnel = TRUE)
)

then I would get:

It seems like 127.0.0.1:5100 is the SSH tunnel and 10.0.0.4:5200 is the ordinary TCP connection for daemons. Behind the scenes, does the host call ssh 10.0.0.4:5100 to SSH into the "remote" over the SSH port 5100?

To implement and document this in crew, I would like to confirm:

1. mirai's SSH features may be appropriate when daemons will be launched from a remote machine accessible over SSH and not directly from the local machine running R interactively.
2. The IP address of the remote machine, the port for daemons, and the port for SSH must all be specified. Of these 3 pieces of information, only the third one (the SSH port) is new. The others are already inputs to crew.
3. crew would translate (2) into daemons(n = 4, url = "tcp://127.0.0.1:SSH_PORT", remote = ssh_config("ssh://REMOTE_IP:DAEMON_PORT", tunnel = TRUE))

Is all this accurate?

[shikokuchuo]

Not entirely accurate I'm afraid. Perhaps the below is clearer:

daemons(
  n = 4,
  url = "tcp://127.0.0.1:5100",
  remote = ssh_config("ssh://10.0.0.4:5200", tunnel = TRUE)
)

1. Host (dispatcher) listens to 127.0.0.1:5100.
2. You establish an SSH connection to 10.0.0.4 over port 5200.
3. A tunnel (within the SSH connection) is created between port 5100 on each machine.
4. 4 daemons are launched dialling 127.0.0.1:5100 on the machine 10.0.0.4.

The tunnel means that the daemons in (4) connect back to host (1).

Note that the default SSH port is 22. If that's in use then there's no need to specify a port for the SSH connection.

[wlandau]

Whether it's SSH or Slurm sbatch or something else wouldn't affect whether a daemon should persist or not. In fact there's no concept of a 'persistent worker that lasts for the lifetime of the compute profile' in mirai.

Is daemons(remote = ssh_config(...), .compute = "profile") responsible for launching the daemons? And is daemons(n = 0, .compute = "profile") responsible for terminating them?

Sorry I realise my reply above was incorrect (now crossed out), you do need to specify n as I gave in my example. I was thinking of launch_remote(), which has a default argument of n = 1L, but this is not the case for daemons(). Perhaps that makes your later questions irrelevant.

In that case, is n equal to the length of ssh_config() URLs? E.g.:

daemons(
  n = 2,
  url = "tcp://127.0.0.1:5100",
  remote = ssh_config(c("ssh://10.0.0.4:5200", "ssh://10.0.0.5:5200") , tunnel = TRUE)
)

[wlandau]

Is daemons(remote = ssh_config(...), .compute = "profile") responsible for launching the daemons? And is daemons(n = 0, .compute = "profile") responsible for terminating them?

If this is the case, I don't think this would actually be a great fit with crew. It would require adding special cases to the crew's client class which is responsible for calling daemons(): https://github.com/wlandau/crew/blob/2ddf92b09d98b072e72cb48b73707df4fab16c06/R/crew_client.R#L258-L265. This call simply sets up the connection on the user's machine but does not launch any daemons yet. It would be great to use this approach but then have the launcher set up the SSH tunnel if it does not already exist, e.g. in https://github.com/wlandau/crew/blob/2ddf92b09d98b072e72cb48b73707df4fab16c06/R/crew_launcher_local.R#L266-L280. That way a new launcher plugin could be developed for this outside crew.

Unfortunately my organization prohibits SSH tunneling into our cluster, and I am having trouble creating a test environment for SSH using a Docker container. So I'm not sure I'll be able to develop this myself.

[shikokuchuo]

Yes maybe it belongs in a plugin package. Conceptually though, if you don't specify the remote argument of daemons() then it won't be different to any other case. You can then implement your own launcher instead of using mirai's built in launch_remote().

[wlandau]

Ah, I see launch_remote() has the same remote = remote_config() argument. I could test this if I had a server capable of SSH tunneling.

[shikokuchuo]

Ok, I think I'll close this for now then if there's no immediate action point.