[vox-tech] Reverse SSH Tunneling

[Chris Jenks]

   I am trying to manage a video recording server that is a hassle to gain 
physical access to, and is behind a firewall. Until a few weeks ago I was
able to have it download and execute a script containing this command:

ssh -p 443 -N -o StrictHostKeyChecking=no -R2222:localhost:22
root at 75.206.132.66 -2

which would connect it to my Dad's Linksys router running OpenWRT. The
reason I used port 443 is that outgoing port 22 is blocked by the firewall.
Then I would shell into the router and connect to the video server with:

ssh -p 2222 chris at localhost

which gave me access. After some problems with kernel versions on the
debian-based video server, I replaced the OS with ubuntu 7.10, to match my
workstations, but I haven't been able to get the reverse SSH working. It
seems that the video server is able to SSH out, but the reverse tunneling is
broken. To make testing easier I set up my server at home to receive the SSH
connection from the video server, with my router translating port 443 to
port 22 on the home server. Before the video server connects, I get
Connection Refused when I try to reverse SSH from the home server, and after
the video server connects I get:

ssh_exchange_identification: Connection closed by remote host

showing that there is an active connection from the video server that lasts
for an hour or two. What is especially odd about this situation is that I
can connect from my ubuntu workstation at work, which is also behind a
firewall, and reverse SSH back to it successfully. The most frequent reason
for the above error is a problem with the hosts.allow or hosts.deny files,
but these files are unaltered (and fully commented out) on both the video
server and the workstation. The other reason I've read is that there are too
many SSH connections, and that the error goes away if the connections are
allowed to time out. I've tried kpill ssh on the video server and waiting
overnight to no avail.

Here is the debug output for the reverse SSH command, when trying to connect
to the video server from my home server:
ssh -vvv -p 2226 chris at localhost:

OpenSSH_4.7p1 Debian-2, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [127.0.0.1] port 2226.
debug1: Connection established.
debug1: identity file /home/chris/.ssh/identity type -1
debug3: Not a RSA1 key file /home/chris/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace (25 times)
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/chris/.ssh/id_rsa type 1
debug1: identity file /home/chris/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host

When I try to connect to my workstation from my home server with an
equivalent command, only the last line above changes:

[...]
debug1: identity file /home/chris/.ssh/id_dsa type -1 (as before)
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.6p1
debug1: Debian-5ubuntu0.1
debug1: match: OpenSSH_4.6p1 Debian-5ubuntu0.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-2
[...]

I searched auth.log on the video server for sshd entries, but there weren't
any. However, I was able to obtain the debug output from the command:

ssh -vvv -p 443 -N -o StrictHostKeyChecking=no -R2226:localhost:22
chris at mydomain.tld

[...]
debug1: Entering interactive session.
debug1: remote forward success for: listen 2226, connect localhost:22
debug1: client_input_channel_open: ctype forwarded-tcpip rchan 2 win 
2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 2226,
originator 127.0.0.1 port 39823
debug2: fd 4 setting O_NONBLOCK
debug2: fd 4 setting TCP_NODELAY
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new [127.0.0.1]
debug1: confirm forwarded-tcpip
debug3: channel 0: waiting for connection
debug1: channel 0: not connected: Connection refused
debug2: channel 0: zombie
debug2: channel 0: garbage collecting
debug1: channel 0: free: 127.0.0.1, nchannels 1
debug3: channel 0: status: The following connections are open:

debug3: channel 0: close_fds r 4 w 4 e -1 c -1
debug3: fd 0 is not O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 228.0 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0

The same command from the workstation gives:

[...]
debug1: confirm forwarded-tcpip (as before)
debug3: channel 0: waiting for connection (as before)
debug1: channel 0: connected
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel 0: send close
debug3: channel 0: will not send data after close
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: 127.0.0.1, nchannels 1
debug3: channel 0: status: The following connections are open:
   #0 127.0.0.1 (t4 r2 i3/0 o3/0 fd 4/4 cfd -1)

debug3: channel 0: close_fds r 4 w 4 e -1 c -1
debug1: Killed by signal 2.

I realized that scp should work if ssh does, and indeed I can copy files
both to and from the video server with scp run on the video server. Any
guesses why the reverse SSH is getting stuck?

   Thanks,

     Chris