How ‘telnet’ and similar tools help you troubleshoot

In Episode 110 of the Podcast there was a brief discussion of “telnet” as a troubleshooting tool (starts around 22:00).

The question arose about why telnet is a troubleshooting tool, what it can (and can’t) do – and I thought I would use this opportunity to detail why this works, how far up the OSI ladder you can use it to troubleshoot and what tools will take it even a step further.

Telnet dates back into the late 60’s, when the Internet was just a bunch of universities and research facilities connecting to each other, known as the ARPAnet, interconnected to exchange data.

You might think now: “The 60’s? Didn’t IP and therefore TCP start out in the 70’s?”
And you’re right. These early RFCs talking about TELNET and the HOST-HOST-Network connection were written before TCP/IP came to light and do not mention it.

They do however mention the most important characteristic of telnet:

* Is "teletype-like", i.e.:
               - ASCII characters are transmitted;
               - Echoes are generated by the remote HOST;
               - The receiving HOST[s] scan for break characters;
               - The transmission rate is slow (less than 20
               characters/sec).

RFC 15

The host-host connection is meant to be “teletype-like” – meaning like the old telegraphy transmission from the 19th century, and also what got telnet it’s name: TELetype NETwork.

I could make this article a long list of subsequent RFCs, but to sum things up: The goal was to type on a keyboard cross-country from the computer you are working on, instead of having a terminal directly connected to it. This was over the course of years and RFCs standardised to be 7-bit ASCII text and a few control characters.

When TCP/IP gained traction and protocols were defined, telnet was moved over, as – beside from a few terminal control infos – just this plain text over a TCP socket, using Port 23.

As you can see in Wireshark, after the TCP handshake – SYN, SYN/ACK, ACK – the telnet data starts flowing.

There is some data exchanged nowadays about the terminal in use and it’s capabilities, but then you see just plain ASCII going back and forth in the TCP segments.

So the answer to the question “What is it about telnet that makes it possible to use it to connect to other services?” is: all telnet does is open a TCP socket, and waiting for either side to send data. Either you start typing, or the server on the other side sends something.

As it is only opening a a connection to a port and waiting for data, there is really nothing from stopping you to connect to other services that talk TCP. Just put a second parameter after the host or IP address as TCP port and off you go. If the TCP handshake is successfully completed, the telnet application either displays the text that the other end sends, or just waits for you to send text. “Text” in this context is meant as data – not necessarily something you can read, depending on the protocol.

% telnet artofnetworkengineering.com 80
Trying 192.0.78.25...
Connected to artofnetworkengineering.com.
Escape character is '^]'.

As we’re connecting on TCP port 80 here, the web server on the other end waits for us to send data, and if we don’t send any, quits the TCP connection after a few seconds.

What have we accomplished with this?
We checked a connection up to layer 4, the TCP handshake was successful when we see “Connected to x”. Not getting a timeout here tells us:

• Our cable, their cable, and the cables in between are plugged in or wirelessly connected (layer 1)
• Data-link is working, whichever is used – probably Ethernet (layer 2)
• IP routing works from end to end, maybe with NAT somewhere (layer 3)
• TCP on a specific port works (layer 4), and an application is listening

So we can infer that no firewall – on one of the hosts, or between them – is blocking our connection, at least not the typical kind (looking at layers 3 & 4). But we also checked another thing – we use the hostname/FQDN here, not an IP address, which means that DNS resolution is also working. “Unknown host” or similar error messages would point to DNS resolution failing, “Operation timed out” to the host not responding on this port or a firewall blocking.

This works with every TCP protocol. We can connect to ssh – it even gives us some plain text with version info before the ciphers are negotiated:

% telnet 172.17.71.210 22
Trying 172.17.71.210...
Connected to 172.17.71.210.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.5

Connection closed by foreign host.

Or DNS (yes, DNS runs on UDP and TCP) – we won’t see anything, because DNS is a binary protocol, but the TCP connection would be established:

% telnet 172.17.71.1 53   
Trying 172.17.71.1...
Connected to 172.17.71.1.
Escape character is '^]'.

Connection closed by foreign host.

If the TCP handshake fails – because nothing is listening, or the connection is blocked – it would lead to a timeout.

% telnet 172.17.71.1 52
Trying 172.17.71.1...
telnet: connect to address 172.17.71.1: Operation timed out
telnet: Unable to connect to remote host

If we are looking at a layer 7 plain text protocol, we can even troubleshoot more. Many of our upper layer protocols came out of the UNIX world, and there were strong proponents to make them plain text protocols – which means there are no complex bit sequences indicating different things, but strings of readable text going back and forth. This has the advantage of relatively easy expansion (your new idea wont fit into the 2 bit value? Too bad. With text, you can just write more), easy to program (just read newlines from the TCP socket), and easy to troubleshoot.

One example of such an plain text protocol is HTTP. Nowadays of course it serves mostly as redirector to HTTPS (which wont work as easy in telnet), but you can check a TCP/IP+HTTP connection that way:

% telnet cisco.com 80      
Trying 2001:420:1101:1::185...
Connected to cisco.com.
Escape character is '^]'.
GET / HTTP/1.1
HOST: cisco.com

HTTP/1.1 301 Moved permanently
Location: https://cisco.com/
Connection: close
Cache-Control: no-cache
Pragma: no-cache

Connection closed by foreign host.

The “GET” line is the HTTP request method, and the “HOST” is needed because we have more than one site on one IP nowadays. Press enter two times, and you get an HTTP answer back. You can see that we would be redirected to the HTTPS site, so HTTP is working fine – even with IPv6 if you have noticed. Without any redirect, we would be getting the HTML code of the site after the HTTP header, directly into our telnet window.

More protocols that work great debugging this way are the mail protocols SMTP, POP3 and IMAP, or the chat protocol IRC.

So what can’t telnet do?

• It has to be a TCP service, as telnet always opens a TCP socket. If you need to test UDP, have a look at netcat – it can do TCP like telnet too, but you can switch it to UDP. But – as there is no connection establishment with UDP – you won’t know if you are really connected. So you need to control the receiving end too (for example, with netcat running there too), or have a UDP service running sending something back (for example, echo).
• If you want to try troubleshooting a layer 7 protocol like above, it has to be unencrypted. A way around that would be stunnel – it handles the SSL/TLS for you, and you can then connect through it with telnet.
• If you suspect that some certificate change or mistake is the culprit, the openssl command is great in troubleshooting that. It is not only to create and manage certificates, but you can use it to connect on any TCP or UDP port and get the certificate, chain, and cipher information when an encrypted connection is expected. openssl s_client -connect artofnetworkengineering.com:443 is all you need to see the public server certificate of this website.

These tools – telnet, netcat, stunnel, openssl – are in part already installed on Linux/UNIX distributions (even macos), or easily installed. To get them on Windows, the easiest way for me is always to install WSL, the Windows Subsystem for Linux. That way you’ll have a great shell too.

Of course, for all of this dedicated tools exist – nmap, curl, swaks, to name a few – but nothing beats telnet for a quick check, as it is easy enough and most of the time already installed.