Comparing curl to telnet

context and whys

why do we care?

Turns out new firewalls ( aka NG or "Next Generation" Firewalls ) like PAN ( Palo Alto Networks ) examine protocols that go by. On a PAN this is called "application" . So you have have a rule that reads:

from: host1
to: host2
tcp port 80
Application ssh
action: deny

note the extra "Application" field. It used to be that when you said "port" you were implying the application that was being used. The truth is you can run any application over any port pretty much.

This rule means : do not allow ssh traffic to go over port 80.

A common trick to get around an old firewall that could not "detect" the application that was blocking port 22 outbound was to "do" ssh over a non-stadard port, say port 80. Which is typically allowed outbound. With an NGFW like a PAN the firewall admin can block that sort of activity.

Ideally the rule would be:

from: host1
to: host2
tcp port 80
Application http
action: allow

So that ONLY http can go over port 80. The pan can tell the difference between http going over port 80 and ssh going over port 80. it inspects the payload.

So let's say I'm troubleshooting an application. Prior to NGFWs I could use telnet to port 80 , or port 22 or port 25, to "test" the application that was having trouble. I know how to talk http and smtp .. so I could pretend to be the application.

In the modern "application aware" firewall world, The PAN can tell that I'm using telnet instead of a proper web browser, or a proper MTA. How can it do that? All it has is the characteristics of the network traffic. How does telnet look different than a "proper" web browser or MTA? That's what this page is about.

steps

capture curl run:

cd /tmp
tcpdump -i lo -s 0 -w curl.out tcp port 80

then in another window:

curl localhost
... output

capture telnet run:

tcpdump -i lo -s 0 -w telnet.out tcp port 80

then in another window:

telnet localhost 80
GET / HTTP/1.0<enter>
<enter>

This results in two files:

1. curl.out
2. telnet.out

we can compare a simple tcpdump parse:

tcpdump -r curl.out > curl.out.parse
tcpdump -r telnet.out > telnet.out.parse

diff:

1,10c1,14 < 17:00:42.394546 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [SEW], seq 1083582039, win 43690, options [mss 65495,sackOK,TS val 178872410 ecr 0,nop,wscale 7], length 0
< 17:00:42.394633 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [S.E], seq 2360024596, ack 1083582040, win 43690, options [mss 65495,sackOK,TS val 178872410 ecr 178872410,nop,wscale 7], length 0
< 17:00:42.394685 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [.], ack 1, win 342, options [nop,nop,TS val 178872410 ecr 178872410], length 0
< 17:00:42.394923 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [P.], seq 1:147, ack 1, win 342, options [nop,nop,TS val 178872410 ecr 178872410], length 146
< 17:00:42.395024 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [.], ack 147, win 350, options [nop,nop,TS val 178872410 ecr 178872410], length 0
< 17:00:42.395612 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [P.], seq 1:1307, ack 147, win 350, options [nop,nop,TS val 178872410 ecr 178872410], length 1306
< 17:00:42.395982 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [.], ack 1307, win 362, options [nop,nop,TS val 178872410 ecr 178872410], length 0
< 17:00:42.398433 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [F.], seq 147, ack 1307, win 362, options [nop,nop,TS val 178872411 ecr 178872410], length 0
< 17:00:42.398591 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [F.], seq 1307, ack 148, win 350, options [nop,nop,TS val 178872411 ecr 178872411], length 0
< 17:00:42.398655 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [.], ack 1308, win 362, options [nop,nop,TS val 178872411 ecr 178872411], length 0
---
> 17:02:41.287111 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [SEW], seq 3898897559, win 43690, options [mss 65495,sackOK,TS val 178884300 ecr 0,nop,wscale 7], length 0
> 17:02:41.287195 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [S.E], seq 4209729007, ack 3898897560, win 43690, options [mss 65495,sackOK,TS val 178884300 ecr 178884300,nop,wscale 7], length 0
> 17:02:41.287245 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [.], ack 1, win 342, options [nop,nop,TS val 178884300 ecr 178884300], length 0
> 17:02:42.686699 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [S.E], seq 4209729007, ack 3898897560, win 43690, options [mss 65495,sackOK,TS val 178884440 ecr 178884300,nop,wscale 7], length 0
> 17:02:42.686791 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [.], ack 1, win 342, options [nop,nop,TS val 178884440 ecr 178884300], length 0
> 17:02:45.447130 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 1:17, ack 1, win 342, options [nop,nop,TS val 178884716 ecr 178884300], length 16
> 17:02:45.447223 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [.], ack 17, win 342, options [nop,nop,TS val 178884716 ecr 178884716], length 0
> 17:02:46.868780 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 17:19, ack 1, win 342, options [nop,nop,TS val 178884858 ecr 178884716], length 2
> 17:02:46.868868 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [.], ack 19, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.869312 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [P.], seq 1:1326, ack 19, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 1325
> 17:02:46.869413 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [.], ack 1326, win 363, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.869583 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [F.], seq 1326, ack 19, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.870309 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [F.], seq 19, ack 1327, win 363, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.870400 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [.], ack 20, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 0

If we now go back to the cap file and dump with ascii:

tcpdump -r curl.out:

GET / HTTP/1.1
User-Agent: curl/7.24.0 (i686-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28
Host: 127.0.0.1
Accept: */*

HTTP/1.1 200 OK
Date: Tue, 25 Mar 2014 22:00:42 GMT
Server: Apache
Last-Modified: Wed, 03 Feb 2010 18:09:42 GMT
ETag: "cf022-43e-47eb61f5a4580"
Accept-Ranges: bytes
Content-Length: 1086
Content-Type: text/html

<html>
...

and the telnet response:

HTTP/1.1 200 OK
Date: Tue, 25 Mar 2014 22:02:45 GMT
Server: Apache
Last-Modified: Wed, 03 Feb 2010 18:09:42 GMT
ETag: "cf022-43e-47eb61f5a4580"
Accept-Ranges: bytes
Content-Length: 1086
Connection: close
Content-Type: text/html

<html>
...

So what is the important difference?

Telnet sends one a packet after each line return. Curl sends the http request in the fewest number of tcp packets ( upto the MTU size)

recall this what we sent:

"GET / HTTP/1.0" ---> 14 characters + 1 for the line return + 1 mystery byte †
"<empty line>" ----> 1 line return + 1 mystery byte

from the parsed telnet tcpdump:

> 17:02:45.447130 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 1:17, ack 1, win 342, options [nop,nop,TS val 178884716 ecr 178884300], length 16

and the final empty line:

> 17:02:46.868780 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 17:19, ack 1, win 342, options [nop,nop,TS val 178884858 ecr 178884716], length 2

the curl request:

GET / HTTP/1.1
User-Agent: curl/7.24.0 (i686-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28
Host: 127.0.0.1
Accept: */*

... 145 characters including line returns + 1 mystery byte.

† - I think the mystery byte is a null character that goes at the end of every tcp packet. ( udp too ? )