Comparing curl to telnet
context and whys
why do we care?
Turns out new firewalls ( aka NG or "Next Generation" Firewalls ) like PAN ( Palo Alto Networks ) examine protocols that go by. On a PAN this is called "application" . So you have have a rule that reads:
from: host1 to: host2 tcp port 80 Application ssh action: deny
note the extra "Application" field. It used to be that when you said "port" you were implying the application that was being used. The truth is you can run any application over any port pretty much.
This rule means : do not allow ssh traffic to go over port 80.
A common trick to get around an old firewall that could not "detect" the application that was blocking port 22 outbound was to "do" ssh over a non-stadard port, say port 80. Which is typically allowed outbound. With an NGFW like a PAN the firewall admin can block that sort of activity.
Ideally the rule would be:
from: host1 to: host2 tcp port 80 Application http action: allow
So that ONLY http can go over port 80. The pan can tell the difference between http going over port 80 and ssh going over port 80. it inspects the payload.
So let's say I'm troubleshooting an application. Prior to NGFWs I could use telnet to port 80 , or port 22 or port 25, to "test" the application that was having trouble. I know how to talk http and smtp .. so I could pretend to be the application.
In the modern "application aware" firewall world, The PAN can tell that I'm using telnet instead of a proper web browser, or a proper MTA. How can it do that? All it has is the characteristics of the network traffic. How does telnet look different than a "proper" web browser or MTA? That's what this page is about.
steps
capture curl run:
cd /tmp tcpdump -i lo -s 0 -w curl.out tcp port 80
then in another window:
curl localhost ... output
capture telnet run:
tcpdump -i lo -s 0 -w telnet.out tcp port 80
then in another window:
telnet localhost 80 GET / HTTP/1.0<enter> <enter>
This results in two files:
- curl.out
- telnet.out
we can compare a simple tcpdump parse:
tcpdump -r curl.out > curl.out.parse tcpdump -r telnet.out > telnet.out.parse
diff:
1,10c1,14
< 17:00:42.394546 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [SEW], seq 1083582039, win 43690, options [mss 65495,sackOK,TS val 178872410 ecr 0,nop,wscale 7], length 0
< 17:00:42.394633 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [S.E], seq 2360024596, ack 1083582040, win 43690, options [mss 65495,sackOK,TS val 178872410 ecr 178872410,nop,wscale 7], length 0
< 17:00:42.394685 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [.], ack 1, win 342, options [nop,nop,TS val 178872410 ecr 178872410], length 0
< 17:00:42.394923 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [P.], seq 1:147, ack 1, win 342, options [nop,nop,TS val 178872410 ecr 178872410], length 146
< 17:00:42.395024 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [.], ack 147, win 350, options [nop,nop,TS val 178872410 ecr 178872410], length 0
< 17:00:42.395612 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [P.], seq 1:1307, ack 147, win 350, options [nop,nop,TS val 178872410 ecr 178872410], length 1306
< 17:00:42.395982 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [.], ack 1307, win 362, options [nop,nop,TS val 178872410 ecr 178872410], length 0
< 17:00:42.398433 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [F.], seq 147, ack 1307, win 362, options [nop,nop,TS val 178872411 ecr 178872410], length 0
< 17:00:42.398591 IP 127.0.0.1.http > 127.0.0.1.38992: Flags [F.], seq 1307, ack 148, win 350, options [nop,nop,TS val 178872411 ecr 178872411], length 0
< 17:00:42.398655 IP 127.0.0.1.38992 > 127.0.0.1.http: Flags [.], ack 1308, win 362, options [nop,nop,TS val 178872411 ecr 178872411], length 0
---
> 17:02:41.287111 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [SEW], seq 3898897559, win 43690, options [mss 65495,sackOK,TS val 178884300 ecr 0,nop,wscale 7], length 0
> 17:02:41.287195 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [S.E], seq 4209729007, ack 3898897560, win 43690, options [mss 65495,sackOK,TS val 178884300 ecr 178884300,nop,wscale 7], length 0
> 17:02:41.287245 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [.], ack 1, win 342, options [nop,nop,TS val 178884300 ecr 178884300], length 0
> 17:02:42.686699 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [S.E], seq 4209729007, ack 3898897560, win 43690, options [mss 65495,sackOK,TS val 178884440 ecr 178884300,nop,wscale 7], length 0
> 17:02:42.686791 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [.], ack 1, win 342, options [nop,nop,TS val 178884440 ecr 178884300], length 0
> 17:02:45.447130 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 1:17, ack 1, win 342, options [nop,nop,TS val 178884716 ecr 178884300], length 16
> 17:02:45.447223 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [.], ack 17, win 342, options [nop,nop,TS val 178884716 ecr 178884716], length 0
> 17:02:46.868780 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 17:19, ack 1, win 342, options [nop,nop,TS val 178884858 ecr 178884716], length 2
> 17:02:46.868868 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [.], ack 19, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.869312 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [P.], seq 1:1326, ack 19, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 1325
> 17:02:46.869413 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [.], ack 1326, win 363, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.869583 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [F.], seq 1326, ack 19, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.870309 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [F.], seq 19, ack 1327, win 363, options [nop,nop,TS val 178884858 ecr 178884858], length 0
> 17:02:46.870400 IP 127.0.0.1.http > 127.0.0.1.39001: Flags [.], ack 20, win 342, options [nop,nop,TS val 178884858 ecr 178884858], length 0
If we now go back to the cap file and dump with ascii:
tcpdump -r curl.out:
GET / HTTP/1.1 User-Agent: curl/7.24.0 (i686-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 Host: 127.0.0.1 Accept: */* HTTP/1.1 200 OK Date: Tue, 25 Mar 2014 22:00:42 GMT Server: Apache Last-Modified: Wed, 03 Feb 2010 18:09:42 GMT ETag: "cf022-43e-47eb61f5a4580" Accept-Ranges: bytes Content-Length: 1086 Content-Type: text/html <html> ...
and the telnet response:
HTTP/1.1 200 OK Date: Tue, 25 Mar 2014 22:02:45 GMT Server: Apache Last-Modified: Wed, 03 Feb 2010 18:09:42 GMT ETag: "cf022-43e-47eb61f5a4580" Accept-Ranges: bytes Content-Length: 1086 Connection: close Content-Type: text/html <html> ...
So what is the important difference?
Telnet sends one a packet after each line return. Curl sends the http request in the fewest number of tcp packets ( upto the MTU size)
recall this what we sent:
"GET / HTTP/1.0" ---> 14 characters + 1 for the line return + 1 mystery byte † "<empty line>" ----> 1 line return + 1 mystery byte
from the parsed telnet tcpdump:
> 17:02:45.447130 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 1:17, ack 1, win 342, options [nop,nop,TS val 178884716 ecr 178884300], length 16
and the final empty line:
> 17:02:46.868780 IP 127.0.0.1.39001 > 127.0.0.1.http: Flags [P.], seq 17:19, ack 1, win 342, options [nop,nop,TS val 178884858 ecr 178884716], length 2
the curl request:
GET / HTTP/1.1 User-Agent: curl/7.24.0 (i686-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 Host: 127.0.0.1 Accept: */*
... 145 characters including line returns + 1 mystery byte.
† - I think the mystery byte is a null character that goes at the end of every tcp packet. ( udp too ? )
