|
cURL, the Swiss Army Knife of Download Utilities
Clever cURL Tricks
Carla Schroder
Tuesday, December 9, 2008 11:55:33 AM
The "client for URLs," or cURL utility, is one of those deceptively simple, handy little programs. The more you dig into it, the more you learn about what it can do. cURL is a download and copying utility that recognizes a host of transfer protocols —
HTTP, HTTPS, FTP, FTPS, TFTP, GopherR, Telnet, DICT, LDAP, and FILE. It also supports, and I quote from the cURL Web site:
"SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos ...), file transfer resume, proxy tunneling and a busload of other useful tricks."
Some folks compare cURL to wget, but they're really different beasts. wget is a mirroring tool and file downloader. cURL can do all manner of clever tricks in addition to merely downloading files. Its simplest invocation fetches Web pages, as this snippet shows:
$ curl http://www.serverwatch.com/
<b style="font-size:10pt;"></a
href="http://cp.jupiterweb.com/index.php/3681_default">Winners announced June 5!
</a></b></p>--> <P><span class=header>New on
ServerWatch</span><p><a
href="http://www.serverwatch.com/trends/article.php/3695711"><span
class=topic>Greening Your Data Center — You May Have No
Choice</span></a><br><span class=body>[August 22,
2007]</span>
|
As you can see, cURL fetches the raw page code. This could be useful if you don't want to use the "View page source" button in a graphical Web browser. But cURL can perform even more useful tasks, such as measuring download performance on a Web site:
$ $ curl -w 'nPreXfer time:t%{time_pretransfer}nStartXfer
time:t%{time_starttransfer}nnTotal time:t%{time_total}n' -o /dev/null -s
http://serverwatch.com/
PreXfer time: 0.138
StartXfer time: 0.286
Total time: 0.286
|
This shows that the file transfer starts almost as soon as the user connects, which is a good thing. You can fetch server banners and see what different sites are running, or see what your own server is revealing to the world:
$ curl -I www.serverwatch.com
HTTP/1.1 200 OK
Date: Thu, 23 Aug 2007 20:12:03 GMT
Server: Apache
Content-Type: application/x-httpd-php
|
That's a good banner. It doesn't give away too much information. Visit cURL for downloads and documentation and to learn hundreds of nifty ways to use cURL.
Article courtesy of Serverwatch.com
|
