cURL: A Powerful Diagnostics Tool

What is cURL?

cURL stands for Client for URLs, a command used for transferring data through various protocols. This tool is supported by a wide range of devices and can be used to test for numerous cases, including:

• Proxy Support
• SSL Communications
• HTTP/HTTPS Functions
• HTTP Headers, etc.

The following protocols are supported by this tool: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP.

This article attempts to investigate common methods of using this tool.

Installing cURL on Windows

If the device that you want to diagnose or test is not based on the Linux/Unix operating system, you will first need to install this tool, as this option is not available in the Windows command prompt by default.

To install this tool on Windows, you will need to download the package from: https://curl.haxx.se/dlwiz/

The steps for downloading the file from this address are as follows:

• Go to Select Type of Package and click curl executable.
• Here, select either Windows / Win32 or Win64 in Select Operating System based on your operating system and click Select.
• Here, in Select for What Flavor, choose Generic and click Select.
• Select and download your desired version of cURL from this page.

Examples of cURL’s General Applications

All commands and examples provided in this article can be entered directly into the terminal.

• HTTP GET Request

One of the simplest applications of the curl command is to use it for sending an HTTP GET Request in exchange for a website’s address (or a specific URL).

curl [http or https]://www.example.com

After running the command, the HTTP Response received form the requested URL will be shown in the output.

• Recovering Just the HTTP Header for a Specific URL

Using the -I option in the curl command makes it possible to only see a specific URL’s HTTP headers (HEAD method).

curl –I [http or https]://www.example.com

Note: Don’t use the –I command while debugging CDN behavior with the curl command, since it sends a HEAD request. Users often send a GET Request instead of a HEAD Request, and CDN behaves differently with HEAD Requests than with GET Requests. You could therefore see meaningless results in the command output.

Always use the –svo option to debug CDN behavior in the curl command (the s options represents silent mode and stops errors from being displayed in the output).

• Saving the cURL Command Output

The curl command output can be saved in file format using the –o and –O options. The difference between these two options is that the –o option saves the output file with a name that we set, while the –O option saves the output file with the name it already has:

curl –o [file name ex: myfile.css] [http or https]://cdn.arvancloud.ir/css/example.css
curl –O [http or https]://cdn.arvancloud.ir/css/example.css

• Modifying the cURL Command Output

It is sometimes only necessary to see some of the information from the cURL command in a URL. The grep (or egrep) option can be used in this command:

curl -o /dev/null http://example.com/img/test.jpg 2>&1 | grep "CF-"

The 1&<2 phrase in the command above represents directing the standard error (stderr) to standard output (stdout).

• Adding an Extra HTTP Header

Extra headers can be added to the GET Request in the cURL command using the –H (or –header) option. This option can be used while testing APIs to send specific request headers, such as: Set-Cookie, HOST or even a custom header:

curl –H "HEADER: VALUE" [http or https]://www.example.com

Note: Whenever Value is not specified in the command above, the semi-colon (;) should be used after setting the intended header.

• Setting a Custom Address for a URL

The cURL command’s –resolve option is used for setting the command to request the URL from a specified address, instead of DNS or the hosts/etc/ file.

curl --resolve hostname:port:DESTINATIONIPADDRESS http(s)://www.example.com

• Receiving More Information

It is sometimes necessary to see more detail regarding a request, like sent request headers and the connection process. In this regard, the –v (or –verbose) option can be used in the cURL command:

curl -v https://www.example.com/

• Saving HTTP Headers

Headers received from the queried URL can be saved using the –D (or –dump-header) option. For example, this option can be used to read headers first, and then read their cookies using the –b (or –cookie) option.

curl –D -  https://www.example.com/

In the command above, – alone represents saving the file in stdout, where out kernels are stored, which is usually the terminal that the command is executed on.

• GET Method

GET is used in HTTP to recover a specific URL’s resources. The GET method is normally used in the cURL command to receive information related to the URL. Regardless, the –GET request (or –GET X) option can be used to explicitly set the method used by HTTP to the GET method:

curl --request GET https://www.example.com/

• POST Method

The POST method is used for sending information to the web server. The –POST request (or –POST X) can be used to set the cURL command to use this method:

curl --request POST https://www.example.com/

• PUT Method

The PUT method can be used to create or update a resource based on the data sent from the user to the server, such as create a new webpage for the website or updating an existing webpage. The –PUT request (or –PUT X) option can be used to set the cURL command to use this method:

curl --request PUT https://www.example.com/

• DELETE Method

A resource related to a particular URL can be deleted from the web server using the DELETE method. You can use this method in the curl command with the –DELETE request (or –DELETE X) option:

curl --request DELETE https://www.example.com

• Using cURL for Sending Data

The –d (or –data) option can be used in the cURL command to send specific data, such as login information, to the server through POST, as shown in the example below:

curl -X POST http://www.example.com/login/ -d 'username=yourusername&password=yourpassword'

• Resuming File Download

If, while downloading a file/source, the process stops for any reason, the –C option can be used to resume the download process:

curl -C - -O https://cdn.arvancloud.ir/img/example.png

• Test Download Time Without Receiving a File

When saving an output file isn’t necessary, for example, when the intention is only to test the download speed without receiving a file, the /dev/null option can be used in the cURL command:

curl -o  https://www.example.com /dev/null

• Setting Maximum Transfer Rate

The –limit-rate option can be used to set the maximum transfer rate for uploads and downloads. This rate is measured by default in bytes per second. The K (kilobyte), M (megabyte) or G (gigabyte) letters can be used after the desired number to change the default behavior:

curl --limit-rate 200K -O https://cdn.arvancloud.ir/img/test.png

• Checking HTTP/2 Support

If you are using the latest version of cURL, you can use the –http2 option to be informed of the URL’s support for HTTP/2. If HTTP/1.1 200 is displayed instead of HTTP/2 200 after executing this command, it means that the website supports HTTP/2:

curl -I --http2 https://www.example.com/

• Recovering Parts of a File

The –r option can be used in the cURL command to recover a specific part of a file. The specific part of a file that needs to be recovered should be specified in a range of bytes, like 0-499:

curl -r 0-20000 -o test.png https://cdn.arvancloud.ir/img/cdn-stats.png

• Running the cURL Command with a Specific User-Agent String

Sometimes, web servers or web applications that the cURL command is sent to have rules that prevents them from accepting the default cURL user agent string. In this case, the –user-agent option can be used to set the user-agent string in the cURL command:

curl --user-agent "USERAGENTSTRING"

HTTP Request Header is the user-agent string that the user agent (e.g. the browser) sends to the web server. It contains information such as the operating system used, browser version, etc.

• Help

The –h option can be used to view a list of various options for use in the curl command, and an explanation for each:

curl -h

• Troubleshooting HTTP Errors

Sometimes a website’s main host server needs to be queried directly to diagnose HTTP problems. In this regard, the website’s main host server IP address should be specified first. If the address is unknown, it can be found by looking at your domain’s DNS records in ArvanCloud user panel and DNS settings. You can then use the –header and –resolve options to send the request directly to the main server.

curl -svo /dev/null --header "Host: example.com" http://ServerIPAddress/
curl -svo /dev/null --resolve example.com:80:ServerIPAddress http://example.com/

Functional Diagnosis

• Time Command

The delayed and weak performance of HTTP/HTTPS requests can be measured using cURL. A simple method is to use the time command with the curl command. Using both these commands at the same time will display the total time required for completing a request:

time curl -svo /dev/null http://example.com/

• The –w or –write-out Option

This option can be used to set the curl command to test the performance of various variables. A complete list of these variables can be found in curl.haxx.se. An example of using these options in the curl command is provided below (/n is used before each variable to show each result more clearly in a separate line in command output):

curl -svo /dev/null https://example.com/ -w "\nContent Type: %{content_type} \
\nHTTP Code: %{http_code} \
\nHTTP Connect:%{http_connect} \
\nNumber Connects: %{num_connects} \
\nNumber Redirects: %{num_redirects} \
\nRedirect URL: %{redirect_url} \
\nSize Download: %{size_download} \
\nSize Upload: %{size_upload} \
\nSSL Verify: %{ssl_verify_result} \
\nTime Handshake: %{time_appconnect} \
\nTime Connect: %{time_connect} \
\nName Lookup Time: %{time_namelookup} \
\nTime Pretransfer: %{time_pretransfer} \
\nTime Redirect: %{time_redirect} \
\nTime Start Transfer: %{time_starttransfer} \
\nTime Total: %{time_total} \
\nEffective URL: %{url_effective}\n" 2>&1

Using cURL for Diagnosing SSL/TLS

cURL can be used for diagnosing SSL/TLS certificates. For example, the following command can be used to obtain valuable information regarding a specific URL’s TLS handshake and information related to the TLS/SSL certificate:

curl -svo /dev/null https://www.example.com/ 2>&1 | egrep -v “^{.*$|^}.*$|^\* http.*$”

The –resolve option can also be used in the command above to investigate the SSL/TLS certificate of the website’s main host server.

TLS Version Test

If there is a need to test the website for supporting a specific version of TLS, the desired TLS version can be specified in the curl command. Versions that can be specified in this command are as follows:

• –tlsv1.0
• –tlsv1.1
• –tlsv1.2
• –tlsv1.3

An example of using one of these options in the curl command is shown below.

curl -svo /dev/null –tls1.2 https://www.example.com/ 2>&1 | egrep -v “^{.*$|^}.*$|^\* http.*$”