Curl Stands for “Client URL”. It is a command tool that allows you to transfer data to and from a server. Curl is available in many Linux distributions and macOS systems. It supports various protocols like HTTP, HTTPS, FTP, and more, making it a versatile tool for testing APIs, downloading files, automating web scraping tasks, etc.
This guide provides a quick cheat sheet for some common usages of curl.
Basic Operations
These are some of the basic ways to use curl to fetch a URL or download a file from the internet/remote servers.
Fetch a URL
To fetch a url, you just pass the url as argument to the curl command. Here is an example:
curl <http://example.com>This will make a GET request to the given URL and print the response to standard output (stdout). It is the simplest usage of curl to retrieve data from a remote resource.
Download a File
To download a file from a remote server and save it you use the -O option:
curl -O <http://example.com/file.zip>The -O option tells curl to save the response to a local file named file.zip.
Follow Redirects
Sometimes you might encounter URLs that redirect to different endpoints. To follow these redirects, the -L options comes in handy. Here is an example:
curl -L <http://example.com>The -L option tells curl to automatically handle and follow any redirect responses like 301, 302 and so on. So if the URL redirects to another location, curl will follow it until it reaches the final destination rather than stopping at the initial redirect.
Data Transfer
The curl command also comes handy when you want to transfer data to and from a remote servers. Here are some examples of doing so.
POST Data
To post a simple key-value form data to the server, use the -d option and pass the data string to send. For example:
curl -d "key1=value1&key2=value2" <http://example.com/post_endpoint>The above command does a POST request to the “/post_endpoint” endpoint while passing the data "key1=value1&key2=value2". This is handy if you want to send data to API endpoints encoded as regular text.
POST JSON Data
To send JSON data, we use the -H option to add a Content-Type header giving it a value of “application/json” and pass the JSON string we want to post to the -d option.
curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" <http://example.com/api>This posts a JSON data payload to the API while setting the expected content type as application/json in the header. The JSON data format is commonly used for passing structured data to APIs and web services.
Upload File
To upload files to a remote server with curl use the -F (—form) option. For example, if to upload a file in the user “james” home directory to the server “http://example.com/upload” which processes file input with a form parameter named file you would run:
curl -F "file=@/home/james/data.txt" <http://example.com/upload>Authentication & Headers
Curl provide you with a way to pass authentication details and custom HTTP headers with your requests.
HTTP Basic Authentication
The -u option allows you pass a username and a password for authenticating to a serve using HTTP basic auth( a common simple aunthentication method supported by curl).
Here is an example of using the -u option to authenticate to a remote server:
curl -u username:password <http://example.com>This authenticates to example.com using the username and password credentials joined by a colon.
Add Headers
Curl lets you add any custom headers required for the remote server by using the -H option. You can also add multiple headers by using multiple -H options. Here is an example of adding custom headers:
curl -H "Authorization: Bearer YOUR_TOKEN" <http://example.com>Here we add an Authorization header with a bearer token which can be used for API authentication.
Debugging & Info
When testing APIs or transfering data, enabling curl's verbose mode can help inspect the requests in detail and debugging issues .
Verbose Mode
To enable the verbose mode use the -v option with curl. For example:
curl -v <http://example.com>Verbose mode prints all request and response headers sent and received by the server. This helps inspect authorization headers, response codes, cookies set etc. to debug requests.
Retrieve Only Headers
The -I options tells tells curl to make a HEAD request, printing only the response headers instead of the full response body.
Example:
curl -I <http://example.com>This returns only the response headers, useful for quickly checking response codes and header values without pulling large response content.
Curl Version & Protocols
To print the curl version number as well as all supported protocols use the —version option:
curl --versionPrinting the version allows you to see which curl release you have installed on your operating system and what features it was compiled with, such as SSL, HTTP/2, brotli compression support etc.
SSL
Skip SSL Certificate Verification
By default, curl will verify SSL certificates when connecting to HTTPS sites. But you can skip the certificate verification and validation for debugging or testing. To skip SSL Certificate verication use the -k option or —insecure option:
curl -k <https://example.com>This will skip the security certificate check, allowing insecure server connections when using HTTPS. It is worth noting that , this should only be used for testing, not production traffic.
Use SSL Certificate
If you want to use your custom certificate for client authentication, use the --cert option followed by the location to your public key pem file.
curl --cert mycert.pem <https://example.com>The --cert allows you to specifies your own certificate file for authenticating to servers that require client SSL certificates to verify your client identity.
Other Useful Options
Here are some other handy curl options you should be aware of.
Limit Rate (e.g., 1MB/s)
To limit the data transfer limit to a maximum rate in bytes per second you use the the —limit-rate option. Here is an example to limit the data transfer speed to 1MB/s:
curl --limit-rate 1M -O <http://example.com/file.zip>This throttles the download rate to a maximum of 1 Megabyte per second. Helpful if you want slower transfers or have limited bandwidth connection. You can replace the M with either K or G to limit the transfer to Kilobytes per second (1KB/s) and Gigabytes per second (1GB/s) respectively.
Resume Broken Download
If a download gets interrupted, use -C - to resume and continue the download at the already downloaded partial file size. For example:
curl -C - -O <http://example.com/file.zip>The -C option allows resuming a failed or broken download from where it stopped and there is no need to re-download the entire file if part of it was already downloaded.
Using a Proxy
To transfer data through a proxy server, use --proxy or -x option followed by the proxy URL and port number.
curl -x <http://proxyserver>:port <http://example.com>This routes the curl request through the configured proxy server instead of connecting directly. This is Useful where a proxy is required to access the internet.
Comments