The Requests Python package is the first choice when we want to perform HTTP post requests in Python. I use it every day in my web-scraping / data-collection journey.
This guide will cover everything you need to know to master sending data with the POST HTTP request.
Quick example: performing a POST request with the Requests module
Here, we make a simple POST request by sending JSON data, and then we print the response.
The data argument is useful when submitting form data or sending payloads that don’t require JSON encoding. This is common when interacting with websites that use traditional HTML forms.
The data argument is useful when submitting form data or sending payloads that don’t require JSON encoding. This is common when interacting with websites that use traditional HTML forms.
In this short section, we’ll see an example of how to specify the data format and the data encoding.
Suppose we have a heavy JSON to send and use the gzip compressing to optimize the internet traffic.
đź’ˇ GZIP is a compression technology frequently used to send and receive data quickly over the internet.
In the following example, we will compress the dictionary {”heavy”: “data”}, using gzip, to send it faster to the endpoint.
Python
import requests
import gzip
import json
url = "https://enxsgnb905ynf.x.pipedream.net"
data = {"heavy": "data"}
# The gzip.compress(...) method takes `bytes` in argument
json_data = json.dumps(data).encode("utf-8")
gzipped_data = gzip.compress(json_data)
# Here we define the headers including 'Content-Encoding'
# to signal gzip compression
headers = {
"Content-Type": "application/json", # The data format
"Content-Encoding": "gzip", # The data is gzipped
}
# Perform the POST request
response = requests.post(url, data=gzipped_data, headers=headers)
print(response.text)
In this code example, we use the compress method from the gzip module for compression.
Which takes in argument, a bytes object. That’s why we have to first encode the dict like this: json_data = json.dumps(data).encode("utf-8").
Now that our json_data is bytes, we can use gzip: gzipped_data = gzip.compress(json_data).
At this point, our data is ready to be sent. But, before performing the request, we must set some headers. They tell the server it is receiving JSON data, encoded and compressed with gzip.
So we set the header "Content-Type": "application/json" to specify that the content format is JSON.
Then, the header "Content-Encoding": "gzip" tells the server to decode the binary data using gzip.
Here is what we received in the request bin page:
We can see that the server successfully read our {”heavy”: “data”}.
Setting cookies with POST requests method
You can also perform POST HTTP requests by passing a cookie dictionary. For example, this is needed to submit a form as an authenticated user.
In this short example, we create a Session object. We import it from the requests library: from requests import Session.
Then, we update the session headers, with Content-Type set to application/json: session.headers.update(headers).
By doing that, we don't need to specify the headers in the requests we perform with the session.
Finally, we perform the request: session.post(url, data=data), which is similar to the requests.post(...) syntax we saw before.
Definition and usage of the POST HTTP method
The POST HTTP method is used when you want to send data inside the request's body.
The post(…) function sends an HTTP request with the POST method. It allows you to encode the request's body content as needed automatically.
Overview of other HTTP requests
If you're not familiar with the HTTP protocol, note that it has different methods for different purposes.
The two most common HTTP methods are the GET and POST methods. Yet, other HTTP methods are helpful in the context of REST APIs.
GET: The most used HTTP method. It’s used to retrieve data. You can also send data directly to the URL using this HTTP method. It’s called query params.
POST: As we discussed, the POST method is used to send data or perform a “create” action.
PUT: The PUT HTTP method is similar to the POST; it performs an “update” action.
PATCH: Here, it also performs an update but a “partial update” action.
DELETE: The name is quite explicit. It is to perform a “delete” action.
OPTIONS: A less common method. Some API endpoints respond to it. They provide info on how to use the endpoint.
HEAD: The usage is quite specific here. It retrieves the metadata of a resource from a server, such as a file name or the file size stored in a server.
Here is a small example of how to use each of those HTTP methods with the Requests package:
Here are all the parameters you can use with the post(…) method:
Parameter
Is required
Description
url
Required
The URL to request
data
Optional
A dictionary, list of tuples, bytes, or file-like object to send in the body to the specified url
json
Optional
A JSON serializable Python object to send in the body of the request
headers
Optional
A dictionary of HTTP headers to send with the request
cookies
Optional
A dictionary or CookieJar object to send with the request
files
Optional
A dictionary of {'name': file-like-objects} (or {'name': file-tuple}) for multipart encoding upload.
auth
Optional
A tuple to enable a certain HTTP authentication.Default None
timeout
Optional
A float, or a tuple, how many seconds to wait for the server to send data before giving up, as a float. As a tuple: (<connect timeout>, <read timeout>) Default None which means the request will continue until the connection is closed
A boolean indication if the response should be immediately downloaded (False) or streamed (True).Default False
verify
Optional, default: True
Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Default True
cert
Optional
If string, path to ssl client cert file (.pem). If tuple, ('cert', 'key') pair. Default None
The return value of all HTTP requests made with the Requests package is a Response object.