How to make a POST with the Python Requests package?

By
Jérémy
Reviewed By
Updated
October 22, 2024
12 min read

The Requests Python package is the first choice when we want to perform HTTP post requests in Python. I use it every day in my web-scraping / data-collection journey.

This guide will cover everything you need to know to master sending data with the POST HTTP request.

Quick example: performing a POST request with the Requests module

Here, we make a simple POST request by sending JSON data, and then we print the response.

Python
    import requests

url = "https://httpbin.org/post"
my_data = {"john": "doe"}

response = requests.post(url, json=my_data)

print(response.json())
  

The output should look like this:

Bash
    {'args': {}, 'data': '{"john": "doe"}', 'files': {}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '15', 'Content-Type': 'application/json', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.32.3', 'X-Amzn-Trace-Id': 'Root=1-66eafda9-03c9f45c5ba30bca01a205c5'}, 'json': {'john': 'doe'}, 'origin': 'IP-CENSORED', 'url': 'https://httpbin.org/post'}
  

POST request with JSON argument

We will cover the json parameter's specifics. It is often used to write data when communicating with an API endpoint.

When communicating with APIs that expect JSON data, you should set some headers like:

We will cover the json parameter's specifics. It is often used to write data when communicating with an API endpoint.

When communicating with APIs that expect JSON data, you should set some headers like:

Python
    headers = {
    "Accept": "application/json",
    "Content-Type": "application/json",
}
  

But with this json parameter, you don’t need to do that. It sets the proper headers and encodes your dictionary or JSON object.

This gives us something straightforward like this:

Python
    headers = {
    "Accept": "application/json",
    "Content-Type": "application/json",
}
  

POST request form data with DATA argument

The data argument is useful when submitting form data or sending payloads that don’t require JSON encoding. This is common when interacting with websites that use traditional HTML forms.

The data argument is useful when submitting form data or sending payloads that don’t require JSON encoding. This is common when interacting with websites that use traditional HTML forms.

Python
    import requests

url = "https://httpbin.org/post"
data = {"username": "jeremylp", "password": "strongpassword"}

response = requests.post(url, data=data)

print(response.text)
  

This sends the data in application/x-www-form-urlencoded format, the default format for form submissions.

Here is the output:

Bash
    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "password": "strongpassword", 
    "username": "jeremylp"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "41", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.32.3", 
    "X-Amzn-Trace-Id": "Root=1-66eb33aa-7af65511275545d24f8a9f17"
  }, 
  "json": null, 
  "origin": "IP_CENSORED", 
  "url": "https://httpbin.org/post"
}
  

Defining Content-Type & Content-Encoding

In this short section, we’ll see an example of how to specify the data format and the data encoding.

Suppose we have a heavy JSON to send and use the gzip compressing to optimize the internet traffic.

đź’ˇ GZIP is a compression technology frequently used to send and receive data quickly over the internet.

In the following example, we will compress the dictionary {”heavy”: “data”}, using gzip, to send it faster to the endpoint.

Python
    import requests
import gzip
import json

url = "https://enxsgnb905ynf.x.pipedream.net"
data = {"heavy": "data"}

# The gzip.compress(...) method takes `bytes` in argument
json_data = json.dumps(data).encode("utf-8")
gzipped_data = gzip.compress(json_data)


# Here we define the headers including 'Content-Encoding'
# to signal gzip compression
headers = {
    "Content-Type": "application/json",  # The data format
    "Content-Encoding": "gzip",  # The data is gzipped
}

# Perform the POST request
response = requests.post(url, data=gzipped_data, headers=headers)

print(response.text)
  

In this code example, we use the compress method from the gzip module for compression.

Which takes in argument, a bytes object. That’s why we have to first encode the dict like this: json_data = json.dumps(data).encode("utf-8").

Now that our json_data is bytes, we can use gzip: gzipped_data = gzip.compress(json_data).

At this point, our data is ready to be sent. But, before performing the request, we must set some headers. They tell the server it is receiving JSON data, encoded and compressed with gzip.

So we set the header "Content-Type": "application/json" to specify that the content format is JSON.

Then, the header "Content-Encoding": "gzip" tells the server to decode the binary data using gzip.

Here is what we received in the request bin page:

example of results of a POST request using Python request

We can see that the server successfully read our {”heavy”: “data”}.

Setting cookies with POST requests method

You can also perform POST HTTP requests by passing a cookie dictionary. For example, this is needed to submit a form as an authenticated user.

Python
    import requests

url = "https://httpbin.org/post"
cookies = {"session_id": "705897e9c4cb41128a272b653bd2e65d"}

response = requests.post(url, json={"full_name": "John Doe"}, cookies=cookies)
  

Performing POST request using the Requests Session object

This last section will show the simple syntax for using the Session object to perform POST requests.

It’s efficient to reuse the same session object when making multiple requests to the same server.

The session object maintains parameters such as cookies and headers across multiple requests.

Python
    from requests import Session

url = "https://httpbin.org/post"
headers = {"Content-Type": "application/json"}
data = {"session": "object"}

session = Session()
session.headers.update(headers)

response = session.post(url, data=data)
  

In this short example, we create a Session object. We import it from the requests library: from requests import Session.

Then, we update the session headers, with Content-Type set to application/json: session.headers.update(headers).

By doing that, we don't need to specify the headers in the requests we perform with the session.

Finally, we perform the request: session.post(url, data=data), which is similar to the requests.post(...) syntax we saw before.

Definition and usage of the POST HTTP method

The POST HTTP method is used when you want to send data inside the request's body.

The post(…) function sends an HTTP request with the POST method. It allows you to encode the request's body content as needed automatically.

Overview of other HTTP requests

If you're not familiar with the HTTP protocol, note that it has different methods for different purposes.

The two most common HTTP methods are the GET and POST methods. Yet, other HTTP methods are helpful in the context of REST APIs.

  • GET: The most used HTTP method. It’s used to retrieve data. You can also send data directly to the URL using this HTTP method. It’s called query params.
  • POST: As we discussed, the POST method is used to send data or perform a “create” action.
  • PUT: The PUT HTTP method is similar to the POST; it performs an “update” action.
  • PATCH: Here, it also performs an update but a “partial update” action.
  • DELETE: The name is quite explicit. It is to perform a “delete” action.
  • OPTIONS: A less common method. Some API endpoints respond to it. They provide info on how to use the endpoint.
  • HEAD: The usage is quite specific here. It retrieves the metadata of a resource from a server, such as a file name or the file size stored in a server.

Here is a small example of how to use each of those HTTP methods with the Requests package:

Python
    import requests

resp = requests.get("https://httpbin.org/get", params={"query": "params"})
resp = requests.post("https://httpbin.org/post", json={"john": "doe"})
resp = requests.put("https://httpbin.org/put", data={"john": "doe"})
resp = requests.delete("https://httpbin.org/delete")
resp = requests.options("https://httpbin.org/options")
resp = requests.head("https://httpbin.org/head")
  

requests.post() parameters

Here are all the parameters you can use with the post(…) method:

Parameter Is required Description
url Required The URL to request
data Optional A dictionary, list of tuples, bytes, or file-like object to send in the body to the specified url
json Optional A JSON serializable Python object to send in the body of the request
headers Optional A dictionary of HTTP headers to send with the request
cookies Optional A dictionary or CookieJar object to send with the request
files Optional A dictionary of {'name': file-like-objects} (or {'name': file-tuple}) for multipart encoding upload.
auth Optional A tuple to enable a certain HTTP authentication.Default None
timeout Optional A float, or a tuple, how many seconds to wait for the server to send data before giving up, as a float.
As a tuple: (<connect timeout>, <read timeout>)
Default None which means the request will continue until the connection is closed
allow_redirects Optional, default: True A boolean to enable/disable redirection.
proxies Optional A dictionary of the protocol to the proxy url.
stream Optional, default: False A boolean indication if the response should be immediately downloaded (False) or streamed (True).Default False
verify Optional, default: True Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Default True
cert Optional If string, path to ssl client cert file (.pem). If tuple, ('cert', 'key') pair. Default None

The return value of all HTTP requests made with the Requests package is a Response object.

Jérémy

Jérémy is a Fullstack Developer and Web Scraping Expert. He has contributed to various projects by designing and scaling their data collection processes. Since his early years, his passion for computer science has driven him to explore and push the limits of digital systems.

Get access to millions of residential and mobile IPs
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Article by
Jérémy

Jérémy is a Fullstack Developer and Web Scraping Expert. He has contributed to various projects by designing and scaling their data collection processes. Since his early years, his passion for computer science has driven him to explore and push the limits of digital systems.

Read more
12 min read
Python Requests Retry: A Complete Guide to Handling Failed HTTP Requests

Enhance your Python applications and learn how to handle HTTP request retries using Requests Retry

12 min read
How to Set Up Proxies with Potatso in iOS: Guide

Discover the ultimate Potatso proxy guide! Learn how to set up and configure proxies on your iOS device effortlessly.

12 min read
How to customize Your User-Agent with Python Requests

Learn how to update and rotate user-agents in Python Requests to avoid detection and improve scraping efficiency.

Ready for Next-Level Proxy Solutions?

Get started now and experience the ultimate in proxy flexibility, speed, and security.

Unlock the Power of anyIP Today!