How to Retry Failed HTTP Requests using Python Request

Performing HTTP requests in Python can sometimes be unreliable due to connection errors. I recommend always including a retry system whether you’re doing scraping or not.

This guide will show you how to set up retries for failed HTTP requests using the Python requests library.

Also, we will cover advanced techniques like exponential backoff, proxy usage, and more.

Mastering retries will make your Python projects more robust.

Implementing retries in Python Requests

In this section, we’ll see how to set up a retry configuration using the existing classes in the Python Requests library.

The requests package uses the library urllib3 behind the scenes. That’s why we will import the object Retry from urllib3.

Let’s say we want a simple retry strategy on the HTTP status codes we mentioned in the last point.

With a backoff factor of 5 seconds and a maximum of 5 retries.

By default, the Retry object retries these HTTP methods: HEAD, GET, PUT, DELETE, OPTIONS, TRACE.

We will customize them in the following example.

Python

    from urllib3.util import Retry

retry_strategy = Retry(
    total=5,  # Here we configure the max retries
    backoff_factor=5,  # And here we are setting the backoff factor
    status_forcelist=[429, 500, 502, 503, 504], # These are the status codes we want to retry
		allowed_methods=["GET", "POST", "PUT", "PATCH"]  # These are the methods we want to retry
)

# We pass our Retry object in the max_retries kwarg
adapter = HTTPAdapter(max_retries=retry_strategy)

# First we create the Session object
session = Session()

# Then we mount the adapter for all http and https traffic
session.mount("http://", adapter)
session.mount("https://", adapter)

session.get("http://httpbin.org/status/429")

Congrats! Now, you can seamlessly retry requests in your data collection process.

Six requests must have been made. The first one, the initial one, is followed by five retries as we configured.

Simplifying retries syntax with the retry-requests package

We can shorten that code for lazy developers like me. There's a small package named retry-requests.

It abstracts the code we saw before. It has a retry() function that returns our ready-to-use Session object.

The package can be installed like this:

Bash

    pip install retry-requests

Using the same retry strategy as the last example, it can be used like this:

Python

    from retry_requests import retry

session = retry(
    retries=5,  # This is the max retries
    backoff_factor=5,  # As its name says, the backoff factor
    status_to_retry=(429, 500, 502, 503, 504),  # The status codes to retry
    allowed_methods=["GET", "POST", "PUT", "PATCH"],  # The methods to retry
)

session.get("http://localhost:8080/status/429")

And it works exactly the same as before.

Why you need retries in Python Requests

Using the requests library in Python to send HTTP requests may cause failures. These issues are often beyond your control, like:

Connection errors: The server might be temporarily unreachable due to network issues.
Timeouts: Servers sometimes take too long to respond, causing your request to hang.
Server errors (5XX status codes): The server might be down or overloaded. This can cause errors like 500, 502, or 503.

Without retries, these failures can crash your app or cause data loss. This is especially true when scraping websites or calling APIs.

That's why a retry strategy is vital. It will help your app handle uncontrolled issues.

It also makes your process trustworthy and reliable.

Understanding retry strategies

In this section, we’ll discuss the common retry strategies. Understanding this section will be helpful for the following ones.

Exponential backoff

When a retry occurs, adding a delay before retrying the request is expected. We can set a base delay that exponentially grows with each attempt.

This prevents flooding the server with requests and reduces the likelihood of bans.

The exponential backoff can be configured with two parameters:

Backoff factor: which is the base delay, for example, if set to 5, it will wait 5 seconds before the first retry.
Growth factor: this is the factor for the exponentiality, usually set to 2 or less.

And the delay can be calculated like this:

Python

    delay = backoff_factor * (growth_factor ** retry_count)

I usually set a backoff factor to 2 or 5 and a growth factor to 2.

Here are the results of delays according to these two examples:

Backoff factor set to 2:

2, 4, 8, 16, 32, 64, 128, 256, 512, 1024
Backoff factor set to 5:

5, 10, 20, 40, 80, 160, 320, 640, 1280, 2560

You can play with these two parameters to have a more or less exponential behavior.

In some specific situations, you would not like an exponential backoff at all. You can disable this behavior simply by setting the growth_factor to 1.

The resulting delay will be the same for all the attempts.

It can be helpful when performing concurrent requests using proxies.

Total retries

The max retries is usually set to 3 in most of the cases.

You can configure this value according to the service or website to which you send HTTP requests.

You can set more or fewer retries depending on the server’s stability or the importance of the data you’re collecting.

Your exponential backoff strategy can also influence the configuration of max retries. This parameter helps to limit the delay exponentiality.

Retry on HTTP status codes

Configuring the HTTP status codes to retry helps with a good retry strategy.

This ensures you only retry when it makes sense. So, retry for temporary issues or rate-limiting errors. Avoid retries for client-side errors that won't resolve.

The most common HTTP errors you can get when scraping websites are:

403 Forbidden: This error usually means you lack permission for the request. Retrying won't help unless you expect temporary authentication or access issues.
429 Too Many Requests: This status indicates that the server is rate-limiting you. Retrying without any changes will likely result in repeated failures. It's essential to implement exponential backoff. Also, consider rotating proxies to distribute requests. This avoids hitting the same rate limits again.
500 Internal Server Error: This error occurs when something fails on the server side, like a bug or a misconfiguration. If the server recovers quickly, retrying can often resolve the issue.
502 Bad Gateway: This error means a server communication issue, often between servers. This is a temporary issue. A retry with backoff is a good way to let the intermediary services recover.
503 Service Unavailable: This error means the server is overloaded or down for maintenance. As with 502, a backoff retry gives the server time to recover.
504 Gateway Timeout: This error occurs when the server takes too long to respond. Retrying after a delay can help with a slow network or an overloaded service.

You might see various HTTP errors when web scraping or using APIs. The MDN web docs provide information about all the HTTP response status codes.

Best Practices when retrying HTTP requests

A retry strategy is vital to make your HTTP requests more robust. However, we must be careful.

We need to avoid overwhelming servers and ensure your app behaves as expected. Here are some best practices to consider when implementing retries:

Limit the number of retries.
While setting a high number of retries to maximize success may be tempting, it's best to limit the number.
A maximum of 3 to 5 retries is generally sufficient for most scenarios.
It prevents too much load on the server. It reduces the risk of being blocked or blacklisted.
Handle max retries exceeded.
Retrying requests doesn’t wholly avoid network errors.
Your retry strategy can still reach the maximum retries and crash for many reasons.
Ensure it doesn’t crash your application or cause data loss on other unrelated requests.
Implement exponential backoff.
As discussed, exponential backoff is an effective strategy for handling retries.
It spreads out the retries over time, allowing temporary issues to fix themselves.
Ensure your backoff strategy adapts to the server's response and conditions.
Be selective with status codes.
Set your retry strategy to apply only to specific HTTP status codes that warrant retries.
Avoid retrying on client-side errors, like 400 status codes.
They mean the request is likely incorrect. Focus on transient errors, such as 429 (Too Many Requests) or 5XX (server errors).
Monitor and log retries
Implement logging to track the retries your application makes.
This can help find patterns or issues.
For example, a specific endpoint that often fails. Knowing when and why requests fail helps troubleshoot them. It may also lead to lasting solutions.
Use contextual information.
Sometimes, you may need to incorporate contextual information into your retry logic.
For example, an API may return rate-limiting headers, which you can use to adjust your backoff strategy.
Make sure to extract and utilize relevant details from the server's response.
Graceful degradation
If all retry attempts fail, make sure your app can degrade gracefully. Provide informative error messages or fallback mechanisms, like cached data. They allow continued operation.
Test your retry logic
Lastly, thoroughly test your retry logic under various conditions. Simulate different types of errors, including network failures, timeouts, and server responses. Ensure that your retry logic behaves as expected and doesn't introduce new issues.

Create your own retry system

In some advanced use cases, you want to customize the retry behavior on specific HTTP status codes or exceptions.

This section will show how to create a custom retry system. We will use it to set a callback for a specific HTTP status code.

For this demo, we will create a simple callback. It will switch our user-agent when the server rate-limits us. This will make the example easy to reproduce.

Note that in an actual use case, we could create a callback that also switches the proxy along the user agent.

Let’s start with the implementation of a basic retry system. We will create a class named RetrySession.

Python

    from typing import Callable, Optional

from requests import Response

class RetrySession:

    def __init__(
        self,
        retries: int = 5,
        backoff_factor: int = 2,
        growth_factor: int = 2,
        max_delay: int = 120,
        status_callbacks: Optional[
            dict[int, Callable[[requests.Response], None]]
        ] = None,
    ):
        self.retries = retries
        self.backoff_factor = backoff_factor
        self.growth_factor = growth_factor
        self.max_delay = max_delay
        self.status_callbacks = status_callbacks or {}
        self.session = requests.Session()

This is our __init__ function. You have already heard about the first three arguments retries, backoff_factor and growth_factor.

We didn't discuss max_delay before. It's important. It limits your exponential backoff so it doesn't wait for hours with a lot of retries. So I decided to implement it here in this example.

Next, we have the argument status_callbacks. It's a dict. It maps HTTP status codes to functions. We'll call the given function when we receive the associated status code.

Now, let’s add the crucial methods to our RetrySession class:

Python

    class RetrySession:

    # [...]

    def make_request(self, method: str, url: str, **kwargs: Any) -> requests.Response:

        attempt = 0
        
        # We do a +1 for the initial request, so we can still configure a session with 0 retries
        for attempt in range(self.retries + 1):
            try:
                response = self.session.request(method, url, **kwargs)
                self._handle_callback(response)
                response.raise_for_status()
                return response
            except requests.exceptions.RequestException as e:
                print(f"Attempt {attempt + 1} failed with error: {e}")

                if attempt == self.retries:
                    raise e

                self.exponential_backoff(attempt)

    def _handle_callback(self, response: Response):
    
		    # First we try to get the callback from our dict
        callback = self.status_callbacks.get(response.status_code)

				# If there is no callback, we have nothing to do
        if callback is None:
            return

				# If there is one, we just call it and send the response as argument
        callback(response)
    
    def _exponential_backoff(self, attempt: int) -> None:

				# You are already familiar with this calculation
				# The new thing here is that keep the smallest value
				# between the backoff result and the self.max_delay
        delay = min(self.backoff_factor * (self.growth_factor**attempt), self.max_delay)
        print(f"Retrying in {delay} seconds...")
        time.sleep(delay)

As you can see, we added 3 methods to our class, make_request, _handle_callback and _exponential_backoff.

Now, our RetrySession is ready to use, but let me get more comfortable and add some functions for the most important HTTP methods.

Python

    class RetrySession:

    # [...]
    
    def get(self, url: str, **kwargs: Any) -> requests.Response:
        """Make a GET request."""
        return self.make_request("GET", url, **kwargs)

    def post(
        self,
        url: str,
        data: Optional[dict[str, Any]] = None,
        json: Optional[dict[str, Any]] = None,
        **kwargs: Any,
    ) -> requests.Response:
        """Make a POST request."""
        return self.make_request("POST", url, data=data, json=json, **kwargs)

    def put(
        self, url: str, data: Optional[dict[str, Any]] = None, **kwargs: Any
    ) -> requests.Response:
        """Make a PUT request."""
        return self.make_request("PUT", url, data=data, **kwargs)

    def delete(self, url: str, **kwargs: Any) -> requests.Response:
        """Make a DELETE request."""
        return self.make_request("DELETE", url, **kwargs)

It’s prettier like this.

It’s time to try our beautiful RetrySession object. First, we must list user agents. Then, we need a callback to switch user agents on HTTP status code 429.

Also, we need to start the httpbin server locally to see the requests from the server:

docker run -p 8080:80 kennethreitz/httpbin gunicorn -b 0.0.0.0:80 --access-logfile '-' httpbin:app

Here is our example code:

Python

    # We set our user-agents examples
user_agents = [
    "User Agent A",
    "User Agent B",
    "User Agent C",
    "User Agent D",
    "User Agent E",
    "User Agent F",
]

# We create the headers for the request
headers = {
    "user-agent": user_agents[0],
}

def handle_rate_limit(response: Response):
    """Callback for handling rate limit exceeded."""
    # Here we set a random user agent in the global variable headers
    headers["user-agent"] = random.choice(user_agents)

# We create our RetrySession object with the handler on 429 HTTP status code 
session = RetrySession(
    retries=5,
    backoff_factor=1,
    status_callbacks={429: handle_rate_limit},
)

# We finally perform the requests to our local httpbin server
session.get("http://localhost:8080/status/200", headers=headers)
session.get("http://localhost:8080/status/429", headers=headers)

In the server’s logs we should see the requests coming:

Bash

    [2024-10-25 10:59:32 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2024-10-25 10:59:32 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
[2024-10-25 10:59:32 +0000] [1] [INFO] Using worker: sync
[2024-10-25 10:59:32 +0000] [9] [INFO] Booting worker with pid: 9
172.17.0.1 - - [25/Oct/2024:10:59:37 +0000] "GET /status/200 HTTP/1.1" 200 0 "-" "User Agent A"
172.17.0.1 - - [25/Oct/2024:10:59:37 +0000] "GET /status/429 HTTP/1.1" 429 0 "-" "User Agent A"
172.17.0.1 - - [25/Oct/2024:10:59:38 +0000] "GET /status/429 HTTP/1.1" 429 0 "-" "User Agent E"
172.17.0.1 - - [25/Oct/2024:10:59:40 +0000] "GET /status/429 HTTP/1.1" 429 0 "-" "User Agent F"
172.17.0.1 - - [25/Oct/2024:10:59:44 +0000] "GET /status/429 HTTP/1.1" 429 0 "-" "User Agent B"
172.17.0.1 - - [25/Oct/2024:10:59:52 +0000] "GET /status/429 HTTP/1.1" 429 0 "-" "User Agent E"
172.17.0.1 - - [25/Oct/2024:11:00:08 +0000] "GET /status/429 HTTP/1.1" 429 0 "-" "User Agent E"

We see that we received the first successful request with HTTP status code 200.

It is followed by six rate-limited requests. This includes the initial request and the five attempts we configured.

We see the random user agent that the handler has updated for each new request.

That’s great. We now have a powerful RetrySession object. It allows us to use a callback on specific HTTP status codes.

Feel free to reuse and customize that class for your own needs. We could have added complex features, like:

changing the backoff values for specific status codes
setting a callback for specific network errors.

‍

Voilà, here is the complete code:

Python

    import random
import time
import requests
from typing import Callable, Any, Optional

from requests import Response


class RetrySession:
    """
    A wrapper around the requests library that adds retry logic with exponential backoff
    and callbacks on HTTP status codes.
    """

    def __init__(
        self,
        retries: int = 5,
        backoff_factor: int = 2,
        growth_factor: int = 2,
        max_delay: int = 120,
        status_callbacks: Optional[
            dict[int, Callable[[requests.Response], None]]
        ] = None,
    ):
        """
        Initialize the RetrySession with retry configurations.

        Args:
        - retries (int): Maximum number of retry attempts.
        - backoff_factor (int): Base delay in seconds for exponential backoff.
        - growth_factor (int): Multiplier for the exponential backoff.
        - max_delay (int): Maximum delay between retries.
        - status_callbacks (Optional[Dict[int, Callable[[requests.Response], None]]]): Optional callbacks for specific status codes.
        """
        assert retries >= 0, "Retries must be a non-negative integer."
        assert backoff_factor >= 0, "Backoff factor must be a non-negative integer."
        assert growth_factor >= 0, "Growth factor must be a non-negative integer."
        self.retries = retries
        self.backoff_factor = backoff_factor
        self.growth_factor = growth_factor
        self.max_delay = max_delay
        self.status_callbacks = status_callbacks or {}
        self.session = requests.Session()

    def exponential_backoff(self, attempt: int) -> None:
        """Calculates the delay for retries based on exponential backoff strategy."""
        delay = min(self.backoff_factor * (self.growth_factor**attempt), self.max_delay)
        print(f"Retrying in {delay} seconds...")
        time.sleep(delay)

    def make_request(self, method: str, url: str, **kwargs: Any) -> requests.Response:
        """
        Make an HTTP request with retry logic.

        Args:
        - method (str): HTTP method (GET, POST, etc.)
        - url (str): The URL for the request.
        - kwargs: Additional arguments to pass to the request.

        Returns:
        - response (requests.Response): The final response after retries.
        """
        for attempt in range(self.retries + 1):
            try:
                response = self.session.request(method, url, **kwargs)
                self._handle_callback(response)
                response.raise_for_status()
                return response
            except requests.exceptions.RequestException as e:
                print(f"Attempt {attempt + 1} failed with error: {e}")

                if attempt == self.retries:
                    raise e

                self.exponential_backoff(attempt)

    def _handle_callback(self, response: Response):
        """Call the appropriate callback based on the response status code."""
        callback = self.status_callbacks.get(response.status_code)

        if callback is None:
            return

        callback(response)

    def get(self, url: str, **kwargs: Any) -> requests.Response:
        """Make a GET request."""
        return self.make_request("GET", url, **kwargs)

    def post(
        self,
        url: str,
        data: Optional[dict[str, Any]] = None,
        json: Optional[dict[str, Any]] = None,
        **kwargs: Any,
    ) -> requests.Response:
        """Make a POST request."""
        return self.make_request("POST", url, data=data, json=json, **kwargs)

    def put(
        self, url: str, data: Optional[dict[str, Any]] = None, **kwargs: Any
    ) -> requests.Response:
        """Make a PUT request."""
        return self.make_request("PUT", url, data=data, **kwargs)

    def delete(self, url: str, **kwargs: Any) -> requests.Response:
        """Make a DELETE request."""
        return self.make_request("DELETE", url, **kwargs)


user_agents = [
    "User Agent A",
    "User Agent B",
    "User Agent C",
    "User Agent D",
    "User Agent E",
    "User Agent F",
]
headers = {
    "user-agent": user_agents[0],
}


def handle_rate_limit(*args, **kwargs):
    """Callback for handling rate limit exceeded."""
    headers["user-agent"] = random.choice(user_agents)


session = RetrySession(
    retries=5,
    backoff_factor=1,
    status_callbacks={429: handle_rate_limit},
)
session.get("http://localhost:8080/status/200", headers=headers)
session.get("http://localhost:8080/status/429", headers=headers)

Python Requests Retry: A Complete Guide to Handling Failed HTTP Requests

Implementing retries in Python Requests

Simplifying retries syntax with the retry-requests package

Why you need retries in Python Requests

Understanding retry strategies

Exponential backoff

Total retries

Retry on HTTP status codes

Best Practices when retrying HTTP requests

Create your own retry system