Residential Proxies

Static Residential Proxies

Unlimited Residential Proxies

Learn more

Youtube Proxies

Residential Proxies

Static Residential Proxies

Unlimited Residential Proxies

Data for AI

Web Scraping

SEO and SERP Scraping

Price Monitoring

Travel Fare Aggregation

Stock Market Data Collection

Swiftproxy’s partners

Gather data at scale

Web Scraping Proxies Free Trial

Gather accurate data worldwide without blocks or interruptions.

Learn more >

Unlimited-Bandwidth Proxy Solution for Large-Scale Video Data Collection

Power Your Business Growth with Swiftproxy

A global network of over 80 million residential proxies, ensuring 99.89% uptime and stable connections, supporting HTTP(S) & SOCKS5 protocols.

Swiftproxy residential proxies with 80M+ IPs, 99.89% uptime, supporting HTTP(S) & SOCKS5 protocols

Affiliate program

30% commission guaranteed

CDK Earning Program

Turn your proxies into profit

How to use Python 3 to rotate proxies and IP addresses?

By - Emily Chan

2025-02-20 18:36:14

In web crawlers and automated tasks, frequent use of the same IP address may lead to the blocking of the target website. This article will explain how to implement proxy and IP rotation with Python 3 through 3 mainstream solutions, and provide detailed code implementation and pitfall avoidance guide.

Basic proxy rotation solution (Requests library)

1. Prepare proxy pool

proxies_pool = [
    {"http": "http://123.45.67.89:8080", "https": "http://123.45.67.89:8080"},
    {"http": "http://112.233.44.55:3128", "https": "http://112.233.44.55:3128"},
    # Expandable with more proxies...
]

2. Implement random rotation

import requests
import random
from time import sleep

def rotate_proxy_request(url):
    while True:
        try:
            proxy = random.choice(proxies_pool)
            response = requests.get(
                url,
                proxies=proxy,
                timeout=10,
                headers={"User-Agent": "Mozilla/5.0"}
            )
            if response.status_code == 200:
                return response.text
        except Exception as e:
            print(f"proxy {proxy} fail: {str(e)}")
            sleep(2)  # Delayed retry after failure

# Usage Examples
data = rotate_proxy_request("https://target-website.com/data")

3. Key parameter description

timeout: Set timeout to avoid long waiting time
Exception capture: Automatically switch to the next proxy
User-Agent rotation: It is recommended to rotate with header information

Advanced rotation scheme (Scrapy middleware)

1. Configure middleware

# middlewares.py
import random

class ProxyMiddleware:
    def process_request(self, request, spider):
        proxy = random.choice(proxies_pool)
        request.meta['proxy'] = proxy['http']
        # Add when authentication is required
        # request.headers['Proxy-Authorization'] = basic_auth_header('user', 'pass')

2. Modify settings.py

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.ProxyMiddleware': 543,
}

Browser Automation Solution (Selenium + Proxy)

1. Chrome Proxy Configuration

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def get_chrome_with_proxy(proxy):
    chrome_options = Options()
    chrome_options.add_argument(f'--proxy-server={proxy}')
    driver = webdriver.Chrome(options=chrome_options)
    return driver

# Usage Examples
driver = get_chrome_with_proxy("123.45.67.89:8080")
driver.get("https://target-site.com")

Notes and optimization suggestions

1. Proxy quality selection

High-anonymous proxy vs. transparent proxy
Recommend paid proxy services (such as Swiftproxy)
Free proxy needs to verify validity

2. Validity verification module

def validate_proxy(proxy):
    try:
        test = requests.get(
            "http://httpbin.org/ip",
            proxies=proxy,
            timeout=5
        )
        return test.json()['origin'] in proxy['http']
    except:
        return False

3. Intelligent switching strategy

Dynamically adjust priority based on response time
Automatically eliminate failures based on threshold

4. Legal compliance

Comply with robots.txt protocol
Control access frequency (recommended ≥5 seconds/time)

Comparison of expansion solutions

Solution	Applicable	Scenario	Advantages
Requests rotation	Simple crawler	Fast implementation	Self-managed sessions required
Scrapy middleware	Large-scale distributed crawler	Good integration	High learning cost
Selenium automation	JS rendering page	Can simulate real browser	High resource consumption

Conclusion

Through the above solutions, developers can choose the appropriate proxy rotation strategy according to specific needs. It is recommended to use paid proxy services in production environments and cooperate with the health check mechanism to ensure the quality of the proxy pool. Pay attention to setting the request interval reasonably and comply with network ethics.

About the author

Emily Chan

Lead Writer at Swiftproxy

Emily Chan is the lead writer at Swiftproxy, bringing over a decade of experience in technology, digital infrastructure, and strategic communications. Based in Hong Kong, she combines regional insight with a clear, practical voice to help businesses navigate the evolving world of proxy solutions and data-driven growth.

The content provided on the Swiftproxy Blog is intended solely for informational purposes and is presented without warranty of any kind. Swiftproxy does not guarantee the accuracy, completeness, or legal compliance of the information contained herein, nor does it assume any responsibility for content on thirdparty websites referenced in the blog. Prior to engaging in any web scraping or automated data collection activities, readers are strongly advised to consult with qualified legal counsel and to review the applicable terms of service of the target website. In certain cases, explicit authorization or a scraping permit may be required.

IN THIS ARTICLE

Top-tier residential proxy solutions