How to Integrate Residential Proxies into Python Scraping Scripts: Requests, HTTPX, AIOHTTP, and Playwright Configuration Guide

Determine Proxy Type and Authentication Method Before Integration

Before writing configuration code, two things must be clarified: whether the task requires rotating or static exit IPs, and whether the provider offers username/password authentication or whitelist authentication.

Dynamic residential proxies are typically billed by traffic or bandwidth, rotating IPs per request or per session, suitable for large-scale scraping, SERP monitoring, and ad verification. Static residential proxies provide fixed IPs, available in short-term (hourly or daily) and long-term (monthly or yearly) options, ideal for store backend logins, social media account operations, and TikTok content management that require stable exit IPs.

Username/password authentication embeds credentials directly in the proxy URL with the format http://username:password@host:port. Whitelist authentication requires binding your server's public IP in the provider's dashboard, after which you can connect to host:port directly without credentials. Some dynamic proxies allow appending parameters to the username suffix to control sticky sessions or target countries, such as user_session_abc123 or user_country_us. Refer to your provider's console extraction instructions for specific formats.

Requests Configuration: Session Binding and SOCKS5h Leak Prevention

Requests is the most commonly used HTTP client, declaring HTTP and HTTPS proxy protocols separately via the proxies dictionary:

import requests

proxies = {
    'http': 'http://username:[email protected]:8000',
    'https': 'http://username:[email protected]:8000'
}

response = requests.get('https://httpbin.org/ip', proxies=proxies)
print(response.json())

If you need to reuse connection pools, maintain cookies, or manage independent sessions for multiple tasks, bind the proxy to a requests.Session():

session = requests.Session()
session.proxies.update(proxies)
response = session.get('https://example.com')

When using SOCKS5 residential proxies, you must first install the support package pip install 'requests[socks]' (based on PySocks). More importantly, use the socks5h:// prefix instead of socks5://:

proxies = {
    'http': 'socks5h://username:[email protected]:1080',
    'https': 'socks5h://username:[email protected]:1080'
}

socks5h forces DNS resolution through the proxy server remotely, preventing local DNS queries from exposing your real IP or network characteristics. If the target site is sensitive to DNS request origins, this step reduces the risk of being identified as proxy traffic.

HTTPX Configuration: Unified Sync and Async Interface

HTTPX supports both synchronous and asynchronous requests. Proxy parameters are passed via proxy when instantiating a Client:

import httpx

proxy_url = 'http://username:[email protected]:8000'

# 同步客户端
with httpx.Client(proxy=proxy_url, follow_redirects=True) as client:
    response = client.get('https://httpbin.org/ip')
    print(response.json())

# 异步客户端
import asyncio

async def fetch():
    async with httpx.AsyncClient(proxy=proxy_url, follow_redirects=True) as client:
        response = await client.get('https://httpbin.org/ip')
        print(response.json())

asyncio.run(fetch())

HTTPX supports HTTP, HTTPS, and SOCKS5 proxies, but does not automatically follow redirects by default—you must explicitly set follow_redirects=True. For tasks with many concurrent requests or requiring persistent connections, adjust limits parameters to control connection pool size and timeouts:

limits = httpx.Limits(max_keepalive_connections=20, max_connections=100)
client = httpx.Client(proxy=proxy_url, limits=limits, timeout=10.0)

AIOHTTP Configuration: Native HTTP and SOCKS5 Extensions

AIOHTTP is an asynchronous HTTP client where proxies are passed via the proxy parameter when making requests. It natively supports HTTP and HTTPS proxies:

import aiohttp
import asyncio

async def fetch():
    proxy = 'http://proxy.example.com:8000'
    proxy_auth = aiohttp.BasicAuth('username', 'password')
    
    async with aiohttp.ClientSession() as session:
        async with session.get(
            'https://httpbin.org/ip',
            proxy=proxy,
            proxy_auth=proxy_auth
        ) as response:
            print(await response.json())

asyncio.run(fetch())

Credentials can be inlined in the URL as http://username:password@host:port or explicitly declared as proxy_auth=aiohttp.BasicAuth('user', 'pass').

To support SOCKS5 proxies, you typically need a third-party connector extension package (such as aiohttp-socks) to inject a custom TCPConnector:

from aiohttp_socks import ProxyConnector

connector = ProxyConnector.from_url('socks5://username:[email protected]:1080')

async with aiohttp.ClientSession(connector=connector) as session:
    async with session.get('https://httpbin.org/ip') as response:
        print(await response.json())

Playwright Configuration: Global and Context-Level Proxy Isolation

Headless browser automation frameworks like Playwright are suitable for scraping tasks requiring JavaScript rendering or realistic browser behavior simulation. Proxies can be configured globally when launching the browser or per-context for task or account isolation.

Global proxy configuration is passed during launch:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy={
            'server': 'http://proxy.example.com:8000',
            'username': 'username',
            'password': 'password'
        }
    )
    page = browser.new_page()
    page.goto('https://httpbin.org/ip')
    print(page.content())
    browser.close()

If different tasks need separate exit IPs, configure per-context within new_context:

browser = p.chromium.launch()

context_1 = browser.new_context(
    proxy={
        'server': 'http://proxy1.example.com:8000',
        'username': 'user1',
        'password': 'pass1'
    }
)
page_1 = context_1.new_page()
page_1.goto('https://httpbin.org/ip')

context_2 = browser.new_context(
    proxy={
        'server': 'http://proxy2.example.com:8000',
        'username': 'user2',
        'password': 'pass2'
    }
)
page_2 = context_2.new_page()
page_2.goto('https://httpbin.org/ip')

This isolation approach suits multi-account operations or parallel scraping tasks requiring different regional exit IPs.

Code Differences Between Whitelist and Username/Password Authentication

After binding your server's public IP, whitelist authentication allows direct connections without credentials, and the proxy URL contains no username or password:

# Requests
proxies = {
    'http': 'http://proxy.example.com:8000',
    'https': 'http://proxy.example.com:8000'
}

# HTTPX
client = httpx.Client(proxy='http://proxy.example.com:8000')

# AIOHTTP
proxy = 'http://proxy.example.com:8000'
async with session.get(url, proxy=proxy) as response:
    pass

# Playwright
proxy={'server': 'http://proxy.example.com:8000'}

Username/password authentication must inline credentials in the URL or pass them via dedicated parameters. Some dynamic proxies allow appending parameters to the username suffix to control sticky sessions or target countries—formats vary by provider, so consult the console extraction instructions.

Static proxies typically connect directly to fixed ports or dedicated addresses, where each IP corresponds to a port or unique hostname, requiring no session identifier in the username.

Exit Verification and Exception Retry

After launching your scraping script or switching proxies, first request an echo endpoint to verify the exit IP matches expectations:

import requests

def verify_proxy(proxies):
    try:
        response = requests.get(
            'https://httpbin.org/ip',
            proxies=proxies,
            timeout=10
        )
        origin_ip = response.json().get('origin')
        print(f"当前出口 IP: {origin_ip}")
        return origin_ip
    except requests.exceptions.ProxyError:
        print("代理连接失败")
        return None
    except requests.exceptions.ConnectTimeout:
        print("代理连接超时")
        return None

Verification includes:

Whether the returned origin IP matches the expected residential exit location
Whether the geographic location aligns with the target country or region
Whether the static IP matches the address provided by the provider

For connection failures possibly caused by temporary proxy node issues or target site rate limiting, use try-except to catch ProxyError, ConnectTimeout, and similar exceptions, combined with exponential backoff retry:

import time

def fetch_with_retry(url, proxies, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, proxies=proxies, timeout=10)
            return response
        except (requests.exceptions.ProxyError, requests.exceptions.ConnectTimeout) as e:
            wait_time = 2 ** attempt
            print(f"第 {attempt + 1} 次重试,等待 {wait_time} 秒")
            time.sleep(wait_time)
    raise Exception("超过最大重试次数")

Selection Recommendations: Task Characteristics Determine Proxy Type and Tool Combination

If the task involves large-scale URL scraping, SERP monitoring, or ad verification, and the target site mainly limits access by IP frequency, choose dynamic residential proxies billed by traffic or bandwidth. Requests or HTTPX with rotating IPs suffices—no browser environment needed. Dynamic bandwidth plans offer unlimited traffic, suitable for sustained high-concurrency scenarios.

If the task requires stable exit IPs—store backend logins, social media account operations, TikTok content management—choose static residential IPs. Short-term suits temporary testing or periodic tasks; long-term suits persistent binding. For TikTok operations, use static residential native IPs with geographic locations strictly matching the account's target region.

If the target site blocks Requests' TLS/JA3 fingerprint or the page depends on JavaScript rendering, use an automation browser like Playwright. In this case, proxy configuration is passed during launch or new_context, with each context binding an independent exit.

When you've determined your task requires a dynamic traffic plan, visit the NexIP Dynamic Traffic Product Page to select region and protocol; for static fixed exits, choose Static Short-Term or Static Long-Term based on lease duration. For recommended solutions by industry or platform, refer to the Solutions Page.