网页爬虫代理IP

http:// 走 HTTP 代理，socks5h:// 走 SOCKS5 并由代理解析域名；IP 白名单认证时去掉 USER:PASS@。用户名或密码里的 @ 要写成 %40。

# HTTP proxy, username and password
curl -x http://USER:PASS@GATEWAY:PORT https://example.com/

# SOCKS5, hostnames resolved by the proxy
curl -x socks5h://USER:PASS@GATEWAY:PORT https://example.com/

# IP whitelist: no credentials
curl -x http://GATEWAY:PORT https://example.com/

proxies 字典按协议指定代理，账号密码写在代理地址里。用 SOCKS5 时先安装 requests[socks]，再把地址换成 socks5h:// 开头。

import requests

proxy = "http://USER:PASS@GATEWAY:PORT"
proxies = {"http": proxy, "https": proxy}
headers = {"User-Agent": "ExampleBot/1.0 (+https://example.com/bot)"}

r = requests.get(
    "https://example.com/",
    proxies=proxies,
    headers=headers,
    timeout=30,
)
print(r.status_code)

Scrapy 中间件设置代理的简洁写法：在请求 meta 里写 proxy，由内置的 HttpProxyMiddleware 处理。示例保留 ROBOTSTXT_OBEY 并打开 AutoThrottle；async def start() 需要 Scrapy 2.13+。

import scrapy


class PageSpider(scrapy.Spider):
    name = "pages"
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,
        "DOWNLOAD_DELAY": 1.0,
    }

    async def start(self):
        yield scrapy.Request(
            "https://example.com/",
            meta={"proxy": "http://USER:PASS@GATEWAY:PORT"},
        )

    def parse(self, response):
        yield {"url": response.url, "status": response.status}

启动浏览器时传入 proxy，也可以用 new_context(proxy=...) 按上下文设置。浏览器走 SOCKS5 时改用 IP 白名单认证。

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://GATEWAY:PORT",
        "username": "USER",
        "password": "PASS",
    })
    page = browser.new_page()
    page.goto("https://example.com/")
    print(page.title())
    browser.close()

ProxyAgent 接受代理地址，账号密码以 Basic 认证发给代理，按请求作为 dispatcher 传入。

import { fetch, ProxyAgent } from "undici";

const dispatcher = new ProxyAgent("http://USER:PASS@GATEWAY:PORT");
const res = await fetch("https://example.com/", { dispatcher });
console.log(res.status);