AI Training Data Proxies

-x sets the proxy and -I fetches headers only, enough to confirm the connection and auth work. socks5h:// uses SOCKS5 with hostnames resolved by the proxy; with IP-allowlist auth, drop USER:PASS@.

# Check connectivity and auth, headers only
curl -I -x http://USER:PASS@GATEWAY:PORT https://example.com/

# SOCKS5, hostnames resolved by the proxy
curl -I -x socks5h://USER:PASS@GATEWAY:PORT https://example.com/

AsyncClient takes the proxy argument (HTTPX 0.26+); the Semaphore caps requests in flight, so tune it to what the target site can take. Name your crawler and a contact in the User-Agent so site owners can identify it.

import asyncio

import httpx

PROXY = "http://USER:PASS@GATEWAY:PORT"
UA = "ExampleCorpusBot/1.0 (+https://example.com/bot)"
URLS = ["https://example.com/a", "https://example.com/b"]


async def fetch(client, sem, url):
    async with sem:
        r = await client.get(url, timeout=30)
        return url, r.status_code, len(r.text)


async def main():
    sem = asyncio.Semaphore(8)
    client = httpx.AsyncClient(proxy=PROXY, headers={"User-Agent": UA})
    async with client:
        jobs = [fetch(client, sem, u) for u in URLS]
        for row in await asyncio.gather(*jobs):
            print(row)


asyncio.run(main())

Follows in-site links and collects body text, with proxy set in each request's meta. ROBOTSTXT_OBEY stays on, AutoThrottle sets the pace and DEPTH_LIMIT bounds the crawl; async def start() needs Scrapy 2.13+.

import scrapy

PROXY = "http://USER:PASS@GATEWAY:PORT"


class CorpusSpider(scrapy.Spider):
    name = "corpus"
    allowed_domains = ["example.com"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,
        "DEPTH_LIMIT": 3,
    }

    async def start(self):
        url = "https://example.com/"
        yield scrapy.Request(url, meta={"proxy": PROXY})

    def parse(self, response):
        text = " ".join(response.css("p::text").getall())
        yield {"url": response.url, "text": text}
        for href in response.css("a::attr(href)").getall():
            yield response.follow(href, meta={"proxy": PROXY})