Web Scraping Proxies

Collect public web data through 100M+ residential IPs: rotate per request or hold a session, pay per GB or by bandwidth.

  • 100M+active residential IPs
  • 195+countries and regions
  • 3protocols: HTTP, HTTPS, SOCKS5
Sign up for NexIP Choose an IP type

Common platforms and tools

  • Python
  • Scrapy
  • Playwright
  • Puppeteer
  • Selenium
  • Node.js
  • Go
  • Java
  • cURL
  • HTTPX
  • Requests
  • undici

Why scraping jobs keep stalling

Requests pile onto a few IPs

When one server or a handful of datacenter IPs carries every request, 429s arrive fast and the job ends up waiting in a queue.

You can't see the target market

Prices, stock and search results change by country and even by city, so data collected from home doesn't match what local users see.

A home-made pool eats engineering time

Collecting, testing and replacing dead IPs needs constant attention, and the hours go into exits instead of the data itself.

Volume swings, so costs are hard to plan

Launches and sales double the requests, quiet weeks barely use any; fixed capacity is either too small or sitting idle.

Why teams choose NexIP proxies for web scraping

A 100M+ residential pool

Residential proxies for web scraping across 195+ countries and regions: requests leave from real home networks, so no handful of IPs carries the load.

Rotating proxies for web scraping, or sticky sessions

Listing and detail pages take a new exit each time; multi-step flows such as paging or setting a region before reading a price keep one IP with a sticky session.

Country and city targeting

Collect each market's prices and pages by country, and use city targeting for local data; ASN targeting is subject to availability.

Unlimited concurrency

Dynamic residential plans put no cap on concurrent connections, so a job scales out as far as it needs and you pay only for the traffic or bandwidth used.

Two ways to pay

Bursty jobs suit the Traffic plan (From $0.8/GB), whose packs don't expire; 24/7 jobs suit the Bandwidth plan, with dedicated bandwidth and no traffic cap.

Standard protocols, existing code

HTTP, HTTPS and SOCKS5 with username-and-password or IP-allowlist auth, so Scrapy, Playwright and requests connect with a one-line change.

How to use a web scraping proxy in your code

After purchase you get a connection address and port; swap them into the placeholders and run. The card shows cURL, requests, Scrapy, Playwright and Node.js.

Protocols
HTTP · HTTPS · SOCKS5
Auth
Username and password · IP allowlist
Concurrency
Unlimited concurrency on dynamic residential
Targeting
Country · city · ASN (subject to availability)

GATEWAY, PORT, USER and PASS are placeholders: swap in the connection details you get after purchase. IP-allowlist auth needs no USER or PASS.

cURL: point -x at the proxy and test

http:// uses an HTTP proxy; socks5h:// uses SOCKS5 with hostnames resolved by the proxy. With IP-allowlist auth, drop USER:PASS@. Write an @ in the username or password as %40.

# HTTP proxy, username and password
curl -x http://USER:PASS@GATEWAY:PORT https://example.com/

# SOCKS5, hostnames resolved by the proxy
curl -x socks5h://USER:PASS@GATEWAY:PORT https://example.com/

# IP whitelist: no credentials
curl -x http://GATEWAY:PORT https://example.com/

Python requests proxy authentication

The proxies dict sets a proxy per scheme, with the credentials in the proxy URL. For SOCKS5, install requests[socks] and switch the address to socks5h://.

import requests

proxy = "http://USER:PASS@GATEWAY:PORT"
proxies = {"http": proxy, "https": proxy}
headers = {"User-Agent": "ExampleBot/1.0 (+https://example.com/bot)"}

r = requests.get(
    "https://example.com/",
    proxies=proxies,
    headers=headers,
    timeout=30,
)
print(r.status_code)

Scrapy proxy middleware: set it in meta

A minimal Scrapy setup: put proxy in the request's meta and the built-in HttpProxyMiddleware handles it. The sample keeps ROBOTSTXT_OBEY on and enables AutoThrottle; async def start() needs Scrapy 2.13+.

import scrapy


class PageSpider(scrapy.Spider):
    name = "pages"
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,
        "DOWNLOAD_DELAY": 1.0,
    }

    async def start(self):
        yield scrapy.Request(
            "https://example.com/",
            meta={"proxy": "http://USER:PASS@GATEWAY:PORT"},
        )

    def parse(self, response):
        yield {"url": response.url, "status": response.status}

Playwright proxy authentication at launch

Pass proxy at launch, or set one per context with new_context(proxy=...). For a browser over SOCKS5, authenticate with an IP allowlist instead.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://GATEWAY:PORT",
        "username": "USER",
        "password": "PASS",
    })
    page = browser.new_page()
    page.goto("https://example.com/")
    print(page.title())
    browser.close()

Node.js: fetch through undici's ProxyAgent

ProxyAgent takes the proxy URL and sends the credentials as Basic auth; pass it per request as the dispatcher.

import { fetch, ProxyAgent } from "undici";

const dispatcher = new ProxyAgent("http://USER:PASS@GATEWAY:PORT");
const res = await fetch("https://example.com/", { dispatcher });
console.log(res.status);

Which proxy fits your scraping job?

Proxy rotation for web scraping comes built into the dynamic plans; fixed exits for allowlisted endpoints come from static IPs.

Recommended

Dynamic residential · Traffic

Batch crawls across markets

Billed per GB with unlimited concurrency, and each request can leave from a different IP in the 100M+ pool. Packs don't expire, which suits uneven workloads.

From $0.8/GB

Learn more · Dynamic residential · Traffic

Dynamic residential · Bandwidth

24/7 monitoring and polling

30-day plans with 100Mbps–1Gbps of dedicated bandwidth and no traffic cap under fair use. Sites in the US, UK, Germany, France, Japan and South Korea need Flagship.

100Mbps / $980/30 days

Learn more · Dynamic residential · Bandwidth

Datacenter IP

Allowlisted APIs and downloads

When a partner allowlists your exit IP, use a dedicated datacenter IP: the address holds for the lease, with high throughput at a low price.

$3/IP/month

Learn more · Datacenter IP

Static residential · Short-term

Short checks and one-off jobs

For checks finished within hours, use short-term IPs: dedicated for 1–24 hours, sold in IP packs that don't expire.

From $0.17/IP

Learn more · Static residential · Short-term

Web scraping proxy FAQ

Choose Traffic for short or uneven batch work: billed per GB (From $0.8/GB) with unlimited concurrency. Choose Bandwidth for 24/7 jobs (100Mbps / $980/30 days, Basic plan): 100Mbps–1Gbps dedicated, with no traffic cap under fair use; sites in the US, UK, Germany, France, Japan, South Korea and similar markets need Flagship. Headless browsers also load images and scripts, so cost browser jobs both ways.

No. Find the cause first. A 429 means you're too fast: wait as Retry-After says and lower the rate. A verification page or a 403 means slow down or pause, and check robots.txt and the terms. For content behind a login or paywall, use the official API. Once a site's anti-scraping measures have refused you, switching IPs to carry on just ignores that refusal. A proxy spreads load and puts you in the right market; it doesn't raise what a site will accept.

No. The Traffic plan is itself a pool of 100M+ active IPs: your code sets only the connection address (GATEWAY in the samples) and port, rotation or sticky sessions are part of the connection setup, and there's no IP list to collect, test or replace.

For most public-page scraping, use dynamic residential IPs: a large pool, country and city targeting, and the pages local users see. For jobs that need a long-term fixed exit but not a home-broadband connection, such as allowlisted APIs and open-data downloads, use the Long-term plan's datacenter IPs for high throughput at a low price.

HTTP, HTTPS and SOCKS5, with username-and-password or IP-allowlist authentication. SOCKS5 also depends on the tool: curl takes socks5h://, requests needs requests[socks], undici uses Socks5ProxyAgent, and Scrapy supports it only through the experimental HttpxDownloadHandler, so stay on HTTP in production.

Yes. Traffic packs start small and don't expire, so prove the workflow on one category or one site first; once the job is steady and runs all day, move it to a Bandwidth plan to keep costs flat. Dynamic residential plans have unlimited concurrency, so scaling up needs no code change.

It depends on what you collect, how, and what for. The basics: collect only public pages that need no login; follow robots.txt, the site's terms and a rate it can handle; and where personal data is involved, collect only the fields you need and have a legal basis. A proxy changes where requests come from, not whether a site allows collection; ask a lawyer about your specific project.

Still have a question?