Scrapy Integration - Rotating Proxies (Middleware Setup)

Overview

Scrapy scales fast, so proxy + retry config matters. This guide shows a clean middleware approach and safe defaults for rotating Proxiesseller proxies.

Installation & Setup

Install Scrapy:

pip install scrapy

Examples

DOWNLOADER_MIDDLEWARES = {
  "scrapy.downloadermiddlewares.retry.RetryMiddleware": 90,
}

RETRY_ENABLED = True
RETRY_TIMES = 3
DOWNLOAD_TIMEOUT = 25
CONCURRENT_REQUESTS = 16

# in your spider:
def start_requests(self):
  yield scrapy.Request(
    url="https://api.ipify.org",
    meta={"proxy": "http://USER:PASS@HOST:PORT"},
    callback=self.parse
  )

Pick a proxy per request (simple rotation).

import random

PROXIES = [
  "http://USER:PASS@HOST1:PORT",
  "http://USER:PASS@HOST2:PORT",
]

def process_request(request, spider):
  request.meta["proxy"] = random.choice(PROXIES)

Troubleshooting

• If requests hang: increase DOWNLOAD_TIMEOUT and reduce concurrency
• If some sites block: enable cookies, use realistic headers, slow down
• If proxy auth fails: verify proxy URL format in meta['proxy']