Scrapy Hitting 429 Too Many Requests?

Blog

September 8.2026

Almost everyone who does data collection has run into a 429. This status code means “too many requests”: the server is telling you that the number of requests sent in a short time exceeds what it can accept. For teams doing <a href="https://www.b2proxy.com/use-case/web" target="_blank">data collection</a> with Scrapy, once 429 shows up frequently, tasks become intermittent, efficiency drops sharply, and it is easy to mistake the problem for a configuration error. <h2 style="line-height: 2;">What Causes 429 Too Many Requests?</h2>429 is an HTTP status code that literally means “too many requests.” It is usually triggered by the server’s traffic management mechanism: the server records how many requests a given source makes within a period of time, and once the threshold is exceeded, it returns 429, asking the requester to slow down.It is worth noting that 429 is not the same as “access denied.” A 429 is more of a temporary pace reminder; the server wants the requester to lower the frequency and try again. It does not usually block access permanently, and requests can often resume once the frequency comes down.The common causes of 429 include: concurrency that is too high, an interval between requests that is too short, requests from a single fixed exit that are too concentrated, and a task whose data volume exceeds what the server tolerates from a single source. Once you understand this, you can see that the core of handling 429 is not to push harder, but to adjust the pace. How to Tune Scrapy Concurrency and Delay to Avoid 429In Scrapy, the two key parameters related to request pacing are the concurrency count and the download delay.The concurrency count determines how many requests are in flight at the same time. The higher the concurrency, the more requests are sent per unit of time, and the easier it is to hit the server’s frequency threshold. Lowering the concurrency is the most direct way to reduce 429 frequency.The download delay determines the interval between two requests. The larger the delay, the slower the pacing and the less pressure on the server. A reasonable delay essentially makes the collection behavior friendlier to the target server, protecting both the other side’s resources and your own task from being interrupted.How to tune is a balancing act: set it too low and 429 appears frequently; set it too high and the task runs too slowly. The usual approach is to start from a conservative value, observe the 429 ratio over a period of time, and fine-tune gradually until you find an acceptable balance between stability and speed. <h2 style="line-height: 2;">How Proxy IP Rotation Helps Reduce 429 Frequency</h2>Besides controlling the pace, another approach is to spread the source of requests more evenly. This is where proxy IP rotation comes in.The principle is that the server’s frequency limits are mostly counted against a single exit source. If all requests go out from the same exit, that exit will quickly accumulate a large number of requests and become a high-frequency source, making 429 more likely. With proxy IP rotation, requests are distributed across different exits, the request density of any single exit drops, and the frequency of 429 falls accordingly.The rotation method also matters. Fully random rotation is simple, but it is not suitable for scenarios that need to keep a session continuous. A more reliable approach is to combine it with sticky sessions: keep the same exit unchanged for a specified window, then switch after the window ends. This spreads the requests while avoiding the interruption of tasks that need to keep state, caused by switching exits too often.When choosing a proxy service, focus on a few capabilities: the size and regional coverage of the address pool, flexible configuration of rotation and sticky sessions, and whether the authentication method is standardized. Professional proxy service providers usually give clear documentation on these dimensions, making it easy to adjust as needed. <h2 style="line-height: 2;">How to Troubleshoot 429 Errors in Scrapy</h2>When 429 keeps showing up, you can troubleshoot in the following order.First, confirm whether it is a pacing problem or something else. Look at the frequency and pattern of 429: does it appear occasionally or constantly in large numbers? Occasional 429 usually means the threshold is just being touched, and fine-tuning the parameters is enough; constant large-volume 429 means you should check whether the concurrency is clearly too high, or whether a single exit has been used at high frequency for too long.Second, check the concurrency and delay. Lower the concurrency and increase the delay, then watch whether the 429 ratio drops. This is the most basic and effective troubleshooting step.Third, check whether the exit is too single. If you use the same exit for a long time, even a moderate pace can trigger 429 because of the accumulated request volume. Introducing rotation to spread the requests usually brings a clear improvement.Fourth, check the data volume of the task. If the target data volume of a single task far exceeds what the server tolerates from a single source, even the best parameters can trigger 429. In this case, consider splitting the task into smaller pieces and running it across time slots, rather than simply adding more requests. <h2 style="line-height: 2;">Conclusion</h2><h2 style="line-height: 2;"> Concurrency + Delay + Proxy Working Together for Stable Collection</h2>The essence of 429 is the server reminding you that the request pace has exceeded an acceptable range. The right way to handle it is to cooperate: use reasonable concurrency and delay to control the pace, and use proxy IP rotation to spread the sources. Only when the three work together can data collection be both stable and continuous.In practice, professional proxy service providers like <a href="https://www.b2proxy.com/pricing/residential-proxies" target="_blank">B2Proxy</a> offer flexible, configurable network exits: you can target by country, state, and city, and switch between rotation and sticky sessions, covering common compliant scenarios such as market research, ad verification, SEO monitoring, and brand protection. Once you combine concurrency, delay, and proxies properly, 429 stops being a roadblock and becomes a variable that can be managed through engineering.

Access B2Proxy's Proxy Network

Just 5 minutes to get started with your online activity

View pricing

Scrapy Hitting 429 Too Many Requests?

You might also enjoy

Proxy lag or disconnects? Speed vs stability ， which matters more?

Overseas Social Media Market Research: Why Does the Data You See Never Match What Locals See?

When Must You Use Proxy IPs for Shopify Operations?

Access B2Proxy's Proxy Network