Making your scraper fail closed when the proxy drops

Every mobile proxy connection drops eventually. A cell tower hands your modem off to a new sector, the carrier renegotiates the session, a SIM hits a throttling threshold, or a port on our end just needs a restart. When that happens mid-scrape, the question that matters is: what does your code do in the half-second before it notices?

Most scraper stacks answer that question badly. They fail open. The HTTP client can’t reach the proxy, so it falls back to a direct connection, or the browser profile silently drops its proxy extension and starts routing through your actual network interface. The scrape keeps running. The logs look fine. And your real IP just made a request to a site that’s now watching for exactly that kind of traffic.

This piece is about the alternative: building scrapers that fail closed, so a dropped proxy stops the request instead of rerouting it.

What fail closed actually means here

Fail closed is a security term borrowed from access control systems. A fail-closed door locks when the power goes out. A fail-open door swings free. For a scraper, the “door” is your proxy connection, and failing closed means: if the proxy isn’t verifiably up and routing traffic the way you expect, the request does not go out at all.

That’s a stricter bar than “handle proxy errors gracefully.” A lot of scraping code already retries on a ProxyError or a timeout. That’s necessary but not sufficient. The dangerous case isn’t the clean error, it’s the silent one, where the connection technically succeeds but not through the path you think it’s going through.

Why mobile proxy connections drop in the first place

If you’re running through a datacenter IP that sits on a static server with a fixed uplink, drops are rare and usually mean something is actually broken. Mobile proxies are different because the exit node is a phone modem on a cellular network, and cellular networks are built to hand you off, not to hold a session open forever.

A few things we deal with running SIM-based exits on SingTel, M1, and StarHub:

• Tower handoff. As the carrier moves your session between cell sectors for load balancing, the underlying IP allocated by carrier-grade NAT can change mid-session. Your TCP connection to the proxy port doesn’t necessarily die, but the egress IP on the other end might.
• Carrier NAT re-lease. Mobile carriers put thousands of subscribers behind shared IP pools. The NAT mapping your session depends on has a lease, and if it expires or gets reassigned, your outbound requests start going out under a different public IP without any error being thrown on your side.
• Data thresholds and throttling. Some SIM plans throttle or briefly interrupt data after a usage threshold in a billing cycle. That interruption can look like a slow timeout rather than a hard failure.
• Modem-level restarts. On our end, if a modem needs a power cycle or a port needs to be reprovisioned to a different device, there’s a window where the port is technically listening but not actually forwarding traffic yet.

None of this is a defect in mobile proxies specifically, it’s the tradeoff for using an exit IP that looks like a real Singapore phone to the site you’re scraping, instead of a datacenter block that gets flagged on sight. But it means your code has to assume the connection is unstable by design, not treat instability as an edge case.

The failure mode that actually hurts you

Say you’re running a multi-account operation, one browser profile per proxy port, scraping or automating against a platform that fingerprints IPs alongside device and behavior signals. The proxy for profile 14 drops for six seconds during a tower handoff. If your automation framework falls back to the host machine’s network interface for that gap, profile 14 just sent a request from your real IP, correlated with a browser fingerprint, cookies, and session state that the platform has been tracking against a Singapore mobile IP the whole time.

That’s not a failed request. That’s a burned identity, and depending on how the platform links accounts by originating network, it can be a burned batch. The cost of a scraper that fails open isn’t “I lost some data for a few seconds,” it’s “I handed the target a fingerprint match between my automation and my real infrastructure.”

For straight scraping without account state, the damage is smaller but still real: your real IP hits the site, gets logged, and if it’s a residential or business IP tied to other work you do, you’ve now associated that IP with scraping traffic on a site that might rate-limit or block it.

Building the fail-closed check

The fix isn’t complicated, but it has to sit at the right layer. Retry logic on the HTTP client catches connection errors. It does not catch “the connection succeeded, but not through the proxy.”

Verify the egress IP before trusting the session. Before starting a batch of requests through a given proxy port, hit a simple IP-echo endpoint through that exact connection and compare the returned IP against the IP you were assigned for that port. If it doesn’t match, or if the request fails, don’t proceed. Re-check periodically during long-running sessions, not just at the start, since the IP can shift mid-session on carrier handoff.

Never configure a system-level fallback route. This is the one that catches people. Some proxy client libraries and some browser automation tools have a “fall back to direct connection on proxy failure” setting, sometimes on by default, sometimes hidden behind a flag meant for local development. Find that setting and turn it off explicitly. The library maintainers built it to make debugging easier, not for production traffic through a proxy you’re relying on for a specific exit IP.

Check for leaks at the browser level, not just the HTTP level. If you’re doing browser automation rather than raw HTTP requests, WebRTC can leak your real IP through STUN even when your HTTP traffic is correctly proxied, because WebRTC negotiates its own connection path outside the browser’s proxy settings. Disable WebRTC or force it through the proxy explicitly in your browser profile config, and confirm with a leak test page through an actual automated run, not just a manual check in a regular browser window.

Treat a proxy health check as a precondition, not a background monitor. A lot of setups ping proxy health every few minutes and alert if something’s down. That catches the outage after the fact. What you want instead is a check immediately before each request or each batch: is this specific port’s egress IP what I expect right now. If the check fails, the request doesn’t fire. This is the actual fail-closed part, everything else is monitoring.

Use a circuit breaker per port, not per pool. If port 14 is flapping, you want that specific profile’s requests to stop while the rest of your farm keeps running normally. A shared circuit breaker across your whole proxy pool means one unstable modem takes down scraping on every other port too, which is its own kind of failure, just a less dangerous one than leaking an IP.

Testing that it actually works

The only way to know your fail-closed logic works is to break the proxy on purpose. Kill the proxy port mid-session and watch what your scraper does. If requests stop and you get a clean error in your logs, good. If a request goes out anyway, you’ve found the leak before a target site did. Do this test after any change to your HTTP client, proxy library, or browser automation framework, since a version bump can silently reintroduce a fallback behavior you’d previously disabled.

It’s worth doing this test against a target you control, like your own echo endpoint, rather than against a live scraping target, so you’re not burning a real session to validate infrastructure.

A dropped proxy connection is not a rare event when your exit nodes are real mobile SIMs on real carrier networks. It’s a normal part of how cellular data works. The scrapers that hold up over time are the ones built to stop cleanly when that happens, not the ones that quietly keep going through whatever connection is available.

If you’re running scraping or multi-account automation through Singapore mobile IPs and want exits on actual SingTel, M1, or StarHub hardware instead of datacenter blocks, take a look at what we run over at Singapore Mobile Proxy.

Get new guides and videos first — join the Telegram channel.