Reliability

Handling timeouts and retries

Public proxies come and go. A short timeout and a fresh proxy on failure keep your script moving.

Why proxies fail.

Public proxies run on other people’s machines. They get busy, restart, change address or close. A proxy that passed its check a few minutes ago can be gone now. Plan for it instead of fighting it.

Set timeouts.

Without a timeout, one dead proxy can hold your script for minutes. Use a short connect timeout and a longer read timeout:

1. Connect: about 5 seconds.
2. Read: 15 to 30 seconds, depending on the site.
3. Whole request: cap it, so a slow drip cannot run forever.

Retry with a new proxy.

When a request through a proxy fails, do not retry the same proxy: ask the API for another one. A few attempts are usually enough.

for attempt in 1 2 3 4; do
  p=$(curl -s 'https://gimmeproxy.com/api/getProxy?curl=true&protocol=http')
  case "$p" in "{"*) echo "pool: $p"; sleep 5; continue;; esac
  curl -sS --connect-timeout 5 -m 30 -x "$p" https://example.com/ && break
done

The same loop in Python is in Using a proxy with Python.

Respect API limits.

Free requests are limited to 240 per IP per day; paid keys have a quota and a per-second limit. Over the limit the API answers 429.

• Read Rate-Limit-Remaining (free) or X-RateLimit-Remaining (key) to see what is left.
• On 429, wait: until Rate-Limit-Reset, or for Retry-After seconds.
• Back off between retries instead of looping as fast as possible.