Integrations

Using a proxy with Python

Fetch a fresh proxy, send traffic through it with requests, and recover when a public proxy fails.

Install.

You need requests. For SOCKS proxies, install its SOCKS extra as well; it adds PySocks.

pip install "requests[socks]"

Get a proxy.

Ask the API for one proxy. With curl=true the answer is a ready-made proxy URL such as http://192.0.2.24:8080.

import requests

API = "https://gimmeproxy.com/api/getProxy"

def get_proxy(**filters):
    r = requests.get(API, params={"curl": "true", **filters}, timeout=10)
    r.raise_for_status()
    if r.text.startswith("{"):          # {"error": "no more proxies left"}
        raise LookupError(r.json()["error"])
    return r.text.strip()

proxy = get_proxy(protocol="http", country="US")

Send traffic.

Pass the proxy URL for both schemes. For a SOCKS proxy, change the scheme to socks5h:// so the proxy also resolves host names.

proxies = {"http": proxy, "https": proxy}
page = requests.get("https://example.com/", proxies=proxies, timeout=(5, 15))
print(page.status_code, len(page.content))

The timeout pair is (connect, read) in seconds. Public proxies are often slow to connect, so a short connect timeout saves the most time.

Handle errors.

Treat a failed proxy as normal: catch the error, fetch another proxy and try again a few times.

from requests.exceptions import ProxyError, ConnectTimeout, ReadTimeout, SSLError

def fetch(url, attempts=4):
    for _ in range(attempts):
        proxy = get_proxy(protocol="http")
        try:
            return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=(5, 15))
        except (ProxyError, ConnectTimeout, ReadTimeout, SSLError):
            continue                     # this proxy is gone; ask for another
    raise RuntimeError("no working proxy after %d attempts" % attempts)