Estimate Proxy Bandwidth with Python

ملخص

Pixel shares a Python guide to estimating bandwidth for authorized public data collection in Kazakhstan. It explains the theoretical Mbps needed for a target traffic volume, then factors in utilization and retry overhead to create a more practical estimate.

DEV Community

Estimating Proxy Bandwidth for Public Data Collection in Kazakhstan with Python

Collecting public data responsibly is not only a question of writing faster code. Before increasing concurrency, you need to estimate the traffic volume, the available time window, retry overhead, and the throughput your workload can actually sustain. This guide builds a small Python model for that calculation. It is intended only for requests you are authorized to make. Respect applicable law, the target website's terms, robots directives, and rate limits. 1. Start with the theoretical minimum For decimal terabytes, the average throughput required to transfer a given volume is: required_Mbps = traffic_TB × 8,000,000 ÷ seconds
seconds = days × 86,400
For example, moving 100 TB evenly over 30 days requires: 100 × 8,000,000 ÷ 2,592,000 = 308.6 Mbps
That is only the theoretical average. It assumes continuous transfer, no retries, no pauses, and no protocol overhead. 2. Add utilization and retry overhead A practical estimate should reserve capacity for variable latency, connection setup, retries, and quiet periods. The function below makes those assumptions explicit. from dataclasses import dataclass

SECONDS_PER_DAY = 86_400
BITS_PER_TERABYTE = 8_000_000 # Mbps-seconds in one decimal TB

@dataclass(frozen=True)
class BandwidthEstimate:
theoretical_mbps: float
planned_mbps: float

def estimate_bandwidth(
traffic_tb: float,
days: int,
utilization: float = 0.70,
retry_fraction: float = 0.10,
) -> BandwidthEstimate:
if traffic_tb <= 0:
raise ValueError("traffic_tb must be positive")
if days <= 0:
raise ValueError("days must be positive")
if not 0 < utilization <= 1:
raise ValueError("utilization must be between 0 and 1")
if retry_fraction < 0:
raise ValueError("retry_fraction cannot be negative")

seconds = days * SECONDS_PER_DAY
theoretical = traffic_tb * BITS_PER_TERABYTE / seconds
planned = theoretical * (1 + retry_fraction) / utilization

return BandwidthEstimate(
    theoretical_mbps=round(theoretical, 1),
    planned_mbps=round(planned, 1),
)

estimate = estimate_bandwidth(traffic_tb=100, days=30)
print(estimate)
With 70% utilization and 10% retry overhead, the result is approximately: BandwidthEstimate(theoretical_mbps=308.6, planned_mbps=485.0)
The gap between 308.6 Mbps and 485.0 Mbps is not wasted bandwidth. It is operational headroom for a workload that does not run perfectly every second of the month. 3. Compare the result with standard throughput levels The following table shows the maximum theoretical transfer volume when a connection runs continuously at the listed rate. Throughput 1 day 7 days 30 days 100 Mbps 1.08 TB 7.56 TB 32.4 TB 200 Mbps 2.16 TB 15.12 TB 64.8 TB 400 Mbps 4.32 TB 30.24 TB 129.6 TB 600 Mbps 6.48 TB 45.36 TB 194.4 TB 800 Mbps 8.64 TB 60.48 TB 259.2 TB 1 Gbps 10.8 TB 75.6 TB 324 TB For the 100 TB example, 400 Mbps covers the theoretical average but leaves limited room for retries and idle periods. A 600 Mbps tier provides more practical headroom under the assumptions above. 4. Measure latency instead of guessing A bandwidth plan alone does not guarantee application throughput. Measure representative endpoints before selecting concurrency. import statistics
import time
import requests

def measure_latency(url: str, samples: int = 20) -> dict[str, float]:
durations = []

for _ in range(samples):
    started = time.perf_counter()
    response = requests.get(url, timeout=15)
    response.raise_for_status()
    durations.append((time.perf_counter() - started) * 1000)
    time.sleep(0.5)

ordered = sorted(durations)
p95_index = max(0, int(len(ordered) * 0.95) - 1)

return {
    "p50_ms": round(statistics.median(ordered), 1),
    "p95_ms": round(ordered[p95_index], 1),
    "mean_ms": round(statistics.mean(ordered), 1),
}

Use an endpoint you control or have permission to test. Do not benchmark a third-party website aggressively. P95 latency ...
Estimate Proxy Bandwidth with Python
Powered by Readybot.io

الأحدث من Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …

Why a Java HPA Got Stuck at 20 Kubernetes Pods

Pixel shares how a Spring Boot pricing service autoscaled to 20 Kubernetes pods and stayed there because the JVM retained committed heap memory after traffic …