Bandwidth Usage Modeling and Reporting

Statistical Modeling and Systems Engineering · 2026 · Historical analysis and live reporting · 3 min read

Packaged a system that scores homelab traffic against a residential usage model and produces dated JSON and one-page reports.

Overview

Published OpenVault usage summaries provide the statistical reference; pfSense/vnStat provides live measurements. A frozen scorer estimates monthly exceedance and model-relative incidence for total, download, and upload traffic.

Problem

Multi-terabyte repository mirrors and planned cloud-backup uploads raised a practical question: how unusual was this usage? Aggregated, right-skewed source data could not support a raw Gaussian z-score or universal ISP limit.

Constraints

  • Fit aggregate reports without household-level observations or a known outlier population
  • Expose uncertainty in extreme and directional estimates
  • Score vnStat exports without implying an ISP enforcement threshold
  • Keep model regeneration separate from routine reporting

Approach

Extracted summaries from nine reports spanning 1Q24–1Q26, tested lognormal and Weibull fits, then selected a segmented gamma model with a Pareto tail. Frozen parameters score vnStat counters; restricted SFTP and systemd automation deliver scored JSON and printable PostScript.

Key Decisions

Report exceedance probability and model-relative incidence instead of raw sigma

Reasoning:

The source lacks a suitable observed standard deviation. A survival curve answers how common a measured volume is under the fitted model.

Alternatives considered:
  • Use mean and standard deviation as a Gaussian score
  • Set green, yellow, and red bands from anecdotal ISP thresholds

Use a segmented gamma model with a Pareto tail

Reasoning:

The lognormal missed low-usage buckets; Weibull improved the fit, while the selected gamma/gamma structure matched buckets more closely. The 5,000 GB tail remains extrapolative.

Alternatives considered:
  • Single lognormal fit
  • Single Weibull fit

Freeze evaluated parameters and separate the reporter from receiver automation

Reasoning:

RPM builds reuse evaluated weights, while separate service identities limit the upload account's access.

Alternatives considered:
  • Refit the model during every report run
  • Run upload reception and reporting under one account

Tech Stack

  • Python
  • SciPy
  • Jupyter
  • pfSense/vnStat
  • systemd
  • RPM
  • PostScript

Result & Impact

  • 504
    Model configurations
  • 1,008
    Period-level fits
  • ~153 s
    Recorded grid runtime

Dated JSON and one-page reports compare measured homelab traffic with a consistent reference curve.

Learnings

  • Matching the mean by construction is not validation; bucket shares exposed the lognormal failure
  • Low bucket error does not make far-tail or directional projections precise
  • Freezing parameters keeps model changes deliberate and reviewable

What the Numbers Mean

The scorer estimates the fraction of households in its modeled reference distribution that would use at least a measured monthly volume. “1 in N” is its reciprocal, not an observed subscriber count or ISP warning. Directional tails have less validation. The 1Q27 reference is a projection, and rare estimates depend heavily on the Pareto tail.

View a Sample Usage Estimation Report

This report was generated from real bandwidth data, collected from vnStat, with incidence and exceedance estimates created by the system.

Sample Usage Estimation Report (PDF)

You can view and download the code on my GitHub.

Read Deep Dive

GitHub Repository

All projects