rl-deploy-bench: a tested cross-deployment pitfall taxonomy

Config-driven CLI / Python API · ONNX, TorchScript, FP16, INT8 · latency P50–P99 + RL action accuracy · v1.1.0

Repository: dafahaha/rl-deploy-bench · every fix below is locked by a regression test in tests/test_core.py unless noted

What the toolkit is — and what it is not

rl-deploy-bench exports policies to ONNX/TorchScript, converts to FP16/INT8, and benchmarks latency (P50/P90/P95/P99), throughput, and RL-specific action accuracy (MSE + cosine similarity vs the FP32 reference), emitting Markdown or HTML reports. This page does not advertise measured speedups: the toolkit was developed and tested on CPU; Jetson/edge is a target deployment path, not a device that has been measured here. The value documented below is the catalog of deployment failure modes that were actually hit, reproduced minimally, detected, fixed, and regression-tested. Each card follows the same four-part structure: phenomenon → minimal trigger → detection → fix/avoidance.

A · Calibration data collection

A1 · Intermittent / consecutive dry episodes hang the calibrator

Phenomenon
An observation_filter rejects every transition in an episode (e.g. an out-of-range or NaN filter). With num_episodes=None the collection loop waits for a fixed sample count and never receives one — an apparent hang. A single dry episode is benign; a run of them means the filter or env is broken.
Minimal trigger
Scripted fake env that always returns transitions the filter rejects; run collect_calibration_data without an episode cap.
Detection
Counter of consecutive dry episodes; raise only on the 3rd consecutive dry episode; any productive episode resets the counter to zero. A bounded num_episodes path finishes without raising.
Fix / avoid
Fail-closed guard in benchmark/calibration.py; filter exceptions propagate instead of being swallowed; np.load(..., allow_pickle=False) for saved datasets. Tests: TestCalibrationData::test_calibration_rejecting_filter_raises_instead_of_looping, test_calibration_intermittent_dry_episode_does_not_raise_with_num_episodes_none, test_calibration_guard_raises_exactly_on_third_consecutive_dry_episode, test_calibration_productive_episode_resets_consecutive_dry_counter, test_calibration_filter_exception_propagates, test_calibration_load_rejects_object_arrays.

A2 · Calibration distribution drift: random noise ≠ deployment observations

Phenomenon
Static INT8 needs activation-range statistics. Calibrating on i.i.d. Gaussian noise when the deployed observation distribution is concentrated (or bounded) biases the min/max range estimate, so quantized activations clip or underflow on real inputs.
Minimal trigger
Static quantization of a policy with the random-data fallback on an env whose observations are not Gaussian.
Detection
Per-dimension calibration statistics recorded and reported; choose policy (real rollouts), random, or mixed explicitly; compare quantized vs FP32 action MSE/cosine on held-out real observations.
Fix / avoid
Prefer EnvironmentCalibrationGenerator rollouts for static INT8; treat random calibration as a smoke test, not a deployment range estimate. Source: benchmark/calibration.py, quantizer/int8.py::static_quantize.

B · Export action semantics

B1 · Missing action-bounds affine map on the verification side

Phenomenon
The ONNX graph maps tanh ∈ [−1, 1] to the env's [low, high], but the verification harness compares against raw tanh outputs. On a Pendulum-style asymmetric bound this produced a max action difference of ~0.43 even though the graph was correct.
Minimal trigger
Export a bounded action model without passing action_low/high to verification.
Detection
Reference-vs-ONNX action comparison on hand-chosen anchor inputs; large systematic offset flags the missing affine map.
Fix / avoid
Pass action_low/action_high through export and verify; anchors include the bounds and interior points. Tests: TestModelExport action-bounds cases in tests/test_core.py; source exporter/onnx_export.py.

B2 · A Python-level branch traced as a constant breaks dynamic batch

Phenomenon
Code like if low.shape[0] == 1: expand(...) is evaluated at trace time and baked as a constant; batch > 1 then takes the wrong branch silently.
Minimal trigger
Export with dynamic batch and compare batch=1 vs batch=4 outputs.
Detection
PyTorch TracerWarning at export; a batch=1/batch=4 equivalence check catches the break.
Fix / avoid
Unconditional expand to the leading dimension inside the traced graph; never branch on tensor shape in Python during tracing. Test: batch=1 vs batch=4 match in TestModelExport.

B3 · Unclamped extrapolation outside [−1, 1]

Phenomenon
Without torch.clamp, numerical drift outside tanh's saturated range maps to actions outside the env's legal bounds.
Minimal trigger
Hand-computed anchor inputs at and beyond the saturation edges.
Detection
Anchor-based verification asserts outputs stay within [low, high].
Fix / avoid
Clamp the rescaled action to [low, high] in the exported graph. Source: exporter/onnx_export.py.

B4 · SB3 double-unscaling continuous actions

Phenomenon
policy._predict already returns actions mapped into the action space; re-applying the tanh-unscale on top produced ~2×-deviating ONNX output.
Minimal trigger
Train a tiny PPO on Pendulum-v1, export, compare ONNX output to model.predict.
Detection
Tight np.allclose(atol=1e-5) against model.predict; a second unscale gives ~0.1+ deviation.
Fix / avoid
Return the already-scaled action as-is in exporter/sb3.py. Test: TestSB3Export::test_sb3_onnx_matches_predict (skipped automatically when the [sb3] extra is absent).

C · Quantization mechanics

C1 · Static quantizer crashes: "Invalid rank: Got 1 Expected 2"

Phenomenon
Random calibration samples are a flat 1-D vector; the onnxruntime quantizer expects a leading batch dim and raises a rank error instead of calibrating.
Minimal trigger
static_quantize(..., calibration_samples=8) on a freshly exported ONNX model.
Detection
Quantized model must load and run; rank error appears immediately on the calibration call.
Fix / avoid
Add obs[np.newaxis] leading dim on the calibration batch. Test: TestQuantization::test_static_quantize_with_random_data.

C2 · Custom input name mismatch during calibration

Phenomenon
A model exported under input name "obs" was calibrated against the hardcoded "observation", so the quantization graph bound the wrong tensor.
Minimal trigger
ExportConfig(input_names=("obs",)) then static quantize without passing input_name="obs".
Detection
Runtime lookup failure or silent binding to a missing tensor; explicit name round-trip test.
Fix / avoid
Thread the model's real input name through the random-calibration path, matching static_quantize_with_dataset. Test: test_static_quantize_respects_custom_input_name.

C3 · Missing shape inference leaves temp artifacts on save failure

Phenomenon
Without onnx.shape_inference, onnxruntime pre-processing warns or errors; and a shared helper that writes a temp model leaked the file when onnx.save raised.
Minimal trigger
Monkeypatch onnx.save to raise inside _shape_inferred_temp_model; check the temp path afterwards.
Detection
Assert the temp file no longer exists after the propagated error.
Fix / avoid
Run shape inference in the shared preprocessing helper; unlink the temp file in an error path. Test: test_shape_inferred_temp_model_cleans_up_when_save_fails.

C4 · FP16 conversion exceptions were swallowed by a blanket except

Phenomenon
An earlier except (ImportError, Exception): pass masked real converter failures, silently emitting whatever the fallback left behind.
Minimal trigger
Force the onnxruntime.transformers.float16 import to raise a non-ImportError failure; the conversion must raise RuntimeError, not return.
Detection
Only ImportError triggers the manual fallback; every other exception re-raises wrapped in RuntimeError. Manual fallback output passes onnx.checker and runs in onnxruntime.
Fix / avoid
Separate "converter unavailable" (warn, fallback) from "converter failed" (raise); manual Method-2 cannot honor op_blocklist, so it warns loudly instead of ignoring the setting; handle both op_blocklist / op_block_list kwarg names across onnxruntime versions. Tests: test_fp16_conversion_roundtrip, test_fp16_manual_fallback_method2_runs, test_fp16_manual_fallback_warns_about_op_blocklist, test_evaluate_fp16_impact_respects_cosine_threshold.

D · Latency measurement hygiene

D1 · No warm-up inflates first-call latency

Phenomenon
Session creation, CUDA context / onnxruntime graph compilation, and memory allocation happen on the first session.run; including it in the sample biases P50/P90 upward.
Minimal trigger
Compare measured latency with and without num_warmup iterations.
Detection
Drop the first num_warmup calls (default 50) before timing; report P50/P90/P95/P99 from the steady-state sample.
Fix / avoid
Warm-up loop in benchmark/latency.py; same warm-up protocol across backends so numbers are comparable.

D2 · Monitor handle leak and non-positive run counts

Phenomenon
If benchmarking raised mid-run, the resource monitor was never stopped (NVML/psutil handle leak); and num_runs ≤ 0 produced NaN percentiles.
Minimal trigger
Force a mid-benchmark exception; assert monitor.stop() ran; pass num_runs=0.
Detection
try/finally around the timed loop; monitor snapshot failure is best-effort and dropped; reject num_runs ≤ 0 up front.
Fix / avoid
Fail-closed validation and guaranteed cleanup in benchmark/latency.py; tests in TestLatencyBenchmark.

D3 · TensorRT per-call cuda.mem_alloc/free dominates the clock

Phenomenon
Allocating and freeing device buffers on every measured call adds allocation overhead that is not part of steady-state inference and dwarfs the compute on small policies.
Detection
Profile allocation calls inside the timed region; compare cached vs per-call buffer latency.
Fix / avoid
Cache/reuse I/O buffers across calls in runtime/tensorrt_runtime.py; free them in close()/__del__. Note: this path is framework-only here (TensorRT not installed on the dev CPU), so it is reviewed code, not unit-tested on device.

D4 · FPS caliber must be stated, not assumed

Phenomenon
Latency measures the internal session.run call only; Python scheduling, copying, and observation preprocessing are excluded. Comparing to end-to-end frame rates elsewhere is invalid.
Detection
State the caliber in every report and README; keep the same caliber for all models in one comparison.
Fix / avoid
Report P50/P90/P95/P99 plus FPS derived from the same internal-call sample; never relabel as end-to-end throughput.

E · Reporting & robustness

E1 · Markdown written into a .html report

Phenomenon
compare --output report.html emitted Markdown content under an HTML suffix, so the viewer rendered raw markdown text.
Minimal trigger
Run rl-deploy-bench compare fp32.onnx quant.onnx --output report.html and inspect the head.
Detection
Branch on file extension: real HTML starts with <!doctype html>/<html> and contains plotly; Markdown does not.
Fix / avoid
Extension-dispatched generator in cli.py. Test: test_compare_html_output_is_real_html (asserts plotly present).

E2 · HTML/Markdown injection from model metadata

Phenomenon
Model name, title, and platform_info flow into reports; HTML tags in a name would inject markup, and | or newlines would break Markdown tables.
Minimal trigger
Name the model <img src=x onerror=1> | a b; generate both reports.
Detection
Inspect emitted files for unescaped tags / broken table rows.
Fix / avoid
html.escape in the HTML reporter; | escaped and newlines folded in the Markdown reporter. Tests: test_html_report_escapes_model_name_and_title, test_markdown_report_escapes_pipe_in_name, test_markdown_report_folds_newlines_in_model_name, test_markdown_report_title_folds_newlines.

E3 · Divide-by-zero on degenerate latency samples

Phenomenon
Empty or zero-duration samples make FPS/MSE denominators zero and crash report generation.
Detection
Guard denominators; render n/a instead of inf/NaN.
Fix / avoid
NaN-safe formatting in the reporters; n/a for undefined ratios.

E4 · NVML session leak when mid-init detection raises

Phenomenon
nvmlInit() succeeded but a later nvmlDeviceGetHandleByIndex raised; without nvmlShutdown() in the error path the NVML session leaked. Same pattern in platform detection and the GPU monitor.
Minimal trigger
Monkeypatch a fake pynvml whose handle call raises; assert nvmlShutdown was called exactly once.
Detection
Mock-based assertion of shutdown calls on the failure path.
Fix / avoid
Shutdown in the except branch of both platform.py detection and monitor/nvidia_gpu.py. Tests: test_detect_nvidia_gpu_shuts_down_nvml_when_middle_step_raises, test_start_shuts_down_nvml_when_get_handle_raises.

E5 · Non-ASCII Windows path breaks TorchScript saving; duck-typing the benchmark seam

Phenomenon
On Windows, torch.jit.save through a Chinese-character (non-ASCII) path can fail; and a custom runtime that omitted the actions/latency_ms/get_provider_info seam crashed benchmark_latency with AttributeError.
Minimal trigger
Export TorchScript to a non-ASCII path; dock a TorchScript runtime into benchmark_latency end-to-end.
Detection
Smoke test matches PyTorch reference and reports backend="torchscript" with a captured input shape.
Fix / avoid
Copy to a unique ASCII temp path inside _safe_load_torchscript before torch.jit.load (the non-ASCII Windows hardening lives on the load side; torch.jit.save writes to the requested path); provide the unified inference duck interface so alternate backends plug into the same loop. Legacy torch.jit emits a FutureWarning on newer torch; ONNX / torch.export are preferred. Test: TestTorchScriptExport::test_torchscript_smoke_matches_pytorch_and_docks_into_benchmark. (The non-ASCII temp-copy path itself is platform hardening without a dedicated hermetic test.)

Reproduction

git clone git@github.com:dafahaha/rl-deploy-bench.git
cd rl-deploy-bench
pip install -e ".[dev,sb3]"

# full regression suite (matrix: Python 3.10 / 3.11 / 3.12 in CI)
pytest tests/ -v

# static checks enforced in CI
black --check src tests
isort --check-only src tests
flake8 src tests

# shipped walkthrough (small MLP, CPU)
python demo.py

# CLI: export an SB3 policy -> quantize -> benchmark -> compare
rl-deploy-bench export ppo_pendulum.zip --algo PPO --output model.onnx
rl-deploy-bench quantize model.onnx --mode dynamic --output model_int8.onnx
rl-deploy-bench benchmark model.onnx --obs-shape 3 --num-runs 2000 --output report.md
rl-deploy-bench compare model.onnx model_int8.onnx --obs-shape 3 --num-samples 100 --output report.html
# FP16 conversion has no CLI subcommand; it is a Python-API call
# (convert_onnx_to_fp16) in quantizer/fp16.py.

CI: .github/workflows/ci.yml runs black/isort/flake8, byte-compile, and pytest on Ubuntu across Python 3.10–3.12 (mypy is a local-only check and is intentionally not run in CI); optional extras are guarded with importorskip so the suite stays green without TensorRT, the SB3 extra, or a GPU.

Scope This is a personal exploratory engineering toolkit (v1.1.0, Beta per pyproject.toml). Pitfalls above are documented from real bug-fix commits and locked by tests; they are not claims about any framework being defective, and no on-device Jetson measurement is reported here.