Research Notes
Daizhi Liao (dafahaha) · Network Engineering, Guangzhou University · Reinforcement Learning & Embodied AI
Two exploratory, independently built artifacts presented with the numbers exactly as measured: a pre-registered three-arm SAC experiment that returned a null result, and a cross-platform RL deployment benchmarking toolkit whose tested failure modes are collected into a deployment-pitfall taxonomy. These are course / independent exploration projects, not published papers and not SOTA benchmarks.
1 · Does critic reward scaling change SAC on Hopper-v4?
A from-scratch PyTorch SAC (twin critics, soft targets, tanh-squashed Gaussian actor, automatic entropy temperature) run on CPU at a 120k-step budget, 3 seeds per arm. The pre-registered question was whether scaling the rewards entering the critic's Bellman backup by 0.1 vs 1.0 changes sample efficiency or asymptotic return. The measured answer at this budget and variance: no detectable effect (primary contrast Baseline vs RewardScaled, Welch p = 0.953; ANOVA F = 0.22, p = 0.811). An apparent ~3× win seen at an early 60k pass did not survive the full budget and a fixed-alpha control arm.
Read the study page · Repository · Pre-registration · Retrospective
2 · rl-deploy-bench: cross-platform deployment pitfalls, tested
A config-driven CLI and Python API that exports policies to ONNX/TorchScript, converts to FP16/INT8, and benchmarks latency (P50/P90/P95/P99), throughput, and RL-specific action accuracy, with auto-generated Markdown/HTML reports. Instead of advertising speedups, the notes below catalog the failure modes actually hit and fixed — calibration dry episodes, action-bounds affine mismatches, missing batch dims in static quantization, silently swallowed converter errors, monitor-handle leaks, and report injection — each with a minimal trigger, a detection method, and the regression test that locks the fix.
Read the pitfall taxonomy · Repository · Changelog v1.1.0 · API docs