Scaling prplOS WiFi

A push-button, agent-driven measurement rig

Build → flash → associate → measure → symbolize → report — orchestrated end-to-end by an LLM agent against the live testbed.

Fully automated build → deploy → measure loop

Flamecharts that actually resolve, plus live counters

Profiles say where; counters say how often and how much. Together they pin the mechanism.

uprobes when perf probe segfaults

• perf probe segfaults on this perf 5.15 musl build — so probes are registered manually via tracefs (/sys/kernel/debug/tracing/uprobe_events) and counted with perf stat -e <probe> -p $(pidof wld).
• Functions live in /usr/lib/libwld.so.7.27.8 (the on-DUT plugin is stripped). Offsets = .dynsym st_value (executable PT_LOAD has p_offset==p_vaddr).
• DM write-txn counted on swl_object_finalizeTransactionOnLocalDm in libswla (2 mappers) — not amxd_trans_apply in libamxd (92 mappers — would int3 the whole box).

The 6 acceptance gates

monotonic CPU 18<60<114 · unknown-frame % within budget · assoc count matches datamodel · governor/CLK_TCK verified · RSS bounded · functional parity (clients held, disassoc cleans up). Every shipped run passed all six first-pass.

The cost is the polling, not the traffic

Why it scales with stations, not bytes

The prplMesh monitor polls getStationStats ~1×/s per radio. Each call ran a full per-station datamodel write-back of mostly-static fields.

 event loop → USP "operate" RPC → amxd_object_invoke_function
   _getStationStats                    41.9%   (≈57% of resolved wld CPU)
    └ s_addStaStatsValues              36.0%
       └ wld_vap_sync_assoclist        27.3%   ← loops ALL N stations / VAP
          └ wld_ad_syncInfo            25.2%   ← per-station write of 40+ params
             └ finalizeTransaction     22.4%   ← one DM write-txn / station / poll
      

Write-txns/call ≈ stations-per-VAP ⇒ one datamodel write-transaction per associated station per poll. ~1.9 ms/station-sync, constant — total scales with density, independent of load.

Retiring the original trace-zone hotspot

One int beats an 80-zone strcmp walk

sahtrace registered ~80 trace zones; every trace check walked the whole list + an "all" fallback, strcmp-ing each — even when the message would never print. The hot swl/pwhm netlink path fires ~1036 INFO traces.

/* libsahtrace fork (patch 0100): short-circuit before the zone walk */
if (lvl > maxEffectiveZoneLevel)   /* one int, recomputed at the 4 zone mutators */
    return;                        /* was: walk ~80 zones + "all" fallback, strcmp each */
      

Caught pre-flight: sah_trace_level is unsigned on this toolchain, so the −1 sentinel promoted to UINT_MAX and silently suppressed all tracing — a perf-only run would have passed. A host equivalence test (now a regression guard) caught it; fixed with a signed cast.

Gate the static datamodel write-back

Why the freshness gate never fired

• Every poll pulls a fresh driver sample → advances lastSampleTime.
• The gate at wld_assocdev.c:1929 compares lastSampleSyncTime != lastSampleTime → always true.
• So all 40+ static params rebuilt + applied every poll, per station — though identity hasn't changed since association.

/* before: any new sample re-applies the whole static block */
if (pAD->lastSampleSyncTime != pAD->lastSampleTime) { rebuild_40_params(); apply(); }

/* after (0901): only on genuine change; otherwise clean, no apply */
if (first_sync || assoc_state_changed) { rebuild_40_params(); apply(); }
else                                   { amxd_trans_clean(&trans); }   /* steady state */
pAD->lastSampleSyncTime = pAD->lastSampleTime;   /* watermark still advanced */
      

Result in the profile: wld_ad_syncInfo 25.2% → 0.5%; finalizeTransaction 22.5% → 0.6%; amxd_trans_apply 21.9% → 0.6%. The readback (swla_dm_getObjectParams, ~13.7%) is now the top slice → Fix #2.

A lean getStationStatsBrief() RPC

Parity by construction, isolated by A/B

Cumulative on the hot path: ~44% → 28.5% → 18.4%

~58% off the getStationStats hot path

Before → after, @114, real symbolized profiles

⤢ Fullscreen for the interactive flamegraph (click a frame to zoom, Esc to close), or open in a new tab. What remains in "after" is the inherent per-station driver pull — the target of Fix #6.

When the data said no

The bulk API is never taken on MaxLinear

mfn_wvap_get_station_stats binds to the MXL HAL, which loops per station — 3 netlink round-trips × N per radio per poll, not one bulk dump.

Fix #6 — collapse the per-station netlink storm

All 14 fields are reproducible from one bulk read

The driver's nl80211 GET_STATION is itself fed by the same tr181 helpers the vendor subcmds use. Every brief key except DownlinkBandwidth derives from mtlk_sta_get_tr181_peer_stats ⇒ a new bulk subcmd is parity-by-construction.

Snapshot-safe: tr181 deltas read read-only; reset only on RESET_STATISTICS — no double-consume. The unused lean GET_ASSOCIATED_STATIONS_STATS covers only ~6/14 with absolute (not delta) semantics → a new subcmd, not reuse.

An LLM agent drove the whole testbed

List-price equivalent — and what caching saved