Driver dies under TLS-1.3 bursts #3

New issue

Open

opened 2026-05-22 05:14:03 -05:00 by dmemarko · 2 comments

dmemarko commented

2026-05-22 05:14:03 -05:00

(Migrated from github.com)

Hardware

Console — PS5 CFI-1108A 01X, FW 4.50
Chip — Marvell (PCIe 1b4b:2b56 (PCIE9097), card_type=0x207)
AP — 5 GHz, 80 MHz, WPA2-PSK, HE-NSS-2, MCS-11
Firmware — PCIEIW620-17.130.3.p28
Kernel — Linux 7.0.8
Distro — Arch Linux. Cross-checked on Ubuntu (powersave disabled): identical failure under the same trigger.
Cross-check — PS5 OS does not suffer from the same burst (verified by running a crafted ELF payload of the trigger).

Bug summary

Concurrent TLS-1.3 ClientHellos with the X25519MLKEM768 PQ key_share (≈1700 B ClientHello, fragmented across TCP segments) fill the FW command queue. Any periodic host-to-FW command issued during that window times out (10 s) and triggers MLAN_EVENT_ID_DRV_DBG_DUMP. The event handler in moal_recv_event() always sets priv->phandle->driver_status = MTRUE ("abnormal driver state"). All later cfg80211 ops are then blocked with Block woal_cfg80211_* in abnormal driver state until a successful fw_reload clears the flag — but neither of the FW reset paths the driver knows about (in-band reset, PCI FLR) reliably recovers.

Reproducer

Prerequisites:

Wifi power save disabled. On Arch: NetworkManager wifi.powersave=2; on Ubuntu: toggle it off in the network settings. With the default wifi.powersave=3 (enabled), the driver dies within seconds of association, before any burst — separate kill path, must be ruled out first to study the burst-triggered one.

Trigger:

while :; do
  for h in www.google.com accounts.google.com clients2.google.com \
           www.bing.com www.duckduckgo.com www.cloudflare.com www.akamai.com \
           rgl.rockstargames.com www.amazon.com ipv4.icanhazip.com; do
    curl -s --max-time 4 --tls-max 1.3 --curves X25519MLKEM768 \
         -o /dev/null "https://$h/" &
  done
  sleep 3
done

Driver dies consistently in ~8–18 s on TLS variant, ~90–120 s on standard HTTPS, ~14 s on real-world Chromium 124+ (Rockstar Games launcher's PSN signin flow). Single-host bursts do not reproduce — concurrent sockets are required.

Failure fingerprint

Timeout cmd id (X.YYYY) <CMD> [0x<id>], act = 0x<act>
Commmand Timeout
num_cmd_timeout = 1
tx_buf_size=3103 max_tx_buf_size=4096
txbd_rdptr=0xRRR txbd_wrptr=0xRRR + 0x80    <-- TX BD ring FULL (128 entries)
rxbd_rdptr=0xRRR rxbd_wrptr=0xRRR + 0x80    <-- RX BD ring FULL
tx_pending = 169   wmm_tx_pending[2] = 169
PCIE DEBUG MODE OUTPUT START
Fail to pull ctrl_data=0x0 host_ready=0x0
Start to process hanging
Send event_hang(0x0) vendor event
Block woal_cfg80211_<op> in abnormal driver state    <-- stuck, permanent until fw_reload

Both TX and RX BD rings full at the same time is consistent across reproductions: FW has stopped processing the rings in both directions.

Cmds that timed out and triggered the trap (observed across runs)

HostCmd ID	Name	Issued by	Caller
`0x00a4`	`HostCmd_CMD_RSSI_INFO`	`woal_get_signal_info`	cfg80211 `dump_station_info` (NM periodic poll)
`0x000b`	`HostCmd_CMD_802_11_GET_LOG`	`woal_get_stats_info`	cfg80211 `dump_survey` and others
`0x0024`	`HostCmd_CMD_802_11_DEAUTHENTICATE`	mlan deauth path	cfg80211 `deauthenticate` (user-initiated)
`0x00a9`	`HostCmd_CMD_FUNC_INIT`	mlan init	post-recovery driver init when FW state stuck

FW recovery paths (`auto_fw_reload` / `indrstcfg`) — none of them reliably work

Mode	Behavior
`auto_fw_reload=0`	No recovery attempted; driver stays in abnormal state. `rmmod + modprobe` alone does not clear it — the chip's scratch register still reports "FW running", the new driver instance prints `WLAN FW already running! Skip FW download`, `Firmware Init Failed`, probe returns `-14`, no iface comes up.
`auto_fw_reload=3` (in-band)	`woal_reset_adma` logs `ADMA reset failed (value:97)`. `woal_pcie_reset_fw` logs `Failed to poll FW reset register 1C94=0x98`. `PCIe In-band Reset Fail`.
`auto_fw_reload=1` (FLR via `pci_reset_function`)	Brick-removes the device until reboot — confirmed across multiple sessions.

Manual recovery sequence	Outcome
`rmmod moal mlan && modprobe moal` (no chip reset)	Always fails. `WLAN FW already running` is logged, init returns -EFAULT, no iface.
`rmmod moal mlan && echo 1 > /sys/bus/pci/devices/0000:40:00.7/reset && modprobe moal` (FLR)	Never fully restored wifi after the bug fired. Two observed outcomes: (a) scan works but auth loops 5x, then the driver's own `"WiFi Reset due to auth timeout cnt 5"` trigger drops it back into abnormal state; (b) `FUNC_INIT` cmd times out during init, driver doesn't register an iface at all.
Full reboot	Always recovers.

Host knobs and trigger variants tried

8 single-variable runs of the TLS burst against the baseline driver.

#	Knob	Result	Triggering cmd
0	baseline (TLS PQ burst)	died in 14 s	RSSI_INFO 0xa4
1	`pcie_aspm=off` on kernel cmdline	died in 8 s (faster)	RSSI_INFO
2	trigger=plain HTTPS GETs (no PQ ClientHello)	died after >90 s (slower, generic burst still kills it)	RSSI_INFO
3	trigger=`ping -f -s 1700` to gateway	survived 240 s	— (single socket, unidirectional)
4	`ps_mode=0 auto_ds=0` (PS disabled)	died in 9 s	RSSI_INFO
5	`max_tx_buf=2048` (half A-MSDU window)	died in 15 s	RSSI_INFO
6	`ring_size=32` (smaller TX BD ring)	died in 18 s	RSSI_INFO
7	`pcie_int_mode=0` (legacy IRQ)	driver fails to probe (-EFAULT — PCIe slot does not deliver legacy INTx)	—

Time-to-die and the failure fingerprint are unaffected by every module-param knob tried, except ASPM-off which made it worse. The only trigger that does not kill the driver is a single-socket one-way flood; the bug needs concurrent sockets with two-way handshake activity.

Root cause

FW cmd queue fills up under PQ-TLS / concurrent-socket bursts. FW stops processing both BD rings (TX and RX go to ring-full) and stops responding to host cmds. This is FW-side, not driver-side: the same trigger against PS5 OS does not break anything.
An over-aggressive recovery trap in the driver. A single cmd timeout sets driver_status = MTRUE permanently. The trap is doing real work — suppressing it deadlocks the kernel — but is the only thing that turns a brief FW hiccup into a permanently broken driver.
No working FW-reset path. In-band reset polls a register that stays stuck at 0x1C94=0x98; FLR removes the device from the bus until reboot.

Ruled out as cause (per the single-variable runs above)

ASPM (turning it off made things worse).
A-MSDU window / TX BD ring depth / IRQ mode (no effect).
Module-level power-save (ps_mode=0 auto_ds=0) — no effect on time-to-die under the burst. (NM-level wifi.powersave is a separate, much faster kill path — see Reproducer prerequisites.)
TLS PQ key_share specifically (standard HTTPS bursts lock the driver up too, slower).
Linux network stack / cfg80211 layer / NetworkManager (USB rtw89 + same NM survives the same burst on the same host).
Trigger shape (any concurrent-socket two-way load kills it; only single-socket one-way doesn't).

Artifacts

iw620-bug-trigger.sh — reproducer that kills the driver and captures dmesg/journal/state.
iw620-bug-20260522-125911.tar.gz — capture bundle from iw620-bug-trigger.sh

## Hardware - **Console** — PS5 CFI-1108A 01X, FW 4.50 - **Chip** — Marvell (PCIe `1b4b:2b56` (PCIE9097), `card_type=0x207`) - **AP** — 5 GHz, 80 MHz, WPA2-PSK, HE-NSS-2, MCS-11 - **Firmware** — `PCIEIW620-17.130.3.p28` - **Kernel** — Linux 7.0.8 - **Distro** — Arch Linux. Cross-checked on Ubuntu (powersave disabled): identical failure under the same trigger. - **Cross-check** — PS5 OS does not suffer from the same burst (verified by running a crafted ELF payload of the trigger). ## Bug summary Concurrent TLS-1.3 ClientHellos with the `X25519MLKEM768` PQ key_share (≈1700 B ClientHello, fragmented across TCP segments) fill the FW command queue. Any periodic host-to-FW command issued during that window times out (10 s) and triggers `MLAN_EVENT_ID_DRV_DBG_DUMP`. The event handler in `moal_recv_event()` always sets `priv->phandle->driver_status = MTRUE` ("abnormal driver state"). All later cfg80211 ops are then blocked with `Block woal_cfg80211_* in abnormal driver state` until a successful `fw_reload` clears the flag — but neither of the FW reset paths the driver knows about (in-band reset, PCI FLR) reliably recovers. ## Reproducer Prerequisites: - Wifi power save disabled. On Arch: NetworkManager `wifi.powersave=2`; on Ubuntu: toggle it off in the network settings. With the default `wifi.powersave=3` (enabled), the driver dies within seconds of association, before any burst — separate kill path, must be ruled out first to study the burst-triggered one. Trigger: ```bash while :; do for h in www.google.com accounts.google.com clients2.google.com \ www.bing.com www.duckduckgo.com www.cloudflare.com www.akamai.com \ rgl.rockstargames.com www.amazon.com ipv4.icanhazip.com; do curl -s --max-time 4 --tls-max 1.3 --curves X25519MLKEM768 \ -o /dev/null "https://$h/" & done sleep 3 done ``` Driver dies consistently in ~8–18 s on TLS variant, ~90–120 s on standard HTTPS, ~14 s on real-world Chromium 124+ (Rockstar Games launcher's PSN signin flow). Single-host bursts do not reproduce — concurrent sockets are required. ## Failure fingerprint ``` Timeout cmd id (X.YYYY) <CMD> [0x<id>], act = 0x<act> Commmand Timeout num_cmd_timeout = 1 tx_buf_size=3103 max_tx_buf_size=4096 txbd_rdptr=0xRRR txbd_wrptr=0xRRR + 0x80 <-- TX BD ring FULL (128 entries) rxbd_rdptr=0xRRR rxbd_wrptr=0xRRR + 0x80 <-- RX BD ring FULL tx_pending = 169 wmm_tx_pending[2] = 169 PCIE DEBUG MODE OUTPUT START Fail to pull ctrl_data=0x0 host_ready=0x0 Start to process hanging Send event_hang(0x0) vendor event Block woal_cfg80211_<op> in abnormal driver state <-- stuck, permanent until fw_reload ``` Both TX and RX BD rings full at the same time is consistent across reproductions: FW has stopped processing the rings in both directions. ## Cmds that timed out and triggered the trap (observed across runs) | HostCmd ID | Name | Issued by | Caller | |---|---|---|---| | `0x00a4` | `HostCmd_CMD_RSSI_INFO` | `woal_get_signal_info` | cfg80211 `dump_station_info` (NM periodic poll) | | `0x000b` | `HostCmd_CMD_802_11_GET_LOG` | `woal_get_stats_info` | cfg80211 `dump_survey` and others | | `0x0024` | `HostCmd_CMD_802_11_DEAUTHENTICATE` | mlan deauth path | cfg80211 `deauthenticate` (user-initiated) | | `0x00a9` | `HostCmd_CMD_FUNC_INIT` | mlan init | post-recovery driver init when FW state stuck | ## FW recovery paths (`auto_fw_reload` / `indrstcfg`) — none of them reliably work | Mode | Behavior | |---|---| | `auto_fw_reload=0` | No recovery attempted; driver stays in abnormal state. `rmmod + modprobe` alone does **not** clear it — the chip's scratch register still reports "FW running", the new driver instance prints `WLAN FW already running! Skip FW download`, `Firmware Init Failed`, probe returns `-14`, no iface comes up. | | `auto_fw_reload=3` (in-band) | `woal_reset_adma` logs `ADMA reset failed (value:97)`. `woal_pcie_reset_fw` logs `Failed to poll FW reset register 1C94=0x98`. `PCIe In-band Reset Fail`. | | `auto_fw_reload=1` (FLR via `pci_reset_function`) | Brick-removes the device until reboot — confirmed across multiple sessions. | | Manual recovery sequence | Outcome | |---|---| | `rmmod moal mlan && modprobe moal` (no chip reset) | Always fails. `WLAN FW already running` is logged, init returns -EFAULT, no iface. | | `rmmod moal mlan && echo 1 > /sys/bus/pci/devices/0000:40:00.7/reset && modprobe moal` (FLR) | Never fully restored wifi after the bug fired. Two observed outcomes: (a) scan works but auth loops 5x, then the driver's own `"WiFi Reset due to auth timeout cnt 5"` trigger drops it back into abnormal state; (b) `FUNC_INIT` cmd times out during init, driver doesn't register an iface at all. | | Full reboot | Always recovers. | ## Host knobs and trigger variants tried 8 single-variable runs of the TLS burst against the baseline driver. | # | Knob | Result | Triggering cmd | |---|---|---|---| | 0 | baseline (TLS PQ burst) | died in 14 s | RSSI_INFO 0xa4 | | 1 | `pcie_aspm=off` on kernel cmdline | died in **8 s** (faster) | RSSI_INFO | | 2 | trigger=plain HTTPS GETs (no PQ ClientHello) | died after >90 s (slower, generic burst still kills it) | RSSI_INFO | | 3 | trigger=`ping -f -s 1700` to gateway | **survived 240 s** | — (single socket, unidirectional) | | 4 | `ps_mode=0 auto_ds=0` (PS disabled) | died in 9 s | RSSI_INFO | | 5 | `max_tx_buf=2048` (half A-MSDU window) | died in 15 s | RSSI_INFO | | 6 | `ring_size=32` (smaller TX BD ring) | died in 18 s | RSSI_INFO | | 7 | `pcie_int_mode=0` (legacy IRQ) | driver fails to probe (-EFAULT — PCIe slot does not deliver legacy INTx) | — | Time-to-die and the failure fingerprint are unaffected by every module-param knob tried, except ASPM-off which made it *worse*. The only trigger that does **not** kill the driver is a single-socket one-way flood; the bug needs concurrent sockets with two-way handshake activity. ## Root cause 1. **FW cmd queue fills up under PQ-TLS / concurrent-socket bursts.** FW stops processing both BD rings (TX *and* RX go to ring-full) and stops responding to host cmds. This is FW-side, not driver-side: the same trigger against PS5 OS does not break anything. 2. **An over-aggressive recovery trap in the driver.** A *single* cmd timeout sets `driver_status = MTRUE` permanently. The trap is doing real work — suppressing it deadlocks the kernel — but is the only thing that turns a brief FW hiccup into a permanently broken driver. 3. **No working FW-reset path.** In-band reset polls a register that stays stuck at `0x1C94=0x98`; FLR removes the device from the bus until reboot. ## Ruled out as cause (per the single-variable runs above) - ASPM (turning it off made things worse). - A-MSDU window / TX BD ring depth / IRQ mode (no effect). - Module-level power-save (`ps_mode=0 auto_ds=0`) — no effect on time-to-die under the burst. (NM-level `wifi.powersave` is a separate, much faster kill path — see Reproducer prerequisites.) - TLS PQ key_share specifically (standard HTTPS bursts lock the driver up too, slower). - Linux network stack / cfg80211 layer / NetworkManager (USB rtw89 + same NM survives the same burst on the same host). - Trigger shape (any concurrent-socket two-way load kills it; only single-socket one-way doesn't). ## Artifacts [iw620-bug-trigger.sh](https://github.com/user-attachments/files/28144102/iw620-bug-trigger.sh) — reproducer that kills the driver and captures dmesg/journal/state. [iw620-bug-20260522-125911.tar.gz](https://github.com/user-attachments/files/28144638/iw620-bug-20260522-125911.tar.gz) — capture bundle from `iw620-bug-trigger.sh`

buzzer-re commented

2026-05-23 14:52:50 -05:00

(Migrated from github.com)

Thanks for reporting it, I will be working on a solution

OnesuchDev commented

2026-06-13 21:58:33 -05:00

(Migrated from github.com)

There is a newer version of the driver here that you could try:
https://github.com/nxp-imx/mwifiex/tree/hotfix/lf-6.18.2_1.0.0_hotfix

I don't know if it will help (I don't even own a PS5), but I thought it was worth pointing out.

There is a newer version of the driver here that you could try: https://github.com/nxp-imx/mwifiex/tree/hotfix/lf-6.18.2_1.0.0_hotfix I don't know if it will help (I don't even own a PS5), but I thought it was worth pointing out.