WIP: macOS: map VideoToolbox frames before falling back to a CPU copy #818

Draft

latifv wants to merge 1 commit from latifv/codex/videotoolbox-direct-mapping into main

latifv commented

2026-09-26 16:03:18 -05:00

(Migrated from github.com)

On macOS, VideoToolbox-decoded frames currently go through av_hwframe_transfer_data before presentation. That allocates a software frame and copies the decoded pixels even when FFmpeg can expose the CVPixelBuffer's CPU planes directly.

This change first tries av_hwframe_map with AV_HWFRAME_MAP_READ | AV_HWFRAME_MAP_DIRECT when the existing Zero-Copy setting is enabled. The mapped AVFrame retains the underlying surface and metadata. Mapping failure preserves the current transfer/copy fallback; disabling Zero-Copy and reconnecting also restores it.

This is a read-only CPU mapping, not Metal texture import: renderer upload and presentation still happen. The production change is confined to the macOS frame-preparation path. No queue sizes, decoder options, renderer defaults, audio, or networking behavior are changed.

Validation

Full Release GUI build on Apple M5 arm64, macOS 26.6.2, Qt 6.11.2, FFmpeg 9.0.1, libplacebo 7.360.1.
ctest --test-dir build-upstream --output-on-failure: 2/2 passed (existing unit suite and the new macOS mapping test).
New test checks NV12/P010 pixels with padded rows, PTS, color and HDR mastering metadata, aspect ratio, invalid inputs, and mapping lifetime after releasing the source and caller's hardware-context references.
The standalone mapping test also passed when compiled with AddressSanitizer and UndefinedBehaviorSanitizer; linked FFmpeg libraries were not sanitizer-instrumented.
Included a reproducible VideoToolbox decode/preparation benchmark. Three runs per codec, 180 frames per run, 1080p60: all nine runs passed pixel equality and source-release checks.

Codec / mapped format	Median of transfer/copy run means	Median of direct-map run means
H.264 / NV12	277.850 us	3.606 us
HEVC / NV12	266.072 us	3.322 us
HEVC / P010	323.367 us	3.528 us

These timings include AVFrame allocation and metadata propagation, but exclude decoding, validation, unmapping/freeing, GPU upload, and presentation. They do not establish an end-to-end Remote Play latency improvement. Reproduction instructions are in doc/videotoolbox-mapping.md.

Remaining validation (draft)

This isolated upstream branch has not had a live PS5 session. Live renderer behavior, decoder-surface retention during a long session, and HDR display output still need testing. Intel Macs, older macOS releases, and Linux/Windows builds have not been tested locally. The local build used Homebrew libraries targeting newer macOS versions and emitted deployment-target warnings, so it does not establish macOS 13 compatibility. No Flatpak validation was performed on this macOS host.

Suggested live check: stream using VideoToolbox with Zero-Copy enabled, repeat with it disabled after reconnecting, and compare correctness, frame pacing, and memory use with H.264 and HEVC (including 10-bit where available).

On macOS, VideoToolbox-decoded frames currently go through `av_hwframe_transfer_data` before presentation. That allocates a software frame and copies the decoded pixels even when FFmpeg can expose the CVPixelBuffer's CPU planes directly. This change first tries `av_hwframe_map` with `AV_HWFRAME_MAP_READ | AV_HWFRAME_MAP_DIRECT` when the existing Zero-Copy setting is enabled. The mapped AVFrame retains the underlying surface and metadata. Mapping failure preserves the current transfer/copy fallback; disabling Zero-Copy and reconnecting also restores it. This is a read-only CPU mapping, not Metal texture import: renderer upload and presentation still happen. The production change is confined to the macOS frame-preparation path. No queue sizes, decoder options, renderer defaults, audio, or networking behavior are changed. ### Validation - Full Release GUI build on Apple M5 arm64, macOS 26.6.2, Qt 6.11.2, FFmpeg 9.0.1, libplacebo 7.360.1. - `ctest --test-dir build-upstream --output-on-failure`: 2/2 passed (existing unit suite and the new macOS mapping test). - New test checks NV12/P010 pixels with padded rows, PTS, color and HDR mastering metadata, aspect ratio, invalid inputs, and mapping lifetime after releasing the source and caller's hardware-context references. - The standalone mapping test also passed when compiled with AddressSanitizer and UndefinedBehaviorSanitizer; linked FFmpeg libraries were not sanitizer-instrumented. - Included a reproducible VideoToolbox decode/preparation benchmark. Three runs per codec, 180 frames per run, 1080p60: all nine runs passed pixel equality and source-release checks. | Codec / mapped format | Median of transfer/copy run means | Median of direct-map run means | | --- | ---: | ---: | | H.264 / NV12 | 277.850 us | 3.606 us | | HEVC / NV12 | 266.072 us | 3.322 us | | HEVC / P010 | 323.367 us | 3.528 us | These timings include AVFrame allocation and metadata propagation, but exclude decoding, validation, unmapping/freeing, GPU upload, and presentation. They do **not** establish an end-to-end Remote Play latency improvement. Reproduction instructions are in `doc/videotoolbox-mapping.md`. ### Remaining validation (draft) This isolated upstream branch has not had a live PS5 session. Live renderer behavior, decoder-surface retention during a long session, and HDR display output still need testing. Intel Macs, older macOS releases, and Linux/Windows builds have not been tested locally. The local build used Homebrew libraries targeting newer macOS versions and emitted deployment-target warnings, so it does not establish macOS 13 compatibility. No Flatpak validation was performed on this macOS host. Suggested live check: stream using VideoToolbox with Zero-Copy enabled, repeat with it disabled after reconnecting, and compare correctness, frame pacing, and memory use with H.264 and HEVC (including 10-bit where available).

This pull request is broken due to missing fork information.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin 'latifv/codex/videotoolbox-direct-mapping:latifv/codex/videotoolbox-direct-mapping'

git switch 'latifv/codex/videotoolbox-direct-mapping'

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 'main'

git merge --no-ff 'latifv/codex/videotoolbox-direct-mapping'

git switch 'latifv/codex/videotoolbox-direct-mapping'

git rebase 'main'

git switch 'main'

git merge --ff-only 'latifv/codex/videotoolbox-direct-mapping'

git switch 'latifv/codex/videotoolbox-direct-mapping'

git rebase 'main'

git switch 'main'

git merge --no-ff 'latifv/codex/videotoolbox-direct-mapping'

git switch 'main'

git merge --squash 'latifv/codex/videotoolbox-direct-mapping'