Supercede RFI protocol with LTR frames

ns6089 commented

2025-12-25 17:22:00 -06:00

(Migrated from github.com)

Presently Reference Frame Invalidation protocol (RFI for short) works by invalidating short-term reference frames, while DPB works as FIFO. LTR or long-term reference frames allow to manually "pin" reference frames in DPB, which exempts them from being pushed out by short-term reference frames. This can give us a number of important properties:

Almost bulletproof error recovery at extremely small DPB size (3 frames total: 2 LTR and 1 short-term). Current RFI doesn't work well with AV1 (DPB is limited to 8 frames), or with high framerate / long distance streams: while RFI request is in-flight server may push all non-invalidated frames out of DPB.
H.264, HEVC and AV1 codecs all support LTR frames. So do NVENC, VPL and AMF encoders. And AMF in particular is unable to do our existing RFI due to API limitations.

Now comes the "but", to do this efficiently the protocol needs to be extended. Servers need to flag LTR frames and clients need to send ACKs for them. So the exchange could look like this:

Server encodes Slot1 LTR frame and in transport flags it as LTR (in addition to our existing IDR/P-frame flag).
Server continues to encode and send typical P-frames.
Client receives LTR frame and validates FEC for it. Then it sends back ACK, control stream is resilient so this ACK shouldn't get lost in transit.
Server receives ACK for Slot1 LTR frame, marks this frame as current recovery frame, then encodes Slot2 LTR frame and the cycle repeats. On the next ACK, server will mark Slot2 LTR frame as recovery and encode new Slot1 LTR frame.
When client loses a frame, it just sends RFI request. Server treats that request as NACK and encodes non-recovery LTR P-frame referencing the recovery LTR frame, then in transport flags the frame as Recovered and LTR. Client doesn't need to request IDR frames after it successfully received a single LTR frame, unless it wants to reset the decoder for some reason.
As far as I can tell it will also be backward-compatible, for both old server / new client and new server / old client combinations. New server can encode LTRs speculatively and invalidate them based on RFI range, and old server will simply ignore ACK control packets.

That's about it. I haven't written any code for this yet, so any thoughts or opinions are welcome.

Presently Reference Frame Invalidation protocol (RFI for short) works by invalidating short-term reference frames, while DPB works as FIFO. LTR or long-term reference frames allow to manually "pin" reference frames in DPB, which exempts them from being pushed out by short-term reference frames. This can give us a number of important properties: 1. Almost bulletproof error recovery at extremely small DPB size (3 frames total: 2 LTR and 1 short-term). Current RFI doesn't work well with AV1 (DPB is limited to 8 frames), or with high framerate / long distance streams: while RFI request is in-flight server may push all non-invalidated frames out of DPB. 2. H.264, HEVC and AV1 codecs all support LTR frames. So do NVENC, VPL and AMF encoders. And AMF in particular is unable to do our existing RFI due to API limitations. Now comes the "but", to do this efficiently the protocol needs to be extended. Servers need to flag LTR frames and clients need to send ACKs for them. So the exchange could look like this: 1. Server encodes Slot1 LTR frame and in transport flags it as LTR (in addition to our existing IDR/P-frame flag). 2. Server continues to encode and send typical P-frames. 3. Client receives LTR frame and validates FEC for it. Then it sends back ACK, control stream is resilient so this ACK shouldn't get lost in transit. 4. Server receives ACK for Slot1 LTR frame, marks this frame as current recovery frame, then encodes Slot2 LTR frame and the cycle repeats. On the next ACK, server will mark Slot2 LTR frame as recovery and encode new Slot1 LTR frame. 5. When client loses a frame, it just sends RFI request. Server treats that request as NACK and encodes non-recovery LTR P-frame referencing the recovery LTR frame, then in transport flags the frame as Recovered and LTR. Client doesn't need to request IDR frames after it successfully received a single LTR frame, unless it wants to reset the decoder for some reason. 6. As far as I can tell it will also be backward-compatible, for both old server / new client and new server / old client combinations. New server can encode LTRs speculatively and invalidate them based on RFI range, and old server will simply ignore ACK control packets. That's about it. I haven't written any code for this yet, so any thoughts or opinions are welcome.

ns6089 commented

2025-12-27 05:59:42 -06:00

(Migrated from github.com)

As far as I can tell it will also be backward-compatible, for both old server / new client and new server / old client combinations. New server can encode LTRs speculatively and invalidate them based on RFI range, and old server will simply ignore ACK control packets.

I've just tested this on VPL encoder (not ffmpeg) and it works as expected. 3-frame DPB and generating LTR frame speculatively every 100ms gives ~200ms of no-IDR recovery. With ACKs in the protocol the no-IDR duration will become virtually infinite and with much shorter temporal distance to LTR recovery frame.

> As far as I can tell it will also be backward-compatible, for both old server / new client and new server / old client combinations. New server can encode LTRs speculatively and invalidate them based on RFI range, and old server will simply ignore ACK control packets. I've just tested this on VPL encoder (not ffmpeg) and it works as expected. 3-frame DPB and generating LTR frame speculatively every 100ms gives ~200ms of no-IDR recovery. With ACKs in the protocol the no-IDR duration will become virtually infinite and with much shorter temporal distance to LTR recovery frame.

cgutman commented

2026-01-02 15:48:59 -06:00

(Migrated from github.com)

I really like the idea of using LTR frames. I don't think it would be difficult to integrate on-demand frame ACKs into the control stream. Maybe the host could indicate that it wants an ACK by setting a flag in the NV_VIDEO_PACKET struct? It seems like either flags or reserved fields would work.

Edit: I think flags won't work because the VideoDepacketizer and RtpVideoQueue is strict about not having additional flags set there. The historical reason is some unusual FEC recovery failures from GFE where a completely corrupt frame would come out of FEC recovery. But it doesn't look like anything touches reserved at all, so we could repurpose that as a second flags field.

I really like the idea of using LTR frames. I don't think it would be difficult to integrate on-demand frame ACKs into the control stream. Maybe the host could indicate that it wants an ACK by setting a flag in the [NV_VIDEO_PACKET struct](https://github.com/moonlight-stream/moonlight-common-c/blob/master/src/Video.h#L21-L33)? It seems like either `flags` or `reserved` fields would work. Edit: I think `flags` won't work because the VideoDepacketizer and RtpVideoQueue is strict about not having additional flags set there. The historical reason is some unusual FEC recovery failures from GFE where a completely corrupt frame would come out of FEC recovery. But it doesn't look like anything touches `reserved` at all, so we could repurpose that as a second flags field.

ns6089 commented

2026-01-03 09:42:34 -06:00

(Migrated from github.com)

But it doesn't look like anything touches reserved at all, so we could repurpose that as a second flags field.

Yeah, I will look into this, thanks.

Oh, and another thing. I'm not sure that relying on UDP packet checksums for frame validation is enough: FEC validation is not being enabled by default and 16-bit checksum makes collisions improbable, but possible. So another, frame level checksum makes sense to me. With some blazing fast algorithm like XXH32. There's only 2 unused bytes left in short frame header, but thankfully we can still utilize long frame header. Sunshine uses version 7.1.431.-1, which gives us 24-byte long header to work with. But this doesn't block LTR RFI and can be done independently.
Actually scratch this. CRC-32 in lower level transports should catch complex collisions, and UDP checksum is enough for bit-flips in the routing hardware. Less work for us, yay.

> But it doesn't look like anything touches `reserved` at all, so we could repurpose that as a second flags field. Yeah, I will look into this, thanks. ~~Oh, and another thing. I'm not sure that relying on UDP packet checksums for frame validation is enough: FEC validation is not being enabled by default and 16-bit checksum makes collisions improbable, but possible. So another, frame level checksum makes sense to me. With some blazing fast algorithm like [XXH32](https://github.com/Cyan4973/xxHash). There's only 2 unused bytes left in short frame header, but thankfully we can still utilize long frame header. Sunshine uses version `7.1.431.-1`, which gives us 24-byte long header to work with. But this doesn't block LTR RFI and can be done independently.~~ Actually scratch this. CRC-32 in lower level transports should catch complex collisions, and UDP checksum is enough for bit-flips in the routing hardware. Less work for us, yay.

ns6089 commented

2026-01-03 14:47:10 -06:00

(Migrated from github.com)

Something like this maybe https://github.com/moonlight-stream/moonlight-common-c/pull/122
LTR ACKs share the queue with RFI

Something like this maybe https://github.com/moonlight-stream/moonlight-common-c/pull/122 LTR ACKs share the queue with RFI

ns6089 commented

2026-01-05 07:48:29 -06:00

(Migrated from github.com)

Ready for review.

ns6089 commented

2026-01-20 03:25:44 -06:00

(Migrated from github.com)

Also this type of RFI can allow multiple clients to share one encoder instance while retaining independent error recovery. And it won't require another protocol extension, or for clients to do anything extra. Whether it's relevant for us is up for debate, but the possibility is there 😅.

Rows
Columns

Supercede RFI protocol with LTR frames #120