Feature Request: Headless CLI stream mode — video frame output + controller input via stdin/stdout for AI agent automation #784

New issue

Open

opened 2026-07-08 14:44:32 -05:00 by galinnanne · 0 comments

galinnanne commented

2026-07-08 14:44:32 -05:00

(Migrated from github.com)

Feature Request: Headless CLI stream mode — video frames + controller input via stdin/stdout

Summary

Extend chiaki-cli with a new stream subcommand that connects to a PS5/PS4, decodes video frames to stdout (raw or PNG), and reads controller inputs from stdin (JSON or line-based protocol). This would enable AI agent automation — letting programs (not just humans) play games via Remote Play.

Motivation

Currently, chiaki-cli only supports discover and wakeup. The full streaming experience (video + audio + controller I/O) is only available through the Qt GUI (chiaki). This makes it impossible to:

Automate gameplay from a headless server or script
Feed game frames to an AI agent (e.g., vision model, reinforcement learning)
Send controller inputs programmatically without a physical controller or GUI

The libchiaki C API already has everything needed:

ChiakiVideoSampleCallback → receives raw H.264/H.265 frames
ChiakiControllerState → full controller state struct (buttons, sticks, touch, motion)
chiaki_feedback_sender_set_controller_state() → pushes inputs to the console
ChiakiEventCallback → session events (connected, quit, rumble, etc.)

A headless CLI mode would unlock the entire libchiaki API for non-GUI use cases.

Proposed Design

CLI Usage

chiaki-cli stream --host 192.168.1.100 --regist-key <KEY> --ps5 \
  --video-format png \    # png | raw-h264 | raw-h265
  --resolution 720p --fps 60 \
  --input-format json     # json | lines

Video Output (stdout or file pipe)

Format	Description
`png`	Decode H.264/H.265 with ffmpeg, output PNG frames (one per frame, frame-delimited)
`raw-h264`	Raw H.264/H.265 NAL units (for external decoding pipeline)
`yuv420p`	Decoded raw YUV frames (no encoding overhead)

Each frame could be prefixed with a header:

FRAME <width> <height> <timestamp> <frame_index>\n
<binary data>\n

Controller Input (stdin)

JSON line-based protocol — one JSON object per line:

{"buttons": ["CROSS", "DPAD_UP"], "left_x": 0, "left_y": -32767, "right_x": 0, "right_y": 0, "l2": 0, "r2": 0}

Where:

buttons: array of button names (CROSS, MOON, BOX, PYRAMID, DPAD_LEFT/RIGHT/UP/DOWN, L1, R1, L3, R3, OPTIONS, SHARE, TOUCHPAD, PS)
left_x, left_y, right_x, right_y: int16 stick values (-32767 to 32767)
l2, r2: uint8 trigger values (0-255)
Optional: touch array, gyro/accel/orient for motion

A special idle input resets the controller:

{"buttons": [], "left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0}

Session Events (stderr)

Events written to stderr as JSON lines for the consuming program:

{"event": "connected", "width": 1280, "height": 720, "fps": 60}
{"event": "rumble", "left": 255, "right": 0}
{"event": "quit", "reason": "stopped"}
{"event": "video_fec_failure", "frame_index": 1234}

Use Cases

AI agent gameplay — Feed video frames to a vision model, let the agent decide actions, send controller inputs back. Same architecture as headless emulator + RAM reads, but for PlayStation Remote Play.
Automated testing — Regression test game builds by scripting inputs and capturing video output.
Streaming bots — Headless stream relay without a desktop environment.
Accessibility — Alternative input methods (eye tracking, voice commands) can pipe into stdin instead of needing a virtual controller.

Implementation Notes

The libchiaki API already exposes all needed callbacks and structs:
- chiaki_session_set_video_sample_cb() for frame capture
- chiaki_session_set_event_cb() for session events
- chiaki_feedback_sender_set_controller_state() for input
ffmpeg is already a dependency for video decoding (used in ffmpegdecoder.c)
A separate thread would read stdin and update ChiakiControllerState at a fixed rate
The CLI already uses argp for argument parsing — new subcommand fits the existing pattern
Could be gated behind a --headless flag to avoid pulling in Qt/SDL dependencies

Alternatives Considered

GUI + Xvfb + screenshot: Run Chiaki GUI on a virtual display, capture with ffmpeg -f x11grab, send inputs via evdev/uinput. Works but is fragile, slow (30-60ms screenshot latency vs direct frame access), and requires a full Qt/graphics stack running.
evdev virtual controller + GUI: Create a virtual evdev device that the GUI picks up. Still requires GUI running and adds kernel dependency.

A native headless mode is cleaner, faster, and more portable.

cli/src/main.c — current CLI with only discover + wakeup
lib/include/chiaki/session.h — ChiakiVideoSampleCallback, ChiakiEventCallback
lib/include/chiaki/controller.h — ChiakiControllerState
lib/include/chiaki/feedbacksender.h — chiaki_feedback_sender_set_controller_state()

Community Interest

This would open up PlayStation game automation to the broader AI agent community (Hermes Agent, Claude Code, OpenAI Codex, etc.) — any tool that can read images and write JSON could play PS5/PS4 games via Remote Play.

## Feature Request: Headless CLI stream mode — video frames + controller input via stdin/stdout ### Summary Extend `chiaki-cli` with a new `stream` subcommand that connects to a PS5/PS4, decodes video frames to stdout (raw or PNG), and reads controller inputs from stdin (JSON or line-based protocol). This would enable **AI agent automation** — letting programs (not just humans) play games via Remote Play. ### Motivation Currently, `chiaki-cli` only supports `discover` and `wakeup`. The full streaming experience (video + audio + controller I/O) is only available through the Qt GUI (`chiaki`). This makes it impossible to: 1. **Automate gameplay** from a headless server or script 2. **Feed game frames to an AI agent** (e.g., vision model, reinforcement learning) 3. **Send controller inputs programmatically** without a physical controller or GUI The `libchiaki` C API already has everything needed: - `ChiakiVideoSampleCallback` → receives raw H.264/H.265 frames - `ChiakiControllerState` → full controller state struct (buttons, sticks, touch, motion) - `chiaki_feedback_sender_set_controller_state()` → pushes inputs to the console - `ChiakiEventCallback` → session events (connected, quit, rumble, etc.) A headless CLI mode would unlock the entire `libchiaki` API for non-GUI use cases. ### Proposed Design #### CLI Usage ```bash chiaki-cli stream --host 192.168.1.100 --regist-key <KEY> --ps5 \ --video-format png \ # png | raw-h264 | raw-h265 --resolution 720p --fps 60 \ --input-format json # json | lines ``` #### Video Output (stdout or file pipe) | Format | Description | |--------|-------------| | `png` | Decode H.264/H.265 with ffmpeg, output PNG frames (one per frame, frame-delimited) | | `raw-h264` | Raw H.264/H.265 NAL units (for external decoding pipeline) | | `yuv420p` | Decoded raw YUV frames (no encoding overhead) | Each frame could be prefixed with a header: ``` FRAME <width> <height> <timestamp> <frame_index>\n <binary data>\n ``` #### Controller Input (stdin) JSON line-based protocol — one JSON object per line: ```json {"buttons": ["CROSS", "DPAD_UP"], "left_x": 0, "left_y": -32767, "right_x": 0, "right_y": 0, "l2": 0, "r2": 0} ``` Where: - `buttons`: array of button names (`CROSS`, `MOON`, `BOX`, `PYRAMID`, `DPAD_LEFT/RIGHT/UP/DOWN`, `L1`, `R1`, `L3`, `R3`, `OPTIONS`, `SHARE`, `TOUCHPAD`, `PS`) - `left_x`, `left_y`, `right_x`, `right_y`: int16 stick values (-32767 to 32767) - `l2`, `r2`: uint8 trigger values (0-255) - Optional: `touch` array, `gyro`/`accel`/`orient` for motion A special `idle` input resets the controller: ```json {"buttons": [], "left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0} ``` #### Session Events (stderr) Events written to stderr as JSON lines for the consuming program: ```json {"event": "connected", "width": 1280, "height": 720, "fps": 60} {"event": "rumble", "left": 255, "right": 0} {"event": "quit", "reason": "stopped"} {"event": "video_fec_failure", "frame_index": 1234} ``` ### Use Cases 1. **AI agent gameplay** — Feed video frames to a vision model, let the agent decide actions, send controller inputs back. Same architecture as headless emulator + RAM reads, but for PlayStation Remote Play. 2. **Automated testing** — Regression test game builds by scripting inputs and capturing video output. 3. **Streaming bots** — Headless stream relay without a desktop environment. 4. **Accessibility** — Alternative input methods (eye tracking, voice commands) can pipe into stdin instead of needing a virtual controller. ### Implementation Notes - The `libchiaki` API already exposes all needed callbacks and structs: - `chiaki_session_set_video_sample_cb()` for frame capture - `chiaki_session_set_event_cb()` for session events - `chiaki_feedback_sender_set_controller_state()` for input - ffmpeg is already a dependency for video decoding (used in `ffmpegdecoder.c`) - A separate thread would read stdin and update `ChiakiControllerState` at a fixed rate - The CLI already uses argp for argument parsing — new subcommand fits the existing pattern - Could be gated behind a `--headless` flag to avoid pulling in Qt/SDL dependencies ### Alternatives Considered - **GUI + Xvfb + screenshot**: Run Chiaki GUI on a virtual display, capture with `ffmpeg -f x11grab`, send inputs via `evdev/uinput`. Works but is fragile, slow (30-60ms screenshot latency vs direct frame access), and requires a full Qt/graphics stack running. - **evdev virtual controller + GUI**: Create a virtual evdev device that the GUI picks up. Still requires GUI running and adds kernel dependency. A native headless mode is cleaner, faster, and more portable. ### Related - `cli/src/main.c` — current CLI with only discover + wakeup - `lib/include/chiaki/session.h` — `ChiakiVideoSampleCallback`, `ChiakiEventCallback` - `lib/include/chiaki/controller.h` — `ChiakiControllerState` - `lib/include/chiaki/feedbacksender.h` — `chiaki_feedback_sender_set_controller_state()` ### Community Interest This would open up PlayStation game automation to the broader AI agent community (Hermes Agent, Claude Code, OpenAI Codex, etc.) — any tool that can read images and write JSON could play PS5/PS4 games via Remote Play.

👎 1