SAOS-3624/dsh-video-storyboard

dsh-video-storyboard is a plugin that lets a text-capable model "watch" a video by reducing it to one numbered grid image (9–16 cells) plus a time-aligned text list: each cell's start/end time, cut count, and spoken line. It aims at agents that need to grasp what a clip contains, not at frame-by-frame analysis.

How it works

Instead of extracting frames as separate images (hundreds of frames at 1 fps, per the README) or feeding only subtitles, it compares downscaled frames (64×36, HSV, hue treated circularly and weighted by saturation) to detect cuts, re-scans 2-second windows at one-frame intervals for fast cuts, and allocates cells by equal information density. A single image is capped at 384 tokens; one call is reported as roughly 1,900 tokens total. The text list carries raw material only — times, cuts, lines — with cut count unreliable here when the numbers are suspect rather than conclusions about the content.

Tools and workflow

Four tools are documented: video_probe (header metadata only, no image tokens), video_storyboard (the grid plus list), video_zoom (a narrower window with denser cells), and video_locate (text query to timestamp). A typical pass is probe → storyboard → zoom on an interesting stretch. Each save also writes <name>.evidence.json with per-cell times and frame timestamps so a human or script can check the grid against the source, which addresses the README's own admission that fine detail is lost.

Requirements

• ffmpeg + ffprobe are mandatory; without them the first step fails. The plugin does not bundle them.
• yt-dlp is optional, needed only for pasting Bilibili/YouTube links; local files work without it.
• ASR is optional: without a transcript the grid still renders, just with no dialogue. Sources are tried in order: embedded subtitle track, a supplied SRT/VTT/JSON via transcriptFile, then ASR (local sherpa/funasr service or any OpenAI-compatible cloud endpoint, with config in ~/.dsh/asr.json). Cloud ASR without a key is silently skipped.
• Login-gated Bilibili videos (HTTP 567, 1080P+) may need an exported cookies file; downloading locally and passing a path avoids this. The README warns that SESSDATA in a cookies file is a login credential.

Documented limits

The README is explicit about where it loses: changes under ~1% of the frame (e.g. one added line on a slide) are invisible at 64×36; dense continuous camera movement distorts cuts=N; frame sharpness selection is unstable on very dark footage; and after sub-cell splitting, the smallest sub-frames are about 100×56. Speaker diarization and burned-in timecodes are deliberately not done. Cell resolution (roughly 331×184) is far below full frames, so reading small text requires a second video_zoom pass at extra cost. The comparison table also lists scenarios the authors say other plugins handle better: multi-round follow-up questions, verifiable conclusions, and per-frame legibility.

HotRepo's take

The project was observed once on the NEW-REPOSITORY board (javascript, 106 total stars, #22) on 2026-10-29; there is no comparable daily-board history, so nothing about momentum or longevity follows. Choose it when the goal is a cheap first pass at what a clip contains; skip it if you need frame-level legibility or repeated question-answering over the same video.

Project source and documentation