SAOS-3624/dsh-see-now

What it does

See Now is a plugin for DSH that gives a vision model access to a live screen, camera, or video file without sending every frame. It scores each frame on a 0–255 scale using a 64×36 HSV grid diff (about 0.2 ms per frame), keeps frames that represent a hard cut or accumulated motion, and packs the kept frames into a single contact-sheet PNG. The point is token cost: a mostly static screen produces almost no sheets, while a busy stretch of interaction often costs only a few sheets instead of dozens.

The package also exports capture, pipeline, and png modules, so it can run as a plain Node library outside DSH.

Install in DSH

The plugin is published on npm and installed through DSH's own dsh plugin command:

dsh plugin --profile web add dsh-see-now

Use --profile desktop for the desktop app; that profile must have been opened once before it is initialized. Restart DSH after installing, since the plugin tree is assembled at process start. To remove or update it, swap add for remove or update.

To build from source for development:

git clone https://github.com/SAOS-3624/dsh-see-now.git
cd dsh-see-now
node scripts/install-dev.mjs

This copies into the local desktop profile rather than symlinking, because Node needs the real path to resolve the host's @deepseek-ai/dsh-tools. Restart DSH afterward.

If the floating window has been closed, reopen it with npx dsh-see-window; --status, --force, and --close inspect, restart, and close it.

Asking the model to look

Once installed, the model calls the tool:

see_now({ action: "watch" })          // blocks until a sheet is ready, then returns it
see_now({ action: "watch", say: "…" }) // also speaks a line in the floating window

The first call returns the current screen in about a second; later calls return what happened since the last one. say is the only channel that shows text in the floating window and gets spoken.

Key parameters include quality (low 800×450, standard, high, ultra; changing it restarts capture and returns the current frame after about 0.6 s), pace (calm / normal / dense, controlling the cut threshold and maximum wake-up interval), scene for one full-frame image per scene change, seconds as the blocking upper bound (default 300, max 860), minFrames to suppress small sheets, and source / path / realtime to pick screen, camera, or file playback. Scene mode is sticky until scene:false or stop.

Using it as a Node library

spawnFrames from dsh-see-now/capture produces frames; createPipeline from dsh-see-now/pipeline scores, selects, and packs them; p.take() returns a PNG plus per-frame timestamps, tier, and source. The README shows a loop writing each sheet to disk.

Setup and constraints

• Windows uses ddagrab with automatic fallback to gdigrab.
• macOS uses avfoundation and requires Screen Recording permission for the app running DSH. Without it the capture exits cleanly with black frames; the probe detects all-black output and says so in the start result.
• Linux is not implemented and will error rather than pretend to work; x11grab or PipeWire support is still needed.
• ffmpeg does not need to be on PATH. The plugin searches DSH_FFMPEG_PATH, FFMPEG_PATH, DSH_FFMPEG_DIR, ~/.dsh/tools/ffmpeg/bin, and ~/.dsh/bin, in that order.
• Change detection is visual only: different content that looks similar may be treated as unchanged, and there is no cross-frame deduplication beyond a coarse dupMin filter.
• Wake-ups are throttled by pace; voice playback is serial so faster intervals just queue.
• No TTS engine is bundled. say is spoken only if you configure an adapter in ~/.dsh/voice.json; a macOS say example is provided via examples/voice-say-mac.mjs.
• The debug window (debug/) is not included in the npm package or plugin.
• The license is Apache-2.0, permitting commercial, closed-source, and redistributed use as long as LICENSE and NOTICE are kept.

HotRepo's take

On the new-project board for JavaScript on 2026-11-02, See Now recorded 111 stars and 5 forks in a window beginning 2026-11-01, its first tracked day. That is a single snapshot, so it supports only the fact that the project drew attention quickly, not a trend. The design choice that makes it worth a look is the token accounting: the README states that an 800×450 sheet is about 233 tokens by the host's patch grid, and packing 4–8 frames into one sheet costs the same as one, so the filter is where the savings actually come from. The limitation is equally concrete: change scoring cannot distinguish two pages with similar layout but different text.