# Vision

> On-device detection, segmentation, pose and mattes as reactive signals — pin layers to subjects, cut them out, reframe, blur, and analyze clips.

framefields runs ONNX vision models on the rendered frames and exposes the results as signals that layers and effects consume like any other. All models are Apache-2.0, pinned to an immutable revision, verified by SHA-256, and downloaded lazily on first use.

## Turn it on

```ts
// Whole composition → a reactive bundle
const vision = comp.withVision({
  enableDetection: true,      // default
  enableSegmentation: true,
  enablePose: true,
  variant: "s",               // "t" | "s" | "m"
  confidence: 0.35,           // default 0.3
  classes: ["person"],        // COCO filter
});

// One layer → render it through a mode
Layer.video("assets/footage/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });
```

| Option | Model | Gives you |
| --- | --- | --- |
| `enableDetection` | RTMDet-Ins | COCO-80 boxes, tracked over time → `vision.objects` |
| `enableSegmentation` | RTMDet-Ins (same pass) | soft per-instance masks → `vision.masks`, `vision.segmentation` |
| `enablePose` | RTMO | 17 COCO keypoints per person → `track.pose`, `vision.poseLandmarks` |
| `enableMatte` | MediaPipe Selfie Segmenter | fast person alpha → `vision.segmentation.matte` |

Layer modes: `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`. For cutouts, `matteSource: "instance"` (default; any class, merges attached parts like a dress or instrument) or `"selfie"` (people filling the frame only, fastest).

## Pin, drive, cut out

```ts
// Pin a label to the main subject
Layer.text("SUBJECT 01", { fontSize: 40, fill: "#f8fafc" }).pinToObject(vision.objects.primary, {
  anchor: "topCenter", offsetY: -48, smoothFrames: 5, hideWhenLost: true,
});

// Drive an effect from mask coverage
Layer.box({ width: 1920, height: 1080, background: "#000" })
  .withEffect(new Vignette({ strength: vision.segmentation.subject.coverage, radius: 0.9 }));

// Editorial one-liners
comp.addSubjectSandwich({ source: "assets/footage/dancer.mp4", behind: [headline], feather: 4, fit: "cover" }); // text behind subject
comp.addSmartFraming({ source: "assets/footage/action.mp4", target: vision.objects.primary, targetAspect: 9 / 16, damping: 0.15 });
comp.addSubjectOutline(vision.segmentation.subject, { source: "assets/footage/character.mp4", color: "#ff5a1f", width: 6, blur: 12 });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });                               // anonymize
```

Anchors: `center`, `topCenter`, `bottomCenter`, `topLeft`, `topRight`, `bottomLeft`, `bottomRight`, `centerLeft`, `centerRight`. `pinToLandmark(coord, { offsetX, offsetY })` pins to a pose keypoint.

## Signals you get

| Group | Highlights |
| --- | --- |
| `objects` | `primary`, `get(trackId)`, `byCategory(cat, rank)`, `count`, `hasCategory` |
| `objects.*.bounds` | `x`, `y`, `width`, `height`, `screenX/Y/Width/Height`, `area`, `aspectRatio` |
| `objects.*.kinematics` | `vx`, `vy`, `speed`, `acceleration`, `headingDeg` |
| `objects.*.pose` | 17 keypoints, `hasPose`, `wristSpeed`, `handRaised`, `bodyTiltAngle` |
| `masks` | `subject`, `get(trackId)`, `count`; per mask `area`, `coverage`, `solidity` |
| `segmentation` | `subject`, `humanSilhouette`, `instanceMasks`, `matte.coverage` |
| `classes` | per class `count`, `maxConfidence`, `present` |

`vision.summary(frame)` returns a deterministic, serializable snapshot — safe inside a frame hook.

## Analyze before authoring

```ts
const report = await comp.analyzeVisionSequence("assets/footage/street.mp4", { tasks: ["detect", "pose"], categories: ["person"] });
report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]
```

A zod-serializable report of what is in the clip, without ffmpeg — use it to decide where to place things.

## Timing: the one-frame delay

Vision reads each layer's **previous** rendered frame. Frame 0 has no results, and masks trail the plate by one frame (invisible at speed).

- `renderVideo` renders in order and handles this.
- For stills, render at least two **consecutive** frames with the same `HeadlessMediaRenderer` and keep the last.
- **Don't verify vision with `renderFrameGrid`**: jumping between frames cuts out pixels from another frame and shows a ghosted double.

## Choosing settings

| Task (CPU, per ~2K frame) | `t` | `s` | `m` |
| --- | --- | --- | --- |
| detect + segment (one pass) | 150–300 ms | 200–340 ms | 420–700 ms |
| pose | ~50 ms | ~120 ms | ~280 ms |
| matte | ~20 ms | ~20 ms | ~20 ms |

- Start with `"s"`; use `"m"` for fast motion or busy backgrounds.
- Raise `confidence` to ~0.5 on stylized footage; restrict `classes` to stop the subject switching.
- Garment drops out of a cutout → `variant: "m"` or `keyBackground: true`. Halo on dark backgrounds → lower `featherRadius` or raise `maskThreshold`.

## Models offline

Models cache in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`). For CI or air-gapped renders, preload once or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror.

```ts
import { VisionRunner } from "@framefields/vision";

const runner = VisionRunner.create({ variant: "s" }); // zero I/O until used
await runner.preload(["detect", "pose"]);             // downloads + verifies SHA-256 (~380 MB for all models)
runner.close();
```