Vision

framefields runs ONNX vision models on the rendered frames and exposes the results as signals that layers and effects consume like any other. All models are Apache-2.0, pinned to an immutable revision, verified by SHA-256, and downloaded lazily on first use.

Turn it on

// Whole composition → a reactive bundle
const vision = comp.withVision({
  enableDetection: true,      // default
  enableSegmentation: true,
  enablePose: true,
  variant: "s",               // "t" | "s" | "m"
  confidence: 0.35,           // default 0.3
  classes: ["person"],        // COCO filter
});

// One layer → render it through a mode
Layer.video("assets/footage/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });

Option	Model	Gives you
enableDetection	RTMDet-Ins	COCO-80 boxes, tracked over time → vision.objects
enableSegmentation	RTMDet-Ins (same pass)	soft per-instance masks → vision.masks, vision.segmentation
enablePose	RTMO	17 COCO keypoints per person → track.pose, vision.poseLandmarks
enableMatte	MediaPipe Selfie Segmenter	fast person alpha → vision.segmentation.matte

Layer modes: passthrough, mask, matte, crop, skeleton, boxes, tracking. For cutouts, matteSource: "instance" (default; any class, merges attached parts like a dress or instrument) or "selfie" (people filling the frame only, fastest).

Pin, drive, cut out

// Pin a label to the main subject
Layer.text("SUBJECT 01", { fontSize: 40, fill: "#f8fafc" }).pinToObject(vision.objects.primary, {
  anchor: "topCenter", offsetY: -48, smoothFrames: 5, hideWhenLost: true,
});

// Drive an effect from mask coverage
Layer.box({ width: 1920, height: 1080, background: "#000" })
  .withEffect(new Vignette({ strength: vision.segmentation.subject.coverage, radius: 0.9 }));

// Editorial one-liners
comp.addSubjectSandwich({ source: "assets/footage/dancer.mp4", behind: [headline], feather: 4, fit: "cover" }); // text behind subject
comp.addSmartFraming({ source: "assets/footage/action.mp4", target: vision.objects.primary, targetAspect: 9 / 16, damping: 0.15 });
comp.addSubjectOutline(vision.segmentation.subject, { source: "assets/footage/character.mp4", color: "#ff5a1f", width: 6, blur: 12 });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });                               // anonymize

Anchors: center, topCenter, bottomCenter, topLeft, topRight, bottomLeft, bottomRight, centerLeft, centerRight. pinToLandmark(coord, { offsetX, offsetY }) pins to a pose keypoint.

Signals you get

Group	Highlights
objects	primary, get(trackId), byCategory(cat, rank), count, hasCategory
objects.*.bounds	x, y, width, height, screenX/Y/Width/Height, area, aspectRatio
objects.*.kinematics	vx, vy, speed, acceleration, headingDeg
objects.*.pose	17 keypoints, hasPose, wristSpeed, handRaised, bodyTiltAngle
masks	subject, get(trackId), count; per mask area, coverage, solidity
segmentation	subject, humanSilhouette, instanceMasks, matte.coverage
classes	per class count, maxConfidence, present

vision.summary(frame) returns a deterministic, serializable snapshot — safe inside a frame hook.

Analyze before authoring

const report = await comp.analyzeVisionSequence("assets/footage/street.mp4", { tasks: ["detect", "pose"], categories: ["person"] });
report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]

A zod-serializable report of what is in the clip, without ffmpeg — use it to decide where to place things.

Timing: the one-frame delay

Vision reads each layer’s previous rendered frame. Frame 0 has no results, and masks trail the plate by one frame (invisible at speed).

• renderVideo renders in order and handles this.
• For stills, render at least two consecutive frames with the same HeadlessMediaRenderer and keep the last.
• Don’t verify vision with renderFrameGrid: jumping between frames cuts out pixels from another frame and shows a ghosted double.

Choosing settings

Task (CPU, per ~2K frame)	t	s	m
detect + segment (one pass)	150–300 ms	200–340 ms	420–700 ms
pose	~50 ms	~120 ms	~280 ms
matte	~20 ms	~20 ms	~20 ms

• Start with "s"; use "m" for fast motion or busy backgrounds.
• Raise confidence to ~0.5 on stylized footage; restrict classes to stop the subject switching.
• Garment drops out of a cutout → variant: "m" or keyBackground: true. Halo on dark backgrounds → lower featherRadius or raise maskThreshold.

Models offline

Models cache in $FRAMEFIELDS_MODELS_DIR (default ~/.cache/framefields/models). For CI or air-gapped renders, preload once or point FRAMEFIELDS_MODELS_BASE_URL at a mirror.

import { VisionRunner } from "@framefields/vision";

const runner = VisionRunner.create({ variant: "s" }); // zero I/O until used
await runner.preload(["detect", "pose"]);             // downloads + verifies SHA-256 (~380 MB for all models)
runner.close();