# Why framefields for your AI agents

Coding agents have fundamentally changed how software gets built. An agent can draft an entire backend service, run unit tests, inspect stack traces, fix regressions, and submit a pull request in minutes.

Yet until now, asking an AI agent to produce a video—whether a product teaser, an onboarding motion graphic, a dynamic data chart, or a localized social clip—has meant hitting an architectural brick wall.

The existing options were never designed for an autonomous loop:

1. **Desktop GUI Suites (After Effects, Premiere, Final Cut)**: Built around human eye-hand coordination, complex mouse interactions, proprietary binary project files, and zero headless programmatic inspection.
2. **Headless Browser Capture (Puppeteer, Playwright, Remotion)**: Wrapping a full Chromium browser to render HTML/CSS and taking screenshots frame by frame.

While headless browsers brought code into video production, they were designed to display interactive web pages to humans—not to be high-performance, deterministic video rendering engines.

**framefields was built to give coding agents a native, deterministic, GPU-accelerated video engine.** Here is why that matters.

---

## 1. The Headless Browser Problem

Headless Chromium is an extraordinary tool for testing web apps. But when used as a video engine driven by an autonomous agent, its limitations compound quickly:

| Bottleneck | Headless Chromium | framefields |
| :--- | :--- | :--- |
| **Frame Capture** | Lay out DOM tree $\to$ composite layers $\to$ read back pixels | Rendered straight into GPU textures via WebGPU (Dawn) |
| **Clock Model** | Wall-clock timers (`setTimeout`, `rAF`) prone to drift | Pure deterministic function: `(comp, frame) → pixels` |
| **Typography** | CPU rasterization; scales blur or re-rasterize | Vector outlines solved per pixel on the GPU; crisp at any scale |
| **Memory Footprint** | 2–4 GB per worker process | 200–400 MB per worker |
| **3D & Shaders** | Separate canvas contexts, difficult layer compositing | Calibrated 3D camera lining up 1:1 with 2D pixels; 50+ GPU effects |
| **On-device AI Vision** | External pipelines or complex WebGL hacks | Embedded ONNX runtime for pose, mattes, and tracking |

When an agent renders a 30-second 4K video using a browser screenshot loop, workers consume gigabytes of RAM, wall-clock timing fluctuates across CI runs, and frame drops ruin motion smoothness. 

Agents need a framework that treats video not as a series of web page screenshots, but as a pure, compiled GPU pipeline.

---

## 2. Determinism: Every Frame is a Function of its Number

In framefields, there are no timers, no `Date.now()`, and no asynchronous rendering race conditions. 

```ts
(composition, frameNumber) => pixels
```

Frame 45 of your video is mathematically guaranteed to render the exact same pixels on an Apple Silicon laptop, a Linux server with Vulkan, or a Cloudflare Worker container. 

This determinism unlocks three critical agent capabilities:

- **Parallel rendering without order dependencies**: Render frames 0–30 on GPU worker A and frames 31–60 on GPU worker B.
- **Repeatable regression testing**: An agent can assert on exact pixel values or frame-to-frame diffs in Vitest.
- **Cheapest-first feedback**: The agent doesn't need to render an entire MP4 file just to know if a transition worked. It can render frame 45 directly to PNG in 8 milliseconds.

---

## 3. The "Cheapest First" Agent Feedback Loop

AI agents work best when the feedback loop is fast, structured, and cheap. When feedback takes five minutes or yields unparsable error logs, agents hallucinate and thrash.

framefields structures the video development process into six progressive tiers:

```
[1. Typecheck] ──> [2. Static Lint] ──> [3. Keyframe PNGs] ──> [4. Frame Grid] ──> [5. Human Preview] ──> [6. Full QA Encode]
   (0.2 sec)          (0.05 sec)            (0.1 sec)             (0.8 sec)         (Interactive)          (GPU render)
```

### Tier 1: Type Safety (`tsc --noEmit`)
Because every layer, effect, transition, and keyframe in framefields is strongly typed in TypeScript, syntax errors, mismatched property names, and invalid color formats are caught by the language server before anything touches the GPU.

### Tier 2: Static Document Lint (`comp.lint()`)
Before allocating GPU memory, the agent runs `comp.lint()`. This executes a document-level static analysis on the composition tree and returns **structured JSON diagnostics**, not unstructured logs:

```ts
const diagnostics = comp.lint();
// [
//   {
//     severity: "warning",
//     code: "animation_truncated",
//     message: "Keyframe at frame 120 exceeds layer duration of 90 frames",
//     layerId: "hero-title"
//   }
// ]
```

Each code maps to an exact entry in the framefields troubleshooting catalog, giving the agent a direct path to self-correction without human intervention.

### Tier 3 & 4: Keyframe Inspection and Contact-Sheet Grids
Once linting passes, the agent renders only the critical inflection points to PNG:

```bash
npm run frames -- 0 30 60 90
```

With multimodal vision capabilities, coding agents can open the resulting PNG images, verify layout alignment, check color contrast, and examine easing. By rendering a contact-sheet grid (`renderFrameGrid`), an agent evaluates transitions and camera moves across time in a single image.

### Tier 5 & 6: Pinned Notes & Automated QA
During interactive review, a human can scrub the WebGPU live preview and drop a note: *"Make the logo enter 10 frames earlier."* The preview saves this to a local `notes.json` file. The agent reads `notes.json`, identifies the layer, adjusts the keyframe frames, and commits the fix.

Finally, `renderVideo({ qa: true })` validates black frames, audio sync, contrast ratios, and playback bitrate before signing off on `qa.passed: true`.

---

## 4. Typography and 3D Built for Motion

When an agent creates motion design, text and spatial depth are paramount. 

### Resolution-Independent Typography
In framefields, glyph outlines are not rasterized to bitmaps at build time. Their vector curves are evaluated per pixel on the GPU. Whether an agent scales text by 10× during a camera zoom, animates letter-by-letter with After Effects–style range selectors, or renders at 8K resolution, text never pixelates or exhibits texture filtering artifacts.

### Pixels That Line Up in 3D
Typical engines require switching between a 2D canvas and a 3D WebGL context with mismatched coordinate systems. 

framefields calibrates its 3D camera so that **$z = 0$ corresponds 1:1 to canvas pixels**:

```ts
import { Composition, Layer, Lighting, Camera } from "framefields";

const comp = new Composition({ width: 1920, height: 1080 });

// 2D title card
comp.add(Layer.text("CYBERNETIC", { x: 960, y: 300, fontSize: 72 }));

// 3D model positioned directly on the 2D grid
comp.add(
  Layer.model("assets/models/device.glb", {
    x: 960,
    y: 600,
    z: 0, // matches 2D coordinates perfectly
    scale: 1.5,
  }).animate((k) => [
    k.fromTo("rotationY", 0, Math.PI * 2, { durationFrames: 120 })
  ])
);

comp.add(Lighting.studio());
```

---

## 5. On-Device Vision: Video That Reacts to Media

One of the most complex tasks for video editors is rotoscoping, object tracking, and reframing. framefields embeds an ONNX runtime directly in the rendering pipeline.

Without writing external Python scripts or making slow API calls to cloud services, your agent can write video code that:
- Detects faces and human poses frame by frame.
- Automatically isolates a subject with a real-time person matte.
- Pins graphical callouts or 3D annotations to a tracked bounding box.
- Automatically reframes a 16:9 landscape clip into 9:16 vertical video for mobile platforms.

```ts
// Extract person matte and pin a dynamic graphics ring to their wrist
const { personMatte, keypoints } = comp.withVision({
  model: "segmentation",
  source: "assets/footage/presenter.mp4",
});

comp.add(
  Layer.box({ width: 60, height: 60, borderRadius: 30, borderColor: "#f5e300" })
    .pinToObject(keypoints.rightWrist)
);
```

---

## 6. Giving Your Agent the Skills

We designed framefields so any coding agent can start building production-ready videos immediately. The instructions teach the agent project structure, timing conventions, layer composition, and the step-by-step review loop.

### Claude Code
```text
/plugin install framefields
```

### Cursor, Codex, Gemini CLI, Windsurf, Copilot
```bash
npx skills add gatewai-dev/framefields
```

Then prompt your agent:

```text
Create a 5-second product teaser for a developer analytics platform in 1080x1920 60fps.
Include an animated bar chart, kinetic headline typography on a beat grid, 
and inspect keyframes before encoding the final MP4.
```

Your agent will scaffold the project, write the composition, lint the layer tree, review the generated PNG frames, and render the MP4 on the GPU.

---

## The Video Engine for the Agentic Era

When coding agents have direct access to a deterministic, high-performance GPU engine, video ceases to be a manual bottleneck. Video becomes code: version-controlled, diffable, automated, and testable.

Explore the [Documentation](/docs/) to see the full API, or browse [framefields on GitHub](https://github.com/gatewai-dev/framefields).