Make a Useful Video Workbench

1Use my brief and selected files for chapter 6 only. Plan one small fixture/UI2increment; wait for approval before applying it. Keep synthetic fixtures separate3from gitignored real media/results. Preserve durationSec validation, provenance,4loading/empty/error states, and visible malformed-vector errors.5Cite current official API contracts for any API assumption; invent no fields/results.6Run local schema/ranking tests and inspect the actual UI with synthetic data;7real provider responses require a separately approved run. Redact debugging errors.8Do not read secrets, commit/push, provision, or make paid calls without my9explicit approval. Do not jump chapters. I review the Git diff and commit.

1Review chapter 6's proposed diff, local tests and actual synthetic UI state.2Check fixture privacy, durationSec schema, dimension checks, visible errors,3loading/empty states and provenance. No real output/media in tracked fixtures.4Cite current official API contracts if relevant; distinguish UI/mock evidence5from actual provider responses. Flag gaps with redacted errors, not invented success.6Do not read secrets, commit/push, provision, or make paid calls without my7explicit approval; no later-chapter work. Show git diff and git diff --cached8for my review. I make the commit.

1# Building with TwelveLabs: From ChatGPT to Bedrock2 3You create this repository yourself (suggested name: `twelvelabs-video-lab`).4Every chapter of the series adds files; this is the cumulative layout.5 6[My companion repository: pfplabs/twelvelabs-video-lab](https://github.com/pfplabs/twelvelabs-video-lab) is the reference implementation for this series. Create your own repository to follow along.7 8Documentation-checked examples; live validation pending. Offline strict doubles test request and parser assumptions, not provider success. Start with Jockey in ChatGPT in chapter one before the code setup.9 10## Layout11 12```13twelvelabs-video-lab/14  README.md                  # this file15  requirements.txt           # Python dependencies16  .env.example               # placeholder names only — copy to .env locally17  .gitignore                 # keeps secrets, media, vectors, state out of Git18  media-manifest.json        # your clips: checksums, durations, ground truth19  media/                     # your recordings (gitignored)20  local-results/             # model outputs + your corrections (gitignored)21  vectors/                   # raw embeddings (gitignored)22  state.json                 # TwelveLabs IDs for resume (gitignored)23  bedrock-state.json         # Bedrock job resume state (gitignored)24  media_manifest.py          # ch2: build the manifest25  managed_search.py          # ch2: Marengo 3.0 managed search26  pegasus_summaries.py       # ch3: streamed Pegasus 1.6 summaries27  pegasus_segments.py        # ch3: async segmentation with validation28  still_image.py             # ch3 exercise: still-frame vs video comparison29  embeddings.py              # ch4: Marengo 3.5 direct embeddings30  rank.py                    # ch4: local cosine retrieval31  composed_query.py          # ch4 exercise: image + text composed query32  jockey_store.py            # ch4: Jockey knowledge store via API33  jockey_responses.py        # ch4: Responses with sessions34  bedrock_analyze.py         # ch5: Pegasus 1.5 via Bedrock35  bedrock_embed.py           # ch5: Marengo 3.0 via Bedrock36  export_fixtures.py         # ch6: synthetic demo fixture exporter37  test_lab_contracts.py      # offline strict contract doubles, no API calls38  workbench/                 # ch6: local fixture-first UI (Vite + React + TS)39```40 41## Setup (chapter 2)42 43```bash44# Create your own empty repository first; replace YOUR-USERNAME.45git clone https://github.com/YOUR-USERNAME/twelvelabs-video-lab.git46cd twelvelabs-video-lab47python3 -m venv .venv && source .venv/bin/activate48pip install -r requirements.txt49cp .env.example .env   # then edit .env locally; it is gitignored50```51 52## Run order53 541. ch2: `python media_manifest.py` → fill in `ground_truth` → `python managed_search.py`552. ch3: `python pegasus_summaries.py` → `python pegasus_segments.py` → `python still_image.py`563. ch4: `python embeddings.py` → `python rank.py` → `python composed_query.py` → `python jockey_store.py` → `python jockey_responses.py`574. ch5: `aws sso login --profile twelvelabs-video-lab` → `python bedrock_analyze.py` → `python bedrock_embed.py`585. ch6: `python export_fixtures.py` → `cd workbench && npm install && npm run dev`59 60## Git discipline61 62Before every commit: `git status`, then `git diff --cached`. If you see63`.env`, `media/`, `local-results/`, `vectors/`, or any `*-state.json`,64unstage it. Only code, the manifest, and the synthetic demo fixture are committed.65 66## Offline verification67 68```bash69python3 -m unittest test_lab_contracts -v70cd workbench71npm install72npm test73```74 75Chapter five ranks only the parsed Bedrock Marengo 3.0 corpus with its own 512-dimensional Bedrock query. Never mix direct 3.5 vectors with Bedrock 3.0 vectors merely because dimensions match.

1"""Export the workbench fixture — synthetic demo only.2 3PRIVACY RULE: real results, media, and vectors stay in gitignored local4paths. This exporter writes ONLY the hand-authored synthetic demo fixture5to workbench/public/demo/fixtures.json. Never point it at local-results/.6"""7import json8from pathlib import Path9 10ROOT = Path(__file__).parent11OUT = ROOT / "workbench" / "public" / "demo" / "fixtures.json"12OUT.parent.mkdir(parents=True, exist_ok=True)13 14 15def fake_vector(seed: int, dim: int = 8) -> list[float]:16    """Deterministic, clearly fake values — tryable with zero API calls."""17    return [round(((seed * (i + 3)) % 17) / 17 - 0.5, 4) for i in range(dim)]18 19 20fixture = {21    "corpus": "synthetic-demo",22    "synthetic": True,23    "clips": [24        {25            "id": "demo-clip-01",26            "title": "Demo: notebook and mug",27            "durationSec": 30,28            "mediaFile": None,29            "groundTruth": [30                {"startSec": 2, "endSec": 6, "label": "person opens notebook"},31                {"startSec": 12, "endSec": 15, "label": "red mug moves right"},32            ],33            "segments": [34                {"startSec": 0, "endSec": 6, "embedding": fake_vector(1), "source": "synthetic"},35                {"startSec": 6, "endSec": 12, "embedding": fake_vector(2), "source": "synthetic"},36                {"startSec": 12, "endSec": 18, "embedding": fake_vector(3), "source": "synthetic"},37            ],38            "summary": "Synthetic summary: a person opens a notebook, then moves a red mug.",39            "corrections": [40                {"startSec": 12, "endSec": 15, "decision": "accept", "note": "matches ground truth"}41            ],42        },43        {44            "id": "demo-clip-02",45            "title": "Demo: pen and clap",46            "durationSec": 24,47            "mediaFile": None,48            "groundTruth": [49                {"startSec": 4, "endSec": 7, "label": "pen rolls off table"},50                {"startSec": 16, "endSec": 17, "label": "single clap"},51            ],52            "segments": [53                {"startSec": 0, "endSec": 8, "embedding": fake_vector(4), "source": "synthetic"},54                {"startSec": 8, "endSec": 16, "embedding": fake_vector(5), "source": "synthetic"},55                {"startSec": 16, "endSec": 24, "embedding": fake_vector(6), "source": "synthetic"},56            ],57            "summary": "Synthetic summary: a pen rolls away; later, one clap.",58            "corrections": [],59        },60    ],61    "queries": [62        {"text": "a person opens a notebook", "embedding": fake_vector(1)},63        {"text": "a single clap", "embedding": fake_vector(6)},64        {"text": "a window opens", "embedding": fake_vector(9)},65    ],66}67 68OUT.write_text(json.dumps(fixture, indent=2))69print(f"wrote synthetic demo fixture -> {OUT}")70print("reminder: real exports stay under local-results/ (gitignored)")

1python export_fixtures.py2cd workbench3npm install4npm run dev     # opens the workbench on the synthetic demo fixture5npm test        # runs the ranking unit tests

1workbench/2  package.json3  tsconfig.json4  vite.config.ts5  index.html6  public/demo/fixtures.json   # written by export_fixtures.py (synthetic)7  src/8    main.tsx9    App.tsx10    styles.css11    types.ts12    lib/13      fixtures.ts14      rank.ts15      rank.test.ts

1{2  "name": "twelvelabs-video-lab",3  "private": true,4  "version": "0.1.0",5  "type": "module",6  "scripts": {7    "dev": "vite",8    "build": "tsc -b && vite build",9    "test": "vitest run"10  },11  "dependencies": {12    "react": "^19.0.0",13    "react-dom": "^19.0.0"14  },15  "devDependencies": {16    "@types/react": "^19.0.0",17    "@types/react-dom": "^19.0.0",18    "@vitejs/plugin-react": "^4.3.0",19    "typescript": "^5.6.0",20    "vite": "^6.0.0",21    "vitest": "^2.1.0"22  }23}

1export interface GroundTruthEvent {2  startSec: number;3  endSec: number;4  label: string;5}6 7export interface Segment {8  startSec: number;9  endSec: number;10  embedding: number[];11  source: string;12}13 14export interface Correction {15  startSec: number;16  endSec: number;17  decision: "accept" | "adjust" | "reject";18  note: string;19}20 21export interface Clip {22  id: string;23  title: string;24  durationSec: number;25  mediaFile: string | null;26  groundTruth: GroundTruthEvent[];27  segments: Segment[];28  summary: string | null;29  corrections: Correction[];30}31 32export interface QueryFixture {33  text: string;34  embedding: number[];35}36 37export interface Fixtures {38  corpus: string;39  synthetic: boolean;40  clips: Clip[];41  queries: QueryFixture[];42}

1import type { QueryFixture, Segment } from "../types";2 3export interface RankedSegment extends Segment {4  clipId: string;5  score: number;6}7 8export function normalize(v: number[]): number[] {9  if (!v.every(Number.isFinite)) throw new Error("non-finite value in vector");10  const norm = Math.sqrt(v.reduce((sum, x) => sum + x * x, 0));11  if (norm === 0) throw new Error("zero-norm vector cannot be ranked");12  return v.map((x) => x / norm);13}14 15/** Cosine similarity ranking — the same math as the Python lab. */16export function rankSegments(17  segments: (Segment & { clipId: string })[],18  queryEmbedding: number[],19  topK = 5,20): RankedSegment[] {21  const q = normalize(queryEmbedding);22  return segments23    .map((seg) => {24      const v = normalize(seg.embedding);25      if (v.length !== q.length) {26        throw new Error(`dimension mismatch: ${v.length} vs query ${q.length}`);27      }28      const score = v.reduce((sum, x, i) => sum + x * q[i], 0);29      return { ...seg, score };30    })31    .sort((a, b) => b.score - a.score)32    .slice(0, topK);33}34/** Rank one query; malformed vectors surface as an error, never as "no results". */35export function rankForQuery(36  segments: (Segment & { clipId: string })[],37  query: QueryFixture | undefined,38): { results: RankedSegment[]; error: string | null } {39  if (!query || segments.length === 0) return { results: [], error: null };40  try {41    return { results: rankSegments(segments, query.embedding), error: null };42  } catch (e) {43    return { results: [], error: e instanceof Error ? e.message : String(e) };44  }45}

1import type { Fixtures } from "../types";2 3export async function loadFixtures(): Promise<Fixtures> {4  const res = await fetch("/demo/fixtures.json");5  if (!res.ok) throw new Error(`fixture load failed: ${res.status}`);6  const data = (await res.json()) as Fixtures;7  return validateFixtures(data);8}9 10/** Schema check: the exporter must emit camelCase durationSec, never duration_sec. */11export function validateFixtures(data: Fixtures): Fixtures {12  if (!Array.isArray(data.clips)) throw new Error("fixture has no clips array");13  for (const clip of data.clips) {14    if (!Number.isFinite(clip.durationSec)) {15      throw new Error(`clip ${clip.id} is missing numeric durationSec`);16    }17  }18  return data;19}

1import { useEffect, useMemo, useRef, useState } from "react";2import { loadFixtures } from "./lib/fixtures";3import { rankForQuery } from "./lib/rank";4import type { Clip, Fixtures } from "./types";5 6type View = "clips" | "search" | "review";7 8function formatTime(sec: number): string {9  const m = Math.floor(sec / 60);10  const s = Math.floor(sec % 60);11  return `${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}`;12}13 14function ClipCard({ clip }: { clip: Clip }) {15  const videoRef = useRef<HTMLVideoElement>(null);16  return (17    <article className="card">18      <h2>{clip.title}</h2>19      <p className="meta">20        {clip.id} · {clip.durationSec}s · {clip.segments.length} segments21      </p>22      {clip.mediaFile ? (23        <video ref={videoRef} src={clip.mediaFile} controls preload="metadata" />24      ) : (25        <p className="meta">Synthetic demo clip — no media file.</p>26      )}27      {clip.summary && <p>{clip.summary}</p>}28      <h3 className="meta">Ground truth</h3>29      <ul>30        {clip.groundTruth.map((e) => (31          <li key={`${e.startSec}-${e.label}`}>32            <button33              type="button"34              onClick={() => {35                if (videoRef.current) {36                  videoRef.current.currentTime = e.startSec;37                  void videoRef.current.play();38                }39              }}40            >41              {formatTime(e.startSec)}–{formatTime(e.endSec)}42            </button>{" "}43            {e.label}44          </li>45        ))}46      </ul>47    </article>48  );49}50 51export function App() {52  const [fixtures, setFixtures] = useState<Fixtures | null>(null);53  const [error, setError] = useState<string | null>(null);54  const [view, setView] = useState<View>("clips");55  const [queryIndex, setQueryIndex] = useState(0);56 57  useEffect(() => {58    let cancelled = false;59    loadFixtures()60      .then((f) => {61        if (!cancelled) setFixtures(f);62      })63      .catch((e: unknown) => {64        if (!cancelled) setError(e instanceof Error ? e.message : String(e));65      });66    return () => {67      cancelled = true;68    };69  }, []);70 71  const allSegments = useMemo(72    () =>73      (fixtures?.clips ?? []).flatMap((c) =>74        c.segments.map((s) => ({ ...s, clipId: c.id })),75      ),76    [fixtures],77  );78 79  const ranking = useMemo(() => rankForQuery(allSegments, fixtures?.queries[queryIndex]), [80    fixtures,81    queryIndex,82    allSegments,83  ]);84  const results = ranking.results;85 86  if (error) return <p className="error">Could not load fixtures: {error}</p>;87  if (!fixtures) return <p className="loading">Loading fixtures…</p>;88  if (fixtures.clips.length === 0)89    return <p className="empty">No clips in the fixture yet.</p>;90 91  return (92    <>93      <header>94        <h1>Video AI workbench</h1>95        <p>96          Corpus: {fixtures.corpus}{" "}97          {fixtures.synthetic && (98            <span className="badge synthetic">synthetic demo data</span>99          )}100        </p>101      </header>102      <nav aria-label="Workbench views">103        {(["clips", "search", "review"] as const).map((v) => (104          <button105            key={v}106            type="button"107            aria-pressed={view === v}108            onClick={() => setView(v)}109          >110            {v === "clips" ? "Clips" : v === "search" ? "Search" : "Review"}111          </button>112        ))}113      </nav>114      <main>115        {view === "clips" && (116          <div className="grid">117            {fixtures.clips.map((clip) => (118              <ClipCard key={clip.id} clip={clip} />119            ))}120          </div>121        )}122        {view === "search" && (123          <>124            <label>125              Query:{" "}126              <select127                value={queryIndex}128                onChange={(e) => setQueryIndex(Number(e.target.value))}129              >130                {fixtures.queries.map((q, i) => (131                  <option key={q.text} value={i}>132                    {q.text}133                  </option>134                ))}135              </select>136            </label>137            <p className="meta">138              Pre-embedded queries only — no provider key ever enters the browser.139            </p>140            {ranking.error ? (141              <p className="error" role="alert">142                Ranking failed: {ranking.error}. Fix or re-export the fixture; no results are shown143                rather than a misleading empty list.144              </p>145            ) : results.length === 0 ? (146              <p className="empty">No rankable segments for this query.</p>147            ) : (148              <table>149                <thead>150                  <tr><th>Score</th><th>Clip</th><th>Start</th><th>End</th><th>Source</th></tr>151                </thead>152                <tbody>153                  {results.map((r) => (154                    <tr key={`${r.clipId}-${r.startSec}`}>155                      <td className="score">{r.score.toFixed(3)}</td>156                      <td>{r.clipId}</td>157                      <td>{formatTime(r.startSec)}</td>158                      <td>{formatTime(r.endSec)}</td>159                      <td><span className="badge">{r.source}</span></td>160                    </tr>161                  ))}162                </tbody>163              </table>164            )}165          </>166        )}167        {view === "review" && (168          <div className="grid">169            {fixtures.clips.map((clip) => (170              <article key={clip.id} className="card">171                <h2>{clip.title}</h2>172                {clip.corrections.length === 0 ? (173                  <p className="meta">No human corrections recorded yet.</p>174                ) : (175                  <table>176                    <thead>177                      <tr><th>Interval</th><th>Decision</th><th>Note</th></tr>178                    </thead>179                    <tbody>180                      {clip.corrections.map((c) => (181                        <tr key={`${c.startSec}-${c.decision}`}>182                          <td>{formatTime(c.startSec)}–{formatTime(c.endSec)}</td>183                          <td><span className="badge">{c.decision}</span></td>184                          <td>{c.note}</td>185                        </tr>186                      ))}187                    </tbody>188                  </table>189                )}190              </article>191            ))}192          </div>193        )}194      </main>195    </>196  );197}

1<!-- index.html -->2<!doctype html>3<html lang="en">4  <head>5    <meta charset="UTF-8" />6    <meta name="viewport" content="width=device-width, initial-scale=1.0" />7    <title>Video AI workbench</title>8  </head>9  <body>10    <div id="root"></div>11    <script type="module" src="/src/main.tsx"></script>12  </body>13</html>14 15// vite.config.ts16import react from "@vitejs/plugin-react";17import { defineConfig } from "vite";18export default defineConfig({ plugins: [react()] });19 20// tsconfig.json21{22  "compilerOptions": {23    "target": "ES2022",24    "lib": ["ES2022", "DOM", "DOM.Iterable"],25    "module": "ESNext",26    "moduleResolution": "bundler",27    "jsx": "react-jsx",28    "strict": true,29    "skipLibCheck": true,30    "noEmit": true31  },32  "include": ["src"]33}34 35// src/main.tsx36import { StrictMode } from "react";37import { createRoot } from "react-dom/client";38import { App } from "./App";39import "./styles.css";40createRoot(document.getElementById("root")!).render(41  <StrictMode><App /></StrictMode>,42);

1import { describe, expect, it } from "vitest";2import { normalize, rankForQuery, rankSegments } from "./rank";3import { validateFixtures } from "./fixtures";4import type { Fixtures } from "../types";5import demo from "../../public/demo/fixtures.json";6 7const seg = (embedding: number[], startSec = 0) => ({8  startSec,9  endSec: startSec + 6,10  embedding,11  source: "test",12  clipId: "clip",13});14 15describe("normalize", () => {16  it("rejects zero-norm vectors", () => {17    expect(() => normalize([0, 0, 0])).toThrow(/zero-norm/);18  });19  it("rejects non-finite values", () => {20    expect(() => normalize([1, Number.NaN])).toThrow(/non-finite/);21  });22  it("produces unit length", () => {23    const v = normalize([3, 4]);24    expect(Math.hypot(...v)).toBeCloseTo(1);25  });26});27 28describe("rankSegments", () => {29  it("ranks identical direction first", () => {30    const hits = rankSegments([seg([1, 0]), seg([0, 1])], [1, 0]);31    expect(hits[0].score).toBeCloseTo(1);32    expect(hits[1].score).toBeCloseTo(0);33  });34  it("throws on dimension mismatch", () => {35    expect(() => rankSegments([seg([1, 0, 0])], [1, 0])).toThrow(/dimension mismatch/);36  });37  it("respects topK", () => {38    const hits = rankSegments(39      [seg([1, 0]), seg([0.9, 0.1]), seg([0, 1])], [1, 0], 2,40    );41    expect(hits).toHaveLength(2);42  });43});44 45describe("rankForQuery", () => {46  it("surfaces malformed vectors as an error instead of empty results", () => {47    const out = rankForQuery([seg([0, 0])], { text: "q", embedding: [1, 0] });48    expect(out.results).toEqual([]);49    expect(out.error).toMatch(/zero-norm/);50  });51});52 53describe("validateFixtures", () => {54  it("accepts the committed synthetic fixture with durationSec", () => {55    expect(validateFixtures(demo as Fixtures).clips.every((c) => c.durationSec > 0)).toBe(true);56  });57  it("rejects snake_case duration_sec", () => {58    const bad = { corpus: "x", synthetic: true, queries: [], clips: [{ id: "c", duration_sec: 5 }] };59    expect(() => validateFixtures(bad as unknown as Fixtures)).toThrow(/durationSec/);60  });61});

1## Optional licensed sample footage (downloaded manually; not bundled)2Media stays in gitignored media/. These are film examples, not tabletop ground truth.3 4Sintel trailer — media/sintel_trailer-720p.mp45© copyright Blender Foundation | www.sintel.org. Licensed under CC BY 3.0.6Source: https://download.blender.org/durian/trailer/sintel_trailer-720p.mp47Rights: https://durian.blender.org/sharing/8License: https://creativecommons.org/licenses/by/3.0/9 10Big Buck Bunny trailer — media/big-buck-bunny-trailer-480p.mp411(c) copyright 2008, Blender Foundation / www.bigbuckbunny.org. Licensed under CC BY 3.0.12Source: https://www.w3.org/2013/Talks/html5/big-buck-bunny-trailer-480p.mp413Source page: https://www.w3.org/2013/Talks/html5/14Rights: https://peach.blender.org/about/15License: https://creativecommons.org/licenses/by/3.0/16 17For each local asset, record modifications before sharing:18- Unmodified download: state "No modifications" only if true.19- Trimmed or edited: record source intervals and describe the changes.20Show these credits, links, and notes beside assets in any film-backed view.21Record observations and timestamps only after watching; no expected model results.

1git status && git diff --cached   # only code + synthetic fixture2git add export_fixtures.py workbench/3git commit -m "Chapter 6: fixture-first video workbench and product chooser"4git push