{
  "sessionId": "cmu6eyacm0012kkiiyf8xnagx",
  "slug": "testing-generative-pipelines",
  "slides": [
    {
      "order": 1,
      "title": "Problem: nondeterminism and scope",
      "bullets": [
        "Same prompt → different outputs; makes testing hard",
        "Use pinning, seeds, and artifact versioning to stabilize",
        "Session uses fictional placeholder material"
      ],
      "imageUrl": null,
      "altText": "Slide titled 'Problem: nondeterminism and scope'. Three bullets: 1) The same prompt can produce different outputs, which complicates testing. 2) Practical stabilizers include pinning model versions, recording seeds where supported, and treating generated artifacts as versioned inputs. 3) A note that the session uses fictional placeholder material. References to determinism and scope are indicated.",
      "targetSeconds": 60,
      "script": "I build and operate generative pipelines, so I’ll start with the core problem: nondeterminism. The same prompt can and will produce different outputs over time, which breaks traditional deterministic unit testing unless you constrain the pipeline. My practical takeaway is explicit: pin model versions, record seeds where the model exposes them, and treat every generated output as a stored, versioned artifact rather than regenerating it on demand — those are the controls that make outputs testable in practice. I also want to be explicit about scope: the examples and guidance here are drawn from placeholder, fictional material for this session and should be replaced before production use. Explanation: that nondeterminism forces us to treat generation as an upstream, versioned input rather than a free, repeatable operation.",
      "sourceReferences": [
        "e27c4ce3-62b7-4f2a-a30e-57fd7789c19c",
        "568787d4-bb6c-4710-aa5e-f44feed55ace"
      ]
    },
    {
      "order": 2,
      "title": "Operational controls: pinning, seeds, immutability",
      "bullets": [
        "Pin model versions used in pipelines",
        "Record seeds where models expose them",
        "Treat each generated artifact as an immutable, versioned input"
      ],
      "imageUrl": null,
      "altText": "Slide titled 'Operational controls: pinning, seeds, immutability'. Three bullets explain operational practices: pin model versions, record seeds if the model supports them, and store each generated artifact as an immutable versioned input rather than regenerating it. Emphasis on these controls as mechanisms to make outputs testable.",
      "targetSeconds": 90,
      "script": "Operational controls are where engineering buys stability. In practice you pin model versions in your pipeline so a change in the underlying model doesn’t silently alter outputs later; this is recommended in the placeholder guidance. When models expose a seed or randomness parameter, record it alongside the artifact; that can reproduce a specific sampled output when the model supports such replay. And treat generated assets as immutable, versioned inputs — check them into artifact storage and reference specific versions in builds rather than regenerating them every time. Explanation: these controls reduce flakiness and allow higher-layer tests to reason about known inputs.",
      "sourceReferences": [
        "e27c4ce3-62b7-4f2a-a30e-57fd7789c19c"
      ]
    },
    {
      "order": 3,
      "title": "Layered testing: unit, statistical, review",
      "bullets": [
        "Unit tests for code-level game logic",
        "Statistical/simulation tests for math and distributions",
        "Human review queue for policy and asset checks"
      ],
      "imageUrl": null,
      "altText": "Slide titled 'Layered testing: unit, statistical, review'. Three bullets: unit tests target code-level game logic; statistical or simulation tests target mathematical properties and distributions; and a human review queue checks generated assets for policy issues before inclusion. This presents a three-layer strategy where each layer addresses different failure modes.",
      "targetSeconds": 90,
      "script": "Testing in generative pipelines needs layers. The fictional test strategy recommends three complementary layers: unit tests for code-level game logic, statistical or simulation tests to validate math and distributional properties, and a human review queue where generated assets are inspected for policy, style and contextual issues before they enter a build. Unit tests catch logic regressions; statistical tests detect distributional drift or broken constraints; human review catches the subjective and policy-sensitive failures that automated checks miss. Explanation: you should view each layer as targeting different failure modes and compose them rather than expecting any single layer to be sufficient.",
      "sourceReferences": [
        "762aa393-76be-421d-8395-3649b8bcd49f"
      ]
    },
    {
      "order": 4,
      "title": "Trade-offs: cost, bottlenecks, review limiter",
      "bullets": [
        "Generation can be cheaper than human review",
        "Human review often becomes the pipeline bottleneck",
        "Optimize tooling/process around review capacity"
      ],
      "imageUrl": null,
      "altText": "Slide titled 'Trade-offs: cost, bottlenecks, review limiter'. Three bullets note that generating prototypes can cost less than the human time to review them, that human review therefore commonly becomes the bottleneck, and recommend focusing optimization and tooling on review capacity. This frames review as the primary operational constraint in generative pipelines.",
      "targetSeconds": 60,
      "script": "Finally, trade-offs. The placeholder material reports that generation cost can be lower than the human time required to review a generated prototype, so human review often becomes the throughput limiter in a pipeline. Explanation: that makes review capacity the primary process bottleneck to optimize — invest in tooling, sampling strategies, triage, or risk-based review rather than just cheaper generation. This is consistent with the layered strategy where review is an explicit gate before assets enter builds.",
      "sourceReferences": [
        "d29c7320-1261-40b6-9586-e6463570f885",
        "762aa393-76be-421d-8395-3649b8bcd49f"
      ]
    }
  ]
}