PITs | Quali analytics

What is it about?

Interview analyser that takes your transcripts and produces insights for product work (or other ends).

Two Main Workflows:

A. Configure your system to PITs

• Start with excellent defaults
• Process 2 or more transcripts to example outputs.
• Transfer your approach and needs - Create Templates with taxonomies, custom prompts, customer analyses and insights
• Iterate your prompts to perfection in the tool

B. Get insights

• Drop your transcripts
• Review if you want
• Get insights

Why I’m building it?

1. I will need it in my work.
2. Its a workflow I know inside-out
3. I build a tool to learn relations between Evals, Prompts and Output Quality

How am I building it?

With a set of AI code agents:

• Crystallise workflows
• Iterate the UI
• Test different approaches for evals and prompt testing in the tool
• Build the tool and connect to OpenAI API

Finishing the transcript coder to get reliable and configurable coded transcripts (vie evals), next up: analyses.

What’s my goal?

• Learn to build a workflow that requires exact mapping from insights to sources.
• Learn to build means to test prompting and evals inside a tool

Tech Corner

// STATUS ──────────────────────────────────────────────────────
transcript coding pipeline done | analyses and insights not yet built

lines_of_code: 38200   |   mvp_estimate: 55% done

// PIPELINE ────────────────────────────────────────────────────
approach        Next.js API pipeline, local only

deterministic   file parsing, paragraph splitting, artifact
                writes — format handling must be exact

llm             gpt-4o-mini for speaker detection, prelabeling,
                excerpt linking

reasoning       gpt-4o-mini for cost ($10 total so far) and
                speed; quali analysis doesn't need GPT-4 depth

// HUMAN IN THE LOOP ───────────────────────────────────────────
where           3-pane review UI — user reviews every coded
                segment, edits labels and excerpts before export

why             analyst judgment is the product; LLM output
                is a first draft, not the output

// EVALUATION ──────────────────────────────────────────────────
how             Zod v4 schema validation on every LLM response
                — wrong shape fails loudly at runtime;
                taxonomy.ts enforced at type level

traces          artifacts written to disk per run (turns,
                questions, excerpts, contexts as JSON);
                SQLite persists review edits; run ID ties all

// CONTEXT ─────────────────────────────────────────────────────
approach        stateless per-step prompts; each step gets
                only what it needs — no conversation history

why             qualitative coding is decomposable; full
                transcript every step is wasteful; schemas
                constrain output so window size is irrelevant

// INTEGRATIONS ─────────────────────────────────────────────────
openai_api      all pipeline steps; OpenAI SDK v6.25.0

sqlite          review state only; synchronous, API routes only

// OTHER TOOLS ──────────────────────────────────────────────────
better_sqlite3  vs Prisma — no ORM overhead for a local single-
                user tool; sync API fits Next.js routes cleanly

mammoth         vs pdf-parse — docx is the primary transcript
                format; better structural parsing than PDF

zod_v4          schema is source of truth, types inferred —
                no manual duplication; validates LLM output