Extracting Action Items and Decisions From Meeting Transcripts With LLMs

Defining the output schema before touching a prompt

Meetings generate decisions and commitments constantly, and organizations lose track of both at a rate that ought to embarrass anyone who has sat through a Monday status call and then a nearly identical one the following Monday. The fix most teams reach for now is an LLM pipeline that reads a transcript and pulls out action items and decisions automatically. Built carelessly, that pipeline produces plausible-looking garbage, confident, well-formatted, and wrong. Built as a staged system, with a defined output schema, a two-stage architecture, and real attention to hallucination and evaluation, it produces something a team can act on without double-checking the recording. The gap between those two outcomes is the entire subject here, and most teams land on the wrong side of it because they skip the schema step.

A large share of workers, in one commonly cited figure, 65%, say meetings get in the way of finishing their actual work. A meaningful chunk of that burden isn't the meeting itself, it's the manual follow-up afterward: someone has to remember who owns what, chase down a deadline nobody wrote down, and turn a recording nobody will rewatch into a task list somebody actually reads. Automated extraction tools built around this workflow have reported real time savings, 62% of users in one study said they saved around four hours. That number matters less as a selling point than as evidence of how expensive the manual version of this task already is.

Before any prompt gets written, the output needs a shape, and teams routinely skip this step and pay for it later. Two things get generated from a transcript, and they are not the same thing, even though most pipelines treat them as interchangeable.

A meeting summary is a high-level, abstractive distillation: what got discussed, what changed, what's different now than it was an hour ago. An action item is something else entirely: a structured, discrete unit of work, not a bullet buried in a summary paragraph but a record with fields. A defensible action item schema includes a description, an owner, a due date when one was actually mentioned, and a link back to the decision or topic that produced it. Decisions get their own record too, with a description, the topic segment they belong to, and a confirmation status.

That confirmation status carries more weight than it looks like. Practical implementations store every extracted action item and decision with a confirmed flag that defaults to false. The LLM's output is a draft until a human looks at it and approves it, at which point it becomes part of the system of record. This is a small design choice with large downstream consequences, since it keeps the pipeline's job limited to reducing human effort rather than replacing human judgment. Skipping this pattern means the first hallucinated deadline that gets treated as ground truth will teach the lesson the hard way, usually in front of a client.

The two-stage architecture: why transcription and extraction are separate problems

The standard architecture splits into two layers. An automatic speech recognition (ASR) layer converts audio into text, and a separate LLM layer analyzes that text. A Zenodo paper documents exactly this split, using Whisper for transcription and locally deployed LLMs run through Ollama for extraction, and that separation now reflects something close to consensus in how these pipelines get built.

Transcription is close to solved, at least for well-recorded, single-language audio. The hard, unsolved work has moved almost entirely to the extraction layer, where a model has to reason about who committed to what and by when. But "solved" is doing quiet work in that sentence, because transcription quality still gates everything downstream. Poor diarization, the system's guess at who said which line, propagates directly into misattributed action items. If the transcript says the wrong person proposed a deadline, the extraction layer inherits that error with no way to catch it. That's a garbage-in problem, and no amount of clever prompting at the extraction stage fixes bad input at the transcription stage.

Platform choice matters too: Google Meet, Microsoft Teams, and Zoom all ship built-in note-taking now, and for teams that live entirely inside one of those ecosystems, that's often good enough, but those tools stay confined to their own platform. Google Meet, Microsoft Teams, and Zoom all ship built-in note-taking now, and for teams that live entirely inside one of those ecosystems, that's often good enough. But those tools stay confined to their own platform, with limited customization and restricted export options. The schema decisions described above get made by the vendor, not the team using the output. A platform-independent pipeline, one that can ingest transcripts from multiple sources, hands practitioners control over both stages of the architecture instead of accepting whatever a single vendor decided the schema should look like.

Building beyond a single monolithic prompt that fails in production

Most teams start the same way: one large prompt, the entire transcript pasted in, asked to return a summary and a list of action items in one shot. It works fine in staging. It breaks in production, and it breaks in a specific, recognizable way.

Consider what happens when a long, complex transcript is fed to a single monolithic prompt: the LLM hallucinates action items and misattributes decisions, and downstream systems inherit those errors without any signal that something went wrong. That isn't a bug in the traditional sense. The model isn't broken so much as overloaded: the transcript it was handed and the task it was asked to do in one pass were both too much for a single inference call to carry reliably. The root cause sits with the input and the framing, not with the model's competence, and treating it as a model problem leads teams to swap models instead of fixing the architecture.

The fix is breaking the single prompt into a modular pipeline, with each stage doing one narrow job. The first stage handles speaker diarization and transcript cleaning. The second stage performs topic segmentation and chunking. Stage three extracts decisions. Stage four extracts action items along with owners and deadlines. Stage five is a linking pass: it connects action items back to the decisions discussed in the same conversational segment, then writes those links into a state object before anything gets routed downstream to a CRM, a task tracker, or a summary email.

Five stages is more engineering than one prompt. It's also what separates a pipeline that degrades gracefully from one that hallucinates with total confidence.

Chunking long transcripts to avoid information loss in the middle

Long transcripts run into a specific, well-documented failure mode: LLMs give less attention to content sitting in the middle of a long input than to content near the beginning or end. The "lost-in-the-middle" effect is a structural property of how these models process long context windows. It's a structural property of how these models process long context windows, and no amount of prompt tuning removes it.

AutoMin 2025, the third Automatic Minuting shared task, noted that transcripts from one-hour meetings routinely require context windows exceeding 16,000 tokens. Whatever got said forty minutes in, often the exact stretch where the real decision got made, is most likely to get underweighted when a model is fed that much text in one pass.

The standard mitigation is topic-level chunking, and it follows a fairly consistent pattern across implementations. One documented approach segments each transcript into contiguous chunks of around 1,024 tokens. Each chunk gets summarized down to two or three bullet points. Those chunk-level summaries then get consolidated into meeting-level topic labels. The result preserves the shape of the full agenda without ever asking a single prompt to hold the entire transcript in working memory at once.

Where you cut the chunks matters as much as the chunk size. Splitting in the middle of a sentence, or worse, in the middle of an argument someone was building toward a conclusion, creates extraction artifacts: half a decision gets summarized as if it were the whole thing. Segmenting along natural topic boundaries or speaker-turn boundaries, rather than fixed token counts, cuts down on that failure considerably.

Keeping the LLM faithful to what was said: hallucination mitigations that work

A hallucinated action item isn't just an accuracy problem. Assign a task to the wrong owner, attach a due date nobody committed to, and you either create real accountability confusion or get the item quietly ignored. Either way, it teaches the whole team not to trust the system's output. Once that trust erodes, the pipeline stops saving time and starts costing it, because someone now has to cross-check every extracted item against the recording anyway, which defeats the point of building the thing.

Structured output enforcement eliminates an entire category of this failure before it ever reaches a human. OpenAI's Structured Outputs feature guarantees that a model's response conforms to a supplied JSON Schema, field types and all. The Instructor library, for Python developers, achieves the same guarantee using Pydantic models, and it retries automatically when a response fails validation. Concretely, this stops a model from returning a deadline field as the string "next week" instead of an actual date like "2026-04-12," a malformed value that will silently break whatever downstream system tries to parse it. A GitHub issue in this space describes a Pydantic v2 schema, wired through Instructor, extracting a summary field, a list of decisions, and a list of action items complete with suggested owners and target dates.

Beyond schema enforcement, context boundary prompting closes a different gap. Explicit instructions, something like "if information is missing, respond with 'Not enough data'" instead of guessing, stop the model from fabricating a plausible-sounding deadline or owner that nobody actually stated. Multi-pass generation helps further: a draft pass followed by a verification pass with internal fact-checking, before a final pass gets produced, cuts down on errors compared to asking for the finished output in one generation. None of this makes hallucination impossible. It makes it rare enough that the confirmed-flag pattern becomes a quick review step instead of a full rewrite.

Inferring owners and due dates: the limits of what the LLM can resolve

Assigning ownership sounds like the easy part. It's actually one of the hardest parts in the whole pipeline. Names spoken aloud in a meeting are frequently partial ("a first name will handle it"), ambiguous (two people sharing the same first name on the same call), or belong to someone who isn't even in the room. Diarization tells you which speaker said which line. It does not tell you which system user in your task tracker that speaker actually is, and conflating the two is where a lot of extraction pipelines quietly fall apart.

A handful of heuristics handle most of this reliably, and the pipelines that skip them are the ones that end up assigning tasks to the wrong person of that name. Cross-referencing the attendee list and the calendar invite maps spoken first names to full accounts with far more precision than the transcript alone provides. Nobody should get assigned a task if they weren't present in the meeting, unless another participant explicitly named them as the intended owner. Directory lookups match a spoken name against system user IDs, and role-based heuristics help too: a task that sounds like a design decision should point to product or design leads as likely candidates before anyone else. Critically, an ambiguous name should trigger a confirmation flow rather than an automatic assignment. Guessing wrong here does more damage than staying silent.

The schema itself needs to accommodate that ambiguity instead of papering over it. A well-built extraction schema stores a suggested_owner_name field, the raw string pulled from the transcript, separately from an owner field, a foreign key to an actual system user that stays nullable. When the name resolves cleanly, the foreign key gets populated. When it doesn't, it stays null, and a human resolves that record rather than the system silently assigning it to whoever the model guessed came closest.

Due dates get the same treatment, and this is where a lot of teams overreach. Dates explicitly stated in conversation get extracted directly. Anything inferred, like "by end of sprint" or "before the client call next week," gets flagged as needing confirmation instead of getting silently converted into a hard calendar date. Turning a vague verbal commitment into a precise deadline without a human checking it first is overreach. It's the exact overreach that erodes trust in the system fastest.

Prompt design choices that move results at the extraction stage

Small prompt-level decisions produce measurable differences in output quality, and a handful of them recur across implementations worth taking seriously.

Assigning the model a persona in the system prompt, and asking it to structure output inside XML tags, cuts down on the redundant throat-clearing models tend to open and close responses with, and nudges output toward a more consistent format. AWS used exactly this approach in its meeting summarization and action item extraction evaluation. One-shot prompting, a single worked example included directly in the prompt, reinforces that consistency further, which matters more than it sounds like it should once a pipeline switches between model families, since different models respond to identical instructions in noticeably different ways.

Chain-of-thought prompting doesn't belong everywhere, and applying it uniformly is a mistake. AWS applied it specifically in the action item extraction prompt, not in the summary prompt, because the two tasks carry different reasoning demands. Summarizing what was said is mostly compression. Extracting an action item requires the model to reason about intent, ownership, and commitment, and that reasoning benefits from being walked through step by step rather than produced in one leap.

On the broader question of prompt engineering versus fine-tuning: for meeting summarization and extraction tasks, prompt engineering is typically the more practical starting point. It allows rapid, domain-specific customization, adapting the same pipeline to a sales team's vocabulary versus an engineering team's, without retraining a model every time the vocabulary shifts. Fine-tuning for this use case is usually solving a problem that a better prompt already solves for less money.

Evaluating what the pipeline produced: why ROUGE is insufficient and what to use instead

Measuring whether any of this worked runs into a separate problem, and it's one most teams underestimate. Established summarization metrics like ROUGE and BERTScore are imperfect proxies when applied to meeting summarization, and they miss the kinds of nuanced errors that matter most here.

Structural factors explain it. ROUGE measures n-gram overlap: how many of the same word sequences appear in the generated output versus a reference text. It cannot verify that the owner listed on an action item is the right person, that the deadline is one anyone actually stated, or that the "action item" was a real commitment rather than a passing suggestion someone floated and then dropped. Those are precisely the dimensions that make a pipeline useful or useless, and they sit entirely outside what overlap-based metrics can see.

The approach gaining ground instead is using LLMs themselves as evaluators. An LLM evaluator brings genuine contextual understanding to the judgment task, and it can adapt its error definitions to a new domain without first needing a large training set of human preference judgments, which was the resource-intensive requirement that made earlier evaluation approaches hard to scale. This isn't fully settled, and evaluation methodology for extraction pipelines remains an active area of comparison across research benchmarks. But the direction is clear enough: judging who owns what, and by when, in a pipeline's output requires a judge that understands meaning, not one that counts overlapping words.