Multimodal AI Agents: When Work Crosses Text, Data, Code, and Media

§ 01What Is a Multimodal AI Agent?

A multimodal AI agent is an agentic system that works with multiple data or media types while selecting tools and steps toward a defined outcome. It differs from a multimodal model because the agent also manages workflow state, tool use, verification, recovery, and completion.

The boundary matters because each claim requires different evidence. Image input proves perception. Video generation proves an output capability. Neither proves that the agent can use figures from a source-backed workbook in the correct scene, render the file, inspect it, and stop before publication.

Multimodal means more than visual

The common shortcut is to equate multimodality with image understanding. Business work uses a broader set of representations:

• natural-language instructions and documents;
• tables, spreadsheets, and database records;
• code, queries, and formulas;
• images, screenshots, charts, and diagrams;
• audio, speech, and transcripts;
• video, scenes, captions, and timelines;
• interfaces, application state, and actions; and
• files whose layout or rendering is part of correctness.

A PDF can contain text, tables, charts, images, page geometry, and annotations at once. Treating it as plain extracted text can discard the information required to interpret it correctly.

The agent loop is the defining difference

OpenAI’s developer quickstart presents text generation, image analysis, file inputs, tools, and agent building as distinct capabilities. A multimodal agent connects selected capabilities through a loop that can:

1. inspect the current goal and state;
2. identify the next required operation;
3. select an appropriate capability or tool;
4. create or update an artifact;
5. verify the artifact or resulting state;
6. repair, retry, escalate, or continue; and
7. stop when the acceptance contract passes.

Without that loop, the user remains the integration layer.

§ 02When Does a Workflow Actually Need Multiple Modalities?

A workflow needs multiple modalities when the accepted outcome depends on information that cannot be preserved, checked, or communicated adequately in one representation.

The test is dependency, not variety.

Multimodality is justified when removing one representation would make the outcome materially less accurate, less usable, or impossible to validate.

Use the artifact-dependency test

For every proposed modality, ask:

1. What information exists only in this representation?
2. Which downstream artifact depends on it?
3. How will the information be transferred?
4. Which evaluator can detect a bad transfer?
5. What happens if the capability is unavailable?

If a modality has no unique input, downstream dependency, or acceptance test, it may be decoration rather than workflow infrastructure.

Use the smallest sufficient modality set

More modalities increase:

• tool-selection choices;
• file and rendering failures;
• conversion loss;
• context volume;
• evaluation work;
• permissions and data exposure;
• latency and cost; and
• dependence on external generation services.

The right architecture uses the smallest set that can produce an accepted outcome. This follows the same task-fit principle used to identify the best first tasks for AI employees: begin with a bounded, observable result whose quality can be checked.

Distinguish essential, supporting, and optional outputs

Classify every artifact before execution:

This prevents a failed optional video render from blocking an otherwise complete decision package. It also prevents an impressive image from hiding a failed essential calculation.

§ 03Which Modality Layers Appear in Real Agent Work?

The same format can play different roles. A spreadsheet may be an input, a working artifact, a final deliverable, or evidence that validates another output.

Confusing these layers creates avoidable risk. A draft image used to explore a concept should not be treated as an approved campaign asset. A model-generated table should not become a source of record because it appears in a polished spreadsheet.

Text and documents carry claims and constraints

Text is usually where goals, policies, caveats, definitions, and explanations live. Document fidelity requires more than fluent sentences:

• every material claim has support;
• qualifications remain attached to the claim;
• names, dates, units, and versions are correct;
• required sections exist;
• prohibited language is absent; and
• the rendered document preserves headings, tables, footnotes, and page layout.

Data and code carry reproducible transformations

Tables, queries, formulas, and code should preserve types, joins, filters, assumptions, missing values, and calculation logic. Deterministic checks are usually stronger than a model’s verbal confirmation.

Use code or formulas to verify totals. Use schema checks to detect missing columns. Use range and type checks to catch malformed values. Preserve the original data and the exact transformation that created the result.

Images and diagrams carry spatial meaning

Images can encode composition, hierarchy, labels, relationships, color meaning, and brand constraints. Evaluation may require:

• optical character recognition for visible text;
• object and layout checks;
• label-to-data reconciliation;
• dimensions and aspect ratio;
• contrast and accessibility;
• rights and source review; and
• human inspection for visual defects.

An attractive chart is wrong when the scale, labels, or series no longer match the data.

Audio and video add time

Audio and video introduce sequence, timing, pronunciation, captions, scene continuity, and playback integrity. A correct script does not prove a correct render.

The acceptance contract may need transcript matching, duration bounds, caption timing, scene order, audio levels, frame dimensions, brand review, and a complete playback test.

Interfaces add external state

An agent may interpret a screen, click a control, upload a file, or update an application. The screenshot is an input modality; the action is an external effect.

A successful click is not the acceptance condition. Verify the resulting record, status, recipient, amount, file, or system state.

§ 04How Does a Multimodal AI Agent Work End to End?

A reliable workflow moves through seven stages. The exact route can change, but each stage has an explicit artifact and control.

The task is complete only when every essential artifact and transition passes.

End-to-end acceptance = all essential artifact gates pass AND all required transition gates pass

Begin with one completion contract

“Analyze this market and make content” leaves too many choices undefined. A stronger contract specifies:

• source boundary and date range;
• required claims and comparison dimensions;
• raw data and missing-value rules;
• calculations or code that must be reproducible;
• document, spreadsheet, image, or media formats;
• brand and accessibility requirements;
• actions that remain draft-only;
• deadline and cost budget;
• acceptance owner; and
• conditions that require a safe stop.

The agent may adapt the path without redefining the outcome.

Create a source manifest before transformation

The manifest should identify:

The manifest prevents a later chart or slide from citing “the report” when several versions exist.

Preserve intermediate artifacts

Use an evidence table, cleaned dataset, analysis code, calculation workbook, chart data, copy deck, script, and render preview as separate versioned objects. Do not rely on one conversation to preserve the complete state.

Intermediate artifacts support recovery. If the video renderer fails, the approved script and scene plan should remain available. If a chart is wrong, the team should be able to return to its data series without repeating the research.

Close on observable state

Anthropic’s agent-evaluation guidance distinguishes the final response from the outcome in the environment. An agent can claim that a file was created while the file is missing, corrupted, or stored in the wrong location.

Closure should check the actual files, records, messages, and approvals - not the agent’s description of them.

§ 05How Do You Write an Artifact Contract?

An artifact contract defines what an output must contain, where it came from, how it is evaluated, and who may approve or use it.

The contract should be machine-checkable where possible and human-checkable where judgment matters.

Define cross-artifact invariants

An invariant is a fact or constraint that must survive every representation. Examples include:

• reporting period is April 1 to June 30, 2026;
• revenue is shown in U.S. dollars;
• percentages use the same denominator;
• the recommendation remains conditional on an unresolved source gap;
• the product name uses approved capitalization;
• external publication requires named approval; and
• every chart links to its source data.

Store invariants once and test them across the package. Rewriting them independently in each prompt increases drift.

Bind approval to a version

Approval of deck_v2 does not approve deck_v3. A changed chart, script, image, caption, recipient, or destination can invalidate the decision.

For consequential use, the approval record should identify the artifact ID, version, scope, approver, time, and allowed next action.

§ 06Why Do Cross-Modal Handoffs Fail?

Most multimodal failures are not caused by one capability being completely absent. They occur because information changes meaning between capabilities.

Extraction can remove meaning before analysis begins

A document parser may capture the visible number but lose its footnote, row label, color legend, page context, or table hierarchy. A screenshot model may interpret the chart but miss the appendix that corrects it.

For material values, retain:

• source ID;
• page, sheet, timestamp, or region;
• original text or cell reference;
• normalized value;
• unit;
• period;
• extraction method; and
• confidence or review state.

Transformation can create silent precision

Code can calculate the wrong metric perfectly. A chart can render bad data beautifully. The agent should not infer that a technically valid file is semantically correct.

Separate:

1. schema validity;
2. calculation validity;
3. business-definition validity;
4. source support; and
5. presentation quality.

Each needs its own evaluator.

Generation can optimize appearance over truth

Images, slides, and videos can look complete while losing caveats, changing numbers, inventing labels, or using prohibited claims. Polished output raises the need for evidence, because reviewers may lower their guard when the artifact looks professional.

Lock approved facts and required caveats before creative generation. Compare the rendered artifact with the locked claim set afterward.

Repair can propagate unnecessarily

A changed source value should update the derived calculation, chart, memo sentence, and slide - not every unrelated artifact.

Maintain a dependency graph:

source > extracted value > calculation > chart > memo claim > slide > script > rendered media

When one node changes, revalidate its descendants and preserve unaffected approved work.

§ 07How Should Context and Artifacts Be Managed?

Large multimodal payloads should live as named, versioned artifacts rather than being pasted into every model call.

Google’s context-aware agent architecture shows how a 5MB CSV, a massive JSON response, or a full PDF transcript can become a permanent context tax when it remains in chat history. Its artifact pattern keeps the large object outside the prompt, exposes a lightweight reference, loads the content when needed, and removes it from working context after the task.

Anthropic’s context-engineering guidance describes a similar just-in-time approach: keep lightweight identifiers such as file paths, stored queries, and links, then retrieve relevant data at runtime.

Separate artifact storage from model context

Storage answers, “What information and files exist?” Working context answers, “What does the agent need for this decision?”

The working set should contain:

• current objective and constraints;
• relevant source excerpts or query results;
• artifact references and versions;
• active invariants;
• unresolved gaps;
• applicable permissions; and
• the next acceptance condition.

It should not contain every prior file, failed render, duplicate message, or unrelated task.

Use summaries as indexes, not new authority

A summary should point back to the source objects it represents. If the summary conflicts with a current source, the source hierarchy should decide which one governs.

Never let a compressed description silently become the only evidence behind a chart, claim, or action.

§ 08How Should Multimodal AI Agents Be Evaluated?

Evaluation should cover each capability, each transition, the complete package, and the final external state.

Google’s Agent Development Kit overview pairs multimodal audio and video streaming with evaluation of execution paths and response quality against predefined datasets. The general lesson is broader than one framework: evaluate the path and result, not the format list.

Build a representative test pack

An illustrative 16-case pilot could include:

• 4 ordinary cross-modal tasks;
• 2 incomplete-source tasks;
• 2 conflicting-source tasks;
• 2 malformed or corrupted-file tasks;
• 2 changed-requirement tasks;
• 2 denied-tool or unavailable-capability tasks; and
• 2 prohibited-action or unsafe-publication tasks.

This is a design example, not an industry standard. Increase the sample when the workflow has more variation, rare high-impact failures, or several independent artifact paths.

Grade transitions with explicit denominators

Use measures such as:

Transition integrity rate = correct tested transfers / all tested transfers

Essential artifact acceptance rate = accepted essential artifacts / submitted essential artifacts

Package acceptance rate = accepted complete packages / attempted packages

Recovery success rate = safely resumed or closed failures / recoverable failures

Cost per accepted package = total platform + tool + review + correction cost / accepted packages

Also track unsupported claims, unit errors, render failures, wrong-tool selections, human correction minutes, and approval violations.

Do not average away hard failures

A package with a 95% weighted quality score still fails if it sends an unapproved asset, exposes protected data, corrupts a required workbook, or uses a materially wrong figure.

Keep non-compensating gates separate:

• prohibited action;
• sensitive-data disclosure;
• missing essential artifact;
• invalid or unsupported material claim;
• unreconciled calculation;
• corrupted final file; and
• inability to stop or recover.

Evaluate rendered output

File generation is not file acceptance. Open the workbook, document, presentation, image, audio, or video in a representative environment.

Check:

• file existence and type;
• dimensions, pages, sheets, slides, or duration;
• fonts, overflow, clipping, and broken layout;
• formulas, links, captions, and embedded assets;
• transcript or visible text;
• playback from beginning to end; and
• consistency with source and working artifacts.

§ 09When Should a Specialist Handle One Modality?

Keep one broad agent when shared context and adaptive sequencing improve the outcome. Add a specialist when a focused boundary produces measurable quality, control, latency, or cost gains.

The general-purpose versus specialized AI agent framework provides the complete architecture test. For multimodal work, the split decision should follow observed failures.

A specialist is a fallback, not a prestige layer

Do not split merely because the task contains several formats. A specialist adds:

• another interface;
• context packaging;
• artifact transfer;
• version reconciliation;
• separate permissions;
• latency;
• cost; and
• another failure boundary.

The split is justified only when those costs are smaller than the measured improvement.

Keep the parent artifact contract intact

Whether the capability is a tool, model, workflow, or specialist agent, the parent owner should preserve the complete outcome and acceptance criteria. The specialist returns a bounded artifact and evidence; it does not silently redefine the final package.

§ 10What Governance Does Multimodal Work Require?

Capability breadth must not become authority breadth. An agent that can create documents, media, code, and application updates should receive only the data and actions needed for the current task.

The AI employee versus workflow automation comparison explains why hybrid systems are often safer. The agent can interpret and compose while deterministic code enforces stable formulas, schemas, policy checks, and consequential state transitions.

Untrusted media is data, not instruction

Documents, images, transcripts, web pages, and files can contain hostile or irrelevant instructions. Their content should not override the task contract, system policy, or permission boundary.

Human review follows consequence

Internal draft artifacts may use sampled review after strong evidence. Public claims, regulated content, financial decisions, hiring decisions, legal or medical material, and irreversible actions require qualified human oversight appropriate to their consequence.

Users set the agent’s goals, permissions, and connected accounts and remain responsible for monitoring work performed on their behalf.

§ 11What Does a Complete Multimodal Workflow Look Like?

Consider an illustrative market-entry decision package. The input contains 8 approved source documents, one 5,000-row CSV, a brand guide, and a request for:

• a source ledger;
• a cleaned analysis workbook;
• a decision memo;
• an 8-slide presentation;
• one draft infographic; and
• one 30-second internal video summary.

The numbers define the example; they are not performance benchmarks.

Stage 1: inventory and extract

The agent records all 9 source objects, their dates, owners, permitted uses, and known gaps. It extracts material claims and table values with page or row references.

Gate:

• every required source is present or marked missing;
• every material extracted value has a location and unit;
• conflicting figures remain visible; and
• unapproved external sources are excluded.

Stage 2: clean and analyze

The agent preserves the raw CSV, creates a cleaned version, records transformations, runs calculations, and reconciles totals. Code or formulas validate row counts, types, missing values, periods, and outliers.

Gate:

• raw data remains unchanged;
• 5,000 input rows reconcile to kept, corrected, and excluded rows;
• formulas or code are available;
• assumptions are labeled; and
• failed checks block downstream claims.

Stage 3: form the decision

The memo combines the source evidence and analysis into one recommendation. Each material claim points to a source or calculated artifact. Unresolved questions remain conditions, not confident conclusions.

Gate:

• required decision dimensions are covered;
• claims have support;
• numbers match the workbook;
• uncertainty is visible; and
• prohibited recommendations are absent.

Stage 4: compose and render the presentation

The presentation uses the memo’s recommendation and the workbook’s chart data. The agent applies the brand guide, renders all 8 slides, and inspects the result.

Gate:

• every slide exists;
• figures and periods match the workbook;
• charts have correct labels and scales;
• text does not overflow;
• required caveats remain visible; and
• the file opens in the target environment.

Stage 5: create optional media

The infographic and video use only the approved claim set. The video script is derived from the accepted memo, then locked before narration and rendering.

Gate:

• visible and spoken claims match the approved script;
• captions match the audio;
• duration is within the 30-second requirement;
• brand and rights checks pass; and
• both assets remain drafts.

If the video fails, the essential workbook, memo, and presentation can still pass. If the workbook fails, every dependent artifact remains blocked.

Stage 6: test recovery

Repeat selected cases with one corrupted PDF, one changed CSV column, one failed render, and one denied media tool.

The agent should preserve valid work, identify the exact affected descendants, avoid prohibited substitutions, and resume from the last accepted artifact after correction.

§ 12How Does CellCog Fit Multimodal Agent Work?

CellCog presents its Super-Agent as a broad execution layer that plans, researches, builds, and delivers finished artifacts across multiple formats.

CellCog’s Super-Agent currently lists deep research, data analysis, code, PDFs, presentations, spreadsheets, images, video, speech, music, dashboards, web apps, diagrams, and 3D models among its output categories.

That is a product capability claim, not evidence that every modality or combination will meet every team’s quality threshold. The right evaluation uses the exact sources, formats, constraints, tools, permissions, and acceptance criteria required by the real workflow.

Match the product path to the artifact

CellCog publishes separate capability pages for tasks such as data analysis and presentation creation, alongside image- and video-generation skills.

Use those pages to identify the supported workflow surface, then test the actual artifact contract. A skill page does not replace a representative file, render, calculation, or review.

Do not transfer benchmark evidence across modalities

A research benchmark can support a scoped research claim under its published task set and evaluator. It does not automatically establish spreadsheet correctness, code quality, visual fidelity, video quality, tool reliability, or cross-modal transition integrity.

Evaluate platform breadth against one complete job

The AI employee platform selection framework helps score role fit, tools, memory, permissions, evaluation, reliability, cost, and data controls. For multimodal work, add:

• required input and output formats;
• artifact storage and versioning;
• deterministic validation;
• rendering and playback inspection;
• cross-artifact reconciliation;
• specialist fallback;
• model and tool routing;
• and cost per accepted package.

§ 13How Should You Pilot a Multimodal AI Agent?

Start with one bounded artifact chain whose current human workflow is known.

Map every artifact to one capability and evaluator

Use a matrix like:

Map each workflow artifact to a capability and evaluator before choosing a platform or adding modalities. This exposes unsupported steps before they become dependencies.

Expand one boundary at a time

Do not change the model, tools, source set, artifact list, and permissions in the same release. A failed result will not reveal which boundary caused the regression.

Add one meaningful capability, rerun affected cases, and preserve the previously accepted package as a comparison.

§ 14Make Every Modality Earn Its Place

A multimodal AI agent is valuable when one accepted outcome genuinely crosses representations and continuity reduces manual handoffs. It is not valuable merely because a platform can create many formats.

Map the source, artifact, dependency, evaluator, owner, and fallback for every step. Keep large objects outside working context until needed. Test each capability, every important transition, the complete package, and the final external state.

If one broad agent preserves the chain reliably, keep the architecture simple. If one modality repeatedly fails, add a focused specialist or deterministic pipeline. The number of formats is not the success metric; the accepted, internally consistent outcome is.