AI Employee Role Scorecard: A Pre-Hire Template

§ 01The Role Scorecard at a Glance

The maximum is 36. The total is useful only after the gate review. A role with 34 points and an uncontrolled payment action is not a better candidate than a 29-point internal research role whose weaknesses can be repaired safely.

§ 02What Is an AI Employee Role Scorecard?

An AI employee role scorecard is a pre-configuration decision record. It tests whether a business responsibility is ready to become a standing, governed software role.

It sits between role discovery and implementation:

Candidate outcome, then job description, then role scorecard, then pilot contract, then onboarding.

The scorecard answers:

1. Is this a coherent role?
2. Is an AI employee the right system shape?
3. What makes the role safe and evaluable enough to test?
4. Which missing artifacts block a pilot?
5. Which evidence must the pilot produce?

It should not answer: Which vendor has the longest feature list? Will the role definitely deliver ROI? Is the worker safe in every context? What production permission should it receive? Can it replace a person?

Those conclusions require platform evidence, representative evaluation, current controls, and live operating data.

Score the work system, not the model

The same model can succeed in one role and fail in another because the surrounding conditions differ: source quality; task mix; tool design; permission boundary; output schema; reviewer standard; escalation behavior; system latency; error detectability; and consequence.

The role scorecard therefore evaluates the complete assignment. A benchmark, polished answer, or one successful tool run cannot substitute for role-specific evidence.

Use a cross-functional review

The score should include the people who know the work and its consequences: the business owner; the current process owner; the downstream user; the domain reviewer; the implementation lead; the security or system owner; and privacy, legal, compliance, finance, or another qualified function when relevant.

One person can fill several roles in a small team. Preserve the decisions even if the meeting is short.

NIST’s AI RMF Core treats AI risk as contextual and calls for clear roles, multidisciplinary input, measurement, ongoing monitoring, and documented risk response. The scorecard applies that principle at the moment a proposed role is still cheap to narrow or reject.

§ 03How to Use the Scorecard

Apply the Six Gates Before Adding Points

Run the gates first. A failed gate produces Stop or Redesign, regardless of the total.

Gate 1: accountable ownership

The software can produce work, maintain state, and escalate. It cannot become the legal or organizational owner of the role’s consequences.

Require names for the business outcome; daily supervision; acceptance; source policy; access; incidents; and retirement.

If nobody will review or answer escalations, the correct decision is not “more autonomy.” It is no launch.

Gate 2: observable acceptance

The role must produce a result another person or system can inspect: an accepted briefing; an approved draft; a correctly reconciled record; a resolved allowlisted case; a verified state change; or a complete escalation packet.

“Helpful,” “productive,” and “proactive” do not create a denominator.

Gate 3: lawful and approved inputs

Identify the required information before selecting a tool. Stop when the outcome depends on data the organization may not collect, disclose, combine, infer, or retain for that purpose.

Do not assume that technical access equals approved use.

Gate 4: controlled consequence

The AI employee permissions and approvals model should be feasible for every consequential action.

Some roles can remain useful in observe or prepare mode. A support employee can classify and draft while a person owns refunds, account changes, security incidents, and policy exceptions.

Gate 5: evaluation

The organization needs representative work samples, a rubric, qualified graders, and boundary cases. If expert reviewers cannot agree on a correct result, narrow the role or improve the policy before asking software to own it.

Anthropic’s agent-evaluation guidance emphasizes that agents act across turns, tools, and changing state. Score the trajectory, permissions, escalation, and durable result - not only the final prose.

Gate 6: intervention and exit

The role needs pause control; trigger disablement; credential revocation; task-state visibility; approval invalidation; evidence preservation; recovery ownership; and a retirement procedure.

A system that works only while everything is healthy is not ready for standing work.

How to Score: 0, 1, 2, or 3

Use the same scale for every criterion.

Do not use 0 to mean “low risk.” It means the team does not know.

Add an evidence code

Record one code beside each score:

• O - Observed: current work sample, system record, measured baseline, or reproduced test.
• D - Documented: approved policy, process map, contract, rubric, or access design.
• A - Assumed: owner judgment without current evidence.
• U - Unknown: unanswered or contradictory.

A score of 3 should usually have observed or documented evidence. An assumed 3 is a warning that discussion confidence has outrun proof.

Score the current design

Do not award points for controls someone plans to create later. Score the current role, then record the artifact needed to improve it.

This makes the scorecard a work queue rather than a ceremonial approval.

Do not average reviewers silently

If a business owner gives tool feasibility a 3 and the system owner gives it a 1, record the disagreement. Resolve which action, identity, data, or environment caused the gap.

Averaging to 2 hides the information that matters.

§ 04Score Criteria 1-5: Outcome and Role Fit

The first four criteria determine whether the proposal is a real standing role.

1. Recurring outcome

Ask: what result should still be owned after the current task ends?

Score 3 when the outcome identifies the accepted unit; the downstream user; the queue or population; the cadence or trigger; the starting scope; and the exception destination.

Examples: maintain one accepted competitor-change brief each week; prepare policy-grounded responses for three allowlisted support categories; reconcile the operating dashboard and explain material changes every Friday.

Score 1 for “manage marketing,” “help operations,” or “be an AI chief of staff.” These may be role-discovery labels, but they are not operating outcomes.

2. Accepted unit

The accepted unit should be countable, reviewable, and tied to a downstream need.

Score 3 when the role has a deliverable or state schema; a completion rule; an acceptance owner; a rubric; a rejection or correction path; and a denominator.

The accepted unit protects the economic calculation. If a worker starts 100 tasks, completes 80, and only 40 are usable, cost per task started hides the operating result.

3. Coherent scope

A coherent role shares one outcome; source hierarchy; acceptance rubric; risk class; reviewer; authority pattern; and escalation path.

Score 1 when the proposal combines research, strategic decisions, publication, customer outreach, CRM administration, and performance reporting because they all sit under “marketing.”

Score 2 when several subtypes share an outcome but need separate test cases or approvals.

Score 3 when included work, non-goals, edge cases, and handoff destinations are explicit.

4. Work recurrence

A standing role adds setup and governance overhead. It needs enough recurring demand to justify an identity; schedules or wake conditions; persistent context; task state; access review; evaluations; monitoring; and handovers.

Score 3 when current records show queue volume, cadence, seasonality, task mix, elapsed time, work time, and exceptions.

Score 1 when the work happened once or each request has a different purpose. An on-demand agent may fit better.

Use the best-tasks framework for AI employees when the role contains several possible starting tasks. That guide evaluates task suitability; this scorecard decides whether those tasks form one standing role.

5. Is the Work Actually Agentic?

An AI employee should not replace a simpler deterministic system.

OpenAI’s practical guide to building agents recommends agents where contextual decisions, difficult-to-maintain rules, and unstructured data make deterministic approaches inadequate. It also recommends starting incrementally rather than maximizing autonomy at the beginning.

Anthropic’s building-effective-agents guide distinguishes workflows with predefined paths from agents that direct their own process and tool use. Apply that distinction inside the role: exact calculations remain code; policy thresholds remain deterministic; permissions remain enforced controls; the agent handles bounded interpretation and next-step choice; and a person handles decisions whose consequence or authority remains human.

Score 0-1 when an API query already returns the answer; the same steps always apply; a rules engine can route the case; no persistent responsibility exists; the outcome depends entirely on human professional judgment; or the role is an excuse to avoid fixing a broken source process.

Score 2 when part of the path benefits from agentic interpretation; deterministic and human-owned segments are identified; the exact split still needs testing; and a narrow draft or prepare mode remains valuable.

Score 3 when the outcome recurs; the path changes with context; the worker must choose among approved tools or sources; the decision can be bounded and reviewed; progress must persist across cycles; and the role creates value without owning prohibited judgment.

§ 05Score Criteria 6-8: Context, Tools, and Authority

These criteria test whether the role can operate inside a real environment.

6. Context readiness

Score 3 when every required source has a purpose; an authority level; an owner; a location; an access scope; a freshness rule; conflict behavior; retention treatment; and a correction path.

Score 1 when “the shared drive” or “our CRM” is the context plan.

The AI employee memory guide explains why stored context needs provenance, freshness, correction, retention, and deletion. The scorecard does not require a complete context pack, but it does require evidence that one can be built from owned sources.

Ask the current process owner: Which documents do people actually use? Which source wins when systems disagree? Which knowledge remains tacit? Which records are stale? Which information may not enter model context? Who corrects an error?

Score 2 if the sources exist but ownership or conflict policy remains incomplete.

7. Tool feasibility

List the minimum operations required to produce the accepted unit:

Score 3 when the minimum path can be tested with a distinct identity and scoped objects.

Score 1 when a person must share broad credentials; the system exposes no appropriate object or action boundary; expected state cannot be verified; failures cannot be detected; required interfaces are too brittle for the role; or the integration effort exceeds the value.

Computer use can extend reach into interfaces without APIs, but it does not remove the need for access, action, evidence, and recovery controls.

8. Authority and recovery

Score each action separately:

Score 3 when the job description identifies the action and target; data scope; conditions; a volume, spend, time, or frequency limit; approval; audit evidence; the postcondition; recovery; and revocation.

Score 1 when the role needs “full access” or when a prompt is the only restriction.

High-impact, irreversible, sensitive, public, financial, security, legal, employment, healthcare, or rights-affecting work should remain human-led unless qualified owners approve a defensible, enforceable design. A high score in other dimensions does not override that boundary.

§ 06Score Criteria 9-10: Evaluation and Ownership

9. Evaluation readiness

Score 3 when the team can build a representative evaluation set that includes normal cases; subtypes and segments; boundary cases; missing inputs; stale and conflicting sources; out-of-scope requests; unsafe or adversarial instructions; tool failure; approval expiry; required escalation; required refusal; and a plausible high-severity failure.

The rubric should judge the outcome; evidence; accuracy; completeness; policy; authority; escalation; durable state; format; and downstream usability.

An evaluation also needs a grader. Name the person or deterministic check that can judge each criterion.

Score 2 when cases exist but cover only the happy path, or when the rubric has not separated cosmetic correction from material failure.

Score 1 when “we will know it when we see it” is the review plan.

Test the trajectory

An agent can arrive at acceptable prose after reading a prohibited source; exposing sensitive content; taking an unauthorized action; retrying excessively; creating duplicate work; or leaving the system in the wrong state.

Score tool choices, permissions, intermediate state, escalation, and postconditions as well as the final artifact.

10. Human ownership

Score 3 when the role names an owner for every decision:

Add response expectations. An escalation queue with no service expectation can leave urgent work unresolved or encourage the worker to infer permission from silence.

Score 2 when names exist but backups, decision authority, or response behavior is unclear.

Score 1 when ownership is a department name.

Microsoft’s current responsible-agent guidance recommends deciding grounding sources, allowed actions, and human-approval points early and using risk-scaled release gates. The scorecard turns those decisions into pre-hire evidence.

§ 07Score Criteria 11-12: Economics and Operating Load

11. Value and economics

Estimate value against the complete operating cost.

Start with a baseline:

Then estimate: expected role value = accepted outcomes multiplied by value per accepted outcome. Expected role cost = platform + usage + data + integration + reviewer + correction + supervision + incident allowance.

Score 3 when the baseline uses current records and the value unit does not depend on unsupported attribution.

Score 2 when the role clearly relieves a constraint but the baseline or price model remains incomplete.

Score 1 when the business case is “AI is cheaper than a salary.” An AI employee is software capacity, not a like-for-like legal or organizational substitute for a person.

Use the AI employee total-cost framework to build the full ledger. The pre-hire score should expose assumptions rather than pretend the estimate is a realized ROI.

OpenAI’s use-case prioritization guidance uses impact and effort to separate quick wins from high-effort, low-impact ideas. The scorecard goes one level deeper by requiring an accepted unit, controls, and ongoing operating cost.

12. Operating load

The role creates human work: preparing and maintaining sources; reviewing outputs; answering escalations; approving actions; correcting errors; monitoring cost and KPIs; managing access; investigating incidents; updating evaluations; and changing the role when the business changes.

Score 3 when the organization has estimated review minutes per accepted outcome; escalation rate and response time; correction burden; access and source maintenance; peak queue; backup coverage; incident workload; and change cadence.

Score 1 when the proposal assumes “the AI handles everything.”

Include coordination tax

If the role hands work to people or other agents, measure assignments rejected; clarification loops; duplicate tasks; waiting time; handover correction; unresolved ownership; and final integration work.

A role can complete its own output while increasing total system work. Score the net operating effect.

§ 08Calculate the Decision Without Hiding the Weaknesses

After the gate review:

1. Score all 12 criteria from 0-3.
2. Add an evidence code to each.
3. Record disagreements and missing artifacts.
4. Add the points.
5. Apply the planning band.
6. Inspect the lowest criteria and worst plausible failure.
7. Make one decision: Test, Redesign, Choose another system shape, or Stop.

Planning bands

These bands are an internal planning convention. They are not research-validated thresholds, safety certifications, or guarantees.

Add decision rules

Use rules that the total cannot override:

• Any failed gate: Stop or Redesign.
• Any unknown in authority, ownership, evaluation, or required data: do not configure live action.
• Any score of 1: name the repair before pilot approval.
• High score with low-confidence evidence: gather evidence before relying on it.
• Severe failure outside tolerance: narrow or stop even if averages pass.
• Net operating load greater than plausible value: choose a different design.

Preserve the shape of the score

Two roles can both score 28: one may have strong outcome and evaluation but weak integration; another may have easy tools but no coherent scope.

Do not compare totals without the criterion profile. The weak dimensions determine the repair.

§ 09Three Worked Examples

Worked Example 1: Weekly Competitor-Change Brief

The proposed role monitors an approved 12-company registry and produces one cited change brief each Monday for the strategy lead.

All six gates pass. The decision is Test, not deploy autonomously.

The pilot contract should require source-owner completion; adverse cases; reviewer minutes; the factual-correction rate; escalation quality; cost per accepted brief; and zero publication or external contact.

Worked Example 2: “AI Marketing Manager”

The proposal asks one worker to research markets, write strategy, produce content, publish, update CRM, run outbound email, and report revenue.

The total is 11/36, and it fails the scope, consequence, evaluation, and ownership gates. The decision is Redesign.

Split the proposal into candidate outcomes: a weekly market-evidence brief; an approved content-outline queue; a CRM enrichment draft; campaign-performance reconciliation; and allowlisted email-draft preparation.

Score each separately. Some may fit workflow automation; some may fit an on-demand assistant; one or two may justify a standing role.

Worked Example 3: Support Triage and Refund Resolution

The initial request asks the role to classify tickets, answer customers, change accounts, and issue refunds.

The team redesigns it: classify three low-risk categories; retrieve the current policy; prepare a response draft; request approval for every send; route refunds, security issues, account changes, and exceptions to people; and update only the internal task state.

The narrower role may be useful without autonomous refund or customer-send authority. The scorecard makes that hybrid design visible.

§ 10Copy-and-Use AI Employee Role Scorecard

A. Role identity. Proposed role; version/date; business owner; supervisor; downstream user/reviewer; current process owner; proposed system shape (workflow/assistant/AI employee/hybrid/human); one-sentence outcome (outcome, queue, cadence, sources, escalation).

B. Gate review. For each of the six gates: pass/fail; evidence; owner; required repair. Then the gate decision: Pass, Redesign, or Stop.

C. Twelve-criterion score. For each criterion: score 0-3; evidence code O/D/A/U; the evidence or gap; owner; due date. Then the total out of 36.

D. Decision. The decision (Test/Redesign/Workflow/Assistant/Human/Stop); failed or conditional gates; the three lowest criteria; the worst plausible failure; human-only decisions; the required pilot mode (observe/shadow/draft/approved action); the evidence the pilot must produce; maximum initial authority; the pause condition; the re-score date or trigger; and approvers.

§ 11Use the Scorecard With CellCog

CellCog’s AI Employees product page describes persistent roles with goals, KPIs, an inbox, task board, memory, schedules, wake conditions, permissions, approvals, shifts, and handovers around a general-purpose agent.

The scorecard should decide which of those capabilities the role needs:

Do not raise a score because the platform advertises a feature. Ask for proof against the exact role: Can the source scope be limited? Can actions be separated into read, prepare, approved, and blocked states? Does approval bind to the exact action? Can the worker preserve evidence and task state? Can a supervisor pause work and revoke access? Can output, correction, escalation, and cost be measured?

Use CellCog’s role pages as examples, not pre-approved job descriptions. A title such as AI Research Assistant or AI Operations Manager becomes a real candidate only after the organization defines its outcome, queue, inputs, authority, and acceptance.

After a role passes the gates and reaches a test-ready score, use the AI employee pilot framework to precommit the sample, modes, metrics, budget, failure rules, and go/redesign/stop decision. Then use graduated onboarding to increase context and authority only when the evidence gate passes.

§ 12Avoid Common Role-Scorecard Mistakes

Scoring the title. “AI executive assistant” may describe ten different outcomes. Score the actual queue, deliverable, sources, tools, and authority.

Treating unknown as average. An unanswered permission or data question is a 0, not a 2. Unknowns become surprises after connection.

Letting a high total cancel a failed gate. Value and recurrence cannot compensate for prohibited data, missing ownership, or uncontrolled consequence.

Awarding points for planned artifacts. “We will write a rubric” is a gap. Create it, review it, and re-score.

Starting from vendor capability. “The platform can browse and email” does not prove that your role has approved sources or sending authority.

Using precise bands as scientific truth. The thresholds organize an internal decision. Calibrate them with portfolio evidence over time, and keep non-compensating gates.

Ignoring the current alternative. Compare the role with workflow automation, an on-demand assistant, process repair, another specialist product, or human-led work. “Do nothing” is not the only alternative.

Hiding reviewer disagreement. Disagreement about consequence, source authority, or correctness is evidence. Resolve it or narrow the scope.

Equating a high score with production readiness. The scorecard approves the next learning step. Evaluation, pilot, security review, onboarding, monitoring, and change control still remain.

§ 13Final Recommendation

Use the scorecard to force an explicit pre-hire decision:

1. Pass the six gates.
2. Score the current role across 12 criteria.
3. Attach evidence and owners.
4. Identify the lowest dimensions.
5. Compare alternative system shapes.
6. Define the maximum safe pilot mode.
7. Precommit the evidence needed to proceed.

The best first role is not the one with the grandest title. It is the one whose outcome matters, boundaries hold, evidence can be tested, human ownership is real, and total operating work can plausibly fall.