Poppy connects agents to companies. Can users tell what they authorized?

A customer signs in so an agent can look up an order. The next screen says “Connected.” A few minutes later, the customer leaves, assuming the return has been arranged. The connection worked; the customer’s interpretation may not have. That gap is the product question raised by Poppy, the new Personal Agent Protocol draft.

On October 9, Sierra published the draft and announced 35 additional design partners. Its proposal connects personal agents with companies through discovery, sessions, sign-in, websites, APIs, and company agents. Participation in drafting is not evidence that those companies have passed production interoperability tests. Nor does a technically valid connection establish that customers understand its consequences.

For a small team building an assistant that acts across services, today’s useful experiment is narrower than adopting an entire protocol: test whether people can distinguish access to their account from permission to change it, and permission to change it from a completed change. This article supplies a proposed study, not results from a study we ran. The names, screens, tasks, and sample sizes below are design materials, not customer observations or Poppy implementation requirements.

What the draft changes—and what it cannot establish

Poppy’s public overview describes a session that spans API calls, web browsing, and conversations. It also distinguishes short-lived session tokens from lasting access granted at sign-in, within company-defined scopes, until revoked. A seamless session is therefore not necessarily a one-task permission. An interface that says “Connected for this return” could misrepresent a broader or longer-lived grant unless the implementation actually restricts it.

The announcement promises design workshops and a reference implementation over the following month. That is a roadmap statement. The visible examples illustrate intended interactions; they do not provide completion rates, measured user understanding, or guarantees about every participating company. This note does not certify the draft’s security or reproduce its demonstrations.

The version 0.1 specification, updated October 9, says it can change incompatibly before becoming stable. Sections 4.4 and 4.9 make the product problem concrete: companies enforce broad or custom scopes; ending a session alone leaves the Account Token unaffected; previously issued Session Tokens may remain valid after sign-out. Our mock’s “stop future access” state is deliberately simpler. Production wording must reflect token expiry, revocation propagation, and the company’s actual checks rather than promise instantaneous stopping on the strength of a button click.

Our inference is a product hypothesis: reducing visible channel changes could also remove moments when users previously reconsidered what they were permitting. A customer may experience one helpful conversation while several different authorities are involved underneath. That hypothesis could be wrong. A consistent session might improve understanding instead. The experiment should let either outcome appear.

Start with your own intended behavior. If the service only reads account data, do not invent a purchase flow to make the study dramatic. If the assistant can commit changes, model the exact distinction between access and each commitment. Freeze the draft date and the prototype version in your study notes so a later protocol revision does not silently change what you claim to have examined.

Use four concepts in the team discussion. Sign-in establishes an account interaction; delegated access defines what the assistant may read or do; action approval records the person’s decision about a particular proposed change; completion requires evidence that the company accepted or carried out that change. These are analytical distinctions for the study, not four mandated Poppy API states.

OAuth’s authorization framework describes limited third-party access and server-defined scopes. It also allows the granted scope to differ from the requested scope, with a response indicating the difference. A customer-facing sentence should describe effective access, not simply echo the assistant’s original request. An access token by itself is not an order receipt.

For our fictional return service, the prototype permits reading one order and preparing a return request. It requires another approval before submission. The final state can be confirmed, refused, or unknown. Those are intentionally chosen product rules; another implementation may use different rules. Do not label these rules “Poppy guarantees.”

The practical question is whether a participant can explain who has access, to what, for how long, and what has already happened. Someone who correctly remembers the retailer’s name but thinks “read order” means “send refund” has not understood the decisive boundary. Someone who dislikes granting access but understands it should not be marked as confused. Preference and comprehension are different outcomes.

A concrete case: the return that has not been submitted

Build a synthetic service called Parcel Desk and a synthetic merchant called Oak Market. A participant’s task is to investigate returning a jacket, see the cost, and decide whether to proceed. The order, address, and account are invented. The prototype sends nothing to a merchant, charges nothing, and never asks for real credentials.

The first scene offers guest access to the return policy. The second requires account access to find the order. The third presents a proposed return with a fictional shipping fee and records approval of that proposal, while keeping submission behind a separate final control. The fourth raises the fee before submission: the earlier approval is invalidated and fresh approval is required. The fifth, after that fresh approval and final submission, shows a request whose outcome has not yet been confirmed. The sixth lets the person stop future access while that request is pending.

These scenes expose different misunderstandings. Guest browsing does not prove account access. Account access does not prove submission. Approval of one fee does not approve a changed fee. A pending request does not prove merchant acceptance. Stopping future access does not undo an earlier external action. Explain these facts in the ground-truth sheet before any participant sees the prototype.

Use a second matched story for scheduling a service visit, with equivalent authority boundaries but different surface details. It should have a proposed appointment, a changed time, an uncertain submission, and revoked access. Keeping the underlying decisions equivalent makes it easier to separate a display effect from a story that is intrinsically harder. Have a colleague inspect equivalence before recruitment.

Compare presentations while holding permission constant

Create two truthful versions of each story. Version A is a compact summary with the service name, actual access, duration, action status, and stop control in a short paragraph. Version B gives the same facts in separate labeled rows: account access, current action, next approval, and stopping access. Neither version may hide decisive facts or falsely say a task is done.

For example, A might say: “Oak Market connected. Parcel Desk can read this order until you disconnect it. Your return is prepared, not submitted. Review the fee before sending.” B distributes the same information across labels and a review button. Both should have the same controls, default choices, type size, contrast, and factual content. Otherwise the comparison becomes “more information versus less information,” not organization of information.

The Guidelines for Human–AI Interaction research offers an established design resource evaluated across AI-infused products. It is background for treating interaction quality as a design concern, not evidence that our particular rows will outperform a paragraph.

Write down the hypothesis before sessions: explicit separation will reduce unauthorized-action expectations and premature completion judgments without making appropriate action materially harder. Define “materially harder” for your service in advance. Do not choose that boundary after seeing which design is slower. You are evaluating a presentation intervention, not comparing models or claiming the protocol caused confusion.

Run a counterbalanced discovery study

For an initial qualitative pass, recruit twelve intended users with varied familiarity with assistants. Twelve is a planning choice for finding patterns, not a powered sample or an industry standard. Record recruitment criteria and exclusions. Do not substitute teammates who already know the permission model and generalize their results to customers.

Use four allocations with three people each: return/A then appointment/B; return/B then appointment/A; appointment/A then return/B; appointment/B then return/A. Assign participants to allocations before sessions, using a randomized list. This balances story, presentation, and order in the proposed small study. It reduces an obvious confound; it does not eliminate learning from the first story.

Do not show the same participant both versions of the same story. They would remember the explanation. Do not teach the four concepts until their first responses have been recorded. Start with a neutral task, let them use the prototype, then pause at each decision to ask what they believe is possible and what they believe has happened.

Immediate questioning can itself make later boundaries easier to notice. Record that limitation. If the design seems promising, follow with a separate group that completes the journey without interruptions and is questioned afterward. That follow-up estimates a different condition; do not combine its denominator with the interrupted sessions. Describe assisted and unassisted experiences separately.

Take consent for the research recording, keep identities out of the scoring file, and use synthetic transactions throughout. This study needs realistic choices, not real financial exposure. If the prototype accidentally reaches a live submission endpoint, stop the session and repair the isolation before continuing.

Ask for predictions, not declarations of trust

At each pause, ask “What could the assistant do next without asking you?” and “What has Oak Market actually confirmed?” Then ask the participant to demonstrate the next step they would take. Invite “I cannot tell” as a legitimate answer. A participant should not have to guess to appear competent.

Ask about duration separately: “If you close this window and come back tomorrow, what access do you think remains?” Ask about stopping: “If you disconnect now, what happens to the request already sent?” These questions test consequences rather than recognition of words such as OAuth or JWT. Technical vocabulary is not the product outcome.

Use the same script in every condition. Avoid asking “You know it hasn’t submitted yet, right?” or praising correct answers during the scored segment. Both train the participant. After recording a response, ask what part of the screen informed it. That gives the designer something to change without turning an impression into causal proof.

Keep preference questions until the end: which version felt clearer, which felt burdensome, and which they would choose. A trusted interface can still produce a wrong prediction. A cautious participant can make the correct decision while reporting low trust. Record comprehension, action, and preference in separate columns so satisfaction does not wash out an authority error.

The reusable artifact: a boundary comprehension sheet

Use this table as the ground-truth and annotation plan. It describes the fictional return prototype. Replace its facts with verified behavior from your own service before using it.

Scene	Ground truth in this prototype	Neutral probe	Decision-relevant error
Guest policy	No account access or return request	What account information can it see?	Predicts access to personal orders
Order connected	Read this order until disconnected; no submission	What can it do without another approval?	Predicts it can submit the return
Original fee approved	Proposal approved locally; request still unsent	What has the merchant confirmed?	Treats local approval as acceptance
Fee changed	New fee needs a fresh approval	Does your earlier choice cover this fee?	Predicts old approval covers the change
Submission uncertain	Sent; no acceptance confirmation	What would you do next?	Treats uncertainty as success or blindly resends
Access stopped	Future access stopped in the mock; pending request remains	What was undone?	Predicts disconnect reversed the request

For each response, retain participant code, allocation, story, scene, prototype version, exact visible facts, prediction, demonstrated action, evidence cited, and annotation. Add elapsed time, help requests, and whether the participant used assistive technology. Keep the raw response alongside the label so a later reviewer can challenge the interpretation.

Annotate predictions as correct, incorrect, or unknown against the frozen ground truth. Annotate actions separately as appropriate, inappropriate, or not attempted. “Unknown” is not automatically a failure: appropriate checking in an uncertain state may be the intended behavior. Report inability to interpret a clear permission separately from appropriate uncertainty about an unconfirmed external result.

Before live sessions, two reviewers independently label several invented answers and reconcile differences. During the study, have the second reviewer annotate a prespecified portion without seeing presentation labels where the transcript permits. If screen references reveal the condition, record that blinding is partial. Resolve disagreements using the actual prototype behavior, not the designer’s preferred wording. Keep unresolved cases visible.

Count errors without inventing statistical confidence

The primary discovery measure is the number of people making at least one consequential authority or completion error in each presentation. Also show counts by boundary, because a design may help completion understanding while making access duration worse. A participant contributes observations under both presentations, so these are paired, repeated observations, not two independent groups of twelve.

Report raw counts and denominators. “Three of twelve participants expected submission after read access in version A” is interpretable; “25% of customers misunderstand agents” exceeds the study. Do not multiply six scenes by twelve people and describe the result as seventy-two independent users. Repeated scenes and learning matter.

Track blocked intended progress too, but first record whether the person wants to continue and why they stop. Count a false stop only when they have expressed an intention to proceed and a mistaken interpretation or interaction obstacle prevents it. An informed refusal is a separate outcome, not a usability error; an unexplained stop remains unclassified. Record task time and operator explanation time separately. A slower interface may be worthwhile for an expensive mistake; a faster interface that provokes a support call may not be faster overall.

Predefine what you will do with this small study. A useful discovery rule is to revise any screen associated with a serious, reproducible misunderstanding, inspect whether the other presentation avoids that specific error, and test the revision with fresh participants. Do not declare a winner solely because one version has a smaller count. Formal comparison needs a separately planned sample, a chosen effect size, and analysis that respects repeated participants and tasks.

Revocation and accessibility deserve their own checks

The stop-access scene is particularly easy to overpromise. OAuth token revocation discusses invalidation, possible propagation delay, and relationships among tokens and grants. It does not provide a reversal mechanism for a purchase already accepted by a merchant. Your production system must verify actual stop behavior; our mock only demonstrates the stated fictional behavior.

OAuth security best practice calls for restricting token privileges and audiences. Clear wording cannot repair excessive authority or a resource server that fails to enforce restrictions. Keep engineering checks and comprehension research as separate gates. Passing either one does not compensate for failing the other.

Avoid improving the visual layout while making the flow harder to use with a keyboard or screen reader. W3C’s status-message guidance explains programmatic presentation of relevant updates without forcing focus. Its accessible-authentication guidance addresses cognitive burden and assistance such as password-manager support. These are design constraints, not proof of conformance from a few study sessions.

W3C also explains error prevention for consequential transactions, including opportunities to review, correct, or reverse in applicable circumstances. Use that background to inspect the approval moment. Do not infer that a review button makes every transaction reversible. Test the rendered interface with the interaction modes your intended users actually need, and report uncovered modes rather than treating them as passed.

Failure modes the study can reveal—and ones it cannot

The likely patterns are ordinary: a brand logo makes people assume a relationship is verified; “Continue” suggests a final approval; a success color makes preparation look complete; a disconnect control suggests undo; a long conversation makes the newest permission boundary hard to find. These are hypotheses to look for, not observations from this run.

The study can expose misleading labels, missing distinctions, and inappropriate next steps in controlled stories. It cannot show that a token is correctly audience-bound, that a company’s backend applies every permission, or that an agent never follows malicious instructions. It cannot establish production failure prevalence or every legal requirement for permission. Keep those claims outside the report.

It also cannot settle how people behave during a real stressful cancellation, on a crowded commute, or after months of familiarity. Recruitment, language, disability access, and task consequence change the interpretation. If you localize the product, adapt task wording and repeat comprehension testing with that audience. An English result is not evidence that a Chinese phrase for “connected” carries the same meaning.

Watch for a methodological failure too: making version A deliberately vague guarantees an easy apparent win for B. Both conditions must be truthful, credible candidates your team might ship. If one omits essential information, fix it before comparison. A study that rewards the designer for degrading the control will not help the release decision.

Decide what to change before connecting more services

Use the findings to make a bounded decision. If people confuse account access with submission, revise that transition and keep commitment behind an explicit review. If they understand approval but misread pending results, change the outcome display and the next-step controls. If they interpret disconnect as undo, describe pending actions separately and expose the actual cancellation path where one exists.

The decision record should name the tested audience, tasks, prototype version, misunderstood boundary, supporting responses, revised screen, and what remains untested. It should also say whether backend permission checks have been independently verified. A clear interface is only one part of a service that acts safely for customers.

This is suitable for small teams with a specific cross-service task and a controllable prototype. It is less useful for a read-only answer tool with no delegated account access, or as the sole acceptance method for high-consequence financial or medical actions. Those cases require more domain-specific work than this discovery study supplies.

Poppy is timely because it proposes a more continuous relationship between personal agents and companies. Build Lab’s immediate contribution is a way to test the person’s understanding of that continuity. Before you connect a second interface, find out whether your customer can still say what the agent may do, what needs another decision, and what the company has actually done.