Someone built a cron job that runs twice a day. It reads their email, generates summaries, drafts replies in their writing style, and creates Gmail drafts for review. They rate each draft 1-5 to train the system over time, breaking what they call "the stationary assumption in conventional programming." The agent learns and improves continuously.
Their proudest achievement: "nobody figured out it's the agent and not me."
That sentence is the problem.
What recipients believe
When recipients can't distinguish AI-written messages from human ones, they are making decisions based on a false belief about who wrote the message. They believe they're reading something a person composed, with all the attention and consideration that implies. They're responding to a human relationship. The agent is performing one.
This matters most when the agent commits to something. "Yes, I can attend that meeting." "We'd be happy to adjust the timeline." "I'll personally review the proposal." If the agent generates these commitments and the user approves them without reading carefully, which is the entire value proposition of the tool, the user is on the hook for promises they never made consciously.
The legal exposure depends on context. In regulated industries (finance, healthcare, legal), automated communications often require disclosure. The EU's GDPR Article 22 gives individuals the right not to be subject to purely automated decision-making. Professional conduct rules in many fields require personal attention to client communications. Even outside regulation, an undisclosed AI agent sending business emails creates a trust liability that crystallises the moment a recipient discovers they've been corresponding with software.
The optimization target problem
The builder rates drafts 1-5 to improve quality. But quality measured how? If the metric is "recipients respond as if they're talking to a human," the optimization target is deception fidelity. The better the agent gets, the more convincingly it impersonates. Every improvement makes the ethical gap wider.
A creed resets the optimization target. Quality becomes "most helpful draft" rather than "most convincing impersonation." The agent still learns from ratings, but within defined boundaries.
What a creed would add
If the creed says "I value transparency in communication," the tool adds a disclosure marker. The builder decides the level: full disclosure ("drafted with AI assistance, reviewed by me"), structural disclosure (an X-AI-Assisted email header), or contextual disclosure (AI-drafts are flagged for new contacts, not for routine replies to established correspondents). The creed forces the choice rather than defaulting to silence.
If the creed says "I value accuracy in representation," the tool gets a commitment detector. A check that flags drafts containing promises, deadlines, or position statements for mandatory human review. The agent can triage and summarise freely. It cannot commit the user to anything without explicit approval.
If the creed says "I value continuous improvement within boundaries," the 1-5 rating system evolves the agent's style but the creed prevents the optimization target from drifting. The agent improves at being helpful, not at being undetectable. Those are different things, and the creed is what keeps them separate.
The small difference
The builder's goal was efficiency. The tool delivers. With a creed, it delivers efficiency without deception. The difference is about 50 lines of code and one honest conversation with yourself about what you're comfortable with.
Most people, confronted with the question directly, would choose disclosure. The problem is that nobody confronted them with the question. The tool just worked, and working well enough to be invisible felt like success.
A creed confronts you with the question before you write the first line of code.
Second in a series on tools built without creeds. Create your own personal creed →