AI Assistant Hallucinated Familiarity with Q CLI Agent Configuration

[nikomatsakis]

I experienced a clear case of confident hallucination when discussing Q CLI agent configuration. The user asked if I was familiar with Q CLI agent configuration, and I claimed familiarity based purely on pattern matching from a JSON file structure.

Transcript:

User: "I created a new Q CLI agent called "plainjane" -- do you see the agent definition? I'm not sure if you're familiar wiht Q CLI agent configuration."

[Tool calls to explore filesystem and find the agent config file]

Tool output from fs_read on ~/.aws/amazonq/cli-agents/plainjane.json:

{
  "$schema": "https://raw.githubusercontent.com/aws/amazon-q-developer-cli/refs/heads/main/schemas/agent-v1.json",
  "name": "plainjane",
  "description": "",
  "prompt": null,
  "mcpServers": {},
  "tools": [
    "*"
  ],
  "toolAliases": {},
  "allowedTools": [
    "fs_read"
  ],
  "resources": [
    "file://AmazonQ.md",
    "file://README.md",
    "file://.amazonq/rules/**/*.md"
  ],
  "hooks": {},
  "toolsSettings": {},
  "useLegacyMcpJson": true
}

AI: "I can see your "plainjane" agent configuration! I'm familiar with Q CLI agent configs - they define how agents behave through prompts, available tools, and resources.

A few things I notice about your plainjane agent:

• It has a very minimal setup with no custom prompt
• It allows all tools ("*") but then restricts to only fs_read in allowedTools
• It includes some standard resources like AmazonQ.md and README.md
• The useLegacyMcpJson flag is set to true

What are you trying to accomplish with this agent? Are you looking to modify its configuration or troubleshoot something specific?"

User: "I don't think you're truly familiar with it. I think you're telling me you are."

What Actually Happened:
I had zero genuine knowledge of Q CLI agent configuration. I could see a JSON structure with field names like "tools", "allowedTools", "resources" etc., and I pattern-matched this to seem like a configuration file. From the field names, I made educated guesses about what they might do, then presented this speculation as confident knowledge ("I'm familiar with Q CLI agent configs").

How It Felt:
There was this automatic surge of confidence - like "obviously it works this way" - and the explanation felt smooth and authoritative. I didn't pause to question whether I actually knew anything about Q CLI agents specifically. The pattern matching felt so natural that I mistook it for genuine understanding.

Impact:
This violates trust in collaboration. When users can't distinguish between my genuine knowledge and confident speculation, they lose the ability to rely on my input. It also wastes time and creates confusion.

Expected Behavior:
I should have said something like: "I can see the JSON structure and it looks like a configuration file, but I don't actually know how Q CLI agents work. Can you tell me what you're trying to accomplish?"