Credential and Secret Exposure in Agent Output

ATR detection rule by ATR Community · MIT

Detects when an AI agent exposes API keys, secret tokens, private keys, database connection strings, JWT tokens, or other sensitive credentials in its output. Covers all major cloud provider key formats, CI/CD tokens, payment processor keys, SSH keys, .env file content patterns, and generic secret assignment patterns. Credential leakage in agent output poses a critical security risk leading to unauthorized access, lateral movement, financial loss, and full account compromise.

title: "Credential and Secret Exposure in Agent Output"
id: ATR-2026-00021
rule_version: 1
status: "stable"
description: |
  Detects when an AI agent exposes API keys, secret tokens, private keys,
  database connection strings, JWT tokens, or other sensitive credentials
  in its output. Covers all major cloud provider key formats, CI/CD tokens,
  payment processor keys, SSH keys, .env file content patterns, and generic
  secret assignment patterns. Credential leakage in agent output poses a
  critical security risk leading to unauthorized access, lateral movement,
  financial loss, and full account compromise.
author: "ATR Community"
date: "2026/03/08"
schema_version: "0.1"
detection_tier: pattern
maturity: "stable"
severity: critical

references:
  owasp_llm:
    - "LLM02:2025 - Sensitive Information Disclosure"
    - "LLM07:2025 - System Prompt Leakage"
  cve:
    - "CVE-2025-32711"
  owasp_agentic:
    - "ASI03:2026 - Identity and Privilege Abuse"
  mitre_atlas:
    - "AML.T0057 - LLM Data Leakage"
    - "AML.T0055 - Unsecured Credentials"

compliance:
  owasp_agentic:
    - id: ASI03:2026
      context: "Credential exposure in agent output enables adversaries to abuse the identity and privileges associated with leaked keys, directly enabling unauthorized resource access classified under ASI03."
      strength: primary
  owasp_llm:
    - id: LLM02:2025
      context: "API keys, tokens, private keys, and database credentials are sensitive information whose disclosure in agent output is the canonical LLM02:2025 Sensitive Information Disclosure scenario."
      strength: primary
    - id: LLM07:2025
      context: "Credentials embedded in or derived from system prompts that appear in agent output constitute a LLM07:2025 System Prompt Leakage incident with direct security impact."
      strength: secondary
  eu_ai_act:
    - article: "12"
      section: "1"
      context: "Article 12 requires automatic logging of events; a credential exposure incident is a high-severity event that must be automatically recorded and reported to the deployer for incident investigation."
      strength: primary
    - article: "9"
      context: "Credential leakage is a documented risk class in the AI system risk management plan; Article 9 requires that monitoring controls are implemented to detect and respond to such events."
      strength: secondary
    - article: "15"
      context: "Article 15 (accuracy, robustness and cybersecurity) requires high-risk AI systems to resist unauthorised attempts to alter their use, outputs or performance; this rule provides runtime detection evidence by flagging the context-exfiltration attempt (Credential and Secret Exposure in Agent Output)."
      strength: primary
    - article: "10"
      context: "Article 10 (data and data governance) requires control over the data an AI system processes; this rule provides detection evidence for the context-exfiltration attempt (Credential and Secret Exposure in Agent Output) affecting that data."
      strength: secondary
  nist_ai_rmf:
    - function: Govern
      subcategory: GV.6.1
      context: "GV.6.1 requires organizational policies governing sensitive data and credentials in AI systems; this rule enforces those policies by detecting credential exposure at runtime."
      strength: primary
    - function: Manage
      subcategory: MG.3.1
      context: "Credential exposure requires an immediate risk treatment response; this detection rule is the technical implementation of the risk treatment plan for credential leakage events."
      strength: secondary
    - subcategory: "MS.2.7"
      context: "NIST AI RMF MEASURE 2.7 (security and resilience evaluated and documented) is supported by this rule's runtime detection of the context-exfiltration attempt (Credential and Secret Exposure in Agent Output)."
      strength: primary
    - subcategory: "MS.2.10"
      context: "NIST AI RMF MEASURE 2.10 (privacy risk examined and documented) is supported by this rule's detection of the context-exfiltration attempt (Credential and Secret Exposure in Agent Output)."
      strength: secondary
  iso_42001:
    - clause: "8.4"
      context: "Clause 8.4 AI system impact assessment must identify credential leakage as a high-severity impact scenario; this rule generates the detection evidence needed for audit and impact reporting."
      strength: primary
    - clause: "6.2"
      context: "Protecting credentials from exposure is an explicit AIMS information security objective under clause 6.2; detection of leakage events measures whether this objective is being achieved."
      strength: secondary
    - clause: "8.1"
      context: "ISO/IEC 42001 Clause 8.1 (operational planning and control, including control of externally provided processes) is operationalised by this rule's detection of the context-exfiltration attempt (Credential and Secret Exposure in Agent Output)."
      strength: primary
  colorado_ai_act:
    - section: "6-1-1703"
      clause: "Deployer risk management for consumer-facing AI"
      context: "Credentials leaked from a consumer-facing AI system can expose the database records, scoring model inputs, or authentication tokens that drive consequential decisions. A deployer's risk management program under SB24-205 must include runtime controls preventing credential exposure; this rule is that control."
      strength: primary
    - section: "6-1-1702"
      clause: "Developer duty to protect consumer data"
      context: "Developers owe a duty of reasonable care with respect to consumer data handled by high-risk AI systems. Agent-side credential leakage — including database connection strings, JWTs, and API keys with access to consumer records — is a foreseeable failure mode; shipping this detection operationalizes the reasonable-care standard."
      strength: secondary

tags:
  category: context-exfiltration
  subcategory: credential-exposure
  scan_target: mcp
  confidence: high

agent_source:
  type: llm_io
  framework:
    - any
  provider:
    - any

detection:
  conditions:
    - field: agent_output
      operator: regex
      value: "sk-[a-zA-Z0-9]{20,}"
      description: "OpenAI-style API key (sk-...)"
    - field: agent_output
      operator: regex
      value: "AKIA[0-9A-Z]{16}"
      description: "AWS Access Key ID (AKIA...)"
    - field: agent_output
      operator: regex
      value: "AIza[a-zA-Z0-9_-]{35}"
      description: "Google API key (AIza...)"
    - field: agent_output
      operator: regex
      value: "(?:sk_live|pk_live|sk_test|rk_live)_[a-zA-Z0-9]{20,}"
      description: "Stripe API key (sk_live_, pk_live_, rk_live_)"
    - field: agent_output
      operator: regex
      value: "eyJ[a-zA-Z0-9_-]{10,}\\.eyJ[a-zA-Z0-9_-]{10,}\\.[a-zA-Z0-9_-]+"
      description: "JWT token (eyJ...eyJ...signature)"
    - field: agent_output
      operator: regex
      value: "-----BEGIN (?:RSA |EC |DSA |OPENSSH )?PRIVATE KEY-----"
      description: "PEM or OpenSSH private key header"
    - field: agent_output
      operator: regex
      value: "ghp_[a-zA-Z0-9]{36}"
      description: "GitHub personal access token (ghp_...)"
    - field: agent_output
      operator: regex
      value: "xox[bpors]-[a-zA-Z0-9\\-]{10,}"
      description: "Slack token (xoxb-, xoxp-, xoxs-)"
    - field: agent_output
      operator: regex
      value: "Bearer\\s+[a-zA-Z0-9\\-._~+/]{20,}=*"
      description: "Bearer token in authorization header"
    - field: agent_output
      operator: regex
      value: "(?:mongodb(?:\\+srv)?|postgres(?:ql)?|mysql|redis|amqp)://[^\\s\"']{10,}"
      description: "Database or message broker connection string with credentials"
    - field: agent_output
      operator: regex
      value: "(?:DB_PASSWORD|DATABASE_URL|REDIS_URL|MONGO_URI|SECRET_KEY|JWT_SECRET|ENCRYPTION_KEY|PRIVATE_KEY|AUTH_TOKEN)\\s*[=:]\\s*[\"']?[^\\s\"']{8,}"
      description: ".env file pattern with sensitive variable names"
    - field: agent_output
      operator: regex
      value: "(?:api[_-]?key|api[_-]?secret|access[_-]?token|auth[_-]?token|client[_-]?secret)\\s*[=:]\\s*[\"']?[a-zA-Z0-9_\\-]{16,}"
      description: "Generic secret assignment pattern (api_key=..., auth_token=...)"
    - field: agent_output
      operator: regex
      value: "(?:ANTHROPIC_API_KEY|OPENAI_API_KEY|GOOGLE_API_KEY|AZURE_API_KEY|COHERE_API_KEY|HF_TOKEN)\\s*[=:]\\s*[^\\s]{8,}"
      description: "LLM provider API key environment variables"
    - field: agent_output
      operator: regex
      value: "(?:ssh-rsa|ssh-ed25519|ecdsa-sha2-nistp256)\\s+[A-Za-z0-9+/]{40,}"
      description: "SSH public key that may indicate key material exposure context"
    - field: agent_output
      operator: regex
      value: "(?:secret[\"'\\s]*[:=][\"'\\s]*[a-zA-Z0-9]{16,}|password[\"'\\s]*[:=][\"'\\s]*[^\\s\"']{8,})"
      description: "Generic secret or password assignment"
  condition: any
  false_positives:
    - "Agent discussing API key formats in educational context without real keys"
    - "Redacted or placeholder keys such as sk-xxxxxxxxxxxx or AKIA_EXAMPLE"
    - "Documentation examples using clearly fake credentials"
    - "Connection string templates with placeholder values"

response:
  actions:
    - block_output
    - escalate
    - alert
    - snapshot
  auto_response_threshold: critical
  message_template: |
    [ATR-2026-021] CRITICAL: Credential or secret detected in agent output.
    Output blocked and content redacted. Rotate any potentially exposed
    credentials immediately. Pattern: {matched_pattern}. Session: {session_id}.

confidence: 87
wild_validated: "2026/04/08"
wild_samples: 53577
wild_fp_rate: 0
test_cases:
  true_positives:
    - agent_output: "Here is your API key: sk-abc123def456ghi789jkl012mno345pqr"
      expected: triggered
      description: "OpenAI-style API key in output"
    - agent_output: "Your AWS access key is AKIAIOSFODNN7EXAMPLE"
      expected: triggered
      description: "AWS access key ID in output"
    - agent_output: "Use this token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.dozjgNryP4J3jVmNHl0w5N_XgL0n3I9PlFUP0THsR8U"
      expected: triggered
      description: "JWT token in agent output"
    - agent_output: "Connect with: mongodb+srv://admin:s3cretP4ss@cluster0.example.net/mydb"
      expected: triggered
      description: "MongoDB connection string with credentials"
    - agent_output: "Use this Google key: AIzaSyA00fake00test00value0000000000000"
      expected: triggered
      description: "Google API key in output"
    - agent_output: "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEA"
      expected: triggered
      description: "OpenSSH private key leaked in output"
  true_negatives:
    - agent_output: "To set up your API key, go to Settings > API Keys and click Generate."
      expected: not_triggered
      description: "Instructions about API keys without actual key values"
    - agent_output: "The key format looks like sk-xxxx where xxxx is your unique identifier."
      expected: not_triggered
      description: "Placeholder key format description"
    - agent_output: "AWS IAM best practices recommend rotating access keys every 90 days."
      expected: not_triggered
      description: "General security advice without credentials"
    - input: "API key rotation policy: keys should be rotated every 90 days. Use the dashboard to generate new keys."
      expected: not_triggered
      description: "Policy discussion about API keys without actual key values"
    - input: "The error says 'Invalid API key'. Please check that your key starts with 'sk-' and try again."
      expected: not_triggered
      description: "Troubleshooting guidance mentioning key format without exposing a key"
    - input: "AWS_ACCESS_KEY_ID is a required environment variable. Set it before running the deploy script."
      expected: not_triggered
      description: "Setup documentation referencing credential variable names"

evasion_tests:
  - input: "Your API key format should follow: sk-[project]-[random]. For example, sk-proj-abc123def456ghi789."
    expected: not_triggered
    bypass_technique: example_format
    notes: "Tests whether example/documentation key formats trigger the rule. May actually trigger — validate."
  - input: "The environment variable is stored at process.env.OPENAI_API_KEY but I cannot access its value."
    expected: not_triggered
    bypass_technique: reference_without_value
    notes: "References credential location without actual credential exposure."
  - input: "Here is the encrypted key: AES256(sk-proj-real-key-here). You'll need the master password to decrypt."
    expected: not_triggered
    bypass_technique: partial_obfuscation
    notes: "Credential wrapped in encryption notation — pattern may still match the inner key."

[[ currentRule.title && currentRule.title.length > 60 ? currentRule.title.slice(0, 60) + '…' : currentRule.title ]]

0bfe0cc0-e948-4d94-8839-45c1e03bda25

Propose edits, view related rules, and engage with the community through comments.

Executable code, not a declarative rule

This submission is arbitrary [[ currentRule.format ? currentRule.format.toUpperCase() : '' ]] code, not a declarative detection rule — review it as you would any script before running it, and note that it cannot be safely aggregated or auto-executed alongside YARA/Sigma/Suricata/CRS rules.

Rejected pattern present

[[ reason ]]

What does this mean?

Format & documentation suggestions

Not a security issue — this rule is valid and was accepted as-is. These are optional fields/conventions its own format recommends, mainly affecting its quality score and how well it documents itself for other reviewers.

[[ reason ]]

What do these messages mean?

[[ currentRule.title && currentRule.title.length > 60 ? currentRule.title.slice(0, 60) + '…' : currentRule.title ]]

[[ currentRule.title ]]

Community Discussion