Unusual Web User Agent
ELASTIC detection rule by Elastic · Elastic License v2
A machine learning job detected a rare and unusual user agent indicating web browsing activity by an unusual process
other than a web browser. This can be due to persistence, command-and-control, or exfiltration activity. Uncommon user
agents coming from remote sources to local destinations are often the result of scanners, bots, and web scrapers, which
are part of common Internet background traffic. Much of this is noise, but more targeted attacks on websites using tools
like Burp or SQLmap can sometimes be discovered by spotting uncommon user agents. Uncommon user agents in traffic from
local sources to remote destinations can be any number of things, including harmless programs like weather monitoring or
stock-trading programs. However, uncommon user agents from local sources can also be due to malware or scanning
activity.
[metadata]
creation_date = "2020/03/25"
integration = ["endpoint", "network_traffic"]
maturity = "production"
updated_date = "2026/07/27"
min_stack_version = "9.4.0"
min_stack_comments = "Use EA (Entity Analytics) fields"
[rule]
anomaly_threshold = 50
author = ["Elastic"]
description = """
A machine learning job detected a rare and unusual user agent indicating web browsing activity by an unusual process
other than a web browser. This can be due to persistence, command-and-control, or exfiltration activity. Uncommon user
agents coming from remote sources to local destinations are often the result of scanners, bots, and web scrapers, which
are part of common Internet background traffic. Much of this is noise, but more targeted attacks on websites using tools
like Burp or SQLmap can sometimes be discovered by spotting uncommon user agents. Uncommon user agents in traffic from
local sources to remote destinations can be any number of things, including harmless programs like weather monitoring or
stock-trading programs. However, uncommon user agents from local sources can also be due to malware or scanning
activity.
"""
false_positives = [
"""
Web activity that is uncommon, like security scans, may trigger this alert and may need to be excluded. A new or
rarely used program that calls web services may trigger this alert.
""",
]
from = "now-45m"
interval = "15m"
license = "Elastic License v2"
machine_learning_job_id = "packetbeat_rare_user_agent_ea"
name = "Unusual Web User Agent"
setup = """## Setup
This rule requires the installation of associated Machine Learning jobs, as well as data coming in from one of the following integrations:
- Elastic Defend
- Network Packet Capture
### Anomaly Detection Setup
Once the rule is enabled, the associated Machine Learning job will start automatically. You can view the Machine Learning job linked under the "Definition" panel of the detection rule. If the job does not start due to an error, the issue must be resolved for the job to commence successfully. For more details on setting up anomaly detection jobs, refer to the [helper guide](https://www.elastic.co/guide/en/kibana/current/xpack-ml-anomalies.html).
### Elastic Defend Integration Setup
Elastic Defend is integrated into the Elastic Agent using Fleet. Upon configuration, the integration allows the Elastic Agent to monitor events on your host and send data to the Elastic Security app.
#### Prerequisite Requirements:
- Fleet is required for Elastic Defend.
- To configure Fleet Server refer to the [documentation](https://www.elastic.co/guide/en/fleet/current/fleet-server.html).
#### The following steps should be executed in order to add the Elastic Defend integration to your system:
- Go to the Kibana home page and click "Add integrations".
- In the query bar, search for "Elastic Defend" and select the integration to see more details about it.
- Click "Add Elastic Defend".
- Configure the integration name and optionally add a description.
- Select the type of environment you want to protect, either "Traditional Endpoints" or "Cloud Workloads".
- Select a configuration preset. Each preset comes with different default settings for Elastic Agent, you can further customize these later by configuring the Elastic Defend integration policy. [Helper guide](https://www.elastic.co/guide/en/security/current/configure-endpoint-integration-policy.html).
- We suggest selecting "Complete EDR (Endpoint Detection and Response)" as a configuration setting, that provides "All events; all preventions"
- Enter a name for the agent policy in "New agent policy name". If other agent policies already exist, you can click the "Existing hosts" tab and select an existing policy instead.
For more details on Elastic Agent configuration settings, refer to the [helper guide](https://www.elastic.co/guide/en/fleet/current/agent-policy.html).
- Click "Save and Continue".
- To complete the integration, select "Add Elastic Agent to your hosts" and continue to the next section to install the Elastic Agent on your hosts.
For more details on Elastic Defend refer to the [helper guide](https://www.elastic.co/guide/en/security/current/install-endpoint.html).
### Network Packet Capture Integration Setup
The Network Packet Capture integration sniffs network packets on a host and dissects known protocols. Monitoring the network traffic is critical to gaining observability and securing your environment — ensuring high levels of performance and security. The Network Packet Capture integration captures the network traffic between your application servers, decodes common application layer protocols and records the interesting fields for each transaction.
#### The following steps should be executed in order to add the Elastic Agent System integration "network_traffic" to your system:
- Go to the Kibana home page and click “Add integrations”.
- In the query bar, search for “Network Packet Capture” and select the integration to see more details about it.
- Click “Add Network Packet Capture”.
- Configure the integration name and optionally add a description.
- Review optional and advanced settings accordingly.
- Add the newly installed “network_traffic” to an existing or a new agent policy, and deploy the agent on your system from which network log files are desirable.
- Click “Save and Continue”.
- For more details on the integration refer to the [helper guide](https://docs.elastic.co/integrations/network_traffic).
"""
references = ["https://www.elastic.co/guide/en/security/current/prebuilt-ml-jobs.html"]
risk_score = 21
rule_id = "91f02f01-969f-4167-8d77-07827ac4cee0"
severity = "low"
tags = [
"Data Source: Elastic Defend",
"Data Source: Network Packet Capture",
"Use Case: Threat Detection",
"Rule Type: ML",
"Rule Type: Machine Learning",
"Tactic: Command and Control",
"Resources: Investigation Guide",
]
type = "machine_learning"
note = """## Triage and analysis
> **Disclaimer**:
> This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
### Investigating Unusual Web User Agent
User agents identify applications interacting with web servers, typically browsers. Adversaries exploit this by using non-standard user agents for malicious activities like data exfiltration or command-and-control. The 'Unusual Web User Agent' detection rule leverages machine learning to identify rare user agents, flagging potential threats from atypical processes, thus aiding in early threat detection.
### Possible investigation steps
- Review the specific user agent string that triggered the alert to determine if it matches known malicious patterns or tools like Burp or SQLmap.
- Identify the source and destination IP addresses involved in the alert to assess whether they are associated with known malicious activity or unusual geographic locations.
- Check the process that generated the unusual user agent to determine if it is a legitimate application or potentially malicious software.
- Analyze network traffic logs for additional context around the time of the alert to identify any related suspicious activities or patterns.
- Investigate any recent changes or installations on the system that could explain the presence of the unusual user agent, such as new software or updates.
- Correlate the alert with other security events or logs to see if there are any related indicators of compromise or ongoing threats.
### False positive analysis
- Non-malicious applications like weather monitoring or stock-trading programs may use uncommon user agents. Regularly review and whitelist these applications to prevent them from triggering false positives.
- Automated tools such as web scrapers or bots can generate unusual user agents. Identify and document these tools if they are part of legitimate business operations, and create exceptions for them in the detection rule.
- Internal scanners or monitoring tools might use non-standard user agents. Ensure these tools are recognized and excluded from alerts by updating the rule's exception list.
- Regularly update the list of known benign user agents to reflect changes in legitimate software and tools used within the organization, reducing unnecessary alerts.
- Collaborate with IT and security teams to maintain an up-to-date inventory of authorized applications and their user agents, ensuring that only truly suspicious activities are flagged.
### Response and remediation
- Isolate the affected system from the network to prevent further data exfiltration or command-and-control communication.
- Conduct a thorough scan of the isolated system using updated antivirus and anti-malware tools to identify and remove any malicious software.
- Review and analyze the logs of the affected system and network traffic to identify the source and scope of the unusual user agent activity.
- Block the identified malicious IP addresses or domains associated with the unusual user agent in the firewall and intrusion prevention systems.
- Update and patch all software and systems to close any vulnerabilities that may have been exploited by the adversary.
- Restore the affected system from a clean backup if malware removal is not feasible or if the system integrity is compromised.
- Report the incident to the appropriate internal teams and, if necessary, escalate to external cybersecurity authorities or partners for further investigation and support."""
[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1071"
name = "Application Layer Protocol"
reference = "https://attack.mitre.org/techniques/T1071/"
[[rule.threat.technique.subtechnique]]
id = "T1071.001"
name = "Web Protocols"
reference = "https://attack.mitre.org/techniques/T1071/001/"
[rule.threat.tactic]
id = "TA0011"
name = "Command and Control"
reference = "https://attack.mitre.org/tactics/TA0011/"