Security monitoring feature parity with Agent V2

[kamalchaturvedi]

Proposed changes

Purpose of this PR is to over the gaps between the current custom securityviolationsprocessor in agent V3, with what level of parsing and transformations we had in agent V2 security-monitoring processor.
Additionally, a protobuf schema is defined to serve as contract between the agent and the downstream consumer.

Commit 1:
The securityviolationsprocessor now processes NGINX App Protect WAF syslog messages, and transforms them into SecurityViolationEvent protobuf messages. This protobuf definition replaces the existing struct definition in /internal folder. This was done to allow management-plane to import this schema as a contract for handling security violations.

Commit 2:
Additionally, added the following capabilities to the parsing the details extraction from raw violations, to bring the feature in parity with Agent V2 implementation:

1. Parses XML violation details with context extraction (parameter, header, cookie, uri, request)
2. Extracts attack signature details

Commit 3:
These changes were thoroughly tested with addition of /testdata in Agent V2 implementation (https://github.com/nginx/agent/tree/dev-v2/src/extensions/nginx-app-protect/monitoring/processor/testdata) and additional variety of violations, to ensure robust coverage.

Commit 4:
Added NAP V5 support: Lookup for docker0 interface IP, which is then utilized to validate for configured syslog IP in NGINX Config.

Commit 5:
Added architecture diagram to describe the custom securityviolations processor.

Agent Config Modifications for Test

features:
  - certificates
  - configuration
  - metrics
  - file-watcher
  - api-action
  - logs-nap 

collector: 
  exporters:
    debug: {}
  processors:
    batch:
      "logs":
        send_batch_size: 1000
        timeout: 30s
        send_batch_max_size: 1000
  pipelines:
   logs:
     "default-security-events":
       receivers: ["tcplog/nginx_app_protect"]
       processors: ["securityviolations/default","batch/logs"]
       exporters: ["debug","otlp/default"]

Testing

Violations Triggered

1. curl -v 'http://127.0.0.1/myfile.tmp'

Expected Violations: VIOL_FILETYPE, VIOL_HTTP_PROTOCOL, VIOL_BOT_CLIENT

Output Payload:

{"resource": {"service.instance.id": "f904b8d3-4d6f-46b8-9b77-ba3926cc3407", "service.name": "otel-nginx-agent", "service.version": "v3.6.2"}, "otelcol.component.id": "securityviolations/default", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/default-security-events", "otelcol.signal": "logs", "protobuf": "policy_name:\"nms_app_protect_strict_policy\"  support_id:\"2378510662008429009\"  request_outcome:REQUEST_OUTCOME_REJECTED  request_outcome_reason:SECURITY_WAF_VIOLATION  blocking_exception_reason:\"N/A\"  method:\"GET\"  protocol:\"HTTP\"  xff_header_value:\"N/A\"  uri:\"/myfile.tmp\"  request:\"GET /myfile.tmp HTTP/1.1\\\\r\\\\nHost: 127.0.0.1\\\\r\\\\nUser-Agent: curl/8.5.0\\\\r\\\\nAccept: */*\\\\r\\\\n\\\\r\\\\n\"  request_status:REQUEST_STATUS_BLOCKED  vs_name:\"1-localhost:1-/\"  remote_addr:\"127.0.0.1\"  destination_port:80  server_port:33336  violations:\"HTTP protocol compliance failed::Illegal file type::Bot Client Detected\"  sub_violations:\"HTTP protocol compliance failed:Host header contains IP address\"  violation_rating:2  sig_set_names:\"N/A\"  sig_cves:\"N/A\"  client_class:\"Untrusted Bot\"  client_application:\"N/A\"  client_application_version:\"N/A\"  severity:SEVERITY_CRITICAL  threat_campaign_names:\"N/A\"  bot_anomalies:\"N/A\"  bot_category:\"HTTP Library\"  enforced_bot_anomalies:\"N/A\"  bot_signature_name:\"curl\"  system_id:\"ea6197e981ac\"  parent_hostname:\"ea6197e981ac\"  violations_data:{violation_data_name:\"VIOL_HTTP_PROTOCOL\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_FILETYPE\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_BOT_CLIENT\"  violation_data_context_data:{}}"}

2. curl -X SEARCH -k -v 'http://127.0.0.1/hello'

Expected Violations: VIOL_METHOD, VIOL_HTTP_PROTOCOL, VIOL_BOT_CLIENT

Output Payload:

{"resource": {"service.instance.id": "f904b8d3-4d6f-46b8-9b77-ba3926cc3407", "service.name": "otel-nginx-agent", "service.version": "v3.6.2"}, "otelcol.component.id": "securityviolations/default", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/default-security-events", "otelcol.signal": "logs", "protobuf": "policy_name:\"nms_app_protect_strict_policy\"  support_id:\"2378510662008429519\"  request_outcome:REQUEST_OUTCOME_REJECTED  request_outcome_reason:SECURITY_WAF_VIOLATION  blocking_exception_reason:\"N/A\"  method:\"SEARCH\"  protocol:\"HTTP\"  xff_header_value:\"N/A\"  uri:\"/hello\"  request:\"SEARCH /hello HTTP/1.1\\\\r\\\\nHost: 127.0.0.1\\\\r\\\\nUser-Agent: curl/8.5.0\\\\r\\\\nAccept: */*\\\\r\\\\n\\\\r\\\\n\"  request_status:REQUEST_STATUS_BLOCKED  vs_name:\"1-localhost:1-/\"  remote_addr:\"127.0.0.1\"  destination_port:80  server_port:33344  violations:\"HTTP protocol compliance failed::Illegal method::Bot Client Detected\"  sub_violations:\"HTTP protocol compliance failed:Host header contains IP address\"  violation_rating:2  sig_set_names:\"N/A\"  sig_cves:\"N/A\"  client_class:\"Untrusted Bot\"  client_application:\"N/A\"  client_application_version:\"N/A\"  severity:SEVERITY_CRITICAL  threat_campaign_names:\"N/A\"  bot_anomalies:\"N/A\"  bot_category:\"HTTP Library\"  enforced_bot_anomalies:\"N/A\"  bot_signature_name:\"curl\"  system_id:\"ea6197e981ac\"  parent_hostname:\"ea6197e981ac\"  violations_data:{violation_data_name:\"VIOL_HTTP_PROTOCOL\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_METHOD\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_BOT_CLIENT\"  violation_data_context_data:{}}"}

• curl -k -v 'http://127.0.0.1/a=<script>getAllMoney()</script>'

Expected Violations: VIOL_ATTACK_SIGNATURE, VIOL_HTTP_PROTOCOL, VIOL_BOT_CLIENT, VIOL_URL_METACHAR, VIOL_RATING_THREAT
Expected Signature IDs ": 200000099, 200000093

Output Payload:

{"resource": {"service.instance.id": "f904b8d3-4d6f-46b8-9b77-ba3926cc3407", "service.name": "otel-nginx-agent", "service.version": "v3.6.2"}, "otelcol.component.id": "securityviolations/default", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/default-security-events", "otelcol.signal": "logs", "protobuf": "policy_name:\"nms_app_protect_strict_policy\"  support_id:\"2378510662008430029\"  request_outcome:REQUEST_OUTCOME_REJECTED  request_outcome_reason:SECURITY_WAF_VIOLATION  blocking_exception_reason:\"N/A\"  method:\"GET\"  protocol:\"HTTP\"  xff_header_value:\"N/A\"  uri:\"/a=<script>getAllMoney()</script>\"  request:\"GET /a=<script>getAllMoney()</script> HTTP/1.1\\\\r\\\\nHost: 127.0.0.1\\\\r\\\\nUser-Agent: curl/8.5.0\\\\r\\\\nAccept: */*\\\\r\\\\n\\\\r\\\\n\"  request_status:REQUEST_STATUS_BLOCKED  vs_name:\"1-localhost:1-/\"  remote_addr:\"127.0.0.1\"  destination_port:80  server_port:33358  violations:\"HTTP protocol compliance failed::Illegal meta character in URL::Attack signature detected::Violation Rating Threat detected::Bot Client Detected\"  sub_violations:\"HTTP protocol compliance failed:Host header contains IP address\"  violation_rating:5  sig_set_names:\"{High Accuracy Signatures;Cross Site Scripting Signatures;Generic Detection Signatures (High/Medium Accuracy)}\"  sig_cves:\"{High Accuracy Signatures;Cross Site Scripting Signatures;Generic Detection Signatures (High/Medium Accuracy)}\"  client_class:\"Untrusted Bot\"  client_application:\"N/A\"  client_application_version:\"N/A\"  severity:SEVERITY_CRITICAL  threat_campaign_names:\"N/A\"  bot_anomalies:\"N/A\"  bot_category:\"HTTP Library\"  enforced_bot_anomalies:\"N/A\"  bot_signature_name:\"curl\"  system_id:\"ea6197e981ac\"  parent_hostname:\"ea6197e981ac\"  violations_data:{violation_data_name:\"VIOL_ATTACK_SIGNATURE\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"/a=<script>getAllMoney()</script>\"}  violation_data_signatures:{sig_data_id:200000099  sig_data_blocking_mask:\"3\"  sig_data_buffer:\"/a=<script>getAllMoney()</script>\"  sig_data_offset:3  sig_data_length:7}  violation_data_signatures:{sig_data_id:200000093  sig_data_blocking_mask:\"3\"  sig_data_buffer:\"/a=<script>getAllMoney()</script>\"  sig_data_offset:4  sig_data_length:7}}  violations_data:{violation_data_name:\"VIOL_HTTP_PROTOCOL\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_URL_METACHAR\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"L2E9PHNjcmlwdD5nZXRBbGxNb25leSgpPC9zY3JpcHQ+\"}}  violations_data:{violation_data_name:\"VIOL_URL_METACHAR\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"L2E9PHNjcmlwdD5nZXRBbGxNb25leSgpPC9zY3JpcHQ+\"}}  violations_data:{violation_data_name:\"VIOL_URL_METACHAR\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"L2E9PHNjcmlwdD5nZXRBbGxNb25leSgpPC9zY3JpcHQ+\"}}  violations_data:{violation_data_name:\"VIOL_BOT_CLIENT\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_RATING_THREAT\"  violation_data_context_data:{}}"}

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING document
• I have run make install-tools and have attached any dependency changes to this pull request
• If applicable, I have added tests that prove my fix is effective or that my feature works
• If applicable, I have checked that any relevant tests pass after adding my changes
• If applicable, I have updated any relevant documentation (README.md)
• If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

[github-actions]

✅ All required contributors have signed the F5 CLA for this PR. Thank you!
_{Posted by the CLA Assistant Lite bot.}

[nginx-nickc]

can we prebuild this map? If we expect to be high-volume, rebuilding the map log parse is probably not needed?

[nginx-nickc]

nit, you can preallocate with length

trimmedParts := make([]string, 0, len(parts))

[nginx-nickc]

did we explore the golang csv parser? https://pkg.go.dev/encoding/csv

[CVanF5]

Hi Kunal, thanks for the PR!

My only comment is that this implementation converts App Protect security violations into a custom/proprietary format that essentially uses OTel as a pipe. It does the job, no doubt.

Taking a step back, one of the key reasons we have OTel in NGINX Agent V3 is to enable customers to view their own telemetry. The goal is to able users to customize Agent's OTel collector and send telemetry to their own, perhaps private collectors. Or they have a plan with Datadog or some cloud provider. If the security violations don't follow the OTel standard for log processing, then it makes it that much harder (if not impossible) for the customer to view the security violations on their own systems. This is a key feature of Agent V3 and something we strive to maintain.