Integration quality: Add tags and improve error.message

[kcreddy]

Proposed commit message

Add pipeline processor tags via elastic-package and expose in on_failure

- Run `elastic-package modify -m pipeline-tag` to add a `tag` key to
  each processor in ingest pipelines. Applied to ti_abusech, o365, 
  sentinel_one, and m365_defender pipelines (part of integration 
  quality phase-1 improvements).
- Update on_failure append to error.message to include
  _ingest.on_failure_processor_tag so pipeline failures can be traced 
  by processor tag. Applied to sentinel_one and m365_defender pipelines.

Note

This change only updates following integrations as part of integration quality phase-1 integrations:

• m365_defender
• o365
• sentinel_one
• ti_abusech

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Tested all 4 packages' processors now have tags:

for pkg in ti_abusech o365 m365_defender sentinel_one; do
  find "packages/$pkg" -path '*/data_stream/*/elasticsearch/ingest_pipeline/*.yml' -type f 2>/dev/null | while read -r f; do
    missing=$(yq '.processors[] | select((to_entries[0].value | has("tag")) | not) | to_entries[0].key' "$f" 2>/dev/null)
    echo "=== $f ==="
    if [ -n "$missing" ]; then
      data_stream=$(echo "$f" | sed 's|.*/data_stream/\([^/]*\)/.*|\1|')
      pipeline=$(basename "$f" .yml)
      echo "data stream: $data_stream, pipeline: $pipeline"
      echo "$missing"
    else
      echo "all processors have tags"
    fi
    echo ""
  done
done

Output

=== packages/ti_abusech/data_stream/sslblacklist/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/ti_abusech/data_stream/malwarebazaar/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/ti_abusech/data_stream/malware/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/ti_abusech/data_stream/url/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/ti_abusech/data_stream/ja3_fingerprints/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/ti_abusech/data_stream/threatfox/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/m365_defender/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/pipeline_alert.yml ===
all processors have tags

=== packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/pipeline_app_and_identity.yml ===
all processors have tags

=== packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/pipeline_email.yml ===
all processors have tags

=== packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/pipeline_device.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/threat_event/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/group/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/activity/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/alert/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/agent/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/threat/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/application/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

=== packages/sentinel_one/data_stream/application_risk/elasticsearch/ingest_pipeline/default.yml ===
all processors have tags

Related issues

• Relates Integration quality: (Phase 1) Add processor tags to reflect failing processor #17436

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e2dca42

cc @kcreddy

[chrisberkhout]

Very nice!

[elastic-vault-github-plugin-prod]

Package m365_defender - 5.7.0 containing this change is available at https://epr.elastic.co/package/m365_defender/5.7.0/

[elastic-vault-github-plugin-prod]

Package o365 - 3.5.0 containing this change is available at https://epr.elastic.co/package/o365/3.5.0/

[elastic-vault-github-plugin-prod]

Package sentinel_one - 2.3.0 containing this change is available at https://epr.elastic.co/package/sentinel_one/2.3.0/

[elastic-vault-github-plugin-prod]

Package ti_abusech - 3.6.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/3.6.0/