[qualys_gav] Fix rate limit headers missing error

[brijesh-elastic]

Proposed commit message

qualys_gav: fix rate limit headers missing error

This removes the rate limit calculations for the `/auth` API call, because we don't receive
the necessary headers from it. It is also mentioned in the [documentation](1) that
rate limits do not apply to the "auth" API (JWT Token Generation API).

This PR updates the Kibana constraint to ^8.19.11 || ^9.2.5 to resolve the rate calculation issue.

[1] https://docs.qualys.com/en/csam/api/get_started/api_rate_limits.htm

Note

After resolving the rate limit missing rate error, I was unable to collect all events from qualys_gav. It only ingested events for 2 intervals. After investigation, it appears that rate was set too low or more throttled than what the server's rate limit headers indicate. (See issue.) This is resolved in PR #119. The fix is included in ^8.19.11 || ^9.2.5.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/qualys_gav directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Closes [qualys_gav]: rate limit missing rate #16517

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package qualys_gav 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
asset	642.67	441.31	-201.36 (-31.33%)	💔

To see the full report comment with /test benchmark fullreport

[ShourieG]

LGTM, nit.

[clement-fouque]

I tested version ⁠0.7.1 but still couldn’t complete a full ingest. The process halted after just four iterations, with a delay of seven seconds between the first two and the last two.

I tested with a batch of 100.

[brijesh-elastic]

Hi @clement-fouque ,

This PR will fix the error you're getting about rate limit missing rate.

I tested version ⁠0.7.1 but still couldn’t complete a full ingest. The process halted after just four iterations, with a delay of seven seconds between the first two and the last two.

During testing, I also came across the issue that the full batch of ingestion never completes. There seems to be an issue with rate calculation. The current behavior of API calls is more throttled than what the server's rate limit headers seem to indicate. Simply put, it seems like due to wrong rate calculation, the qualys API calling rate is 60 times lower than what it should be. We're looking into it in this issue.

[clement-fouque]

When can we expect a resolution? The current version has issues, so I’m considering whether reverting the entire rate-limiting functionality would be the best approach to allow us to address the problem.

[brijesh-elastic]

The mito PR fix has been merged. It will be included in the next stack release.

[brijesh-elastic]

@ShourieG, I'm putting the PR into draft for now. Once the next stack is released, I'll update the PR with that constraint and we can merge it afterwards.

[efd6]

The mito fix has been merged, but it is not yet available in the input. This is done in elastic/beats#48500 for 8.19, elastic/beats#48526 for 9.2, and elastic/beats#48527 for 9.3 (elastic/beats#48499 for main) and will be available when the respective releases are made public.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1dd8947

History

• 💚 Build #36213 succeeded aa0c325
• 💚 Build #36135 succeeded 023a139
• 💚 Build #36133 succeeded 4643ec7

cc @brijesh-elastic

[ShourieG]

LGTM

[elastic-vault-github-plugin-prod]

Package qualys_gav - 0.7.1 containing this change is available at https://epr.elastic.co/package/qualys_gav/0.7.1/