o365: Fix the handling of duplicated field QueryTime

[chemamartinez]

Proposed commit message

For some unknown reason we couldn't find out, some events from O365 introduces a malformed JSON object by adding a duplicated QueryTime field. The pattern this issue follows is:

• It only affects events with RecordType == 64
• The undesired QueryTime field doesn't follow ISO8601 format.

So the pipeline fails with:

"error": {
      "message": [
        "Duplicate field 'QueryTime'\n at [Source: (String)\"{\"Version\":\"3.0\",\"VendorName\":\"Microsoft\"...\"[truncated 11497 chars]; line: 1, column: 5099]"
      ]
}

To fix the error it was added a gsub processor that attempts to find the undesired QueryTime field and removed from the whole Data object. In addition, to prevent other similar issues, added a on_failure case when decoding the Data field which generally contains unknown fields.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Closes [Microsoft 365] Pipeline failure resulting in incorrect root-level fields #9920

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4c836ca

cc @chemamartinez

[efd6]

They are not technically wrong; RFC 8259 only says that "The names within an object SHOULD be unique." (section 4). Not nice though. (RFC 7493 does prohibit this.)

[elastic-vault-github-plugin-prod]

Package o365 - 2.6.5 containing this change is available at https://epr.elastic.co/search?package=o365