Skip to content

[SPARK-52582][SQL] Improve the memory usage of XML parser #51287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
+554 −436
Open

[SPARK-52582][SQL] Improve the memory usage of XML parser #51287

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
05e975c
draft
xiaonanyang-db Jun 26, 2025
854d3de
add test
xiaonanyang-db Jun 26, 2025
6d2a960
fix
xiaonanyang-db Jun 30, 2025
57c9685
address comments
xiaonanyang-db Jun 30, 2025
f8f8216
u
xiaonanyang-db Jul 1, 2025
f787809
u
xiaonanyang-db Jul 1, 2025
6baa722
u
xiaonanyang-db Jul 1, 2025
02b09f6
u
xiaonanyang-db Jul 1, 2025
57d3105
u
xiaonanyang-db Jul 2, 2025
2cb368f
u
xiaonanyang-db Jul 2, 2025
5fe3877
u
xiaonanyang-db Jul 2, 2025
26a8565
u
xiaonanyang-db Jul 2, 2025
3e959a8
u
xiaonanyang-db Jul 3, 2025
b52e78c
use optimized parser in schema inerence
xiaonanyang-db Jul 3, 2025
74ecf46
address comments
xiaonanyang-db Jul 7, 2025
fcf4af8
u
xiaonanyang-db Jul 7, 2025
92c0e87
minor
xiaonanyang-db Jul 7, 2025
5c7164e
support XSD validation
xiaonanyang-db Jul 8, 2025
b137fd8
u
xiaonanyang-db Jul 8, 2025
bedd7c5
address comments - 1
xiaonanyang-db Jul 8, 2025
39f27b7
address comments - 2
xiaonanyang-db Jul 8, 2025
5742935
address comments - 3
xiaonanyang-db Jul 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/XmlExpressionEvalUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ case class XmlToStructsEvaluator(
val xsdSchema = Option(parsedOptions.rowValidationXSDPath).map(ValidatorUtil.getSchema)

new FailureSafeParser[String](
input => rawParser.doParseColumn(input, mode, xsdSchema),
input => rawParser.doParseColumn(input, xsdSchema),
mode,
schema,
parsedOptions.columnNameOfCorruptRecord)
Expand Down
Loading