Skip to content

Request change F.reduce to F.aggregate for better backwards spark version support #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tongqqiu opened this issue May 13, 2025 · 2 comments
Closed

Request change F.reduce to F.aggregate for better backwards spark version support #31

tongqqiu opened this issue May 13, 2025 · 2 comments

Comments

@tongqqiu
Copy link
Contributor

tongqqiu commented May 13, 2025 •
edited
Loading

Hi, It is not a bug, but reduce is a function introduced in 3.5+. Many places 3.5.0 is approved spark version to use. reduce and aggregate have the same functionality, can we use aggregate instead. I think that will make the tool be used in more places. Let me know if you need me to create a PR for that.
@flitzpiepe93
Copy link
Contributor

Hi, thanks for pointing that out! You're absolutely right – using aggregate instead of reduce would increase compatibility with older Spark versions like 3.5.0.

Feel free to open a PR with your suggested changes. I'm happy to take any help to improve the tool!

@tongqqiu tongqqiu mentioned this issue May 13, 2025
Merged
5 tasks
@flitzpiepe93
Copy link
Contributor

Issue solved with the following MR: #32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tongqqiu @flitzpiepe93