Skip to content

[SPARK-51796] [SQL] Disallow Sort order expressions under non-Sort operators #50582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mihailoale-db
Copy link
Contributor

What changes were proposed in this pull request?

I propose that we disallow SortOrder expressions under non-Sort operators.

Why are the changes needed?

Following Dataframe program throw an internal error:

val df = sql("SELECT 1 AS name")
df.select(desc("name"))

In this PR we fix is so we throw a more user friendly one.

Does this PR introduce any user-facing change?

Different (more user friendly) error is thrown.

How was this patch tested?

Added test.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the SQL label Apr 14, 2025
@mihailoale-db
Copy link
Contributor Author

@cloud-fan should we do it like this or should we move checkExpressionUnderUnsupportedOperator check to the begging of the operator match? Thanks

@mihailoale-db mihailoale-db force-pushed the mihailoale-db/sortordercheck branch 2 times, most recently from dd8d02f to 5ab8227 Compare April 14, 2025 22:32
@mihailoale-db mihailoale-db force-pushed the mihailoale-db/sortordercheck branch from 5ab8227 to 2bb8519 Compare April 14, 2025 23:08
@cloud-fan
Copy link
Contributor

cloud-fan commented Apr 15, 2025

there are cases that an expression containing SortOrder (e.g. ListAgg) can appear in non-sort operators. We need to find out all these cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants