[SPARK-52575][SQL] Introduce contextIndependentFoldable attribute for Expressions #51282

gengliangwang · 2025-06-25T19:58:12Z

What changes were proposed in this pull request?

Introduces a method to determine whether an expression can be folded without relying on any external context (e.g., time zone, session configurations, or catalogs). If an expression is context-independent foldable, it can be safely evaluated during DDL operations such as creating tables, views, or constraints. This enables systems to store the computed value rather than the original expression, simplifying implementation and improving performance. By default, expressions are not considered context-independent foldable to ensure explicit annotation of context independence.

Examples of context-independent foldable:

1 + 1: Always produces 2, regardless of context.
1 > 0: Always evaluates to true.
CAST('2025-08-01' AS DATE): Always gives the same date value.

Examples of not context-independent foldable:

CURRENT_DATE: Result depends on the current date and time zone.
CURRENT_USER: Result depends on the session/user context.
CAST('2025-08-01 00:00:00' AS TIMESTAMP): Result can vary depending on the session time zone setting

Why are the changes needed?

This allows catalogs and connectors to store the computed value rather than the expression itself, improving both simplicity and performance.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit tests

Was this patch authored or co-authored using generative AI tooling?

No

gengliangwang · 2025-06-25T20:00:48Z

I will create a follow-up to use this new method in the V2 expression conversion

 class V2ExpressionBuilder(e: Expression, isPredicate: Boolean = false) extends Logging  {
 
-  def build(): Option[V2Expression] = generateExpression(e, isPredicate)
+  def build(): Option[V2Expression] = {
+    val exprToBuild = if (e.foldable && e.contextIndependentFoldable) {
+      ConstantFolding.constantFolding(e)
+    } else {
+      e
+    }
+    generateExpression(exprToBuild, isPredicate)
+  }

cc @cloud-fan @beliefer @aokolnychyi

aokolnychyi

This would be super helpful to simplify expressions before converting them to DSv2. I only have a question about marking expressions context-free by default for all unary and binary expressions.

aokolnychyi · 2025-06-25T22:22:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

+   *
+   * Default is false to ensure explicit marking of context independence.
+   */
+  def contextIndependentFoldable: Boolean = false


Technically, we could consider splitting contextIndependent and foldable into separate methods. Are there use cases where we want to know if the expression is context independent without checking it is foldable?

I can't think of such special cases. I think we can always check foldable before checking contextIndependentFoldable, which seems very safe to me. The syntax for this method is when it is foldable, does it depend on context?

I guess some unfoldable expression could be context independent. Such as: 1 + a', the type of attribute a' is int.
But I don't know should we consider these cases.
I'm +1 for @aokolnychyi if we must consider the case I mentioned.
I'm +1 for @gengliangwang if we not.

Such as: 1 + a', the type of attribute a' is int

@beliefer in such a case, column a' is not contextIndependent

I means the context that is related to the Spark context or Session context. The value of attribute is not.

Or the time zone etc.

The key is how to define the context ?

Context-independent means that the expression is independent of contexts(current user, current database, current time, etc) when it is foldable.
For such expressions, we can evaluate them before storing in column default values, table constraints, etc.

So the foldable is the prerequisite of context-independent. Thank you for the explanation.

How about
lazy val contextIndependentFoldable: Boolean = children.forall(_. contextIndependentFoldable)
here ?
So we can avoid change everywhere.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/DataTypeUtils.scala

cloud-fan · 2025-06-30T14:06:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

@@ -493,6 +493,11 @@ case class Cast(

  final override def nodePatternsInternal(): Seq[TreePattern] = Seq(CAST)

+  override def contextIndependentFoldable: Boolean = {
+    child.contextIndependentFoldable && !DataTypeUtils.containsTimestampOrUDT(dataType) &&


I don't think UDT matters here. The CAST implementation only allows pass-through of UDT casting, there is no operation and no context is needed.

also, let's be more specific about LTZ

oh wait, can't we simply call needsTimeZone here?

ok, updated

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

beliefer · 2025-07-02T08:47:19Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/DataTypeUtils.scala

+   */
+  def containsTimestampOrUDT(dataType: DataType): Boolean = {
+    matchesPattern(dataType,
+      dt => dt.isInstanceOf[TimestampType] || dt.isInstanceOf[UserDefinedType[_]])


I think we can add another function containsUserDefinedType and remove the dt.isInstanceOf[UserDefinedType[_]] from here. It looks weird.

I removed containsTimestampOrUDT in the latest code. In the current code, the matchesPattern method is only used here. I prefer to make it more general.

cloud-fan · 2025-07-03T08:06:05Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala

@@ -27,7 +27,7 @@ import org.apache.spark.sql.catalyst.expressions.Cast.{toSQLId, toSQLType}
 import org.apache.spark.sql.catalyst.expressions.codegen._
 import org.apache.spark.sql.catalyst.expressions.codegen.Block._
 import org.apache.spark.sql.catalyst.trees.TreePattern.{BINARY_ARITHMETIC, TreePattern}
-import org.apache.spark.sql.catalyst.types.{PhysicalDecimalType, PhysicalFractionalType, PhysicalIntegerType, PhysicalIntegralType, PhysicalLongType}
+import org.apache.spark.sql.catalyst.types.{DataTypeUtils, PhysicalDecimalType, PhysicalFractionalType, PhysicalIntegerType, PhysicalIntegralType, PhysicalLongType}


unused import

cloud-fan · 2025-07-03T08:07:15Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala

@@ -415,6 +415,11 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
  Literal.validateLiteralValue(value, dataType)

  override def foldable: Boolean = true
+
+  override def contextIndependentFoldable: Boolean = {
+    !DataTypeUtils.matchesPattern(dataType, dt => dt.isInstanceOf[UserDefinedType[_]])


How can a literal be context dependent? It's not an operation.

Looking at Literal.validateLiteralValue:

case udt: UserDefinedType[_] => doValidate(v, udt.sqlType)

So the stored value is the same as UDT's underlying SQL type. I don't think it requires any context

gengliangwang added 5 commits June 24, 2025 15:55

introduce contextIndependentFoldable

d9d7626

improve comment

2b8bb31

fix and add test

7993b28

add test for literals and math

7635371

add test for predicates

71439ac

github-actions bot added the SQL label Jun 25, 2025

gengliangwang requested review from cloud-fan and beliefer June 25, 2025 19:58

aokolnychyi approved these changes Jun 25, 2025

View reviewed changes

move method under DataTypeUtils.scala

ff22162

cloud-fan mentioned this pull request Jun 26, 2025

[SPARK-52551][SQL] Add a new v2 Predicate BOOLEAN_EXPRESSION #51247

Closed

gengliangwang added 6 commits June 26, 2025 15:27

override unary expressions

049665c

override binary expressions

412fa08

arithmetic and string

88c157a

update literal

6676539

update StringExpressionsSuite

d74393f

fix cast

21fbdf9

cloud-fan reviewed Jun 27, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/DataTypeUtils.scala Outdated Show resolved Hide resolved

gengliangwang added 2 commits June 27, 2025 10:30

fix test failure

5e7de91

rename

fbc1bad

cloud-fan reviewed Jun 30, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Jun 30, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Jun 30, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala Outdated Show resolved Hide resolved

gengliangwang added 2 commits July 1, 2025 10:26

address comments

76f4c79

address more

67398a1

beliefer reviewed Jul 2, 2025

View reviewed changes

gengliangwang added 2 commits July 2, 2025 12:53

address more comments

e6e914b

fix style

0b939bf

cloud-fan reviewed Jul 3, 2025

View reviewed changes

[SPARK-52575][SQL] Introduce contextIndependentFoldable attribute for Expressions #51282

Are you sure you want to change the base?

[SPARK-52575][SQL] Introduce contextIndependentFoldable attribute for Expressions #51282

Uh oh!

Conversation

gengliangwang commented Jun 25, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

gengliangwang commented Jun 25, 2025

Uh oh!

aokolnychyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

beliefer Jul 3, 2025 •

edited

Loading