48 Pull requests opened by 38 people

• [SPARK-51796] [SQL] Disallow Sort order expressions under non-Sort operators

  #50582 opened Apr 14, 2025

• [SPARK-51765][DOCS] Docs for SQL Scripting

  #50592 opened Apr 15, 2025

• [SPARK-51596][SS] Fix concurrent StateStoreProvider maintenance and closing

  #50595 opened Apr 15, 2025

• [SPARK-51554][SQL] Add the time_trunc() function for TIME datatype

  #50607 opened Apr 16, 2025

• [SPARK-51830] Exception handling for partition datatype conversion call

  #50610 opened Apr 16, 2025

• [SPARK-51699][BUILD] Upgrade to Apache parent pom 34

  #50627 opened Apr 17, 2025

• [SPARK-51272][CORE] Aborting instead of continuing partially completed indeterminate result stage at ResubmitFailedStages

  #50630 opened Apr 17, 2025

• [SPARK-51842][SQL] Remove unnecessary judgement from fillDefaultValue

  #50633 opened Apr 18, 2025

• [SPARK-51847][PYTHON] Extend PySpark testing framework util functions with basic data tests

  #50644 opened Apr 18, 2025

• [SPARK-51886][SQL]Part2.a Adds support for decorrelating nested correlated subqueries in Optimizer

  #50668 opened Apr 22, 2025

• [SPARK-51883][DOCS][PYTHON] Python Data Source user guide for filter pushdown

  #50684 opened Apr 23, 2025

• Test rockdbsjni 10

  #50688 opened Apr 24, 2025

• [SPARK-51710][PYTHON] Passing an empty array to PySpark Dataframe.dropDuplicates should behave the same as passing no arguments

  #50714 opened Apr 25, 2025

• [SPARK-51918][CORE] Executor exit wait for out/err appenders to stop + flush remaining data

  #50715 opened Apr 25, 2025

• [SPARK-51919][PYTHON] Allow overwriting statically registered Python Data Source

  #50716 opened Apr 25, 2025

• [SPARK-51671][SQL] Add column pruning ro Recursive CTEs

  #50717 opened Apr 25, 2025

• [SPARK-51920][SS][PYTHON] Fix composite/nested StructType in value state for python

  #50718 opened Apr 25, 2025

• [DO NOT MERGE]Testing nested correlations handling

  #50720 opened Apr 25, 2025

• [Python] (PySpark) Support for subclasses in type_verifier

  #50726 opened Apr 26, 2025

• [SPARK-51927][BUILD] Upgrade jackson to 2.19.0

  #50730 opened Apr 27, 2025

• [SPARK-51955] Adding release() to ReadStateStore interface and reusing ReadStore for Streaming Aggregations

  #50742 opened Apr 28, 2025

• [SPARK-51949][CONNECT] Bump up the default value of `CONNECT_GRPC_MARSHALLER_RECURSION_LIMIT`

  #50756 opened Apr 29, 2025

• [SPARK-51016][SQL] Non-deterministic SQL expressions should set indeterminate map stage output level

  #50757 opened Apr 29, 2025

• [WIP][PYTHON][CONNECT] Introduce Scalar Arrow UDF

  #50759 opened Apr 30, 2025

• [WIP][SPARK-52011][SQL] Reduce HDFS NameNode RPC on vectorized Parquet reader

  #50765 opened Apr 30, 2025

• [SPARK-52041][CORE] Add better support for integrating with external cluster manager

  #50770 opened Apr 30, 2025

• [SPARK-51972][SS] State Store file integrity verification using checksum

  #50773 opened May 1, 2025

• [SPARK-51966][PYTHON] Replace select.select() with select.poll() when running on POSIX os

  #50774 opened May 1, 2025

• [SPARK-51880][FOLLOW-UP] Fix LDAModel.toLocal

  #50788 opened May 5, 2025

• [SPARK-52020][TEST] Build hive-test-udfs.jar from source

  #50790 opened May 5, 2025

• [SPARK-51987][SQL] DSv2 expressions in column defaults on write

  #50792 opened May 5, 2025

• [SPARK-52008] [SS] Add StateStore TaskCompletionListener to abort store and throw error

  #50795 opened May 5, 2025

• [SPARK-52012][CORE][SQL] Restore IDE Index with type annotations

  #50798 opened May 6, 2025

• [SPARK-52014][SQL] Support FoldableUnevaluable in HiveGenericUDFEvaluator

  #50800 opened May 6, 2025

• [SPARK-51821][CORE] Call interrupt() without holding uninterruptibleLock to avoid possible deadlock

  #50810 opened May 7, 2025

• [SPARK-51867][ML][FOLLOW-UP] Make funtions in `ReadWriteUtils` private

  #50811 opened May 7, 2025

• [SPARK-52024][SQL] Support cancel ShuffleQueryStage when propagate empty relations

  #50814 opened May 7, 2025

• [SPARK-52029][SQL] Support external field-id to column-name mapping for Parquet

  #50819 opened May 7, 2025

• [WIP][SPARK-50585][PYTHON][DOCS] Render interactive PySpark plot examples

  #50820 opened May 7, 2025

• [SPARK-52031][SQL]Part 3. Subquery generator for testing queries with nested correlations

  #50821 opened May 8, 2025

• [SPARK-52023][SQL] Fix data corruption/segfault returning Option[Product] from udaf

  #50827 opened May 8, 2025

• Properly handle table existence check for jdbc dialects

  #50835 opened May 8, 2025

• [SPARK-51903][SQL] Validate data on adding a CHECK constraint

  #50839 opened May 9, 2025

• [SPARK-52060][SQL] Make `OneRowRelationExec` node

  #50849 opened May 9, 2025

• [SPARK-46685][DOCS] Refine PySpark profiler user guide for clarity

  #50850 opened May 9, 2025

• [SPARK-52065][SQL] Produce another plan tree with output columns (name, data type, nullability) in plan change logging

  #50852 opened May 10, 2025

• [SPARK-52077][PYTHON][CONNECT][TEST] Skip ArrowUDTFParityTests in Spark Connect compatibility test

  #50856 opened May 12, 2025

• [SPARK-52078][TEST] Rewrite ZStandardBenchmark with TPC-DS data

  #50857 opened May 12, 2025

41 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-51756][CORE] Computes RowBasedChecksum in ShuffleWriters

  #50230 commented on May 2, 2025 • 30 new comments

• [SPARK-51885][SQL]Part 1.b Add analyzer support for nested correlated subqueries

  #50548 commented on Apr 29, 2025 • 23 new comments

• [SPARK-49386][SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill

  #47856 commented on May 6, 2025 • 12 new comments

• [SPARK-51745] Enforce State Machine for RocksDBStateStore

  #50497 commented on May 6, 2025 • 11 new comments

• [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys

  #50033 commented on Apr 16, 2025 • 9 new comments

• [SPARK-49872][CORE] allow unlimited json size again

  #49163 commented on May 8, 2025 • 4 new comments

• Enable -Xsource:3 compiler flag

  #50474 commented on Apr 15, 2025 • 3 new comments

• [SPARK-51562][SQL] Add the time() function

  #50557 commented on Apr 15, 2025 • 2 new comments

• [SPARK-47573][K8S] Support custom driver log url

  #45728 commented on Apr 28, 2025 • 2 new comments

• [SPARK-51400] Replace ArrayContains nodes to InSet

  #50170 commented on May 3, 2025 • 2 new comments

• [SPARK-51069][SQL] Add big-endian support to UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl

  #49773 commented on Apr 28, 2025 • 1 new comment

• [SPARK-51568][SQL] Introduce isSupportedExtract to prevent happening unexpected behavior

  #50333 commented on Apr 15, 2025 • 1 new comment

• [SPARK-35564][SQL] Support subexpression elimination for conditionally evaluated expressions

  #32987 commented on May 5, 2025 • 0 new comments

• [SPARK-51016][SQL] Stage.isIndeterminate gives wrong result in case the shuffle partitioner uses an inDeterministic attribute or expression.

  #50029 commented on May 2, 2025 • 0 new comments

• [SPARK-51301][BUILD] Bump zstd-jni 1.5.7-1

  #50057 commented on May 12, 2025 • 0 new comments

• [WIP][SPARK-51348][BUILD][SQL] Upgrade Hive to 4.0

  #50213 commented on Apr 17, 2025 • 0 new comments

• Update supported_api_gen.py: remove an invalid escape sequence "\_" using a raw string

  #50243 commented on May 7, 2025 • 0 new comments

• [SPARK-51585][SQL] Oracle dialect supports pushdown datetime functions

  #50353 commented on Apr 18, 2025 • 0 new comments

• [SPARK-51068][SQL] Canonicalized CTEs to avoid cached result not being used and recomputed

  #50360 commented on Apr 27, 2025 • 0 new comments

• [Spark-50873][SQL] Prune column after RewriteSubquery rule for DSV2

  #50399 commented on May 9, 2025 • 0 new comments

• [Core]Convert shuffleWriteTime from Nanoseconds to Milliseconds for Consistency with Other Metrics

  #50418 commented on Apr 13, 2025 • 0 new comments

• [SPARK-51740][SQL] Allow get_json_object to consider leading spaces in paths

  #50533 commented on May 4, 2025 • 0 new comments

• [SPARK-51728][SQL] Add SELECT EXCEPT Support

  #50536 commented on Apr 29, 2025 • 0 new comments

• [SPARK-37019][SQL] Add codegen support to array higher-order functions

  #34558 commented on May 5, 2025 • 0 new comments

• [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC using MERGE INTO with temp table

  #41611 commented on Apr 28, 2025 • 0 new comments

• [SPARK-44639][SS][YARN] Use Java tmp dir for local RocksDB state storage on Yarn

  #42301 commented on May 5, 2025 • 0 new comments

• [SPARK-22876][YARN] Respect YARN AM failure validity interval

  #42570 commented on May 5, 2025 • 0 new comments

• [SPARK-47618][CORE] Use `Magic Committer` for all S3 buckets by default

  #45740 commented on May 9, 2025 • 0 new comments

• [MINOR][INFRA] Do not upload docker build record

  #48012 commented on Apr 28, 2025 • 0 new comments

• [SPARK-49984][CORE] Fix duplicate JVM options

  #48488 commented on May 5, 2025 • 0 new comments

• [SPARK-50137][HIVE] Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception

  #48668 commented on May 12, 2025 • 0 new comments

• [SPARK-49547][SQL][PYTHON] Add iterator of `RecordBatch` API to `applyInArrow`

  #49005 commented on May 6, 2025 • 0 new comments

• [SPARK-50531][SQL] Update `StringType` proto to support constraints

  #49130 commented on May 12, 2025 • 0 new comments

• [MINOR][CORE] Place the comments in the appropriate location

  #49216 commented on Apr 24, 2025 • 0 new comments

• [SPARK-50639][SQL] Improve warning logging in CacheManager

  #49276 commented on Apr 16, 2025 • 0 new comments

• [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC

  #49528 commented on Apr 28, 2025 • 0 new comments

• [BUILD] Upgrade `RoaringBitmap` to 1.3.19

  #49710 commented on Apr 24, 2025 • 0 new comments

• [SPARK-51149][CORE] Log classpath in SparkSubmit on ClassNotFoundException

  #49870 commented on Apr 16, 2025 • 0 new comments

• [WIP][SPARK-51180][BUILD] Upgrade Arrow to 19.0.0

  #49909 commented on May 1, 2025 • 0 new comments

• [SPARK-51243][CORE][ML] Configurable allow native BLAS

  #49986 commented on Apr 21, 2025 • 0 new comments

• [SPARK-51250][K8S] Add Support for K8s PriorityClass Configuration fo…

  #49998 commented on Apr 17, 2025 • 0 new comments