24 Pull requests opened by 19 people

• [SPARK-51966][PYTHON] Replace select.select() with select.poll() when running on POSIX os

  #50774 opened May 1, 2025

• [SPARK-51981][SS] Add JobTags to queryStartedEvent

  #50780 opened May 1, 2025

• [SPARK-51880][FOLLOW-UP] Fix LDAModel.toLocal

  #50788 opened May 5, 2025

• [SPARK-52020][TEST] Build hive-test-udfs.jar from source

  #50790 opened May 5, 2025

• [SPARK-51987][SQL] DSv2 expressions in column defaults on write

  #50792 opened May 5, 2025

• [SPARK-52008] [SS] Add StateStore TaskCompletionListener to abort store and throw error

  #50795 opened May 5, 2025

• [SPARK-52012][CORE][SQL] Restore IDE Index with type annotations

  #50798 opened May 6, 2025

• [SPARK-52013] [CONNECT] [ML] Fix SparkConnectClient ml_caches

  #50799 opened May 6, 2025

• [SPARK-52014][SQL] Support FoldableUnevaluable in HiveGenericUDFEvaluator

  #50800 opened May 6, 2025

• [SPARK-52017][SQL] Enable multiple self-references and self-references from a Subquery inside rCTEs

  #50803 opened May 6, 2025

• [SPARK-51821][CORE] Call interrupt() without holding uninterruptibleLock to avoid possible deadlock

  #50810 opened May 7, 2025

• [SPARK-51867][ML][FOLLOW-UP] Make funtions in `ReadWriteUtils` private

  #50811 opened May 7, 2025

• [SPARK-52006][SQL][CORE] Exclude CollectMetricsExec accumulator from Spark UI + event logs + metric heartbeats

  #50812 opened May 7, 2025

• [SPARK-52024][SQL] Support cancel ShuffleQueryStage when propagate empty relations

  #50814 opened May 7, 2025

• Add `spark-core` as compile dependency for `sql-api`

  #50815 opened May 7, 2025

• [SPARK-52026][PS] Block pandas API on Spark on ANSI mode by default

  #50816 opened May 7, 2025

• [SPARK-52029][SQL] Support external field-id to column-name mapping for Parquet

  #50819 opened May 7, 2025

• [SPARK-50585][PYTHON][DOCS] Render interactive PySpark plot examples

  #50820 opened May 7, 2025

• [SPARK-52031][SQL]Part 3. Subquery generator for testing queries with nested correlations

  #50821 opened May 8, 2025

• [SPARK-51884][SQL]Part 1.a Add outer scope attributes for SubqueryExpression

  #50822 opened May 8, 2025

• [SPARK-52033][SQL] Fix Generate node bug fix where the output of the child node can have multiple copies of the same Attribute

  #50823 opened May 8, 2025

• [SPARK-52035][ML] Decouple LinearRegressionTrainingSummary and LinearRegressionModel

  #50825 opened May 8, 2025

• [SPARK-52036][SQL] Add SQLConfHelper trait to v2.FileScan

  #50826 opened May 8, 2025

• [SPARK-52023][SQL] Fix data corruption/segfault returning Option[Product] from udaf

  #50827 opened May 8, 2025

35 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-51972][SS] State Store file integrity verification using checksum

  #50773 commented on May 7, 2025 • 29 new comments

• [SPARK-51756][CORE] Computes RowBasedChecksum in ShuffleWriters

  #50230 commented on May 2, 2025 • 20 new comments

• [SPARK-51955] Adding release() to ReadStateStore interface and reusing ReadStore for Streaming Aggregations

  #50742 commented on May 8, 2025 • 20 new comments

• [SPARK-51291] [SS] Reclassify validation errors thrown from state store loading

  #50045 commented on May 7, 2025 • 17 new comments

• [SPARK-51906][SQL] Dsv2 expressions in alter table add columns

  #50701 commented on May 8, 2025 • 17 new comments

• [SPARK-51596][SS] Fix concurrent StateStoreProvider maintenance and closing

  #50595 commented on May 6, 2025 • 13 new comments

• [SPARK-51016][SQL] Non-deterministic SQL expressions should set indeterminate map stage output level

  #50757 commented on May 5, 2025 • 6 new comments

• [SPARK-49386][SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill

  #47856 commented on May 6, 2025 • 5 new comments

• [SPARK-51847][PYTHON] Extend PySpark testing framework util functions with basic data tests

  #50644 commented on May 2, 2025 • 4 new comments

• [SPARK-51883][DOCS][PYTHON] Python Data Source user guide for filter pushdown

  #50684 commented on May 8, 2025 • 4 new comments

• [SPARK-51272][CORE] Aborting instead of re-submitting of partially completed indeterminate result stage

  #50630 commented on May 7, 2025 • 3 new comments

• [SPARK-51842][SQL] Remove unnecessary judgement from fillDefaultValue

  #50633 commented on May 8, 2025 • 1 new comment

• [WIP][SPARK-52011][SQL] Reduce HDFS NameNode RPC on vectorized Parquet reader

  #50765 commented on May 6, 2025 • 1 new comment

• [SPARK-51765][DOCS] Docs for SQL Scripting

  #50592 commented on May 7, 2025 • 1 new comment

• Test rockdbsjni 10

  #50688 commented on May 2, 2025 • 0 new comments

• [SPARK-51710][PYTHON] Passing an empty array to PySpark Dataframe.dropDuplicates should behave the same as passing no arguments

  #50714 commented on May 7, 2025 • 0 new comments

• [SPARK-51554][SQL] Add the time_trunc() function for TIME datatype

  #50607 commented on May 6, 2025 • 0 new comments

• [WIP][PYTHON][CONNECT] Introduce Scalar Arrow UDF

  #50759 commented on May 6, 2025 • 0 new comments

• [SPARK-51740][SQL] Allow get_json_object to consider leading spaces in paths

  #50533 commented on May 4, 2025 • 0 new comments

• [SPARK-51745] Enforce State Machine for RocksDBStateStore

  #50497 commented on May 6, 2025 • 0 new comments

• [Spark-50873][SQL] Prune column after RewriteSubquery rule for DSV2

  #50399 commented on May 8, 2025 • 0 new comments

• Update supported_api_gen.py: remove an invalid escape sequence "\_" using a raw string

  #50243 commented on May 7, 2025 • 0 new comments

• [SPARK-51400] Replace ArrayContains nodes to InSet

  #50170 commented on May 3, 2025 • 0 new comments

• [SPARK-51301][BUILD] Bump zstd-jni 1.5.7-1

  #50057 commented on May 7, 2025 • 0 new comments

• [SPARK-51016][SQL] Stage.isIndeterminate gives wrong result in case the shuffle partitioner uses an inDeterministic attribute or expression.

  #50029 commented on May 2, 2025 • 0 new comments

• [WIP][SPARK-51180][BUILD] Upgrade Arrow to 19.0.0

  #49909 commented on May 1, 2025 • 0 new comments

• [SPARK-50991][SQL] Align the table schema column position beginning index

  #49671 commented on May 8, 2025 • 0 new comments

• [WIP]Refactor metrics

  #49629 commented on May 8, 2025 • 0 new comments

• [SPARK-49872][CORE] allow unlimited json size again

  #49163 commented on May 7, 2025 • 0 new comments

• [SPARK-49547][SQL][PYTHON] Add iterator of `RecordBatch` API to `applyInArrow`

  #49005 commented on May 6, 2025 • 0 new comments

• [SPARK-49984][CORE] Fix duplicate JVM options

  #48488 commented on May 5, 2025 • 0 new comments

• [SPARK-22876][YARN] Respect YARN AM failure validity interval

  #42570 commented on May 5, 2025 • 0 new comments

• [SPARK-44639][SS][YARN] Use Java tmp dir for local RocksDB state storage on Yarn

  #42301 commented on May 5, 2025 • 0 new comments

• [SPARK-37019][SQL] Add codegen support to array higher-order functions

  #34558 commented on May 5, 2025 • 0 new comments

• [SPARK-35564][SQL] Support subexpression elimination for conditionally evaluated expressions

  #32987 commented on May 5, 2025 • 0 new comments