Skip to content

Commit bebfca4

Browse files
Reuse or Create SparkSession
1 parent 94f001d commit bebfca4

File tree

1 file changed

+38
-1
lines changed

1 file changed

+38
-1
lines changed

docs/jvm/SparkSession.Builder.md

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ create(): SparkSession
2626

2727
`create` [setDefaultAndActiveSession](#setDefaultAndActiveSession) and then [applyOptions](#applyOptions).
2828

29-
## Try to Create New Session from SparkConnectClient { #tryCreateSessionFromClient }
29+
## Try to Create New Session for SparkConnectClient { #tryCreateSessionFromClient }
3030

3131
```scala
3232
tryCreateSessionFromClient(): Option[SparkSession]
@@ -47,3 +47,40 @@ Otherwise, `tryCreateSessionFromClient` returns no `SparkSession`.
4747
`tryCreateSessionFromClient` is used when:
4848

4949
* `SparkSession.Builder` is requested to [create a new SparkSession](#create) and [getOrCreate a SparkSession](#getOrCreate)
50+
51+
## Reuse or Create SparkSession { #getOrCreate }
52+
53+
??? note "SparkSessionBuilder"
54+
55+
```scala
56+
getOrCreate(): SparkSession
57+
```
58+
59+
`getOrCreate` is part of the `SparkSessionBuilder` ([Spark SQL]({{ book.spark_sql }}/SparkSessionBuilder.md#getOrCreate)) abstraction.
60+
61+
??? note "Public API"
62+
`getOrCreate` is part of the public API.
63+
64+
???+ warning "Generative AI"
65+
This description was co-authored using Generative AI tools, namely [JetBrains AI Assistant](https://www.jetbrains.com/ai/) (with the [openai-gpt-4o](https://openai.com/index/hello-gpt-4o/) model).
66+
67+
Prompt: "What does getOrCreate do? Please explain every line."
68+
69+
`getOrCreate` either returns an existing `SparkSession` (that matches the configuration provided) or creates a new `SparkSession` instance if no suitable session exists.
70+
71+
`getOrCreate` is used to ensure that a `SparkSession` is always available without having to worry whether a session already exists.
72+
73+
`getOrCreate` uses a caching mechanism alongside proper configuration and session updates.
74+
75+
---
76+
77+
`getOrCreate` [starts a local Spark Connect server](SparkSession.md#withLocalConnectServer) unless already started.
78+
79+
`getOrCreate` [attempts to create a session by directly reusing the current client (if it exists and is valid)](#tryCreateSessionFromClient).
80+
If a valid client exists, a new `SparkSession` is built and returned immediately.
81+
82+
Otherwise (if no valid client exists or the session is not reusable), `getOrCreate` finds the `SparkSession` (in the [sessions](SparkSession.md#sessions) by the [Configuration](#configuration)) or creates a session.
83+
84+
`getOrCreate` updates the global default and/or active `SparkSession` in the application.
85+
86+
`getOrCreate` applies [`spark.sql`-prefixed options](SparkSession.md#sparkOptions) and the builder's options to the new or existing session.

0 commit comments

Comments
 (0)