You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/jvm/SparkSession.Builder.md
+38-1Lines changed: 38 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ create(): SparkSession
26
26
27
27
`create`[setDefaultAndActiveSession](#setDefaultAndActiveSession) and then [applyOptions](#applyOptions).
28
28
29
-
## Try to Create New Session from SparkConnectClient { #tryCreateSessionFromClient }
29
+
## Try to Create New Session for SparkConnectClient { #tryCreateSessionFromClient }
30
30
31
31
```scala
32
32
tryCreateSessionFromClient():Option[SparkSession]
@@ -47,3 +47,40 @@ Otherwise, `tryCreateSessionFromClient` returns no `SparkSession`.
47
47
`tryCreateSessionFromClient` is used when:
48
48
49
49
*`SparkSession.Builder` is requested to [create a new SparkSession](#create) and [getOrCreate a SparkSession](#getOrCreate)
50
+
51
+
## Reuse or Create SparkSession { #getOrCreate }
52
+
53
+
??? note "SparkSessionBuilder"
54
+
55
+
```scala
56
+
getOrCreate(): SparkSession
57
+
```
58
+
59
+
`getOrCreate` is part of the `SparkSessionBuilder` ([Spark SQL]({{ book.spark_sql }}/SparkSessionBuilder.md#getOrCreate)) abstraction.
60
+
61
+
??? note "Public API"
62
+
`getOrCreate` is part of the public API.
63
+
64
+
???+ warning "Generative AI"
65
+
This description was co-authored using Generative AI tools, namely [JetBrains AI Assistant](https://www.jetbrains.com/ai/) (with the [openai-gpt-4o](https://openai.com/index/hello-gpt-4o/) model).
66
+
67
+
Prompt: "What does getOrCreate do? Please explain every line."
68
+
69
+
`getOrCreate` either returns an existing `SparkSession` (that matches the configuration provided) or creates a new `SparkSession` instance if no suitable session exists.
70
+
71
+
`getOrCreate` is used to ensure that a `SparkSession` is always available without having to worry whether a session already exists.
72
+
73
+
`getOrCreate` uses a caching mechanism alongside proper configuration and session updates.
74
+
75
+
---
76
+
77
+
`getOrCreate`[starts a local Spark Connect server](SparkSession.md#withLocalConnectServer) unless already started.
78
+
79
+
`getOrCreate`[attempts to create a session by directly reusing the current client (if it exists and is valid)](#tryCreateSessionFromClient).
80
+
If a valid client exists, a new `SparkSession` is built and returned immediately.
81
+
82
+
Otherwise (if no valid client exists or the session is not reusable), `getOrCreate` finds the `SparkSession` (in the [sessions](SparkSession.md#sessions) by the [Configuration](#configuration)) or creates a session.
83
+
84
+
`getOrCreate` updates the global default and/or active `SparkSession` in the application.
85
+
86
+
`getOrCreate` applies [`spark.sql`-prefixed options](SparkSession.md#sparkOptions) and the builder's options to the new or existing session.
0 commit comments