[SPARK-44740][CONNECT][FOLLOW] Fix metadata values in Python client

### What changes were proposed in this pull request?
This is a follow-up PR that avoids putting the session ID into the metadata of the GRPC channel.

### Why are the changes needed?
Behavhior

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
UT

Closes apache#42452 from grundprinzip/SPARK-44740_v2.

Authored-by: Martin Grund <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>

Author: grundprinzip

python/pyspark/sql/connect/client/core.py

@@ -285,6 +285,7 @@ def metadata(self) -> Iterable[Tuple[str, str]]:
                 ChannelBuilder.PARAM_USE_SSL,
                 ChannelBuilder.PARAM_USER_ID,
                 ChannelBuilder.PARAM_USER_AGENT,
+                ChannelBuilder.PARAM_SESSION_ID,
             ]
         ]
 

python/pyspark/sql/tests/connect/test_connect_basic.py

@@ -3528,6 +3528,21 @@ def test_metadata(self):
         chan = ChannelBuilder(f"sc://host/;session_id={id}")
         self.assertEqual(id, chan.session_id)
 
+        chan = ChannelBuilder(f"sc://host/;session_id={id};user_agent=acbd;token=abcd;use_ssl=true")
+        md = chan.metadata()
+        for kv in md:
+            self.assertNotIn(
+                kv[0],
+                [
+                    ChannelBuilder.PARAM_SESSION_ID,
+                    ChannelBuilder.PARAM_TOKEN,
+                    ChannelBuilder.PARAM_USER_ID,
+                    ChannelBuilder.PARAM_USER_AGENT,
+                    ChannelBuilder.PARAM_USE_SSL,
+                ],
+                "Metadata must not contain fixed params",
+            )
+
         with self.assertRaises(ValueError) as ve:
             chan = ChannelBuilder("sc://host/;session_id=abcd")
             SparkConnectClient(chan)