[CORE] Let LocalSparkContext clear active context in beforeAll

[price-qian]

What changes were proposed in this pull request?

This change is to ensure no active SparkContext remain from previous test runs. This prevents potential resource leaks by cleaning up any lingering context before test execution begins. By having this call in the trait, we stabilize all tests that mix in this trait.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Ran all unit tests on 3.5 branch and all passed.

Was this patch authored or co-authored using generative AI tooling?

No.

[sarutak]

@price-qian Thank you for opening this PR.
Unfortunately, I don't think this change needed because resetSparkContext in afterEach calls clearActiveContext indirectly. So, do you mind closing this PR?

Also, please note:

• Give a proper tag to the title of PR like [SPARK-XXXXX] ([MINOR] is acceptable if a change is very small like typo-fix).
• Activate your GitHub Actions to run CI. See this error message.
• Follow our contribution guide.

[price-qian]

Thanks @sarutak for the review. Tests that mix in this trait will clean up SparkContext properly. The issue is when some tests do not mix in it and forget to clean up SparkContext. When running alone, they can all pass but when running together, the lingering SparkContext can cause issues when a different test tries to do new SparkContext because SparkContext doesn't allow multiple active instances. This PR is to make it more defensive in this shared trait. I can adjust based on your notes too but just want to share this thought with you first.

[price-qian]

I've activated my Github Actions but I don't see a way to rerun this workflow.

[sarutak]

@price-qian
I understand the issue you argue. But I have never encountered such an issue when I run tests, so could you show the way to reproduce this issue for us to judge if the suggested solution is the best way?
Also, please file this issue in JIRA and note the issue and solution more precisely.

I've activated my Github Actions but I don't see a way to rerun this workflow.

If you configured correctly, GA should run after the next push.