FV changes

[mansijain52]

• Supported spark 3.4
  Added scala code for spark 3.4
  Made changes to it compatible for spark 3.4

• Changes to support c# in latest synapse runtime
  Renamed DotnetRunner class to Runner and its references.
  This change is mainly needed for using dotnet for spark library in Synapse latest runtime. As we observed that SYnapse has a check at their end to look for this keyword and block the execution.
  With this,we are able to run dotnet job in Synapse latest runtime using custom spark worker library

[wudanzy]

Hi @mansijain52, are you still working on this?

[mansijain52]

Hi @mansijain52, are you still working on this?

Yes. We are done with changes. Waiting to get this merged

[mansijain52]

@mansijain52 please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@dotnet-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@dotnet-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@dotnet-policy-service agree company="Microsoft"

Contributor License Agreement

@dotnet-policy-service agree company="Microsoft"

[wudanzy]

/AzurePipelines run

[azure-pipelines]

Pull request contains merge conflicts.

[wudanzy]

Thanks for confirming! Let wait for #1197

[wudanzy]

Hi @mansijain52, please resolve your conflicts and proceed with this PR, thanks!

[wudanzy]

Hi @mansijain52, will you continue work on this?

[mansijain52]

Yes. I will complete it this week

…

On Wed, 26 Feb, 2025, 09:50 Dan Wu, ***@***.***> wrote: Hi @mansijain52 <https://github.com/mansijain52>, will you continue work on this? — Reply to this email directly, view it on GitHub <#1177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALRCDNQZJ4RDR6UMNHEEYG32RU6IXAVCNFSM6AAAAABIYAAKA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTHA2TKOBTGE> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: wudanzy]*wudanzy* left a comment (dotnet/spark#1177) <#1177 (comment)> Hi @mansijain52 <https://github.com/mansijain52>, will you continue work on this? — Reply to this email directly, view it on GitHub <#1177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALRCDNQZJ4RDR6UMNHEEYG32RU6IXAVCNFSM6AAAAABIYAAKA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTHA2TKOBTGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mansijain52]

Hi @mansijain52, will you continue work on this?

Ready for review

[wudanzy]

/AzurePipelines run

[wudanzy]

Thanks for the change!

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[wudanzy]

src\csharp\Microsoft.Spark\TaskContext.cs#L28

src\csharp\Microsoft.Spark\TaskContext.cs(28,22): error CS0102: (NETCORE_ENGINEERING_TELEMETRY=Build) The type 'TaskContext' already contains a definition for 'CPUs'

[wudanzy]

Maybe a conflict, because previous upgrades (3.3, 3.5) already added this field.

[mansijain52]

Maybe a conflict, because previous upgrades (3.3, 3.5) already added this field.

Updated

[mansijain52]

@wudanzy @SparkSnail Pending review!

[azure-pipelines]

Commenter does not have sufficient privileges for PR 1177 in repo dotnet/spark

[wudanzy]

Hi @mansijain52, the spark 3.4 was already added and a new release was made. Please check if that already meets your need.