Support for writing to existing Iceberg tables in S3 from C# service in ECS

[MPeli]

Hello .NET for Apache Spark team,

We're exploring the possibility of using .NET for Apache Spark to optimize our current data pipeline. Our scenario is as follows:

Current setup:

• C# service running in Amazon ECS writes data to an Amazon Kinesis stream
• AWS Glue job reads from this stream and writes to Iceberg tables in S3

Desired setup:

• C# service running in Amazon ECS writes directly to the same Iceberg tables in S3 that our Glue jobs are currently writing to, bypassing Kinesis and Glue

Our main question:

Is it possible to use .NET for Apache Spark within a C# service running in ECS to write directly to existing Iceberg tables in S3? If so, could you provide a high-level overview of how this might be implemented?

We appreciate any insights or documentation you can provide to help us evaluate the feasibility of this approach.

Thank you for your time and assistance!

[wudanzy]

I think currently that's not supported, to use Spark you will need to deploy it into a cluster management framework, such as Yarn or Kubernetes, or use the spark standalone cluster. And then you can use it.

In our case, I don't have much knowledge on ECS, but from the name it sounds like a framework that can allow you to run arbitrary services. In such case, you may bring up a spark standalone cluster, e.g. spark master on some machines and spark workers on others.

You will need to prototype on the idea to see whether it works.