Consider integrating open3fs into K8s ?

[linuxflow-hqukz]

This article , Setup 3FS on Kubernetes.
https://github.com/AliyunContainerService/alibabacloud-3FS/blob/main/deploy/container/README.md

[diabloneo]

We considered this when we started this project. We found that using k8s brings a problem about RDMA network: the pod of a service must be able to access a RDMA nic. To achieve this, you can use NVIDIA network-operator or DaoCloud Spiderpool to pass an additional RDMA NIC dedicatedly to a pod. But, in a on-prem environment, each server has limited number of physical NIC cards, which means in most cases every server can only run one 3FS service because it will use all RDMA NIC on that server.

Back to this Aliyun's solution, the reason they can deploy 3FS on ACK (Alibaba Cloud Container Service For Kubernetes) without worrying this problem is because ACK can provide ERDMA resource for each pod. Take a look at the resources section of the yaml files, pay attention to the aliyun/erdma configuration:

    spec:
      containers:
      - name: mgmtd
        image: registry-vpc.cn-beijing.aliyuncs.com/huweiwen-test/3fs-mgmtd:dev
        imagePullPolicy: Always
        args:
        - --cfg=/etc/mgmtd/mgmtd_main.toml
        - --app_config.node_id=$(POD_INDEX)
        - --launcher_config.allow_dev_version=true # FIXME
        - --launcher_config.cluster_id={{.Values.clusterID}}
        resources:
          requests:
            aliyun/erdma: "1"
            memory: "512Mi"
            cpu: "500m"
          limits:
            aliyun/erdma: "1"
            memory: "2Gi"
            cpu: "2"

So, my opinions now is: If you want to deploy 3FS in a on-prem environment, I prefer not introduce k8s until the k8s platform provides similar network features like ACK's ERDMA.

[linuxflow-hqukz]

Can Soft RoCE be used in K8S to complete the deployment of Open3FS? I have previously used Soft RoCE and csi-driver-nvmf to achieve RDMA mounting of hard disks. I don’t know if this is helpful to you.

the pod of a service must be able to access a RDMA nic

我们在开始这个项目时考虑了这一点。我们发现使用k8s带来了一个关于RDMA网络的问题：服务的pod必须能够访问RDMA NIC。为了实现这一点，您可以使用NVIDIA网络运营商或DaoCloud Spiderpool将额外的RDMA NIC专门传递给pod。但是，在本地环境中，每台服务器的物理NIC卡数量有限，这意味着在大多数情况下，每台服务器只能运行一个3FS服务，因为它将使用该服务器上的所有RDMA NIC。

ACK can provide ERDMA resource for each pod****aliyun/erdma

回到阿里云的解决方案，他们之所以可以在确认字符上部署3FS（阿里云Container Service For库伯内特斯）而不用担心这个问题，是因为确认字符可以为每个pod提供ERDMA资源。看看yaml文件的资源部分，注意阿里云/erdma配置：

    spec:
      containers:
      - name: mgmtd
        image: registry-vpc.cn-beijing.aliyuncs.com/huweiwen-test/3fs-mgmtd:dev
        imagePullPolicy: Always
        args:
        - --cfg=/etc/mgmtd/mgmtd_main.toml
        - --app_config.node_id=$(POD_INDEX)
        - --launcher_config.allow_dev_version=true # FIXME
        - --launcher_config.cluster_id={{.Values.clusterID}}
        resources:
          requests:
            aliyun/erdma: "1"
            memory: "512Mi"
            cpu: "500m"
          limits:
            aliyun/erdma: "1"
            memory: "2Gi"
            cpu: "2"

So, my opinions now is: If you want to deploy 3FS in a on-prem environment, I prefer not introduce k8s until the k8s platform provides similar network features like ACK's ERDMA.

所以，我现在的观点是：如果您想在本地环境中部署3FS，我更愿意在k8s平台提供类似的网络功能（如确认字符的ERDMA）之前不引入k8s。

Can Soft RoCE be used in K8S to complete the deployment of Open3FS? I have previously used Soft RoCE and csi-driver-nvmf to achieve RDMA mounting of hard disks. I don’t know if this is helpful to you.

[diabloneo]

I think soft RoCE can be used. You can use it as a normal RoCE link. But you still have to pass the link to the pod.