[feature] ASIC-focused multicast replication and dendrite API #14

zeeshanlakhani · 2025-04-01T17:03:14Z

This brings multicast group management and hardware-accelerated forwarding to Dendrite, providing the foundation for our multicast networking stack.

The PR includes:

API Layer:

Multicast group operations and lifecycle management
Integrated Source-Specific Multicast (SSM) support
Bulk reset functionality for clearing multicast state
Endpoints designed for integration with higher-level orchestration

Hardware Integration:

Updated sidecar.p4 for native Tofino ASIC multicast replication
ASIC table management for efficient multicast forwarding
Geneve option parsing for multicast tagging
Egress table handling for decapped packets and MAC rewriting
Range-based port bitmap matching for multicast egress (for decap)

Network Processing:

Leverages the Tofino packet replication engine for replication
Handles proper encapsulation/decapsulation flows
Bifurcated multicast handling: Separate processing paths for internal admin-scoped IPv6 groups vs external IPv4/IPv6 groups
Multiple replication scenarios:
- Unencapped external multicast → encapsulated for underlay transport
- Encapped IPv6 multicast → underlay-only replication (infrastructure traffic)
- Encapped IPv6 multicast → external-only replication with decapsulation
- Bifurcated replication: Both external members (decapped) and underlay members (encapped) from a group
Admin-scoped NAT target validation: External groups properly reference internal admin-scoped groups

Validation:

Extensive iperf3 testing shows no performance regression between baseline and multicast-enhanced versions
Comprehensive integration test suite covering:
- All replication scenarios (external-only, underlay-only, bifurcated)
- Encapsulation/decapsulation flows with Geneve multicast tagging
- Source-Specific Multicast (SSM) filtering with exact and prefix matching
- Admin-scoped NAT target validation and lifecycle management
- Performance validation with concurrent packet replication
- Edge cases including TTL handling, MAC derivation, and VLAN propagation

Context:

Implements the dendrite side of the bifurcated multicast design, handling both internal admin-scoped IPv6 groups (admin-local, site-local, and organization-local scopes) and external IPv4/IPv6 groups with NAT encapsulation
External multicast traffic gets encapsulated for underlay transport, then properly decapped and MAC-rewritten for local delivery
Internal groups stay within the admin scope for rack-local multicast communication
Supports complex replication patterns where single groups can serve both customer traffic (external, decapped) and infrastructure traffic (underlay, encapsulated)

Associated PRs

[dendrite] Bump to c44dd2504f2a623e640d94955376cdb5cadd7ed6 omicron#8576

Includes: * Multicast Group API management: add, modify, delete, reset, SSM handling * sidecar.p4 tofino_asic updates to integrate multicast packet replication * Table mangagement for ASIC switch tables for multicast * integration tests and test utility additions

zeeshanlakhani · 2025-04-15T19:38:25Z

I'll add some notes to the PR tomorrow morning (to explain through pieces).

dpd-client/tests/integration_tests/common.rs

dpd/src/counters.rs

dpd/p4/constants.p4

rcgoodfellow

Thanks Zeeshan. I've provided some comments on the API and am about half way through the P4 code. Have to run, but wanted to get these initial comments in.

dpd/src/api_server.rs

dpd/p4/sidecar.p4

rcgoodfellow · 2025-04-17T17:44:54Z

dpd/p4/sidecar.p4

+			( 0, false, true, true, _, USER_SPACE_SERVICE_PORT, true, _, _, _, _, _, _ ) : forward_from_userspace;
+			( 0, false, false, _, _, _, false, true, _, _, _, _, _ ) : forward_to_userspace;
+			( 0, false, false, _, true, _, _, _, _, _, _, _, _ ) : forward_to_userspace;
+			// Link-local multicast


Since we're changing quite a bit around link-local multicast, we need to test this e2e with ddm.

dpd/p4/sidecar.p4

FelixMcFelix

Thanks for this Zeeshan, some notes so far. I haven't yet made it through dpd/src/mcast.rs and the table management code yet.

dpd/src/api_server.rs

dpd-client/tests/integration_tests/mcast.rs

dpd-client/tests/integration_tests/table_tests.rs

dpd-client/tests/integration_tests/mcast.rs

dpd/p4/constants.p4

dpd/p4/parser.p4

dpd/p4/sidecar.p4

Includes: * make sure to avoid switch table apply on drop-specific code in sidecar

rcgoodfellow · 2025-07-16T05:24:41Z

Unless @Nieuwejaar has any more feedback I'd say let's get this landed.

Nieuwejaar · 2025-04-24T14:58:16Z

dpd/src/api_server.rs

+    let group_id = query_id.into_inner().group_id;
+
+    // If a group ID is provided, get the group by ID
+    if let Some(group_id) = group_id {


Oh, I see. This was a special case so you could get by id, since we already have an endpoint for ip. I missed that there was a similar special case for DELETE. That's pretty messy.

Is there a reason we can't have /multicast/groups/{group_id}? We already have a MulticastGroupIdParam::to_identifier() operation that could convert it from either format.

Nieuwejaar · 2025-07-16T12:40:32Z

.gitignore

 # Editor config
 .vscode
 .dir-locals.el
+bacon.toml


Should this be removed?

I had been using it (and it is a tool, but fine to remove it).

It's harmless. I just wasn't sure if you meant to leave it in or not. If you want to keep it, feel free.

Nieuwejaar · 2025-07-16T12:50:08Z

dpd/p4/constants.p4

+const int IPV6_NEIGHBOR_SIZE        = 512;  // ipv6 neighbor cache
+const int SWITCH_IPV4_ADDRS_SIZE    = 512;  // ipv4 addrs assigned to our ports
+const int SWITCH_IPV6_ADDRS_SIZE    = 512;  // ipv6 addrs assigned to our ports
+const int IPV4_MULTICAST_TABLE_SIZE = 1024; // multicast routing table(s) for IPv4


How did you arrive at a size of 1024 for all of the multicast tables? I suspect that when we add customer support for IPv6, we're going to have to bump up the existing IPv6 tables, and we don't have a lot of headroom. Unless we know that we need the multicast tables to be this big, I would be inclined to start with something much smaller and grow them if the customer hits the limit. That will be much lower risk than trying to shrink them after this is in use at a customer site.

@Nieuwejaar I thought 1024 was starting small, based on what mcast-interested customers had been bringing up (which was larger). I could break up the V6 constants as they're used for different table components, but maybe @rcgoodfellow has thoughts here?

If this is the size we need, then I guess it's the size we need.

I suspect we're going to need to start packaging multiple sidecar binaries, optimized for different customer sites. We're just not going to be able to support a large number of IPv4 routes, a large number of IPv6 routes, and large scale multicast all at the same time.

Yeah, I was a bit worried this would be the direction we have to move into.

It's not our only option. One of the challenges with big monolithic tables is that they have to fit in a single hardware stage. If we break up tables and allow them to be distributed across stages using approaches like algorithmic TCAM then we can potentially achieve a higher upper bound on total usable table space.

I think a table size of 1024 is a reasonable starting point, as we know this number will need to grow significantly to meet customer requirements.

Commit on main: c44dd2504f2a623e640d94955376cdb5cadd7ed6

Updates to include oxidecomputer/dendrite#14.

Introduce end-to-end multicast support across control plane and sled-agent, and integrate IP pool model extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances. Highlights: - DB: new multicast_group tables; member lifecycle management; pool_type/mvlan/switch_port_uplinks - API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback - Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking - Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests - Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present - Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/ Components: - Database schema: external and underlay multicast groups; member/instance association tables; IP pool enhancements (pool_type, mvlan, switch_port_uplinks) - Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence - API layer: endpoints and validation; default-VNI semantics when VPC not provided - Sled agent: OPTE stubs and compatibility shims for older agents Workflows Implemented: 1. Instance lifecycle integration: - "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW - "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack - "Stop" -> deactivate dataplane membership; retain DB membership for fast restart - "Delete" -> remove instance memberships; group deletion is explicit - "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees - Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming 2. RPW reconciliation: - ensure dataplane switches match database state - handle sled migrations and state transitions - Eventual consistency with retry logic Migrations: - Apply schema changes in schema/crdb/multicast-support/up01.sql (and update dbinit.sql) - Bump schema versions accordingly API/Compatibility: - OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json - Regenerate clients where applicable References: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 Follow-ups include: - OPTE integration - commtest extension - omdb commands are tracked in issues - pool and group stats

This work introduces multicast IP pool capabilities to support external multicast traffic routing through the rack's switching infrastructure. Includes: - Add IpPoolType enum (unicast/multicast) with unicast as default - Add multicast pool fields: switch_port_uplinks (UUID[]), mvlan (VLAN ID) - Add database migration (multicast-support/up01.sql) with new columns and indexes - Add ASM/SSM range validation for multicast pools to prevent mixing - Add pool type-aware resolution for IP allocation - Add custom deserializer for switch port uplinks with deduplication - Update external API params/views for multicast pool configuration - Add SSM constants (IPV4_SSM_SUBNET, IPV6_SSM_FLAG_FIELD) for validation Database schema updates: - ip_pool table: pool_type, switch_port_uplinks, mvlan columns - Index on pool_type for efficient filtering - Migration preserves existing pools as unicast type by default This provides the foundation for multicast group functionality while maintaining full backward compatibility with existing unicast pools. References (for review): - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14

Introduce end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances. Highlights: - DB: new multicast_group tables; member lifecycle management - API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback - Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking - Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests - Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present - Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/ Components: - Database schema: external and underlay multicast groups; member/instance association tables - Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence - API layer: endpoints and validation; default-VNI semantics when VPC not provided - Sled agent: OPTE stubs and compatibility shims for older agents Workflows Implemented: 1. Instance lifecycle integration: - "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW - "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack - "Stop" -> deactivate dataplane membership; retain DB membership for fast restart - "Delete" -> remove instance memberships; group deletion is explicit - "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees - Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming 2. RPW reconciliation: - ensure dataplane switches match database state - handle sled migrations and state transitions - Eventual consistency with retry logic Migrations: - Apply schema changes in schema/crdb/multicast-support/up01.sql (and update dbinit.sql) - Bump schema versions accordingly API/Compatibility: - OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json - Regenerate clients where applicable References: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 Follow-ups include: - OPTE integration - commtest extension - omdb commands are tracked in issues - pool and group stats

Introduces end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances. Highlights: - DB: new multicast_group tables; member lifecycle management - API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback - Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking - Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests - Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present - Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/ Components: - Database schema: external and underlay multicast groups; member/instance association tables - Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence - API layer: endpoints and validation; default-VNI semantics when VPC not provided - Sled agent: OPTE stubs and compatibility shims for older agents Workflows Implemented: 1. Instance lifecycle integration: - "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW - "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack - "Stop" -> deactivate dataplane membership; retain DB membership for fast restart - "Delete" -> remove instance memberships; group deletion is explicit - "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees - Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming 2. RPW reconciliation: - ensure dataplane switches match database state - handle sled migrations and state transitions - Eventual consistency with retry logic Migrations: - Apply schema changes in schema/crdb/multicast-support/up01.sql (and update dbinit.sql) - Bump schema versions accordingly API/Compatibility: - OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json - Regenerate clients where applicable References: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - IP Pool extensions: #9084 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 Follow-ups include: - OPTE integration - commtest extension - omdb commands are tracked in issues - pool and group stats

Introduces end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances. Highlights: - DB: new multicast_group tables; member lifecycle management - API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback - Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking - Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests - Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present - Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/ Components: - Database schema: external and underlay multicast groups; member/instance association tables - Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence - API layer: endpoints and validation; default-VNI semantics when VPC not provided - Sled agent: OPTE stubs and compatibility shims for older agents Workflows Implemented: 1. Instance lifecycle integration: - "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW - "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack - "Stop" -> deactivate dataplane membership; retain DB membership for fast restart - "Delete" -> remove instance memberships; group deletion is explicit - "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees - Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming 2. RPW reconciliation: - ensure dataplane switches match database state - handle sled migrations and state transitions - Eventual consistency with retry logic Migrations: - Apply schema changes in schema/crdb/multicast-group-support/up01.sql (and update dbinit.sql) - Bump schema versions accordingly API/Compatibility: - OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json - Contains a version change (to v5) as InstanceEnsureBody has been modified to include multicast_groups associated with an instance in the underlying sled config - Regenerate clients where applicable References: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - IP Pool extensions: #9084 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 Follow-ups include: - OPTE integration - commtest extension - omdb commands are tracked in issues - pool and group stats

zeeshanlakhani force-pushed the zl/p4-mcast branch 3 times, most recently from 4e2fc4b to 1ce4908 Compare April 2, 2025 02:40

zeeshanlakhani force-pushed the zl/p4-mcast branch from 1ce4908 to df62508 Compare April 15, 2025 18:10

zeeshanlakhani changed the title ~~wip~~ [feature] ASIC-focused 1st draft of multicast PRE Apr 15, 2025

[fix] change stage max

4a945a8

zeeshanlakhani force-pushed the zl/p4-mcast branch from 2426e18 to 4a945a8 Compare April 15, 2025 19:36

zeeshanlakhani requested review from rcgoodfellow and Nieuwejaar April 15, 2025 19:37

zeeshanlakhani marked this pull request as ready for review April 15, 2025 19:37

zeeshanlakhani requested a review from FelixMcFelix April 15, 2025 19:38

zeeshanlakhani commented Apr 16, 2025

View reviewed changes

dpd-client/tests/integration_tests/common.rs Show resolved Hide resolved

update chaos tables, revert any p4 formatting for review

bb876fa

zeeshanlakhani commented Apr 16, 2025

View reviewed changes

dpd/src/counters.rs Show resolved Hide resolved

[minor] comments/align

2dcb98e

zeeshanlakhani commented Apr 16, 2025

View reviewed changes

dpd/p4/constants.p4 Outdated Show resolved Hide resolved

rcgoodfellow reviewed Apr 17, 2025

View reviewed changes

FelixMcFelix reviewed Apr 17, 2025

View reviewed changes

zeeshanlakhani added 3 commits April 21, 2025 20:05

Merge remote-tracking branch 'origin/multicast' into zl/p4-mcast-3

3ef78f0

[review] address first pass of review

37498e4

[review+] meta -> metadata, fix test issues

d4953df

Includes: * make sure to avoid switch table apply on drop-specific code in sidecar

zeeshanlakhani requested review from FelixMcFelix and rcgoodfellow April 21, 2025 21:43

zeeshanlakhani and others added 2 commits April 21, 2025 23:34

[stages] test back to 14

dbe684f

..

5537794

This was referenced Apr 22, 2025

Moving to Multicast: Look into Warnings found in Tofinio Simulator #34

Open

Egress (port) counters return 0 even when counters are updated in Tofino Simulator #35

Open

zeeshanlakhani requested a review from taspelund April 22, 2025 18:39

Nieuwejaar reviewed Jul 16, 2025

View reviewed changes

Nieuwejaar approved these changes Jul 16, 2025

View reviewed changes

zeeshanlakhani merged commit c44dd25 into main Jul 16, 2025
6 checks passed

zeeshanlakhani deleted the zl/p4-mcast branch July 16, 2025 15:51

zeeshanlakhani added a commit to oxidecomputer/omicron that referenced this pull request Jul 17, 2025

[dendrite] update for merged-in oxidecomputer/dendrite#14

ff0e3c6

Commit on main: c44dd2504f2a623e640d94955376cdb5cadd7ed6

zeeshanlakhani added a commit to oxidecomputer/omicron that referenced this pull request Jul 17, 2025

[dendrite] update for merged-in oxidecomputer/dendrite#14

2379efb

Commit on main: c44dd2504f2a623e640d94955376cdb5cadd7ed6

zeeshanlakhani mentioned this pull request Jul 17, 2025

[dendrite] Bump to c44dd2504f2a623e640d94955376cdb5cadd7ed6 oxidecomputer/omicron#8576

Merged

zeeshanlakhani added a commit to oxidecomputer/omicron that referenced this pull request Jul 17, 2025

[dendrite] Bump to c44dd2504f2a623e640d94955376cdb5cadd7ed6 (#8576)

3ea2ffd

Updates to include oxidecomputer/dendrite#14.

zeeshanlakhani mentioned this pull request Sep 24, 2025

[feat] Multicast groups and IP pool extensions oxidecomputer/omicron#9069

Closed

zeeshanlakhani mentioned this pull request Sep 25, 2025

[feat, multicast] Add IP pool multicast support oxidecomputer/omicron#9084

Open

1 task

zeeshanlakhani mentioned this pull request Sep 25, 2025

[feat, multicast] Multicast Group Support oxidecomputer/omicron#9091

Open

[feature] ASIC-focused multicast replication and dendrite API #14

[feature] ASIC-focused multicast replication and dendrite API #14

Uh oh!

Conversation

zeeshanlakhani commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeeshanlakhani commented Apr 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

FelixMcFelix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow commented Jul 16, 2025

Uh oh!

Nieuwejaar Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

Nieuwejaar Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nieuwejaar Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Nieuwejaar Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nieuwejaar Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rcgoodfellow Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Apr 1, 2025 •

edited

Loading

zeeshanlakhani Jul 16, 2025 •

edited

Loading

zeeshanlakhani Jul 16, 2025 •

edited

Loading

zeeshanlakhani Jul 16, 2025 •

edited

Loading