Diagram editor

[koonpeng]

New feature implementation

Implemented feature

This still needs more testing before I would consider it stable, but I thought this is a good time to merge to main.

Even though the change is huge, most of the changes is in diagram-editor so there shouldn't be much conflicts.

GenAI Use

We follow OSRA's policy on GenAI tools

• I used a GenAI tool in this PR.
• I did not use GenAI

Generated-by: Gemini

[koonpeng]

Discrepancies between the displayed state and the actual state

Turns out this is due to reactflow only returning the new values on the next render, fixed by avoiding use of reactflow's state.

Confusing Join input

Added labels to buffer edges.

Unable to choose Key for Join connection

fixed

The icons for Fork Result and Split should be flipped upside-down. I would also recommend using a question mark in some way for Fork Result as a reference to its similarity to the Rust question mark operator.

updated the icons.

I like that buffer "outputs" and join/listen "inputs" have the same color to indicate that they can be connected to each other. I think we should do the same for ordinary inputs and outputs. I don't think it makes sense that ordinary outputs are white and ordinary inputs are orange when they can connect to each other. If we want to distinguish inputs from outputs then maybe output icons can be little downward pointing triangles instead of circles.

Which operation did you see the inputs to be orange? Both ordinary inputs and outputs should be white-ish. Orange is for stream outputs.

[mxgrey]

Which operation did you see the inputs to be orange? Both ordinary inputs and outputs should be white-ish. Orange is for stream outputs.

In the videos you can see that mul has an orange output icon, but it doesn't produce any streams.

Would ReactFlow be able to support having streams come out of the side (maybe right side) of the box instead of the bottom?

[koonpeng]

Which operation did you see the inputs to be orange? Both ordinary inputs and outputs should be white-ish. Orange is for stream outputs.

In the videos you can see that mul has an orange output icon, but it doesn't produce any streams.

Would ReactFlow be able to support having streams come out of the side (maybe right side) of the box instead of the bottom?

mul is a node operation, it can output either ordinary edges or "stream out" edges hence it is both white and orange. Unless I misunderstood how streams are connected, node operations and node builders do not report the streams they output so we must assume all node operations may output a stream. But even if we know which node operations may output streams, we can't display it on the UI atm as the edge handles are fixed based on the type of operation atm.

Would ReactFlow be able to support having streams come out of the side (maybe right side) of the box instead of the bottom?

done.

[mxgrey]

I've tried out the editor with the recent fixes, and the behaviors are really good now!

node operations and node builders do not report the streams they output so we must assume all node operations may output a stream.

You're right, this is something I forgot to follow up on when I added the ability for diagrams to support streams. I'll try to get streams into the node builder registry ASAP.

But even if we know which node operations may output streams, we can't display it on the UI atm as the edge handles are fixed based on the type of operation atm.

It's not urgent but I think this is worth following up on in a future PR. I'll open an issue ticket for this.

While expanding on the calculator example, I added a fibonacci node builder that streams out numbers of the Fibonacci sequence. It seems the UI does not allow you to connect a stream to an input slot:

stream_connection_bug.mp4

Furthermore if I create the diagram manually with the intended "stream_out" setup and then upload it to the editor, the UI displays it as connected to the node output:

When I look at what kind of diagram gets generated from this, the print operation is now connected to the output of fibonacci instead of connected to its stream.

I would suggest we do one of the following before merging:

• Since there are still some loose ends with streams (i.e. the stream keys aren't being registered), we could just remove stream support from the diagram editor and leave it for a follow-up.
• If it's not too difficult to fix the current bugs, the first draft of support for streams could allow users to manually type in the key string of each connection coming out of a stream output. The user would have to know the names of the stream outputs, but at least a roundtrip of uploading a diagram and then re-exporting it would work correctly.

[koonpeng]

I mistakenly thought that stream outputs and only connect to stream outs. It should be fixed now.

[mxgrey]

This is an excellent first release for the diagram editor! I think any remaining follow-up items can be addressed in incremental PRs.