Segment union (mcmonkeyprojects#894)

* Add syntax for merging multiple segment masks into a single segment.

`<segment:face || hair || yolo-skin_yolov8m-seg_60.pt> prompt for face and hair and skin`

Also supports a threshold for each part of the mask, by appending a semicolon and the threshold

`<segment:face;0.5 || hair, 0.4, 0.6>` will apply 0.5 threshold to the face mask and the overall segment threshold (0.6) to the hair mask.

* Update UI to support multiple segment masks

* Update Prompt Syntax with the new segment syntax

* Revert segment modal window changes to keep it simple for beginners.

* Update SwarmClipSeg node to emit mask with 3 dimensions.

* Update SwarmYolo node to emit masks with 3 dimensions (instead of sometimes 2)

* simplify the PR to just the focal point

* single-pipe

---------

Co-authored-by: Alex "mcmonkey" Goodwin <[email protected]>

Author: bman654

docs/Features/Prompt Syntax.md

@@ -150,6 +150,11 @@
         - To control the creativity/threshold with a yolo model just append `,<creativity>,<threshold>`, for example `<segment:yolo-face_yolov8m-seg_60.pt-1,0.8,0.25>` sets a `0.8` creativity and `0.25` threshold.
             - Note the default "confidence threshold" for Yolo models is `0.25`, which is different than is often used with ClipSeg, and does not have a "max threshold" like ClipSeg does.
         - If you have a yolo model with multiple supported classes, you can filter specific classes by appending `:<classes>:` to the model name where `<classes>` is a comma-separated list of class IDs or names, e.g., `<segment:yolo-modelnamehere:0,apple,2:,0.8,0.25>`
+    - You can also combine multiple areas into a single segment to refine them as a single group.
+        - Separate the areas with `|` in `texthere`
+        - For example, `<segment:face|hair>` will find all the faces and hair in the image and refine them as a single group.
+        - This works with YOLOv8 models as well.
+            - `<segment:yolo-face_yolov8m-seg_60.pt | yolo-hair_yolov8m-seg_60.pt | fingers>` will refine the group of faces and hair (found by YOLO) and fingers (found by CLIPSeg) as a single group.
     - There's an advanced parameter under `Segment Refining` named `Segment Model` to customize the base model used for segment processing
     - There's also a parameter named `Save Segment Mask` to save a preview copy of the generated mask
 

src/BuiltinExtensions/ComfyUIBackend/ExtraNodes/SwarmComfyCommon/SwarmClipSeg.py

@@ -63,7 +63,7 @@ def seg(self, images, match_text, threshold):
             mask /= max
         while mask.ndim < 4:
             mask = mask.unsqueeze(0)
-        mask = torch.nn.functional.interpolate(mask, size=(images.shape[1], images.shape[2]), mode="bilinear").squeeze(0).squeeze(0)
+        mask = torch.nn.functional.interpolate(mask, size=(images.shape[1], images.shape[2]), mode="bilinear").squeeze(0)
         return (mask,)
 
 NODE_CLASS_MAPPINGS = {

src/BuiltinExtensions/ComfyUIBackend/ExtraNodes/SwarmComfyExtra/SwarmYolo.py

@@ -83,9 +83,9 @@ def seg(self, image, model_name, index, class_filter=None, sort_order="left-righ
             result = masks[0]
             for i in range(1, len(masks)):
                 result = torch.max(result, masks[i])
-            return (result, )
+            return (result.unsqueeze(0), )
         elif index > len(masks):
-            return (torch.zeros_like(masks[0]), )
+            return (torch.zeros_like(masks[0]).unsqueeze(0), )
         else:
             sortedindices = []
             for mask in masks:

src/BuiltinExtensions/ComfyUIBackend/WorkflowGeneratorSteps.cs

@@ -1342,45 +1342,73 @@ JArray doMaskShrinkApply(WorkflowGenerator g, JArray imgIn)
                 for (int i = 0; i < parts.Length; i++)
                 {
                     PromptRegion.Part part = parts[i];
-                    string segmentNode;
-                    if (part.DataText.StartsWith("yolo-"))
-                    {
-                        string fullname = part.DataText.After("yolo-");
-                        string[] modelParts = fullname.Split(':');
-                        fullname = modelParts[0];
-                        string classFilter = modelParts.Length > 1 ? modelParts[1] : "";
-                        (string mname, string indexText) = fullname.BeforeAndAfterLast('-');
-                        if (!string.IsNullOrWhiteSpace(indexText) && int.TryParse(indexText, out int index))
-                        {
-                            fullname = mname;
+                    string[] segmentSections = part.DataText.Split('|', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
+                    string segmentNode = null;
+                    foreach (string dataText in segmentSections)
+                    {
+                        string newSegmentNode = null;
+                        if (dataText.StartsWith("yolo-"))
+                        {
+                            string fullname = dataText.After("yolo-");
+                            string[] modelParts = fullname.Split(':');
+                            fullname = modelParts[0];
+                            string classFilter = modelParts.Length > 1 ? modelParts[1] : "";
+                            (string mname, string indexText) = fullname.BeforeAndAfterLast('-');
+                            if (!string.IsNullOrWhiteSpace(indexText) && int.TryParse(indexText, out int index))
+                            {
+                                fullname = mname;
+                            }
+                            else
+                            {
+                                index = 0;
+                            }
+                            if (part.Strength > 0.999)
+                            {
+                                Logs.Warning($"Yolo confidence threshold is set to 1. This was recommended syntax before yolo thresholds were supported, but is no longer valid. Swarm will automatically reset the value to default (0.25) instead.");
+                                part.Strength = 0.25;
+                            }
+                            newSegmentNode = g.CreateNode("SwarmYoloDetection", new JObject()
+                            {
+                                ["image"] = g.FinalImageOut,
+                                ["model_name"] = fullname,
+                                ["index"] = index,
+                                ["class_filter"] = classFilter,
+                                ["sort_order"] = g.UserInput.Get(T2IParamTypes.SegmentSortOrder, "left-right"),
+                                ["threshold"] = Math.Abs(part.Strength)
+                            });
                         }
                         else
                         {
-                            index = 0;
+                            newSegmentNode = g.CreateNode("SwarmClipSeg", new JObject()
+                            {
+                                ["images"] = g.FinalImageOut,
+                                ["match_text"] = dataText,
+                                ["threshold"] = Math.Abs(part.Strength)
+                            });
                         }
-                        if (part.Strength > 0.999)
+                        if (segmentSections.Length > 1 && g.UserInput.Get(T2IParamTypes.SaveSegmentMask, false))
                         {
-                            Logs.Warning($"Yolo confidence threshold is set to 1. This was recommended syntax before yolo thresholds were supported, but is no longer valid. Swarm will automatically reset the value to default (0.25) instead.");
-                            part.Strength = 0.25;
+                            string imageNode = g.CreateNode("MaskToImage", new JObject()
+                            {
+                                ["mask"] = new JArray() { newSegmentNode, 0 }
+                            });
+                            g.CreateImageSaveNode([imageNode, 0], g.GetStableDynamicID(50000, 0));
                         }
-                        segmentNode = g.CreateNode("SwarmYoloDetection", new JObject()
+                        if (segmentNode is null)
                         {
-                            ["image"] = g.FinalImageOut,
-                            ["model_name"] = fullname,
-                            ["index"] = index,
-                            ["class_filter"] = classFilter,
-                            ["sort_order"] = g.UserInput.Get(T2IParamTypes.SegmentSortOrder, "left-right"),
-                            ["threshold"] = Math.Abs(part.Strength)
-                        });
-                    }
-                    else
-                    {
-                        segmentNode = g.CreateNode("SwarmClipSeg", new JObject()
+                            segmentNode = newSegmentNode;
+                        }
+                        else
                         {
-                            ["images"] = g.FinalImageOut,
-                            ["match_text"] = part.DataText,
-                            ["threshold"] = Math.Abs(part.Strength)
-                        });
+                            segmentNode = g.CreateNode("MaskComposite", new JObject()
+                            {
+                                ["destination"] = new JArray() { segmentNode, 0 },
+                                ["source"] = new JArray() { newSegmentNode, 0 },
+                                ["operation"] = "add",
+                                ["x"] = 0,
+                                ["y"] = 0
+                            });
+                        }
                     }
                     if (part.Strength < 0)
                     {

src/wwwroot/js/genpage/gentab/prompttools.js

@@ -76,7 +76,7 @@ class PromptTabCompleteClass {
                     return this.getOrderedMatches(yolomodels.map(m => `yolo-${m}`), prefixLow);
                 }
             }
-            return ['\nSpecify before the ">" some text to match against in the image, like "<segment:face>".', '\nCan also do "<segment:text,creativity,threshold>" eg "face,0.6,0.5" where creativity is InitImageCreativity, and threshold is mask matching threshold for CLIP-Seg.', '\nYou can use a negative threshold value like "<segment:face,0.6,-0.5>" to invert the mask.', '\nYou may use the "yolo-" prefix to use a YOLOv8 seg model,', '\nor format "yolo-<model>-1" to get specifically the first result from a YOLOv8 match list.', '\n Additionally, you can apply a class filter by appending "yolo-<model>:<class_ids>:" where <class_ids> is a comma-separated list of class IDs or names to filter the detection results.'];
+            return ['\nSpecify before the ">" some text to match against in the image, like "<segment:face>".', '\nCan also do "<segment:text,creativity,threshold>" eg "face,0.6,0.5" where creativity is InitImageCreativity, and threshold is mask matching threshold for CLIP-Seg.', '\nYou can use a negative threshold value like "<segment:face,0.6,-0.5>" to invert the mask.', '\nYou may use the "yolo-" prefix to use a YOLOv8 seg model,', '\nFor more advanced usages and a link to relevant docs, click the "+" button next to the prompt box, then "Auto Segment Refinement".'];
         });
         this.registerPrefix('setvar[var_name]', 'Store text for reference later in the prompt', (prefix) => { 
             return ['\nSave the content of the tag into the named variable. eg "<setvar[colors]: red and blue>", then use like "<var:colors>"', '\nVariables can include the results of other tags. eg "<setvar[expression]: <random: smiling|frowning|crying>>"', '\nReference stored values later in the prompt with the <var:> tag', '\nThe setvar tag emits a copy the variable value in place. You can not do this with eg "<setvar[colors,false]: red and blue>"'];