Fix support for running multiple web flows

[jkronborg]

Proposed changes

Support running multiple web flows in one test command. This used to work when using the --device=chromium option, but broke in version 1.39.0 (commit).

Testing

# Workspace containing web flows
maestro test path/to/web-workspace

# Multiple web flows in a directory
maestro test some/directory/web-*.yaml

# Single web flow
maestro test some/directory/web-flow1.yaml

# Workspace containing iOS flows
maestro test path/to/ios-workspace

# Multiple iOS flows in a directory
maestro test some/directory/ios-*.yaml

# Single web flow
maestro test some/directory/ios-flow1.yaml

# A mix of web flows and iOS flows (fails)
maestro test path/to/mixed-workspace

I'm not really familiar with the codebase, so I don't know if this change has any unintended consequences not caught by the tests above.

Automated test coverage of this could be added in #2195.

Issues fixed

#2269

[jkronborg]

Hi @proksh, thanks for the review. Can I do anything to move this fix forward? This bug has prevented us from updating Maestro since version 1.39.0 which is now more than 5 months old.

[Fishbowler]

Rebased.

[Fishbowler]

When you say the mixed test fails, does it fail because of this check?

[jkronborg]

The iOS flow seems to fail because ChromeDriver tries to launch the app:

[Failed] iOS flow (2s) (Unable to launch app dk.monsenso.green: invalid argument
(Session info: chrome=135.0.7049.84)
Build info: version: '4.26.0', revision: '8ccf0219d7'
System info: os.name: 'Mac OS X', os.arch: 'aarch64', os.version: '15.4', java.version: '17.0.12'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [53aba667c9efd35da55aab0bc57c99b5, get {url=dk.monsenso.green}]

[jkronborg]

Thanks for the review @Fishbowler! After the rebase, running a workspace of web flows is broken again due to fa3eb42.

My sample workspace looks like this:

• 📁 web-workspace
  • 📄 config.yaml
  • 📁 feature1
    • 📄 flow1.yaml
    • 📄 flow2.yaml
  • 📁 feature2
    • 📄 flow3.yaml
    • 📄 flow4.yaml

And config.yaml like this:

flows:
  - '*/*'

The change in the commit I linked above does not traverse further than 1 level and thus never finds the flow files. Do you want me to push a fix for that?

[Fishbowler]

Do you want me to push a fix for that?

That'd be ace!

Moving the extra checks out of the recursion loop, and above the readConfig would solve your problem whilst maintaining the bugfix.

[jkronborg]

Do you want me to push a fix for that?

That'd be ace!

Moving the extra checks out of the recursion loop, and above the readConfig would solve your problem whilst maintaining the bugfix.

Alright, done. Feel free to change the code for style reasons or otherwise.

[jkronborg]

Rebased.

[smnatale]

Has this been merged in?

[jkronborg]

Has this been merged in?

Nope. Don't know what's holding it up.

[proksh]

Hey, quick question — what problem is this change intended to solve?

Right now, web flows aren't running, and iOS flows are picked up by default.

With this change, only web flows will be picked up (if any exist), and iOS flows will be skipped.

Is this the desired behavior? If so, can you clarify the reasoning behind it?

[jkronborg]

Thanks for getting back to me @proksh.

Hey, quick question — what problem is this change intended to solve?

The problem of maestro test not being able to execute a workspace of web flows or just multiple web flows. This used to work before version 1.39.0, and your documentation states that it's supported.

Right now, web flows aren't running, and iOS flows are picked up by default.

Not sure what you exactly mean, can you elaborate? In my testing using the latest from main, when you execute a workspace consisting of a mix of web flows and Android/iOS flows, no tests are run, unless an Android emulator or iOS simulator is running, in which case the Android/iOS flows are run while the web flows fail.

With this change, only web flows will be picked up (if any exist), and iOS flows will be skipped. Is this the desired behavior? If so, can you clarify the reasoning behind it?

It's true that the behavior of executing a mixed workspace while having an emulator/simulator running changes from "Android/iOS flows are executed, web flows fail" to "web flows are executed, Android/iOS flows fail". But the scenario is already broken today (the overall test run is guaranteed to fail).

My thinking was that it would be a net win to fix the bug, the behavior change mentioned above being a small trade-off.

[proksh]

Got it. I think there are a few issues here.

Currently, if someone is working with iOS/Android flows and adds a web flow, the system stops picking up any iOS/Android flows unless the user explicitly specifies a deviceId. That’s definitely inconvenient.

The existing approach checks only for a single web flow. If none is found, it defaults to iOS/Android flows, which is a better fallback and avoids blocking mobile flows.

What might work better is extending this logic: just like we check for web flows, we could also check for iOS and Android flows and automatically include those in the deviceIds.

Hey @dmitry-zaitsev , could you take a look and suggest any improvements here? You probably have more context around this.

[jkronborg]

Yeah, it would definitely be ideal to add support for mixed workspaces. Alternatively you could add chromium as a device id (like you had in older versions), and support web workspaces when that's set.

[proksh]

Yeah, it would definitely be ideal to add support for mixed workspaces. Alternatively you could add chromium as a device id (like you had in older versions), and support web workspaces when that's set.

Agree, I will wait for @dmitry-zaitsev response on this.