Can the SDK test if Mockito/Build Runner works at ToT?

[matanlurey]

Could help prevent things like:

• Mockito+Build are unusable at tip of tree (including in the Flutter SDK) #60566
• Analyzer 7.4.0 source-incompatibility #60545

I imagine with the cancellation of Macros and the fact Build Runner is being improved this might have gone from a P3ish need to P1ish?

[srawlins]

I suppose we could wire up a lot of infrastructure that tests analyzer at HEAD of the main branch against the latest pub release versions of a selection of packages (mockito, build_runner). That is a sizeable task.

[matanlurey]

I guess I sort of assumed that already happened for other packages and it would be the "simple" matter of just += mockito 😅

[davidmorgan]

I'm interested in better supporting our generators; we could run e2e tests of them in the build repo, there are already benchmarks of mockito, json_serializable and built_value. We could override the pubspec to run against the latest dev analyzer, which should be plenty of notice to catch this kind of issue.

It won't help right now though, ToT analyzer is the breaking removal of element1 and the generators can't use that yet. Work in progress :)

[feinstein]

Sorry if I am contributing with something that might be irrelevant, but I thought it would be nice to share these 2 cents:

I think this is the kind of task that can be implemented a lot quicker if you use AI.

I have used AI for this kind of stuff lately and yeah, it's not perfect, but I can get lots of boilerplate and test cases and automation tools done in a couple of days, that would take weeks normally, or would never be made at all because of their sizing.

So I would like to nudge that this might be a route the dart team would like to explore to not get trapped into this turning into a big task.

[davidmorgan]

So I would like to nudge that this might be a route the dart team would like to explore to not get trapped into this turning into a big task.

I don't think it will be too bad :) either way, as Matan suggested, it makes sense as part of investing more in build_runner.

[athomas]

It would be trivial to add them to this list:
https://github.com/dart-lang/sdk/blob/main/tools/bots/test_matrix.json#L3130

They are currently not dependencies of the SDK, however, so they'd also need to be added to DEPS and then someone would have to roll them continuously.

Alternatively, the packages in question could setup their GitHub Actions' setup-dart to run with the main channel. The latter is probably a better idea than artificially inflate the dependencies of the SDK repo. Though there is also an argument that some tests already use those non-hermetically through pub and would benefit from a pin in the SDK.

[davidmorgan]

Alternatively, the packages in question could setup their GitHub Actions' setup-dart to run with the main channel.

I think that's not quite sufficient, as it's not the analyzer-in-the-SDK that they're using but the analyzer-via-pub. So I think what they need is a CI run with an override of their pubspecs--not sure if there's an easy way to do that.

But it can't be too hard, I'll look at setting up something along those lines in the build repo. But there's no point for now as there are no generators that can use analyzer 8.0.0 yet: that's blocked on an element2 release of package:build. So, I'll take a look once I've done that package:build release :)

build already pulls in third party generators for benchmarking+testing so I think it's a good place for it. It won't get you presubmit in the SDK repo, but it will get you an email not too soon after commit if something looks problematic :)