[script/tool] speed up the pub get portion of the analyze command #3982

devoncarew · 2021-05-27T16:27:45Z

This PR speeds up the analyze command for the plugins tool. We were running pub get twice for all 42 example directories (flutter pub get automatically runs pub get for an example/ child directory when running pub get for the parent). The analyze command was spending 60% of it's time running flutter pub get; after this change the overall command should be about 25% faster.

List which issues are fixed by this PR. You must list at least one issue.

work towards speed up the analysis CI phase of the flutter/packages repo flutter#83261 (and the flutter-analyze try bot should also speculatively analyze flutter/plugins dart-lang/sdk#45873)

If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides. (Note that unlike the flutter/flutter repo, the flutter/plugins repo does use dart format. See plugin_tool format)
I signed the CLA.
The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
I listed at least one issue that this PR fixes in the description above.
[-] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[-] I updated CHANGELOG.md to add a description of the change.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test exempt.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

stuartmorgan-g

Nice, just minor comments.

stuartmorgan-g · 2021-05-27T16:51:17Z

script/tool/lib/src/analyze_command.dart

@@ -64,6 +65,14 @@ class AnalyzeCommand extends PluginCommand {
    }

    final List<Directory> packageDirectories = await getPackages().toList();
+    final List<String> packagePaths =
+        packageDirectories.map((Directory dir) => dir.path).toList();


toSet()? You're only using this for inclusion testing.

will change

stuartmorgan-g · 2021-05-27T16:51:56Z

script/tool/lib/src/analyze_command.dart

+      // runs 'pub get' there as part of handling the parent directory.
+      return directory.basename == 'example' &&
+          packagePaths.contains(directory.parent.path);
+    });


I was hoping we could just use getPlugins instead of doing the filtering, but it looks like flutter/packages has some other nested constructions.

Yeah, I would prefer something more semantic here as well. The logic for how these are computed - getPlugins() and getPackages() - seems complicated, and I wasn't comfortable modifying it.

It looks like plugins actually returns packages? And packages returns packages plus their example directories?

I would have assumed that getPlugins() would return the directory containing the n dart packages that make up a plugin - like url_launcher/, url_launcher_platform_interface/, url_launcher_macos/, ...

I'll want a call like that - the top level plugin directory - for a follow up PR, where I'm going to try and speed up the analysis portion of the analyze call.

It looks like plugins actually returns packages? And packages returns packages plus there example directories?

I suspect what happened is that this was written for flutter/plugins, then expanded for use in flutter/packages without renaming. Fixing the fact that "plugins" is used to refer to packages in general in the tool (including the --plugins argument) is on my list.

stuartmorgan-g · 2021-05-27T16:54:12Z

script/tool/test/analyze_command_test.dart

+              plugin1Dir.path),
+        ]));
+  });
+


Could you also add a test of a package with a sub-package that's not example, to make sure nobody makes the "simplification" that I was about to suggest before checking flutter/packages?

We have one like that already - the analyzes all packages test - but I can add one where the 2nd package is named 'example' but not contained by the 1st package.

devoncarew

Some comments - code changes to follow separately.

devoncarew · 2021-05-27T17:30:22Z

script/tool/lib/src/analyze_command.dart

@@ -64,6 +65,14 @@ class AnalyzeCommand extends PluginCommand {
    }

    final List<Directory> packageDirectories = await getPackages().toList();
+    final List<String> packagePaths =
+        packageDirectories.map((Directory dir) => dir.path).toList();


will change

devoncarew · 2021-05-27T17:35:01Z

script/tool/lib/src/analyze_command.dart

+      // runs 'pub get' there as part of handling the parent directory.
+      return directory.basename == 'example' &&
+          packagePaths.contains(directory.parent.path);
+    });


Yeah, I would prefer something more semantic here as well. The logic for how these are computed - getPlugins() and getPackages() - seems complicated, and I wasn't comfortable modifying it.

It looks like plugins actually returns packages? And packages returns packages plus their example directories?

I would have assumed that getPlugins() would return the directory containing the n dart packages that make up a plugin - like url_launcher/, url_launcher_platform_interface/, url_launcher_macos/, ...

I'll want a call like that - the top level plugin directory - for a follow up PR, where I'm going to try and speed up the analysis portion of the analyze call.

devoncarew · 2021-05-27T17:36:07Z

script/tool/test/analyze_command_test.dart

+              plugin1Dir.path),
+        ]));
+  });
+


We have one like that already - the analyzes all packages test - but I can add one where the 2nd package is named 'example' but not contained by the 1st package.

stuartmorgan-g

LGTM

…mand (flutter/plugins#3982)

* master: (131 commits) [in_app_purchase] fix "autoConsume" param in "buyConsumable" (flutter#3957) [video_player_web] fix: set autoplay to false during initialization (flutter#3985) [multiple_web] Adapt web PlatformView widgets to work slotted. (flutter#3964) [url_launcher] Add iOS unit and UI tests (flutter#3987) [image_picker] Change storage location for camera captures to internal cache on Android, to comply with new Google Play storage requirements. (flutter#3956) [script/tool] Use 'dart pub' instead of deprecated 'pub' (flutter#3991) [video_player] Add iOS unit and UI tests (flutter#3986) Add pubspec convention checks (flutter#3984) Enable pubspec dependency sorting lint (flutter#3983) [image_picker] Migrate maven repo from jcenter to mavenCentral (flutter#3915) [video_player] Update README.md (flutter#3975) [script/tool] speed up the pub get portion of the analyze command (flutter#3982) Revert commit e742a7b (flutter#3976) Added support to request list of purchases (flutter#3944) [google_maps_flutter] Add iOS unit and UI tests (flutter#3978) Added Windows to the description (flutter#3936) use 'flutter pub get' for both dart and flutter packages (flutter#3973) [camera] android-rework part 3: Android exposure related features (flutter#3797) Remove exoplayer workaround from everything but video_player (flutter#3980) Allow reverts when checking versions (flutter#3981) ...

…utter#3982)

speed up the pub get portion of the analyze command

90674e4

google-cla bot added the cla: yes label May 27, 2021

devoncarew requested a review from stuartmorgan-g May 27, 2021 16:44

stuartmorgan-g suggested changes May 27, 2021

View reviewed changes

devoncarew commented May 27, 2021

View reviewed changes

review comments; add additional test

8364bf0

stuartmorgan-g approved these changes May 27, 2021

View reviewed changes

devoncarew added the waiting for tree to go green (Use "autosubmit") This PR is approved and tested, but waiting for the tree to be green to land. label May 27, 2021

fluttergithubbot merged commit 2807531 into flutter:master May 27, 2021

engine-flutter-autoroll mentioned this pull request May 27, 2021

Roll Plugins from 4ef31b640d14 to c6d067987252 (2 revisions) flutter/flutter#83511

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 27, 2021

2807531 [script/tool] speed up the pub get portion of the analyze com…

37bda6c

…mand (flutter/plugins#3982)

fotiDim pushed a commit to fotiDim/plugins that referenced this pull request Sep 13, 2021

[script/tool] speed up the pub get portion of the analyze command (fl…

5226830

…utter#3982)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[script/tool] speed up the pub get portion of the analyze command #3982

[script/tool] speed up the pub get portion of the analyze command #3982

devoncarew commented May 27, 2021

stuartmorgan-g left a comment

stuartmorgan-g May 27, 2021

devoncarew May 27, 2021

stuartmorgan-g May 27, 2021

devoncarew May 27, 2021 •

edited

Loading

stuartmorgan-g May 27, 2021

stuartmorgan-g May 27, 2021

devoncarew May 27, 2021

devoncarew left a comment

devoncarew May 27, 2021

devoncarew May 27, 2021 •

edited

Loading

devoncarew May 27, 2021

stuartmorgan-g left a comment

[script/tool] speed up the pub get portion of the analyze command #3982

[script/tool] speed up the pub get portion of the analyze command #3982

Conversation

devoncarew commented May 27, 2021

Pre-launch Checklist

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devoncarew May 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devoncarew left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devoncarew May 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuartmorgan-g left a comment

Choose a reason for hiding this comment

devoncarew May 27, 2021 •

edited

Loading

devoncarew May 27, 2021 •

edited

Loading