Skip to content

Loading CPU profile in JIT doesn't work #8866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
derekxu16 opened this issue Feb 6, 2025 · 13 comments
Open

Loading CPU profile in JIT doesn't work #8866

derekxu16 opened this issue Feb 6, 2025 · 13 comments
Labels
cost: medium Medium engineering cost to implement P2 important to work on, but not at the top of the work list. screen: cpu profiler Issues related to the CPU Profiler screen

Comments

@derekxu16
Copy link
Contributor

The following is a copy of the bug report that @jensjoha had initially filed in the SDK repo:

Say I run

out/ReleaseX64/dart-sdk/bin/dart --enable-vm-service --pause_isolates_on_exit --profiler pkg/_fe_analyzer_shared/test/scanner_benchmark.dart pkg/_fe_analyzer_shared/lib/src/parser/parser_impl.dart --bytes

and launch devtools. I go to the "CPU Profiler" tab, click "Load all CPU samples". Then it just sits there, "Fetching CPU samples" forever.
In the javascript console I can see this stuff:

completePageInitializationIfReady_  {message: 'The message port closed before a response was received.'}

Attempting to load DevTools with canvaskit renderer.

Version: 2.43.0, Renderer: canvaskit

WebSocket connection to 'ws://127.0.0.1:46349/uqClOtdj8j8=' failed: 

[Dart Tooling Daemon connection failed.]: WebSocketChannelException: Instance of 'minified:OT'

DevTools log level changed to INFO

Could not find release notes for DevTools version 2.43.0.

Attempted to call extension 'ext.flutter.inspector.structuredErrors', but no service with that name exists

Could not time async operation "cpuProfileProcessingTime" because an exception was thrown:
Null check operator used on a null value
TypeError: Cannot read properties of undefined (reading 'd')
    at bNN.$2$current$parent (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:89924:44)
    at bNN.$2$current$parent (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:89928:83)
    at Object.cAi (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:14693:32)
    at http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:14601:129
    at bOY.a (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:5131:63)
    at bOY.$2 (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:62218:14)
    at Object.k (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:5117:10)
    at Object.To (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:14631:10)
    at http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:50796:14
    at bOY.a (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:5131:63)

Uncaught Error: Null check operator used on a null value
    at bNN.$2$current$parent (cpu_profile_model.dart:1261:39)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at Object.cAi (cpu_profile_model.dart:1380:5)
    at cpu_profile_model.dart:562:50
    at bOY.a (async_patch.dart:311:19)
    at bOY.$2 (async_patch.dart:336:23)
    at Object.k (async_patch.dart:241:3)
    at Object.To (cpu_profile_model.dart:556:11)
    at cpu_profile_service.dart:64:42
    at bOY.a (async_patch.dart:311:19)

[zoneGuarded]: Null check operator used on a null value
../../../lib/src/screens/profiler/cpu_profile_model.dart 1261:39                   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9                    _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1380:5                    CpuSamples.<anonymous extension>.generateStackFramesJson
../../../lib/src/screens/profiler/cpu_profile_model.dart 562:50                    CpuProfileData.generateFromCpuSamples
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 311:19  _wrapJsFunctionForAsync
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 336:23  _wrapJsFunctionForAsync.<anonymous function>
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 241:3   _asyncStartSync
../../../lib/src/screens/profiler/cpu_profile_model.dart 556:11                    CpuProfileData.generateFromCpuSamples
../../../lib/src/screens/profiler/cpu_profile_service.dart 64:42                   CpuProfilerExtension.getCpuProfile
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 311:19  _wrapJsFunctionForAsync

I am running through a ssh tunnel with port 8181 open btw.

Worth noting though is that if I instead do

pkg/vm/tool/precompiler2 pkg/_fe_analyzer_shared/test/scanner_benchmark.dart test.aot
out/ReleaseX64/dartaotruntime --enable-vm-service --pause_isolates_on_exit --profiler test.aot pkg/_fe_analyzer_shared/lib/src/parser/parser_impl.dart --bytes

it works. It loads the profile and there are no errors in the console.
@derekxu16 derekxu16 mentioned this issue Feb 6, 2025
Closed
@elliette elliette added screen: cpu profiler Issues related to the CPU Profiler screen P2 important to work on, but not at the top of the work list. cost: medium Medium engineering cost to implement labels Feb 13, 2025
@elliette
Copy link
Member

@jakemac53 and another one!

@jakemac53 jakemac53 self-assigned this Feb 24, 2025
@jakemac53
Copy link
Contributor

@derekxu16 This looks related to the fact that you are running through SSH - note that web socket URL that it is trying to connect to WebSocket connection to 'ws://127.0.0.1:46349/uqClOtdj8j8=' failed: - this is not the port you have open (it is a different random one).

Using VsCode and the proxy server its creates I was able to load a profile (even when running devtools on a remote machine through ssh). I had to do some manual munging of the connection URL to use the vscode proxy URL, prior to that I saw the same errors you see.

I did however still see several errors in the JavaScript console, but was ultimately able to load the profile.

@jakemac53
Copy link
Contributor

It looks like when running via the AOT command you provided, it always uses port 8181. So that is why it works.

@jakemac53
Copy link
Contributor

I believe you could work around this by passing explicitly --enable-vm-service=8181 to specify that port.

@jakemac53
Copy link
Contributor

The remaining errors I do see are related to the dart tooling daemon connection which fails (but gracefully). It is the same core issue, it tries to connect to dtd using a local host/port which isn't being forwarded. I think it probably gets the connection URI from a file or something. I am not sure there is much we can do there.

@jakemac53 jakemac53 removed their assignment Feb 24, 2025
@derekxu16
Copy link
Contributor Author

This issue was reported by @jensjoha, I just transferred it from the SDK repo to here.

@jakemac53 jakemac53 mentioned this issue Feb 24, 2025
Merged
@jakemac53
Copy link
Contributor

jakemac53 commented Feb 24, 2025 •
edited
Loading

cc @jensjoha can you confirm things work if you provide an explicit port that you have forwarded?

I do expect you will see the error connecting to the dart tooling daemon still though.

@jensjoha
Copy link

Specifying a port doesn't change things.
--enable-vm-service already implies 8181:

--enable-vm-service[=<port>[/<bind-address>]]
  Enables the VM service and listens on specified port for connections
  (default port number is 8181, default bind address is localhost).

You'd have to specify --enable-vm-service=0 to get a random port.

It's weird that it's trying to connect to a different port, the output is

The Dart VM service is listening on http://127.0.0.1:8181/jBuyQl3qlK0=/
The Dart DevTools debugger and profiler is available at: http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/?uri=ws://127.0.0.1:8181/jBuyQl3qlK0=/ws

so it's seemingly asking webtools to connect to port 8181.

That being said, it does stop complaining about not being able to connect if chromoting in, but it still doesn't load and I still get null errors:

Successfully connected to DTD at: ws://127.0.0.1:42383/uojO5f1dPJI=

DevTools log level changed to INFO

[GroupMarkerNotSet(crbug.com/242999)!:A0E01200F4070000]Automatic fallback to software WebGL has been deprecated. Please use the --enable-unsafe-swiftshader flag to opt in to lower security guarantees for trusted content.

[GroupMarkerNotSet(crbug.com/242999)!:A0C06606F4070000]Automatic fallback to software WebGL has been deprecated. Please use the --enable-unsafe-swiftshader flag to opt in to lower security guarantees for trusted content.

Attempted to call extension 'ext.flutter.inspector.structuredErrors', but no service with that name exists

Could not find release notes for DevTools version 2.43.0.

Could not time async operation "cpuProfileProcessingTime" because an exception was thrown:
Null check operator used on a null value
TypeError: Cannot read properties of undefined (reading 'd')
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89924:44)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)

Uncaught Error: Null check operator used on a null value
    at bNN.$2$current$parent (cpu_profile_model.dart:1261:39)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)

[zoneGuarded]: Null check operator used on a null value
../../../lib/src/screens/profiler/cpu_profile_model.dart 1261:39  _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame

@jakemac53
Copy link
Contributor

I did send out a fix for the null check exception fwiw, #8940. But, I doubt its really going to help your situation, as this indicates a not useful response anyways. But, it might get you a different or better error.

jakemac53 added a commit that referenced this issue Feb 26, 2025
@jakemac53
handle null functions key in cpu samples (#8940)
7a8c65d 
Fixes one of the errors shown in #8866.
@kenzieschmoll
Copy link
Member

@jensjoha would you be able to try again with the latest DevTools code and see if you get a better error now that #8940 is fixed?

@jensjoha
Copy link

jensjoha commented May 2, 2025

I still get null-errors in the console and the actual page is just standing there saying "Fetching CPU samples".

@jakemac53
Copy link
Contributor

@jensjoha can you provide a CPU profile which fails to load?

@jensjoha
Copy link

jensjoha commented May 5, 2025

I don't understand. The reproduction is in the first message?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cost: medium Medium engineering cost to implement P2 important to work on, but not at the top of the work list. screen: cpu profiler Issues related to the CPU Profiler screen
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jakemac53 @derekxu16 @jensjoha @elliette @kenzieschmoll