crossgen2 fails with 139 exit code on Arm64

[mthalman]

This is causing a failure in the VMR for .NET 9 Preview 1 when building the installer repo. It fails in the Ubuntu2204Arm64_Offline_MsftSdk_arm64 job (example build [internal link]).

Error:

/vmr/src/installer/src/redist/targets/Crossgen.targets(163,5): error MSB6006: "crossgen2" exited with code 139. [/vmr/src/installer/src/redist/redist.csproj]

Here is a binlog:
sourcebuild.installer.zip

We need this resolved for .NET 9 Preview 1.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[mthalman]

@jkoritzinsky - Can you help investigate? This was working recently and regressed.

[jkoritzinsky]

I can try but I don't have a Linux ARM64 machine. Do you know the first VMR build that this broke in? I'll see if I can narrow down where the failure from.

[jkoritzinsky]

It looks like the most recent passing build was https://dev.azure.com/dnceng/internal/_build/results?buildId=2359219&view=results and the first failing build was https://dev.azure.com/dnceng/internal/_build/results?buildId=2359821&view=results.

The runtime range of commits between these two is dotnet/runtime@8accd80...e99836a

The commits that pop out to me as possibly causing crossgen2 crashes are:

• dotnet/runtime@205ef03
• dotnet/runtime@d00587d

I think the second is more likely than the first.

cc: @EgorBo

[EgorBo]

Checking now

[mthalman]

It's getting a different error now and failing with error code 134:

/vmr/src/installer/src/redist/targets/Crossgen.targets(172,5): *** stack smashing detected ***: terminated [/vmr/src/installer/src/redist/redist.csproj]

binlog

Failing build (internal link)

[mthalman]

It's consistently failing but not always failing with the same error. The two errors mentioned in this issue above are all I've seen so far.

[mthalman]

Interestingly, the build succeeded in this recent run (internal link). But I'm not sure I trust that the issue is resolved. Is it perhaps an intermittent issue? The source changes that exist between the two builds (failing and passing) don't include any runtime changes at all.

[jkoritzinsky]

Yes, it is possible that the failure is intermittent given your sample with the 134 exit code. It's likely a native corruption somewhere in crossgen2. If this is using a live crossgen2 (which I think it is), then it could be anywhere in the JIT, GC, or CoreCLR runtime. This also means it could have appeared before the first instance during the time frame where the build was failing before getting to installer, which makes this harder to diagnose...

[mthalman]

There's been 3 other builds since that one that passed and they've all failed.