roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.follower-reads/mixed-version/survival=region/locality=global/reads=strong failed with artifacts on release-24.3 @ af8aef87935b9de78646824015c3d8898db0b75e:

(follower_reads.go:763).verifySQLLatency: 23 latency values ([167.227937ms 90.572478ms 90.571343ms 90.562641ms 90.557317ms 90.55923ms 90.563677ms 90.564632ms 90.563914ms 90.557448ms 90.558936ms 90.563245ms 90.558489ms 90.563821ms 90.562576ms 90.040559ms 89.314362ms 88.575197ms 71.656127ms 71.654623ms 71.656561ms 71.653117ms 71.654623ms]) are above target latency 25ms, 4 permitted
(mixedversion.go:782).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/survival=region/locality=global/reads=strong/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• mvtDeploymentMode=shared-process
• mvtVersions=v23.1.5 → v23.2.8 → v24.1.8 → v24.2.7 → release-24.3
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed #141357 roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed [C-test-failure O-roachtest O-robot T-testeng branch-master release-blocker]
• roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed #140967 roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed [B-runtime-assertions-enabled C-test-failure GA-blocker O-roachtest O-robot T-jobs branch-release-25.1 release-blocker]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-47820

[tbg]

This test just highlighted a GA blocker over on release-25.1, but this is a different failure mode and 24.3 does not include that migration that caused the issue. Given that this is a multi-region test and it's a latency violation and not some catastrophic failure, removing release-blocker.

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.follower-reads/mixed-version/survival=region/locality=global/reads=strong failed with artifacts on release-24.3 @ 5be83d7c3461add218e04a15e158139ef945ea6c:

(follower_reads.go:763).verifySQLLatency: 5 latency values ([147.737765ms 147.737765ms 147.737765ms 147.737765ms 92.491472ms]) are above target latency 25ms, 4 permitted
(mixedversion.go:778).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/survival=region/locality=global/reads=strong/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• mvtDeploymentMode=shared-process
• mvtVersions=v23.1.13 → v23.2.11 → v24.1.7 → release-24.3
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[miraradeva]

The deflake in #139944 didn't quite work; it's the same issue here (though it did work for 25.1 and master). I'll take another look.