[LV] Improve code in selectInterleaveCount (NFC) #128002
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvm/pr-subscribers-vectorizers Author: Ramkumar Ramachandra (artagnon) ChangesFull diff: https://github.com/llvm/llvm-project/pull/128002.diff 1 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 8c41f896ad622..0c3afab724b34 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -4922,7 +4922,6 @@ LoopVectorizationCostModel::selectInterleaveCount(ElementCount VF,
if (Legal->hasUncountableEarlyExit())
return 1;
- auto BestKnownTC = getSmallBestKnownTC(PSE, TheLoop);
const bool HasReductions = !Legal->getReductionVars().empty();
// If we did not calculate the cost for VF (because the user selected the VF)
@@ -4998,51 +4997,53 @@ LoopVectorizationCostModel::selectInterleaveCount(ElementCount VF,
}
unsigned EstimatedVF = getEstimatedRuntimeVF(VF, VScaleForTuning);
- unsigned KnownTC = PSE.getSE()->getSmallConstantTripCount(TheLoop);
- if (KnownTC > 0) {
- // At least one iteration must be scalar when this constraint holds. So the
- // maximum available iterations for interleaving is one less.
- unsigned AvailableTC =
- requiresScalarEpilogue(VF.isVector()) ? KnownTC - 1 : KnownTC;
-
- // If trip count is known we select between two prospective ICs, where
- // 1) the aggressive IC is capped by the trip count divided by VF
- // 2) the conservative IC is capped by the trip count divided by (VF * 2)
- // The final IC is selected in a way that the epilogue loop trip count is
- // minimized while maximizing the IC itself, so that we either run the
- // vector loop at least once if it generates a small epilogue loop, or else
- // we run the vector loop at least twice.
-
- unsigned InterleaveCountUB = bit_floor(
- std::max(1u, std::min(AvailableTC / EstimatedVF, MaxInterleaveCount)));
- unsigned InterleaveCountLB = bit_floor(std::max(
- 1u, std::min(AvailableTC / (EstimatedVF * 2), MaxInterleaveCount)));
- MaxInterleaveCount = InterleaveCountLB;
-
- if (InterleaveCountUB != InterleaveCountLB) {
- unsigned TailTripCountUB =
- (AvailableTC % (EstimatedVF * InterleaveCountUB));
- unsigned TailTripCountLB =
- (AvailableTC % (EstimatedVF * InterleaveCountLB));
- // If both produce same scalar tail, maximize the IC to do the same work
- // in fewer vector loop iterations
- if (TailTripCountUB == TailTripCountLB)
- MaxInterleaveCount = InterleaveCountUB;
- }
- } else if (BestKnownTC && *BestKnownTC > 0) {
+
+ // Try to get the exact trip count, or an estimate based on profiling data or
+ // ConstantMax from PSE, failing that.
+ if (auto BestKnownTC = getSmallBestKnownTC(PSE, TheLoop)) {
// At least one iteration must be scalar when this constraint holds. So the
// maximum available iterations for interleaving is one less.
unsigned AvailableTC = requiresScalarEpilogue(VF.isVector())
? (*BestKnownTC) - 1
: *BestKnownTC;
- // If trip count is an estimated compile time constant, limit the
- // IC to be capped by the trip count divided by VF * 2, such that the vector
- // loop runs at least twice to make interleaving seem profitable when there
- // is an epilogue loop present. Since exact Trip count is not known we
- // choose to be conservative in our IC estimate.
- MaxInterleaveCount = bit_floor(std::max(
+ unsigned InterleaveCountLB = bit_floor(std::max(
1u, std::min(AvailableTC / (EstimatedVF * 2), MaxInterleaveCount)));
+
+ if (PSE.getSE()->getSmallConstantTripCount(TheLoop) > 0) {
+ // If the estimated trip count is actually an exact one we select between
+ // two prospective ICs, where
+ //
+ // 1) the aggressive IC is capped by the trip count divided by VF
+ // 2) the conservative IC is capped by the trip count divided by (VF * 2)
+ //
+ // The final IC is selected in a way that the epilogue loop trip count is
+ // minimized while maximizing the IC itself, so that we either run the
+ // vector loop at least once if it generates a small epilogue loop, or
+ // else we run the vector loop at least twice.
+
+ unsigned InterleaveCountUB = bit_floor(std::max(
+ 1u, std::min(AvailableTC / EstimatedVF, MaxInterleaveCount)));
+ MaxInterleaveCount = InterleaveCountLB;
+
+ if (InterleaveCountUB != InterleaveCountLB) {
+ unsigned TailTripCountUB =
+ (AvailableTC % (EstimatedVF * InterleaveCountUB));
+ unsigned TailTripCountLB =
+ (AvailableTC % (EstimatedVF * InterleaveCountLB));
+ // If both produce same scalar tail, maximize the IC to do the same work
+ // in fewer vector loop iterations
+ if (TailTripCountUB == TailTripCountLB)
+ MaxInterleaveCount = InterleaveCountUB;
+ }
+ } else {
+ // If trip count is an estimated compile time constant, limit the
+ // IC to be capped by the trip count divided by VF * 2, such that the
+ // vector loop runs at least twice to make interleaving seem profitable
+ // when there is an epilogue loop present. Since exact Trip count is not
+ // known we choose to be conservative in our IC estimate.
+ MaxInterleaveCount = InterleaveCountLB;
+ }
}
assert(MaxInterleaveCount > 0 &&
|
|
@llvm/pr-subscribers-llvm-transforms Author: Ramkumar Ramachandra (artagnon) ChangesFull diff: https://github.com/llvm/llvm-project/pull/128002.diff 1 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 8c41f896ad622..0c3afab724b34 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -4922,7 +4922,6 @@ LoopVectorizationCostModel::selectInterleaveCount(ElementCount VF,
if (Legal->hasUncountableEarlyExit())
return 1;
- auto BestKnownTC = getSmallBestKnownTC(PSE, TheLoop);
const bool HasReductions = !Legal->getReductionVars().empty();
// If we did not calculate the cost for VF (because the user selected the VF)
@@ -4998,51 +4997,53 @@ LoopVectorizationCostModel::selectInterleaveCount(ElementCount VF,
}
unsigned EstimatedVF = getEstimatedRuntimeVF(VF, VScaleForTuning);
- unsigned KnownTC = PSE.getSE()->getSmallConstantTripCount(TheLoop);
- if (KnownTC > 0) {
- // At least one iteration must be scalar when this constraint holds. So the
- // maximum available iterations for interleaving is one less.
- unsigned AvailableTC =
- requiresScalarEpilogue(VF.isVector()) ? KnownTC - 1 : KnownTC;
-
- // If trip count is known we select between two prospective ICs, where
- // 1) the aggressive IC is capped by the trip count divided by VF
- // 2) the conservative IC is capped by the trip count divided by (VF * 2)
- // The final IC is selected in a way that the epilogue loop trip count is
- // minimized while maximizing the IC itself, so that we either run the
- // vector loop at least once if it generates a small epilogue loop, or else
- // we run the vector loop at least twice.
-
- unsigned InterleaveCountUB = bit_floor(
- std::max(1u, std::min(AvailableTC / EstimatedVF, MaxInterleaveCount)));
- unsigned InterleaveCountLB = bit_floor(std::max(
- 1u, std::min(AvailableTC / (EstimatedVF * 2), MaxInterleaveCount)));
- MaxInterleaveCount = InterleaveCountLB;
-
- if (InterleaveCountUB != InterleaveCountLB) {
- unsigned TailTripCountUB =
- (AvailableTC % (EstimatedVF * InterleaveCountUB));
- unsigned TailTripCountLB =
- (AvailableTC % (EstimatedVF * InterleaveCountLB));
- // If both produce same scalar tail, maximize the IC to do the same work
- // in fewer vector loop iterations
- if (TailTripCountUB == TailTripCountLB)
- MaxInterleaveCount = InterleaveCountUB;
- }
- } else if (BestKnownTC && *BestKnownTC > 0) {
+
+ // Try to get the exact trip count, or an estimate based on profiling data or
+ // ConstantMax from PSE, failing that.
+ if (auto BestKnownTC = getSmallBestKnownTC(PSE, TheLoop)) {
// At least one iteration must be scalar when this constraint holds. So the
// maximum available iterations for interleaving is one less.
unsigned AvailableTC = requiresScalarEpilogue(VF.isVector())
? (*BestKnownTC) - 1
: *BestKnownTC;
- // If trip count is an estimated compile time constant, limit the
- // IC to be capped by the trip count divided by VF * 2, such that the vector
- // loop runs at least twice to make interleaving seem profitable when there
- // is an epilogue loop present. Since exact Trip count is not known we
- // choose to be conservative in our IC estimate.
- MaxInterleaveCount = bit_floor(std::max(
+ unsigned InterleaveCountLB = bit_floor(std::max(
1u, std::min(AvailableTC / (EstimatedVF * 2), MaxInterleaveCount)));
+
+ if (PSE.getSE()->getSmallConstantTripCount(TheLoop) > 0) {
+ // If the estimated trip count is actually an exact one we select between
+ // two prospective ICs, where
+ //
+ // 1) the aggressive IC is capped by the trip count divided by VF
+ // 2) the conservative IC is capped by the trip count divided by (VF * 2)
+ //
+ // The final IC is selected in a way that the epilogue loop trip count is
+ // minimized while maximizing the IC itself, so that we either run the
+ // vector loop at least once if it generates a small epilogue loop, or
+ // else we run the vector loop at least twice.
+
+ unsigned InterleaveCountUB = bit_floor(std::max(
+ 1u, std::min(AvailableTC / EstimatedVF, MaxInterleaveCount)));
+ MaxInterleaveCount = InterleaveCountLB;
+
+ if (InterleaveCountUB != InterleaveCountLB) {
+ unsigned TailTripCountUB =
+ (AvailableTC % (EstimatedVF * InterleaveCountUB));
+ unsigned TailTripCountLB =
+ (AvailableTC % (EstimatedVF * InterleaveCountLB));
+ // If both produce same scalar tail, maximize the IC to do the same work
+ // in fewer vector loop iterations
+ if (TailTripCountUB == TailTripCountLB)
+ MaxInterleaveCount = InterleaveCountUB;
+ }
+ } else {
+ // If trip count is an estimated compile time constant, limit the
+ // IC to be capped by the trip count divided by VF * 2, such that the
+ // vector loop runs at least twice to make interleaving seem profitable
+ // when there is an epilogue loop present. Since exact Trip count is not
+ // known we choose to be conservative in our IC estimate.
+ MaxInterleaveCount = InterleaveCountLB;
+ }
}
assert(MaxInterleaveCount > 0 &&
|
|
Can you add more description to this PR explaining why the newer version is better? |
Done. |
david-arm
reviewed
Feb 27, 2025
artagnon
force-pushed
the
lv-ic-count-nfc
branch
from
c0a966b to
c49546b
Compare
|
Gentle ping. |
fhahn
reviewed
Mar 25, 2025
|
Gentle ping. |
2 similar comments
|
Gentle ping. |
|
Gentle ping. |
david-arm
approved these changes
May 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use the fact that getSmallBestKnownTC returns an exact trip count, if possible, and falls back to returning an estimate, to factor some code in selectInterleaveCount.