SelectionDAG: Improve expandFP_TO_INT_SAT #139217
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It is used to mark a value that we are sure that it is not some fcType. The examples include: * An arguments of a function is marked with nofpclass * Output value of an intrinsic can be sure to not be some type So that the following operation can make some assumptions. fix typo assernofpclass Add isKnownNeverNaN test Mention operands Fix typo Fix AssertNoFPClass assert Fix code styles 1. Use Op.getConstantOperandVal to get Constant value 2. (NoFPClass & fcNan) == fcNan 3. Use N2->getAsZExtValue() to get Constant value 4. Use getTargetConstant with ptr width to fix 32bit target 5. Don't propagate the flags 6. Add MIPSr6 testcase for 32bit target tests Add Vector test case Add DemandedElts Use 32bit target Fix MERGE_VALUE cases Add aggregate tests with fmin Fix comment Add snan and qnan test cases Test both array and struct Fix WidenVectorResult Fix PromoteHalf support Use seperate function for assertnofpclass Fix wrong dead code from SoftPromoteHalfRes_UnaryOp Fix internal error
Currently, expandFP_TO_INT_SAT uses FMAXNUM and FMINNUM, which is not correct if the Src is sNaN. Let's try all 3 flavor of Max/Min for it: 1) FMAXIMUMNUM/FMINIMUMNUM See test/CodeGen/RISCV/bfloat-convert.ll 2) FMAXNUM/FMAXNUM See test/CodeGen/Mips/Half2Int16.ll 3) FMAXIMUM/FMAXIMUM See test/CodeGen/WebAssembly/Half2Int16.ll
llvmbot
added
backend:ARM
backend:AArch64
backend:WebAssembly
backend:X86
llvm:SelectionDAG
|
@llvm/pr-subscribers-llvm-selectiondag @llvm/pr-subscribers-backend-x86 Author: YunQiang Su (wzssyqa) ChangesCurrently, expandFP_TO_INT_SAT uses FMAXNUM and FMINNUM, which is Let's try all 3 flavor of Max/Min for it:
Patch is 44.47 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/139217.diff 16 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index 80ef32aff62ae..1042318343987 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -67,6 +67,15 @@ enum NodeType {
/// poisoned the assertion will not be true for that value.
AssertAlign,
+ /// AssertNoFPClass - These nodes record if a register contains a float
+ /// value that is known to be not some type.
+ /// This node takes two operands. The first is the node that is known
+ /// never to be some float types; the second is a constant value with
+ /// the value of FPClassTest (casted to uint32_t).
+ /// NOTE: In case of the source value (or any vector element value) is
+ /// poisoned the assertion will not be true for that value.
+ AssertNoFPClass,
+
/// Various leaf nodes.
BasicBlock,
VALUETYPE,
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 41fed692c7025..b28a8b118de7a 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -875,6 +875,7 @@ def SDT_assert : SDTypeProfile<1, 1,
[SDTCisInt<0>, SDTCisInt<1>, SDTCisSameAs<1, 0>]>;
def assertsext : SDNode<"ISD::AssertSext", SDT_assert>;
def assertzext : SDNode<"ISD::AssertZext", SDT_assert>;
+def assertnofpclass : SDNode<"ISD::AssertNoFPClass", SDTFPUnaryOp>;
def assertalign : SDNode<"ISD::AssertAlign", SDT_assert>;
def convergencectrl_anchor : SDNode<"ISD::CONVERGENCECTRL_ANCHOR",
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
index 432209e8ecb0a..ea0f56d7741ac 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
@@ -168,6 +168,7 @@ void DAGTypeLegalizer::SoftenFloatResult(SDNode *N, unsigned ResNo) {
case ISD::POISON:
case ISD::UNDEF: R = SoftenFloatRes_UNDEF(N); break;
case ISD::VAARG: R = SoftenFloatRes_VAARG(N); break;
+ case ISD::AssertNoFPClass: R = GetSoftenedFloat(N->getOperand(0)); break;
case ISD::VECREDUCE_FADD:
case ISD::VECREDUCE_FMUL:
case ISD::VECREDUCE_FMIN:
@@ -2582,6 +2583,7 @@ bool DAGTypeLegalizer::PromoteFloatOperand(SDNode *N, unsigned OpNo) {
case ISD::LLROUND:
case ISD::LRINT:
case ISD::LLRINT: R = PromoteFloatOp_UnaryOp(N, OpNo); break;
+ case ISD::AssertNoFPClass: R = PromoteFloatOp_UnaryOpExt1(N, OpNo); break;
case ISD::FP_TO_SINT_SAT:
case ISD::FP_TO_UINT_SAT:
R = PromoteFloatOp_FP_TO_XINT_SAT(N, OpNo); break;
@@ -2640,6 +2642,13 @@ SDValue DAGTypeLegalizer::PromoteFloatOp_UnaryOp(SDNode *N, unsigned OpNo) {
return DAG.getNode(N->getOpcode(), SDLoc(N), N->getValueType(0), Op);
}
+// Convert the promoted float value to the desired integer type
+SDValue DAGTypeLegalizer::PromoteFloatOp_UnaryOpExt1(SDNode *N, unsigned OpNo) {
+ SDValue Op = GetPromotedFloat(N->getOperand(0));
+ return DAG.getNode(N->getOpcode(), SDLoc(N), N->getValueType(0), Op,
+ N->getOperand(1));
+}
+
SDValue DAGTypeLegalizer::PromoteFloatOp_FP_TO_XINT_SAT(SDNode *N,
unsigned OpNo) {
SDValue Op = GetPromotedFloat(N->getOperand(0));
@@ -2804,6 +2813,9 @@ void DAGTypeLegalizer::PromoteFloatResult(SDNode *N, unsigned ResNo) {
case ISD::FTAN:
case ISD::FTANH:
case ISD::FCANONICALIZE: R = PromoteFloatRes_UnaryOp(N); break;
+ case ISD::AssertNoFPClass:
+ R = PromoteFloatRes_UnaryOpExt1(N);
+ break;
// Binary FP Operations
case ISD::FADD:
@@ -2996,10 +3008,18 @@ SDValue DAGTypeLegalizer::PromoteFloatRes_UnaryOp(SDNode *N) {
EVT VT = N->getValueType(0);
EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
SDValue Op = GetPromotedFloat(N->getOperand(0));
-
return DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Op);
}
+// Unary operation with a more non-float operand where the result and the
+// operand have PromoteFloat type action. Construct a new SDNode with the
+// promoted float value of the old operand.
+SDValue DAGTypeLegalizer::PromoteFloatRes_UnaryOpExt1(SDNode *N) {
+ EVT VT = N->getValueType(0);
+ EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
+ SDValue Op = GetPromotedFloat(N->getOperand(0));
+ return DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Op, N->getOperand(1));
+}
// Binary operations where the result and both operands have PromoteFloat type
// action. Construct a new SDNode with the promoted float values of the old
// operands.
@@ -3281,6 +3301,9 @@ void DAGTypeLegalizer::SoftPromoteHalfResult(SDNode *N, unsigned ResNo) {
case ISD::FTAN:
case ISD::FTANH:
case ISD::FCANONICALIZE: R = SoftPromoteHalfRes_UnaryOp(N); break;
+ case ISD::AssertNoFPClass:
+ R = SoftPromoteHalfRes_UnaryOpExt1(N);
+ break;
// Binary FP Operations
case ISD::FADD:
@@ -3607,6 +3630,21 @@ SDValue DAGTypeLegalizer::SoftPromoteHalfRes_UnaryOp(SDNode *N) {
return DAG.getNode(GetPromotionOpcode(NVT, OVT), dl, MVT::i16, Res);
}
+SDValue DAGTypeLegalizer::SoftPromoteHalfRes_UnaryOpExt1(SDNode *N) {
+ EVT OVT = N->getValueType(0);
+ EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), OVT);
+ SDValue Op = GetSoftPromotedHalf(N->getOperand(0));
+ SDLoc dl(N);
+
+ // Promote to the larger FP type.
+ Op = DAG.getNode(GetPromotionOpcode(OVT, NVT), dl, NVT, Op);
+
+ SDValue Res = DAG.getNode(N->getOpcode(), dl, NVT, Op, N->getOperand(1));
+
+ // Convert back to FP16 as an integer.
+ return DAG.getNode(GetPromotionOpcode(NVT, OVT), dl, MVT::i16, Res);
+}
+
SDValue DAGTypeLegalizer::SoftPromoteHalfRes_BinOp(SDNode *N) {
EVT OVT = N->getValueType(0);
EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), OVT);
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
index 720393158aa5e..f4d5fa230f821 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
@@ -80,7 +80,9 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
/// Pretend all of this node's results are legal.
bool IgnoreNodeResults(SDNode *N) const {
return N->getOpcode() == ISD::TargetConstant ||
- N->getOpcode() == ISD::Register;
+ N->getOpcode() == ISD::Register ||
+ (N->getOpcode() == ISD::AssertNoFPClass &&
+ IgnoreNodeResults(N->getOperand(0).getNode()));
}
// Bijection from SDValue to unique id. As each created node gets a
@@ -772,6 +774,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue PromoteFloatRes_SELECT(SDNode *N);
SDValue PromoteFloatRes_SELECT_CC(SDNode *N);
SDValue PromoteFloatRes_UnaryOp(SDNode *N);
+ SDValue PromoteFloatRes_UnaryOpExt1(SDNode *N);
SDValue PromoteFloatRes_UNDEF(SDNode *N);
SDValue BitcastToInt_ATOMIC_SWAP(SDNode *N);
SDValue PromoteFloatRes_XINT_TO_FP(SDNode *N);
@@ -785,6 +788,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue PromoteFloatOp_FP_EXTEND(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_STRICT_FP_EXTEND(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_UnaryOp(SDNode *N, unsigned OpNo);
+ SDValue PromoteFloatOp_UnaryOpExt1(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_FP_TO_XINT_SAT(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_STORE(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_ATOMIC_STORE(SDNode *N, unsigned OpNo);
@@ -796,6 +800,8 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
//===--------------------------------------------------------------------===//
SDValue GetSoftPromotedHalf(SDValue Op) {
+ while (Op.getNode()->getOpcode() == ISD::AssertNoFPClass)
+ Op = Op.getNode()->getOperand(0);
TableId &PromotedId = SoftPromotedHalfs[getTableId(Op)];
SDValue PromotedOp = getSDValue(PromotedId);
assert(PromotedOp.getNode() && "Operand wasn't promoted?");
@@ -820,6 +826,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue SoftPromoteHalfRes_SELECT(SDNode *N);
SDValue SoftPromoteHalfRes_SELECT_CC(SDNode *N);
SDValue SoftPromoteHalfRes_UnaryOp(SDNode *N);
+ SDValue SoftPromoteHalfRes_UnaryOpExt1(SDNode *N);
SDValue SoftPromoteHalfRes_XINT_TO_FP(SDNode *N);
SDValue SoftPromoteHalfRes_UNDEF(SDNode *N);
SDValue SoftPromoteHalfRes_VECREDUCE(SDNode *N);
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
index d0b69b88748a9..834f0e2b8038d 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
@@ -61,6 +61,7 @@ void DAGTypeLegalizer::ScalarizeVectorResult(SDNode *N, unsigned ResNo) {
case ISD::AssertZext:
case ISD::AssertSext:
case ISD::FPOWI:
+ case ISD::AssertNoFPClass:
R = ScalarizeVecRes_UnaryOpWithExtraInput(N);
break;
case ISD::INSERT_VECTOR_ELT: R = ScalarizeVecRes_INSERT_VECTOR_ELT(N); break;
@@ -1276,6 +1277,7 @@ void DAGTypeLegalizer::SplitVectorResult(SDNode *N, unsigned ResNo) {
case ISD::UINT_TO_FP:
case ISD::VP_UINT_TO_FP:
case ISD::FCANONICALIZE:
+ case ISD::AssertNoFPClass:
SplitVecRes_UnaryOp(N, Lo, Hi);
break;
case ISD::ADDRSPACECAST:
@@ -2614,7 +2616,7 @@ void DAGTypeLegalizer::SplitVecRes_UnaryOp(SDNode *N, SDValue &Lo,
const SDNodeFlags Flags = N->getFlags();
unsigned Opcode = N->getOpcode();
if (N->getNumOperands() <= 2) {
- if (Opcode == ISD::FP_ROUND) {
+ if (Opcode == ISD::FP_ROUND || Opcode == ISD::AssertNoFPClass) {
Lo = DAG.getNode(Opcode, dl, LoVT, Lo, N->getOperand(1), Flags);
Hi = DAG.getNode(Opcode, dl, HiVT, Hi, N->getOperand(1), Flags);
} else {
@@ -4879,6 +4881,7 @@ void DAGTypeLegalizer::WidenVectorResult(SDNode *N, unsigned ResNo) {
case ISD::FREEZE:
case ISD::ARITH_FENCE:
case ISD::FCANONICALIZE:
+ case ISD::AssertNoFPClass:
Res = WidenVecRes_Unary(N);
break;
case ISD::FMA: case ISD::VP_FMA:
@@ -5623,6 +5626,9 @@ SDValue DAGTypeLegalizer::WidenVecRes_Unary(SDNode *N) {
SDValue InOp = GetWidenedVector(N->getOperand(0));
if (N->getNumOperands() == 1)
return DAG.getNode(N->getOpcode(), SDLoc(N), WidenVT, InOp, N->getFlags());
+ if (N->getOpcode() == ISD::AssertNoFPClass)
+ return DAG.getNode(N->getOpcode(), SDLoc(N), WidenVT, InOp,
+ N->getOperand(1), N->getFlags());
assert(N->getNumOperands() == 3 && "Unexpected number of operands!");
assert(N->isVPOpcode() && "Expected VP opcode");
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index effe08cdd44f8..dc92b66eca639 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -5832,6 +5832,15 @@ bool SelectionDAG::isKnownNeverNaN(SDValue Op, const APInt &DemandedElts,
return false;
return true;
}
+ case ISD::AssertNoFPClass: {
+ FPClassTest NoFPClass =
+ static_cast<FPClassTest>(Op.getConstantOperandVal(1));
+ if ((NoFPClass & fcNan) == fcNan)
+ return true;
+ if (SNaN && (NoFPClass & fcSNan) == fcSNan)
+ return true;
+ return isKnownNeverNaN(Op.getOperand(0), DemandedElts, SNaN, Depth + 1);
+ }
default:
if (Opcode >= ISD::BUILTIN_OP_END || Opcode == ISD::INTRINSIC_WO_CHAIN ||
Opcode == ISD::INTRINSIC_W_CHAIN || Opcode == ISD::INTRINSIC_VOID) {
@@ -7491,6 +7500,16 @@ SDValue SelectionDAG::getNode(unsigned Opcode, const SDLoc &DL, EVT VT,
N2.getOpcode() == ISD::TargetConstant && "Invalid FP_ROUND!");
if (N1.getValueType() == VT) return N1; // noop conversion.
break;
+ case ISD::AssertNoFPClass: {
+ assert(N1.getValueType().isFloatingPoint() &&
+ "AssertNoFPClass is used for a non-floating type");
+ assert(isa<ConstantSDNode>(N2) && "NoFPClass is not Constant");
+ FPClassTest NoFPClass = static_cast<FPClassTest>(N2->getAsZExtVal());
+ assert(llvm::to_underlying(NoFPClass) <=
+ BitmaskEnumDetail::Mask<FPClassTest>() &&
+ "FPClassTest value too large");
+ break;
+ }
case ISD::AssertSext:
case ISD::AssertZext: {
EVT EVT = cast<VTSDNode>(N2)->getVT();
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 9d138d364bad7..c5f65cee23c5b 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -11803,9 +11803,18 @@ void SelectionDAGISel::LowerArguments(const Function &F) {
else if (Arg.hasAttribute(Attribute::ZExt))
AssertOp = ISD::AssertZext;
- ArgValues.push_back(getCopyFromParts(DAG, dl, &InVals[i], NumParts,
- PartVT, VT, nullptr, NewRoot,
- F.getCallingConv(), AssertOp));
+ SDValue OutVal =
+ getCopyFromParts(DAG, dl, &InVals[i], NumParts, PartVT, VT, nullptr,
+ NewRoot, F.getCallingConv(), AssertOp);
+
+ FPClassTest NoFPClass = Arg.getNoFPClass();
+ if (NoFPClass != fcNone) {
+ SDValue SDNoFPClass = DAG.getTargetConstant(
+ static_cast<uint64_t>(NoFPClass), dl, MVT::i32);
+ OutVal = DAG.getNode(ISD::AssertNoFPClass, dl, OutVal.getValueType(),
+ OutVal, SDNoFPClass);
+ }
+ ArgValues.push_back(OutVal);
}
i += NumParts;
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
index 8faf97271d99e..6f846bedf3c82 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
@@ -124,6 +124,7 @@ std::string SDNode::getOperationName(const SelectionDAG *G) const {
case ISD::TokenFactor: return "TokenFactor";
case ISD::AssertSext: return "AssertSext";
case ISD::AssertZext: return "AssertZext";
+ case ISD::AssertNoFPClass: return "AssertNoFPClass";
case ISD::AssertAlign: return "AssertAlign";
case ISD::BasicBlock: return "BasicBlock";
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
index 1bc30336a02bf..586728a44571e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
@@ -3264,6 +3264,7 @@ void SelectionDAGISel::SelectCodeCommon(SDNode *NodeToMatch,
return;
case ISD::AssertSext:
case ISD::AssertZext:
+ case ISD::AssertNoFPClass:
case ISD::AssertAlign:
ReplaceUses(SDValue(NodeToMatch, 0), NodeToMatch->getOperand(0));
CurDAG->RemoveDeadNode(NodeToMatch);
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index ba34c72156228..78a6e9f458a75 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -11542,22 +11542,47 @@ SDValue TargetLowering::expandFP_TO_INT_SAT(SDNode *Node,
// If the integer bounds are exactly representable as floats and min/max are
// legal, emit a min+max+fptoi sequence. Otherwise we have to use a sequence
// of comparisons and selects.
- bool MinMaxLegal = isOperationLegal(ISD::FMINNUM, SrcVT) &&
+ bool MinMax2019NumLegal = isOperationLegal(ISD::FMINIMUMNUM, SrcVT) &&
+ isOperationLegal(ISD::FMAXIMUMNUM, SrcVT);
+ bool MinMax2019Legal = isOperationLegal(ISD::FMINIMUM, SrcVT) &&
+ isOperationLegal(ISD::FMAXIMUM, SrcVT);
+ bool MinMax2008Legal = isOperationLegal(ISD::FMINNUM, SrcVT) &&
isOperationLegal(ISD::FMAXNUM, SrcVT);
- if (AreExactFloatBounds && MinMaxLegal) {
- SDValue Clamped = Src;
- // Clamp Src by MinFloat from below. If Src is NaN the result is MinFloat.
- Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, MinFloatNode);
- // Clamp by MaxFloat from above. NaN cannot occur.
- Clamped = DAG.getNode(ISD::FMINNUM, dl, SrcVT, Clamped, MaxFloatNode);
+ if (AreExactFloatBounds && (MinMax2019NumLegal || MinMax2019Legal || MinMax2008Legal)) {
+ SDValue Clamped = Src;
+ bool Use2019 = false;
+
+ if (MinMax2019NumLegal) {
+ // Clamp Src by MinFloat from below. If Src is NaN the result is MinFloat.
+ Clamped = DAG.getNode(ISD::FMAXIMUMNUM, dl, SrcVT, Clamped, MinFloatNode);
+ // Clamp by MaxFloat from above. NaN cannot occur.
+ Clamped = DAG.getNode(ISD::FMINIMUMNUM, dl, SrcVT, Clamped, MaxFloatNode);
+ } else if (MinMax2008Legal) {
+ // Try 2008 first as it has better performance for converting SNaN to unsigned.
+ if (!IsSigned && !DAG.isKnownNeverSNaN(Clamped)) {
+ Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, Clamped);
+ }
+ // Clamp Src by MinFloat from below. If Src is NaN the result is MinFloat.
+ Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, MinFloatNode);
+ // Clamp by MaxFloat from above. NaN cannot occur.
+ Clamped = DAG.getNode(ISD::FMINNUM, dl, SrcVT, Clamped, MaxFloatNode);
+ } else if (MinMax2019Legal) {
+ // Clamp Src by MinFloat from below. If Src is NaN the result is qNaN.
+ Clamped = DAG.getNode(ISD::FMAXIMUM, dl, SrcVT, Clamped, MinFloatNode);
+ // Clamp by MaxFloat from above. NaN may occur.
+ Clamped = DAG.getNode(ISD::FMINIMUM, dl, SrcVT, Clamped, MaxFloatNode);
+ Use2019 = true;
+ } else {
+ llvm_unreachable("No Min/Max supported?");
+ }
// Convert clamped value to integer.
SDValue FpToInt = DAG.getNode(IsSigned ? ISD::FP_TO_SINT : ISD::FP_TO_UINT,
dl, DstVT, Clamped);
// In the unsigned case we're done, because we mapped NaN to MinFloat,
// which will cast to zero.
- if (!IsSigned)
+ if ((!IsSigned && !Use2019) || DAG.isKnownNeverNaN(Src))
return FpToInt;
// Otherwise, select 0 if Src is NaN.
diff --git a/llvm/test/CodeGen/AArch64/nofpclass.ll b/llvm/test/CodeGen/AArch64/nofpclass.ll
new file mode 100644
index 0000000000000..3139aa0ef0bf6
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/nofpclass.ll
@@ -0,0 +1,182 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc --mtriple=aarch64-linux-gnu < %s | FileCheck %s
+
+define float @f(float nofpclass(nan) %a, float nofpclass(nan) %b) {
+; CHECK-LABEL: f:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm s0, s0, s1
+; CHECK-NEXT: ret
+entry:
+ %cond = tail call float @llvm.maximumnum.f32(float %a, float %b)
+ ret float %cond
+}
+
+define <4 x float> @fv4f32(<4 x float> nofpclass(nan) %a, <4 x float> nofpclass(nan) %b) {
+; CHECK-LABEL: fv4f32:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm v0.4s, v0.4s, v1.4s
+; CHECK-NEXT: ret
+entry:
+ %c = call <4 x float> @llvm.maximumnum.v4f32(<4 x float> %a, <4 x float> %b)
+ ret <4 x float> %c
+}
+
+define {float, float} @m({float, float} nofpclass(nan) %a0, {float, float} nofpclass(nan) %a1) {
+; CHECK-LABEL: m:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm s1, s1, s3
+; CHECK-NEXT: fmaxnm s0, s0, s2
+; CHECK-NEXT: ret
+entry:
+ %a0f0 = extractvalue {float, float} %a0, 0
+ %a0f1 = extractvalue {float, float} %a0, 1
+ %a1f0 = extractvalue {float, float} %a1, 0
+ %a1f1 = extractvalue {float, float} %a1, 1
+ %max0 = tail call float @llvm.maximumnum.f32(float %a0f0, float %a1f0)
+ %max1 = tail call float @llvm.maximumnum.f32(float %a0f1, float %a1f1)
+ %ret0 = insertvalue {float, float} poison, float %max0, 0
+ %ret1 = insertvalue {float, float} %ret0, float %max1, 1
+ ret {float, float} %ret1
+}
+
+define [2 x float] @mA([2 x float] nofpclass(nan) %a0, [2 x float] nofpclass(nan) %a1) {
+; CHECK-LABEL: mA:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm s1, s1, s3
+; CHECK-NEXT: fmaxnm s0, s0, s2
+; CHECK-NEXT: ret
+entry:
+ %a0f0 = extractvalue [2 x float] %a0, 0
+ %a0f1 = extractvalue [2 x float] %a0, 1
+ %a1f0 = extractvalue [2 x float] %a1, 0
+ %a1f1 = extractvalue [2 x float] %a1, 1
+ %max0 = tail call floa...
[truncated]
|
|
@llvm/pr-subscribers-backend-aarch64 Author: YunQiang Su (wzssyqa) ChangesCurrently, expandFP_TO_INT_SAT uses FMAXNUM and FMINNUM, which is Let's try all 3 flavor of Max/Min for it:
Patch is 44.47 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/139217.diff 16 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index 80ef32aff62ae..1042318343987 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -67,6 +67,15 @@ enum NodeType {
/// poisoned the assertion will not be true for that value.
AssertAlign,
+ /// AssertNoFPClass - These nodes record if a register contains a float
+ /// value that is known to be not some type.
+ /// This node takes two operands. The first is the node that is known
+ /// never to be some float types; the second is a constant value with
+ /// the value of FPClassTest (casted to uint32_t).
+ /// NOTE: In case of the source value (or any vector element value) is
+ /// poisoned the assertion will not be true for that value.
+ AssertNoFPClass,
+
/// Various leaf nodes.
BasicBlock,
VALUETYPE,
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 41fed692c7025..b28a8b118de7a 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -875,6 +875,7 @@ def SDT_assert : SDTypeProfile<1, 1,
[SDTCisInt<0>, SDTCisInt<1>, SDTCisSameAs<1, 0>]>;
def assertsext : SDNode<"ISD::AssertSext", SDT_assert>;
def assertzext : SDNode<"ISD::AssertZext", SDT_assert>;
+def assertnofpclass : SDNode<"ISD::AssertNoFPClass", SDTFPUnaryOp>;
def assertalign : SDNode<"ISD::AssertAlign", SDT_assert>;
def convergencectrl_anchor : SDNode<"ISD::CONVERGENCECTRL_ANCHOR",
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
index 432209e8ecb0a..ea0f56d7741ac 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
@@ -168,6 +168,7 @@ void DAGTypeLegalizer::SoftenFloatResult(SDNode *N, unsigned ResNo) {
case ISD::POISON:
case ISD::UNDEF: R = SoftenFloatRes_UNDEF(N); break;
case ISD::VAARG: R = SoftenFloatRes_VAARG(N); break;
+ case ISD::AssertNoFPClass: R = GetSoftenedFloat(N->getOperand(0)); break;
case ISD::VECREDUCE_FADD:
case ISD::VECREDUCE_FMUL:
case ISD::VECREDUCE_FMIN:
@@ -2582,6 +2583,7 @@ bool DAGTypeLegalizer::PromoteFloatOperand(SDNode *N, unsigned OpNo) {
case ISD::LLROUND:
case ISD::LRINT:
case ISD::LLRINT: R = PromoteFloatOp_UnaryOp(N, OpNo); break;
+ case ISD::AssertNoFPClass: R = PromoteFloatOp_UnaryOpExt1(N, OpNo); break;
case ISD::FP_TO_SINT_SAT:
case ISD::FP_TO_UINT_SAT:
R = PromoteFloatOp_FP_TO_XINT_SAT(N, OpNo); break;
@@ -2640,6 +2642,13 @@ SDValue DAGTypeLegalizer::PromoteFloatOp_UnaryOp(SDNode *N, unsigned OpNo) {
return DAG.getNode(N->getOpcode(), SDLoc(N), N->getValueType(0), Op);
}
+// Convert the promoted float value to the desired integer type
+SDValue DAGTypeLegalizer::PromoteFloatOp_UnaryOpExt1(SDNode *N, unsigned OpNo) {
+ SDValue Op = GetPromotedFloat(N->getOperand(0));
+ return DAG.getNode(N->getOpcode(), SDLoc(N), N->getValueType(0), Op,
+ N->getOperand(1));
+}
+
SDValue DAGTypeLegalizer::PromoteFloatOp_FP_TO_XINT_SAT(SDNode *N,
unsigned OpNo) {
SDValue Op = GetPromotedFloat(N->getOperand(0));
@@ -2804,6 +2813,9 @@ void DAGTypeLegalizer::PromoteFloatResult(SDNode *N, unsigned ResNo) {
case ISD::FTAN:
case ISD::FTANH:
case ISD::FCANONICALIZE: R = PromoteFloatRes_UnaryOp(N); break;
+ case ISD::AssertNoFPClass:
+ R = PromoteFloatRes_UnaryOpExt1(N);
+ break;
// Binary FP Operations
case ISD::FADD:
@@ -2996,10 +3008,18 @@ SDValue DAGTypeLegalizer::PromoteFloatRes_UnaryOp(SDNode *N) {
EVT VT = N->getValueType(0);
EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
SDValue Op = GetPromotedFloat(N->getOperand(0));
-
return DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Op);
}
+// Unary operation with a more non-float operand where the result and the
+// operand have PromoteFloat type action. Construct a new SDNode with the
+// promoted float value of the old operand.
+SDValue DAGTypeLegalizer::PromoteFloatRes_UnaryOpExt1(SDNode *N) {
+ EVT VT = N->getValueType(0);
+ EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
+ SDValue Op = GetPromotedFloat(N->getOperand(0));
+ return DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Op, N->getOperand(1));
+}
// Binary operations where the result and both operands have PromoteFloat type
// action. Construct a new SDNode with the promoted float values of the old
// operands.
@@ -3281,6 +3301,9 @@ void DAGTypeLegalizer::SoftPromoteHalfResult(SDNode *N, unsigned ResNo) {
case ISD::FTAN:
case ISD::FTANH:
case ISD::FCANONICALIZE: R = SoftPromoteHalfRes_UnaryOp(N); break;
+ case ISD::AssertNoFPClass:
+ R = SoftPromoteHalfRes_UnaryOpExt1(N);
+ break;
// Binary FP Operations
case ISD::FADD:
@@ -3607,6 +3630,21 @@ SDValue DAGTypeLegalizer::SoftPromoteHalfRes_UnaryOp(SDNode *N) {
return DAG.getNode(GetPromotionOpcode(NVT, OVT), dl, MVT::i16, Res);
}
+SDValue DAGTypeLegalizer::SoftPromoteHalfRes_UnaryOpExt1(SDNode *N) {
+ EVT OVT = N->getValueType(0);
+ EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), OVT);
+ SDValue Op = GetSoftPromotedHalf(N->getOperand(0));
+ SDLoc dl(N);
+
+ // Promote to the larger FP type.
+ Op = DAG.getNode(GetPromotionOpcode(OVT, NVT), dl, NVT, Op);
+
+ SDValue Res = DAG.getNode(N->getOpcode(), dl, NVT, Op, N->getOperand(1));
+
+ // Convert back to FP16 as an integer.
+ return DAG.getNode(GetPromotionOpcode(NVT, OVT), dl, MVT::i16, Res);
+}
+
SDValue DAGTypeLegalizer::SoftPromoteHalfRes_BinOp(SDNode *N) {
EVT OVT = N->getValueType(0);
EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), OVT);
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
index 720393158aa5e..f4d5fa230f821 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
@@ -80,7 +80,9 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
/// Pretend all of this node's results are legal.
bool IgnoreNodeResults(SDNode *N) const {
return N->getOpcode() == ISD::TargetConstant ||
- N->getOpcode() == ISD::Register;
+ N->getOpcode() == ISD::Register ||
+ (N->getOpcode() == ISD::AssertNoFPClass &&
+ IgnoreNodeResults(N->getOperand(0).getNode()));
}
// Bijection from SDValue to unique id. As each created node gets a
@@ -772,6 +774,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue PromoteFloatRes_SELECT(SDNode *N);
SDValue PromoteFloatRes_SELECT_CC(SDNode *N);
SDValue PromoteFloatRes_UnaryOp(SDNode *N);
+ SDValue PromoteFloatRes_UnaryOpExt1(SDNode *N);
SDValue PromoteFloatRes_UNDEF(SDNode *N);
SDValue BitcastToInt_ATOMIC_SWAP(SDNode *N);
SDValue PromoteFloatRes_XINT_TO_FP(SDNode *N);
@@ -785,6 +788,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue PromoteFloatOp_FP_EXTEND(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_STRICT_FP_EXTEND(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_UnaryOp(SDNode *N, unsigned OpNo);
+ SDValue PromoteFloatOp_UnaryOpExt1(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_FP_TO_XINT_SAT(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_STORE(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_ATOMIC_STORE(SDNode *N, unsigned OpNo);
@@ -796,6 +800,8 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
//===--------------------------------------------------------------------===//
SDValue GetSoftPromotedHalf(SDValue Op) {
+ while (Op.getNode()->getOpcode() == ISD::AssertNoFPClass)
+ Op = Op.getNode()->getOperand(0);
TableId &PromotedId = SoftPromotedHalfs[getTableId(Op)];
SDValue PromotedOp = getSDValue(PromotedId);
assert(PromotedOp.getNode() && "Operand wasn't promoted?");
@@ -820,6 +826,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue SoftPromoteHalfRes_SELECT(SDNode *N);
SDValue SoftPromoteHalfRes_SELECT_CC(SDNode *N);
SDValue SoftPromoteHalfRes_UnaryOp(SDNode *N);
+ SDValue SoftPromoteHalfRes_UnaryOpExt1(SDNode *N);
SDValue SoftPromoteHalfRes_XINT_TO_FP(SDNode *N);
SDValue SoftPromoteHalfRes_UNDEF(SDNode *N);
SDValue SoftPromoteHalfRes_VECREDUCE(SDNode *N);
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
index d0b69b88748a9..834f0e2b8038d 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
@@ -61,6 +61,7 @@ void DAGTypeLegalizer::ScalarizeVectorResult(SDNode *N, unsigned ResNo) {
case ISD::AssertZext:
case ISD::AssertSext:
case ISD::FPOWI:
+ case ISD::AssertNoFPClass:
R = ScalarizeVecRes_UnaryOpWithExtraInput(N);
break;
case ISD::INSERT_VECTOR_ELT: R = ScalarizeVecRes_INSERT_VECTOR_ELT(N); break;
@@ -1276,6 +1277,7 @@ void DAGTypeLegalizer::SplitVectorResult(SDNode *N, unsigned ResNo) {
case ISD::UINT_TO_FP:
case ISD::VP_UINT_TO_FP:
case ISD::FCANONICALIZE:
+ case ISD::AssertNoFPClass:
SplitVecRes_UnaryOp(N, Lo, Hi);
break;
case ISD::ADDRSPACECAST:
@@ -2614,7 +2616,7 @@ void DAGTypeLegalizer::SplitVecRes_UnaryOp(SDNode *N, SDValue &Lo,
const SDNodeFlags Flags = N->getFlags();
unsigned Opcode = N->getOpcode();
if (N->getNumOperands() <= 2) {
- if (Opcode == ISD::FP_ROUND) {
+ if (Opcode == ISD::FP_ROUND || Opcode == ISD::AssertNoFPClass) {
Lo = DAG.getNode(Opcode, dl, LoVT, Lo, N->getOperand(1), Flags);
Hi = DAG.getNode(Opcode, dl, HiVT, Hi, N->getOperand(1), Flags);
} else {
@@ -4879,6 +4881,7 @@ void DAGTypeLegalizer::WidenVectorResult(SDNode *N, unsigned ResNo) {
case ISD::FREEZE:
case ISD::ARITH_FENCE:
case ISD::FCANONICALIZE:
+ case ISD::AssertNoFPClass:
Res = WidenVecRes_Unary(N);
break;
case ISD::FMA: case ISD::VP_FMA:
@@ -5623,6 +5626,9 @@ SDValue DAGTypeLegalizer::WidenVecRes_Unary(SDNode *N) {
SDValue InOp = GetWidenedVector(N->getOperand(0));
if (N->getNumOperands() == 1)
return DAG.getNode(N->getOpcode(), SDLoc(N), WidenVT, InOp, N->getFlags());
+ if (N->getOpcode() == ISD::AssertNoFPClass)
+ return DAG.getNode(N->getOpcode(), SDLoc(N), WidenVT, InOp,
+ N->getOperand(1), N->getFlags());
assert(N->getNumOperands() == 3 && "Unexpected number of operands!");
assert(N->isVPOpcode() && "Expected VP opcode");
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index effe08cdd44f8..dc92b66eca639 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -5832,6 +5832,15 @@ bool SelectionDAG::isKnownNeverNaN(SDValue Op, const APInt &DemandedElts,
return false;
return true;
}
+ case ISD::AssertNoFPClass: {
+ FPClassTest NoFPClass =
+ static_cast<FPClassTest>(Op.getConstantOperandVal(1));
+ if ((NoFPClass & fcNan) == fcNan)
+ return true;
+ if (SNaN && (NoFPClass & fcSNan) == fcSNan)
+ return true;
+ return isKnownNeverNaN(Op.getOperand(0), DemandedElts, SNaN, Depth + 1);
+ }
default:
if (Opcode >= ISD::BUILTIN_OP_END || Opcode == ISD::INTRINSIC_WO_CHAIN ||
Opcode == ISD::INTRINSIC_W_CHAIN || Opcode == ISD::INTRINSIC_VOID) {
@@ -7491,6 +7500,16 @@ SDValue SelectionDAG::getNode(unsigned Opcode, const SDLoc &DL, EVT VT,
N2.getOpcode() == ISD::TargetConstant && "Invalid FP_ROUND!");
if (N1.getValueType() == VT) return N1; // noop conversion.
break;
+ case ISD::AssertNoFPClass: {
+ assert(N1.getValueType().isFloatingPoint() &&
+ "AssertNoFPClass is used for a non-floating type");
+ assert(isa<ConstantSDNode>(N2) && "NoFPClass is not Constant");
+ FPClassTest NoFPClass = static_cast<FPClassTest>(N2->getAsZExtVal());
+ assert(llvm::to_underlying(NoFPClass) <=
+ BitmaskEnumDetail::Mask<FPClassTest>() &&
+ "FPClassTest value too large");
+ break;
+ }
case ISD::AssertSext:
case ISD::AssertZext: {
EVT EVT = cast<VTSDNode>(N2)->getVT();
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 9d138d364bad7..c5f65cee23c5b 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -11803,9 +11803,18 @@ void SelectionDAGISel::LowerArguments(const Function &F) {
else if (Arg.hasAttribute(Attribute::ZExt))
AssertOp = ISD::AssertZext;
- ArgValues.push_back(getCopyFromParts(DAG, dl, &InVals[i], NumParts,
- PartVT, VT, nullptr, NewRoot,
- F.getCallingConv(), AssertOp));
+ SDValue OutVal =
+ getCopyFromParts(DAG, dl, &InVals[i], NumParts, PartVT, VT, nullptr,
+ NewRoot, F.getCallingConv(), AssertOp);
+
+ FPClassTest NoFPClass = Arg.getNoFPClass();
+ if (NoFPClass != fcNone) {
+ SDValue SDNoFPClass = DAG.getTargetConstant(
+ static_cast<uint64_t>(NoFPClass), dl, MVT::i32);
+ OutVal = DAG.getNode(ISD::AssertNoFPClass, dl, OutVal.getValueType(),
+ OutVal, SDNoFPClass);
+ }
+ ArgValues.push_back(OutVal);
}
i += NumParts;
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
index 8faf97271d99e..6f846bedf3c82 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
@@ -124,6 +124,7 @@ std::string SDNode::getOperationName(const SelectionDAG *G) const {
case ISD::TokenFactor: return "TokenFactor";
case ISD::AssertSext: return "AssertSext";
case ISD::AssertZext: return "AssertZext";
+ case ISD::AssertNoFPClass: return "AssertNoFPClass";
case ISD::AssertAlign: return "AssertAlign";
case ISD::BasicBlock: return "BasicBlock";
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
index 1bc30336a02bf..586728a44571e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp
@@ -3264,6 +3264,7 @@ void SelectionDAGISel::SelectCodeCommon(SDNode *NodeToMatch,
return;
case ISD::AssertSext:
case ISD::AssertZext:
+ case ISD::AssertNoFPClass:
case ISD::AssertAlign:
ReplaceUses(SDValue(NodeToMatch, 0), NodeToMatch->getOperand(0));
CurDAG->RemoveDeadNode(NodeToMatch);
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index ba34c72156228..78a6e9f458a75 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -11542,22 +11542,47 @@ SDValue TargetLowering::expandFP_TO_INT_SAT(SDNode *Node,
// If the integer bounds are exactly representable as floats and min/max are
// legal, emit a min+max+fptoi sequence. Otherwise we have to use a sequence
// of comparisons and selects.
- bool MinMaxLegal = isOperationLegal(ISD::FMINNUM, SrcVT) &&
+ bool MinMax2019NumLegal = isOperationLegal(ISD::FMINIMUMNUM, SrcVT) &&
+ isOperationLegal(ISD::FMAXIMUMNUM, SrcVT);
+ bool MinMax2019Legal = isOperationLegal(ISD::FMINIMUM, SrcVT) &&
+ isOperationLegal(ISD::FMAXIMUM, SrcVT);
+ bool MinMax2008Legal = isOperationLegal(ISD::FMINNUM, SrcVT) &&
isOperationLegal(ISD::FMAXNUM, SrcVT);
- if (AreExactFloatBounds && MinMaxLegal) {
- SDValue Clamped = Src;
- // Clamp Src by MinFloat from below. If Src is NaN the result is MinFloat.
- Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, MinFloatNode);
- // Clamp by MaxFloat from above. NaN cannot occur.
- Clamped = DAG.getNode(ISD::FMINNUM, dl, SrcVT, Clamped, MaxFloatNode);
+ if (AreExactFloatBounds && (MinMax2019NumLegal || MinMax2019Legal || MinMax2008Legal)) {
+ SDValue Clamped = Src;
+ bool Use2019 = false;
+
+ if (MinMax2019NumLegal) {
+ // Clamp Src by MinFloat from below. If Src is NaN the result is MinFloat.
+ Clamped = DAG.getNode(ISD::FMAXIMUMNUM, dl, SrcVT, Clamped, MinFloatNode);
+ // Clamp by MaxFloat from above. NaN cannot occur.
+ Clamped = DAG.getNode(ISD::FMINIMUMNUM, dl, SrcVT, Clamped, MaxFloatNode);
+ } else if (MinMax2008Legal) {
+ // Try 2008 first as it has better performance for converting SNaN to unsigned.
+ if (!IsSigned && !DAG.isKnownNeverSNaN(Clamped)) {
+ Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, Clamped);
+ }
+ // Clamp Src by MinFloat from below. If Src is NaN the result is MinFloat.
+ Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, MinFloatNode);
+ // Clamp by MaxFloat from above. NaN cannot occur.
+ Clamped = DAG.getNode(ISD::FMINNUM, dl, SrcVT, Clamped, MaxFloatNode);
+ } else if (MinMax2019Legal) {
+ // Clamp Src by MinFloat from below. If Src is NaN the result is qNaN.
+ Clamped = DAG.getNode(ISD::FMAXIMUM, dl, SrcVT, Clamped, MinFloatNode);
+ // Clamp by MaxFloat from above. NaN may occur.
+ Clamped = DAG.getNode(ISD::FMINIMUM, dl, SrcVT, Clamped, MaxFloatNode);
+ Use2019 = true;
+ } else {
+ llvm_unreachable("No Min/Max supported?");
+ }
// Convert clamped value to integer.
SDValue FpToInt = DAG.getNode(IsSigned ? ISD::FP_TO_SINT : ISD::FP_TO_UINT,
dl, DstVT, Clamped);
// In the unsigned case we're done, because we mapped NaN to MinFloat,
// which will cast to zero.
- if (!IsSigned)
+ if ((!IsSigned && !Use2019) || DAG.isKnownNeverNaN(Src))
return FpToInt;
// Otherwise, select 0 if Src is NaN.
diff --git a/llvm/test/CodeGen/AArch64/nofpclass.ll b/llvm/test/CodeGen/AArch64/nofpclass.ll
new file mode 100644
index 0000000000000..3139aa0ef0bf6
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/nofpclass.ll
@@ -0,0 +1,182 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc --mtriple=aarch64-linux-gnu < %s | FileCheck %s
+
+define float @f(float nofpclass(nan) %a, float nofpclass(nan) %b) {
+; CHECK-LABEL: f:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm s0, s0, s1
+; CHECK-NEXT: ret
+entry:
+ %cond = tail call float @llvm.maximumnum.f32(float %a, float %b)
+ ret float %cond
+}
+
+define <4 x float> @fv4f32(<4 x float> nofpclass(nan) %a, <4 x float> nofpclass(nan) %b) {
+; CHECK-LABEL: fv4f32:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm v0.4s, v0.4s, v1.4s
+; CHECK-NEXT: ret
+entry:
+ %c = call <4 x float> @llvm.maximumnum.v4f32(<4 x float> %a, <4 x float> %b)
+ ret <4 x float> %c
+}
+
+define {float, float} @m({float, float} nofpclass(nan) %a0, {float, float} nofpclass(nan) %a1) {
+; CHECK-LABEL: m:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm s1, s1, s3
+; CHECK-NEXT: fmaxnm s0, s0, s2
+; CHECK-NEXT: ret
+entry:
+ %a0f0 = extractvalue {float, float} %a0, 0
+ %a0f1 = extractvalue {float, float} %a0, 1
+ %a1f0 = extractvalue {float, float} %a1, 0
+ %a1f1 = extractvalue {float, float} %a1, 1
+ %max0 = tail call float @llvm.maximumnum.f32(float %a0f0, float %a1f0)
+ %max1 = tail call float @llvm.maximumnum.f32(float %a0f1, float %a1f1)
+ %ret0 = insertvalue {float, float} poison, float %max0, 0
+ %ret1 = insertvalue {float, float} %ret0, float %max1, 1
+ ret {float, float} %ret1
+}
+
+define [2 x float] @mA([2 x float] nofpclass(nan) %a0, [2 x float] nofpclass(nan) %a1) {
+; CHECK-LABEL: mA:
+; CHECK: // %bb.0: // %entry
+; CHECK-NEXT: fmaxnm s1, s1, s3
+; CHECK-NEXT: fmaxnm s0, s0, s2
+; CHECK-NEXT: ret
+entry:
+ %a0f0 = extractvalue [2 x float] %a0, 0
+ %a0f1 = extractvalue [2 x float] %a0, 1
+ %a1f0 = extractvalue [2 x float] %a1, 0
+ %a1f1 = extractvalue [2 x float] %a1, 1
+ %max0 = tail call floa...
[truncated]
|
|
Depends on: #138839 It is needed for RISC-V to migrate FMAXNUM to FMAXIMUMNUM, so that we don't need |
You can test this locally with the following command:git-clang-format --diff HEAD~1 HEAD --extensions h,cpp -- llvm/include/llvm/CodeGen/ISDOpcodes.h llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp llvm/lib/CodeGen/SelectionDAG/TargetLowering.cppView the diff from clang-format here.diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index 78a6e9f45..f198e3bef 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -11543,13 +11543,14 @@ SDValue TargetLowering::expandFP_TO_INT_SAT(SDNode *Node,
// legal, emit a min+max+fptoi sequence. Otherwise we have to use a sequence
// of comparisons and selects.
bool MinMax2019NumLegal = isOperationLegal(ISD::FMINIMUMNUM, SrcVT) &&
- isOperationLegal(ISD::FMAXIMUMNUM, SrcVT);
+ isOperationLegal(ISD::FMAXIMUMNUM, SrcVT);
bool MinMax2019Legal = isOperationLegal(ISD::FMINIMUM, SrcVT) &&
- isOperationLegal(ISD::FMAXIMUM, SrcVT);
+ isOperationLegal(ISD::FMAXIMUM, SrcVT);
bool MinMax2008Legal = isOperationLegal(ISD::FMINNUM, SrcVT) &&
- isOperationLegal(ISD::FMAXNUM, SrcVT);
+ isOperationLegal(ISD::FMAXNUM, SrcVT);
- if (AreExactFloatBounds && (MinMax2019NumLegal || MinMax2019Legal || MinMax2008Legal)) {
+ if (AreExactFloatBounds &&
+ (MinMax2019NumLegal || MinMax2019Legal || MinMax2008Legal)) {
SDValue Clamped = Src;
bool Use2019 = false;
@@ -11559,7 +11560,8 @@ SDValue TargetLowering::expandFP_TO_INT_SAT(SDNode *Node,
// Clamp by MaxFloat from above. NaN cannot occur.
Clamped = DAG.getNode(ISD::FMINIMUMNUM, dl, SrcVT, Clamped, MaxFloatNode);
} else if (MinMax2008Legal) {
- // Try 2008 first as it has better performance for converting SNaN to unsigned.
+ // Try 2008 first as it has better performance for converting SNaN to
+ // unsigned.
if (!IsSigned && !DAG.isKnownNeverSNaN(Clamped)) {
Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, Clamped);
}
|
arsenm
reviewed
May 9, 2025
Comment on lines
+11545
to
+11546
| bool MinMax2019NumLegal = isOperationLegal(ISD::FMINIMUMNUM, SrcVT) && | ||
| isOperationLegal(ISD::FMAXIMUMNUM, SrcVT); |
There was a problem hiding this comment.
probably should be isLegalOrCustom
There was a problem hiding this comment.
No. Normally the custom of FMAXIMUMNUM is quite expensive.
Fallback to fcmp is much cheaper.
| Clamped = DAG.getNode(ISD::FMAXNUM, dl, SrcVT, Clamped, MinFloatNode); | ||
| // Clamp by MaxFloat from above. NaN cannot occur. | ||
| Clamped = DAG.getNode(ISD::FMINNUM, dl, SrcVT, Clamped, MaxFloatNode); | ||
| if (AreExactFloatBounds && (MinMax2019NumLegal || MinMax2019Legal || MinMax2008Legal)) { |
There was a problem hiding this comment.
Can we just stick to one, and let separate combines take care of the expansion to one of the other pairs
There was a problem hiding this comment.
I guess no, since it may be more expensive than fallback to fcmp.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, expandFP_TO_INT_SAT uses FMAXNUM and FMINNUM, which is
not correct if the Src is sNaN.
Let's try all 3 flavor of Max/Min for it:
FMAXIMUMNUM/FMINIMUMNUM
See test/CodeGen/RISCV/bfloat-convert.ll
FMAXNUM/FMAXNUM
See test/CodeGen/Mips/Half2Int16.ll
FMAXIMUM/FMAXIMUM
See test/CodeGen/WebAssembly/Half2Int16.ll