Skip to content

[VPlan] Move FOR splice cost into VPInstruction::FirstOrderRecurrenceSplice #129645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 14, 2025
Merged

[VPlan] Move FOR splice cost into VPInstruction::FirstOrderRecurrenceSplice #129645

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a3d6cf4
Precommit test
lukel97 Mar 4, 2025
eaf1947
[VPlan] Move FOR splice cost into VPInstruction::FirstOrderRecurrence…
lukel97 Mar 4, 2025
6e4fd8f
Add assertion message
lukel97 Mar 5, 2025
508ebe9
Merge branch 'main' of github.com:llvm/llvm-project into loop-vectori…
lukel97 Mar 13, 2025
c5f5681
Remove getVPSingleValue
lukel97 Mar 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -754,6 +754,17 @@ InstructionCost VPInstruction::computeCost(ElementCount VF,
return Cost + Ctx.TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy,
Ctx.CostKind);
}
case VPInstruction::FirstOrderRecurrenceSplice: {
assert(VF.isVector() && "Scalar FirstOrderRecurrenceSplice?");
SmallVector<int> Mask(VF.getKnownMinValue());
std::iota(Mask.begin(), Mask.end(), VF.getKnownMinValue() - 1);
Type *VectorTy =
toVectorTy(Ctx.Types.inferScalarType(this->getVPSingleValue()), VF);

return Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Splice,
cast<VectorType>(VectorTy), Mask,
Ctx.CostKind, VF.getKnownMinValue() - 1);
}
default:
// TODO: Compute cost other VPInstructions once the legacy cost model has
// been retired.
Expand Down Expand Up @@ -3472,14 +3483,7 @@ VPFirstOrderRecurrencePHIRecipe::computeCost(ElementCount VF,
if (VF.isScalable() && VF.getKnownMinValue() == 1)
return InstructionCost::getInvalid();

SmallVector<int> Mask(VF.getKnownMinValue());
std::iota(Mask.begin(), Mask.end(), VF.getKnownMinValue() - 1);
Type *VectorTy =
toVectorTy(Ctx.Types.inferScalarType(this->getVPSingleValue()), VF);

return Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Splice,
cast<VectorType>(VectorTy), Mask, Ctx.CostKind,
VF.getKnownMinValue() - 1);
return 0;
}

#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
Expand Down
3 changes: 2 additions & 1 deletion llvm/test/Transforms/LoopVectorize/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ define void @first_order_recurrence(ptr noalias %A, ptr noalias %B, i64 %TC) {
; IF-EVL-NEXT: EMIT vp<[[RESUME_EXTRACT:%.+]]> = extract-from-end ir<[[LD]]>, ir<1>
; IF-EVL-NEXT: EMIT branch-on-cond ir<true>
; IF-EVL-NEXT: Successor(s): ir-bb<for.end>, scalar.ph

; IF-EVL: Cost of 0 for VF vscale x 4: FIRST-ORDER-RECURRENCE-PHI ir<[[FOR_PHI]]> = phi ir<33>, ir<[[LD]]>
; IF-EVL: Cost of 4 for VF vscale x 4: WIDEN-INTRINSIC vp<[[SPLICE]]> = call llvm.experimental.vp.splice(ir<[[FOR_PHI]]>, ir<[[LD]]>, ir<-1>, ir<true>, vp<[[PREV_EVL]]>, vp<[[EVL]]>)
entry:
br label %for.body

Expand Down
Loading