Extend the Llama-Nemotron-Nano-8B perf-integration-tests (cpp)

[venkywonka]

Description

Expand release-perf-regession-testing converage of llama_v3.1_nemotron_nano_8b to map to NIM benchmarking configs only for the cpp backend (as the PyT backend seems to be having issues that need further debugging).

---

What’s inside

Cross product of the following:

- model: llama_v3.1_nemotron_nano_8b
- runtime: bench
- backend: [trt]
- con: [1, 250] # concurrency
- input_output_len:
  - [5000, 500]
  - [500, 2000]
  - [1000, 1000]
  - [20000, 2000]
- quant: [none, fp8]
- dtype: [bfloat16]
- maxbs: [64]

Total: 16 newly added cpp-backend perf teststests/integration/test_lists/qa/trt_llm_release_perf_test.yml, removing previous cpp-backend perf tests, preserving the pyt-backend perf tests.

📊 Performance Benchmark Summary (subset)

As sanity check, below is perf overview of subset (8) of the 32 cases - that span the TRT flow on 1 H100.

Input/Output Length	Concurrency	Precision	Request Throughput (req/s)	Output TPS (tokens/s)	P50 Latency (ms)
5000 / 500	1	BF16	0.2312	115.60	4328.22
5000 / 500	1	FP8	0.4163	208.13	2397.91
500 / 2000	1	BF16	0.0616	123.26	16203.74
500 / 2000	1	FP8	0.1051	210.18	9516.47
5000 / 500	250	BF16	2.0712	1035.62	117121.77
5000 / 500	250	FP8	3.6120	1805.98	67227.43
500 / 2000	250	BF16	0.8633	1726.57	278001.38
500 / 2000	250	FP8	1.4663	2932.65	163683.41

🧵 Observations

• FP8 consistently improves both request throughput and token throughput across all configurations, with up to 2× higher output TPS in high concurrency scenarios.
• Latency (P50) significantly reduces with FP8, especially under low concurrency, showing strong inference-time gains.
• Concurrency scaling: Increasing concurrency to 250 leads to significantly higher overall throughput, though with expected increases in tail latencies.

📂 Dataset Details

Each configuration used synthetic datasets generated with consistent parameters (512 sequences per run) using the script in TensorRT-LLM/benchmarks/cpp/prepare_dataset.py.

[Copilot]

Pull Request Overview

This PR extends the performance integration tests for llama_v3.1_nemotron_nano_8b so that the test cases reflect the updated NIM benchmarking configurations.

• Removed previous torch and TRT backend tests
• Added 32 new test scenarios covering both C++ (cpp) and Python (pyt) backends with various precision, batch, and sequence length configurations

[venkywonka]

I'm locally seeing issue with the above configurations when using pyt backend, hence limiting this PR to only cpp backend. Don't want to check-in tests that break.

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #4892 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #4892 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3544 completed with status: 'FAILURE'

[venkywonka]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #4901 [ run ] triggered by Bot

[venkywonka]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #4904 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #4901 [ run ] completed with state ABORTED

[tensorrt-cicd]

PR_Github #4904 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3556 completed with status: 'FAILURE'

[venkywonka]

@LarryXFly @kaiyux can this be merged?

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #5348 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #5348 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3901 completed with status: 'FAILURE'

[venkywonka]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #5531 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #5531 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4032 completed with status: 'FAILURE'

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #5545 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #5545 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4046 completed with status: 'FAILURE'

[chzblych]

/bot skip --comment "Irrelevant TOT failures"

[tensorrt-cicd]

PR_Github #5573 [ skip ] triggered by Bot

[tensorrt-cicd]

PR_Github #5573 [ skip ] completed with state SUCCESS
Skipping testing for commit 74ff4d8