[Question] Int8 Gemm's perf degraded in real models.

[foreverlms]

I have encountered a problem that whe benchmark the

TensorRT-LLM/cpp/tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm_template.h

Lines 62 to 63 in a65dba7

	void genericInt8GemmKernelLauncher(int8_t const* A, int8_t const* B, tk::QuantMode quantOption, float const* alphaCol,
	float const* alphaRow, T* C, int m, int n, int k, tkc::CutlassGemmConfig gemmConfig, char* workspace,

Int8 cutlass kernel seperately, the kernel really beats the fp16 gemm kernel.
But when it comes to real model, the int8 gemm kernel's perf degraded a lot. (I did use gemmprofilerplugin).

Two pictures from the nsight:

Int8 Gemm (above: seperate kernels, below: in real models)

int8 in benchmark:

int8 in models:

Device: A100 SXM-80GB

For 16, 6144, 4096; 14us -> 24us, almost doubbled. The config is exactly the same as the nsight shows.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

[foreverlms]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

!

[karljang]

I'm wondering if this issue is still relevant to you, if not, please let us know so we can close it.
While the INT8 GEMM path hasn't changed significantly, the focus seems to have shifted toward FP8, which offers better performance on recent architectures. Most recent optimizations are targeting FP8 rather than INT8. Thank you!

[foreverlms]

OK, close this.

[karljang]

Thank you!