Verify the GPU architecture

Oct 13, 2025 by wbrum.org 20 views

Troubleshooting TensorRT-LLM Error: Arch unsupported for CUTLASS FP4 GEMM

TensorRT-LLM is a powerful library for large language models, but it can be challenging to troubleshoot errors. In this article, we will explore a specific error that occurs when trying to run a model in FP4 format using the LLM API code. The error message is [TensorRT-LLM Error][CutlassFp4GemmRunner][GEMM Dispatch] Arch unsupported for CUTLASS FP4 GEMM.

GPU: 5090D
Project branch: main stream
Docker image: nvcr.io/nvidia/pytorch:25.02-py3
Model conversion: scripts from tensor-model-opimizer
Testing code: provided in the question

The error log is quite extensive, but we will focus on the key parts that are relevant to the issue.

2025-03-10 10:48:56,898 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.18.0.dev2025022500
Using MpiPoolSession to spawn MPI processes
2025-03-10 10:49:05,423 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.18.0.dev2025022500
[TensorRT-LLM][INFO] Refreshed the MPI local session
Loading weights: 100%|██████████| 401/401 [00:27<00:00, 14.85it/s]
Model init total -- 27.48s
2025-03-10 10:49:33,204 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-03-10 10:49:53,309 - INFO - flashinfer.jit: Finished loading JIT ops: norm
[ERROR    | TRT-LLM            ]: [TRT-LLM] [E] Failed to initialize executor on rank 0: [TensorRT-LLM Error][CutlassFp4GemmRunner][GEMM Dispatch] Arch unsupported for CUTLASS FP4 GEMM
[03/10/2025-10:49:53] [TRT-LLM] [E] Failed to initialize executor on rank 0: [TensorRT-LLM Error][CutlassFp4GemmRunner][GEMM Dispatch] Arch unsupported for CUTLASS FP4 GEMM
[ERROR    | TRT-LLM            ]: [TRT-LLM] [E] Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 581, in worker_main
    worker: ExecutorBindingsWorker = worker_cls(
                                     ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 126, in __init__
    self.engine = _create_engine()
                  ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 122, in _create_engine
    return create_executor(executor_config=executor_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 106, in create_py_executor
    kv_cache_max_tokens = estimate_max_kv_cache_tokens(model_engine,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/_util.py", line 118, in estimate_max_kv_cache_tokens
    model_engine.forward(req, resource_manager)
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 990, in forward
    return self._forward_step(inputs, gather_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1030, in _forward_step
    logits = self.model.forward(**inputs,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_utils.py", line 234, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 168, in forward
    hidden_states, residual = decoder_layer(position_ids=position_ids,
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 113, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/attention.py", line 113, in forward
    qkv = self.qkv_proj(hidden_states)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 367, in forward
    output = self.apply_linear(input, self.weight, self.bias)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 328, in apply_linear
    output = self.profiler.run_gemm(act_fp4, self.weight, act_sf,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: [TensorRT-LLM Error][CutlassFp4GemmRunner][GEMM Dispatch] Arch unsupported for CUTLASS FP4 GEMM

Check the model format: Ensure that the model is in the correct format (FP4) and that the conversion scripts are up-to-date.
Verify the GPU architecture: Confirm that the GPU architecture is compatible with the CUTLASS FP4 GEMM kernel.
Update the TensorRT-LLM library: Make sure that the TensorRT-LLM library is up-to-date and includes the latest fixes for the CUTLASS FP4 GEMM kernel.
Check the configuration files: Verify that the configuration files (e.g., config.json) are correctly set up and that the model parameters are correctly specified.
Run the model in debug mode: Run the model in debug mode to get more detailed error messages and to identify the specific issue.

**Q&A: Troubleshooting TensorRT-LLM Error: Arch unsupported for CUTLASS FP4 GEMM**

Q: What is the cause of the "Arch unsupported for CUTLASS FP4 GEMM" error?

A: The "Arch unsupported for CUTLASS FP4 GEMM" error is caused by a compatibility issue between the GPU architecture and the CUTLASS FP4 GEMM kernel. This error can occur when the model is converted to FP4 format and the GPU architecture is not compatible with the CUTLASS FP4 GEMM kernel.

Q: How can I verify the GPU architecture?

A: To verify the GPU architecture, you can use the nvidia-smi command to check the GPU model and architecture. For example:

nvidia-smi -L

This will display the GPU model and architecture, which you can then use to verify compatibility with the CUTLASS FP4 GEMM kernel.

Q: How can I update the TensorRT-LLM library?

A: To update the TensorRT-LLM library, you can use the following command:

pip install --upgrade tensorrt-llm

This will update the TensorRT-LLM library to the latest version, which may include fixes for the CUTLASS FP4 GEMM kernel.

Q: What are the configuration files that I need to check?

A: The configuration files that you need to check are the config.json file and any other configuration files that are specific to your model. These files contain the model parameters and settings, which may need to be adjusted to resolve the "Arch unsupported for CUTLASS FP4 GEMM" error.

Q: How can I run the model in debug mode?

A: To run the model in debug mode, you can add the --debug flag to the command that runs the model. For example:

python -m tensorrt_llm --debug

This will enable debug mode, which will provide more detailed error messages and help you identify the specific issue.

Q: What are the system requirements for running the TensorRT-LLM library?

A: The system requirements for running the TensorRT-LLM library are:

NVIDIA GPU with CUDA support
Python 3.6 or later
TensorRT 7.0 or later
CUDA 10.2 or later

Q: Can I use the TensorRT-LLM library with other deep learning frameworks?

A: Yes, the TensorRT-LLM library can be used with other deep learning frameworks, such as TensorFlow and PyTorch. However, you may need to modify the configuration files and model parameters to work with these frameworks.

Q: How can I get support for the TensorRT-LLM library?

A: You can get support for the TensorRT-LLM library by:

Checking the official documentation and tutorials
Joining the NVIDIA Developer Community forum
Submitting a support request through the NVIDIA website
Contacting a NVIDIA support representative directly