Successful Test with Triton Server

Oct 13, 2025 by wbrum.org 31 views

System Information

GPU: 1 * A100 80G
TensorRT: 10.6.0
TensorRT LLM: 0.15.0

Who Can Help?

Unfortunately, there has been no response to this issue.

Information

Official Example Scripts: The official example scripts have been reviewed, but they do not provide a solution to this issue.
Modified Scripts: My own modified scripts have also been reviewed, but they do not provide a solution to this issue.

Tasks

Officially Supported Task: An officially supported task in the examples folder (such as GLUE/SQuAD, ...) has been attempted, but it does not provide a solution to this issue.
Custom Task: A custom task or dataset has been created, but it does not provide a solution to this issue.

Reproduction

The issue has been reproduced by following the instructions at:

https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html#Draft-Target-Model

The draft and target models are Qwen2.5 7b/32b, both quantized as w8a16.

Successful Test

A successful test has been performed using the following script:

export BASE_MODEL_PATH=<path to work dir>

TENSORRT_LLM_DRAFT_MODEL_NAME="tensorrt_llm_draft"
TENSORRT_LLM_MODEL_NAME="tensorrt_llm"

DRAFT_ENGINE_PATH=$BASE_MODEL_PATH/llm_engines_draft
TARGET_ENGINE_PATH=$BASE_MODEL_PATH/llm_engines
TOKENIZER_PATH=$BASE_MODEL_PATH/tokenizer

python3 /app/tensorrt_llm/examples/run.py \
    --tokenizer_dir $TOKENIZER_PATH \
    --draft_engine_dir $DRAFT_ENGINE_PATH \
    --engine_dir $TARGET_ENGINE_PATH \
    --draft_target_model_config="[4,[0],[0],False]" \
    --max_output_len=256 \
    --kv_cache_enable_block_reuse \
    --kv_cache_free_gpu_memory_fraction=0.1 \
    --input_text="How does Draft-Sampling work?"

Successful Test with Triton Server

A successful test has also been performed using the following script:

export BASE_MODEL_PATH=<some local dir>

DRAFT_ENGINE_PATH=$BASE_MODEL_PATH/llm_engines_draft
TARGET_ENGINE_PATH=$BASE_MODEL_PATH/llm_engines
TOKENIZER_PATH=$BASE_MODEL_PATH/tokenizer

ACCUMULATE_TOKEN="false"
BACKEND="tensorrtllm"
BATCH_SCHEDULER_POLICY="guaranteed_no_evict"
BATCHING_STRATEGY="inflight_fused_batching"
BLS_INSTANCE_COUNT="1"
DECODING_MODE="top_k_top_p"
DECOUPLED_MODE="False"
DRAFT_GPU_DEVICE_IDS="0"
E2E_MODEL_NAME="ensemble"
ENABLE_KV_CACHE_REUSE="true"
ENGINE_PATH=$TARGET_ENGINE_PATH
EXCLUDE_INPUT_IN_OUTPUT="false"
KV_CACHE_FREE_GPU_MEM_FRACTION="0.1"
MAX_ATTENTION_WINDOW_SIZE=""
MAX_BEAM_WIDTH="1"
MAX_QUEUE_DELAY_MICROSECONDS="0"
MAX_TOKENS_IN_KV_CACHE=""
NORMALIZE_LOG_PROBS="true"
POSTPROCESSING_INSTANCE_COUNT="1"
PREPROCESSING_INSTANCE_COUNT="1"
TARGET_GPU_DEVICE_IDS="0"
TENSORRT_LLM_DRAFT_MODEL_NAME="tensorrt_llm_draft"
TENSORRT_LLM_MODEL_NAME="tensorrt_llm"
# TOKENIZER_TYPE=llama
TRITON_GRPC_PORT="8001"
TRITON_HTTP_PORT="8000"
TRITON_MAX_BATCH_SIZE="16"
TRITON_METRICS_PORT="8002"
TRITON_REPO="tritonllm_repo"
USE_DRAFT_LOGITS="false"
LOGITS_DATATYPE="TYPE_FP32" # Replace by TYPE_FP16 for FP8 model

BASEDIR=`cd "$(dirname $0)"; pwd`
TRITON_SCRIPTS_DIR=$BASEDIR/configs/triton_trtllm_0.15/scripts
FILL_TEMPLATE=$TRITON_SCRIPTS_DIR/fill_template.py

# Make a copy of triton repo and replace the fields in the configuration files
# cd /app/tensorrtllm_backend/
# apt-get update && apt-get install -y build-essential cmake git-lfs
# pip3 install git-lfs tritonclient grpcio
rm -rf ${TRITON_REPO}
cp -R configs/triton_trtllm_0.15/inflight_batcher_llm ${TRITON_REPO}
python3 $FILL_TEMPLATE -i ${TRITON_REPO}/ensemble/config.pbtxt triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},logits_datatype:${LOGITS_DATATYPE}
python3 $FILL_TEMPLATE -i ${TRITON_REPO}/preprocessing/config.pbtxt tokenizer_dir:${TOKENIZER_PATH},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},preprocessing_instance_count:${PREPROCESSING_INSTANCE_COUNT}
python3 $FILL_TEMPLATE -i ${TRITON_REPO}/postprocessing/config.pbtxt tokenizer_dir:${TOKENIZER_PATH},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},postprocessing_instance_count:${POSTPROCESSING_INSTANCE_COUNT},logits_datatype:${LOGITS_DATATYPE}
python3 $FILL_TEMPLATE -i ${TRITON_REPO}/tensorrt_llm_bls/config.pbtxt triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},decoupled_mode:${DECOUPLED_MODE},accumulate_tokens:${ACCUMULATE_TOKEN},bls_instance_count:${BLS_INSTANCE_COUNT},tensorrt_llm_model_name:${TENSORRT_LLM_MODEL_NAME},tensorrt_llm_draft_model_name:${TENSORRT_LLM_DRAFT_MODEL_NAME},logits_datatype:${LOGITS_DATATYPE}

# Make a copy of tensorrt_llm as configurations of draft / target models.
cp -R ${TRITON_REPO}/tensorrt_llm ${TRITON_REPO}/tensorrt_llm_draft
sed -i 's/name: "tensorrt_llm"/name: "tensorrt_llm_draft"/g' ${TRITON_REPO}/tensorrt_llm_draft/config.pbtxt
python3 $FILL_TEMPLATE -i ${TRITON_REPO}/tensorrt_llm/config.pbtxt          triton_backend:${BACKEND},engine_dir:${TARGET_ENGINE_PATH},decoupled_mode:${DECOUPLED_MODE},max_tokens_in_paged_kv_cache:${MAX_TOKENS_IN_KV_CACHE},max_attention_window_size:${MAX_ATTENTION_WINDOW_SIZE},batch_scheduler_policy:${BATCH_SCHEDULER_POLICY},batching_strategy:${BATCHING_STRATEGY},kv_cache_free_gpu_mem_fraction:${KV_CACHE_FREE_GPU_MEM_FRACTION},exclude_input_in_output:${EXCLUDE_INPUT_IN_OUTPUT},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},max_queue_delay_microseconds:${MAX_QUEUE_DELAY_MICROSECONDS},max_beam_width:${MAX_BEAM_WIDTH},enable_kv_cache_reuse:${ENABLE_KV_CACHE_REUSE},normalize_log_probs:${NORMALIZE_LOG_PROBS},enable_chunked_context:${ENABLE_CHUNKED_CONTEXT},gpu_device_ids:${TARGET_GPU_DEVICE_IDS},decoding_mode:${DECODING_MODE},encoder_input_features_data_type:TYPE_FP16,logits_datatype:${LOGITS_DATATYPE}
python3 $FILL_TEMPLATE -i ${TRITON_REPO}/tensorrt_llm_draft/config.pbtxt    triton_backend:${BACKEND},engine_dir:${DRAFT_ENGINE_PATH},decoupled_mode:${DECOUPLED_MODE},max_tokens_in_paged_kv_cache:${MAX_TOKENS_IN_KV_CACHE},max_attention_window_size:${MAX_ATTENTION_WINDOW_SIZE},batch_scheduler_policy:${BATCH_SCHEDULER_POLICY},batching_strategy:${BATCHING_STRATEGY},kv_cache_free_gpu_mem_fraction:${KV_CACHE_FREE_GPU_MEM_FRACTION},exclude_input_in_output:${EXCLUDE_INPUT_IN_OUTPUT},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},max_queue_delay_microseconds:${MAX_QUEUE_DELAY_MICROSECONDS},max_beam_width:${MAX_BEAM_WIDTH},enable_kv_cache_reuse:${ENABLE_KV_CACHE_REUSE},normalize_log_probs:${NORMALIZE_LOG_PROBS},enable_chunked_context:${ENABLE_CHUNKED_CONTEXT},gpu_device_ids:${DRAFT_GPU_DEVICE_IDS},decoding_mode:${DECODING_MODE},encoder_input_features_data_type:TYPE_FP16,logits_datatype:${LOGITS_DATATYPE}

python3 $TRITON_SCRIPTS_DIR/launch_triton_server.py \
    --model_repo=${TRITON_REPO} \
    --tensorrt_llm_model_name "${TENSORRT_LLM_MODEL_NAME},${TENSORRT_LLM_DRAFT_MODEL_NAME}" \
    --multi-model

Failed Test

However, a failed test has been performed using the following script:

TENSORRT_LLM_DRAFT_MODEL_NAME="tensorrt_llm_draft"
TENSORRT_LLM_MODEL_NAME="tensorrt_llm"

python3 /app/tensorrtllm_backend/tools/inflight_batcher_llm/speculative_decoding_test.py \
    --max-input-len 2048 \
    --dataset=input_data.json \
    --url-target=localhost:8001 \
    --url-draft=localhost:8001 \
    --url-control=localhost:8001 \
    --draft-tensorrt-llm-model-name="${TENSORRT_LLM_DRAFT_MODEL_NAME}" \
    --target-tensorrt-llm-model-name="${TENSORRT_LLM_MODEL_NAME}" \
    --bls-speculative-tensorrt-llm-model-name="tensorrt_llm_bls" \
    --execute-bls-speculative-decoding \
    --disable-output-comparison \
    --num-draft-tokens=<br/>
# **Q&A: Fail to Test DraftTarget Model with Triton Server TensorRTLLM Backend**

## **Q: What is the issue with testing the DraftTarget model with Triton Server TensorRTLLM backend?**

A: The issue is that the test fails with an error message indicating that the embedding bias shape is not as expected. This error occurs when trying to run the speculative decoding test using the `inflight_batcher_llm` script.

## **Q: What are the system requirements for this issue?**

A: The system requirements for this issue are:

* **GPU:** 1 * A100 80G
* **TensorRT:** 10.6.0
* **TensorRT LLM:** 0.15.0

## **Q: What are the steps to reproduce this issue?**

A: The steps to reproduce this issue are:

1. Follow the instructions at: https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html#Draft-Target-Model
2. Use the Qwen2.5 7b/32b model as the draft and target models, both quantized as w8a16.
3. Run the `inflight_batcher_llm` script with the `--draft-target-model-config` option set to `[4,[0],[0],False]`.
4. Run the `speculative_decoding_test.py` script with the `--url-draft` and `--url-target` options set to `localhost:8001`.

## **Q: What are the possible causes of this issue?**

A: The possible causes of this issue are:

* **Incorrect model configuration:** The model configuration may be incorrect, leading to an error in the embedding bias shape.
* **Incorrect tensorrt_llm model:** The tensorrt_llm model may be incorrect, leading to an error in the embedding bias shape.
* **Incorrect triton server configuration:** The triton server configuration may be incorrect, leading to an error in the embedding bias shape.

## **Q: How can this issue be resolved?**

A: This issue can be resolved by:

* **Checking the model configuration:** Verify that the model configuration is correct and that the embedding bias shape is as expected.
* **Checking the tensorrt_llm model:** Verify that the tensorrt_llm model is correct and that the embedding bias shape is as expected.
* **Checking the triton server configuration:** Verify that the triton server configuration is correct and that the embedding bias shape is as expected.

## **Q: What are the next steps to resolve this issue?**

A: The next steps to resolve this issue are:

* **Review the model configuration:** Review the model configuration to ensure that it is correct and that the embedding bias shape is as expected.
* **Review the tensorrt_llm model:** Review the tensorrt_llm model to ensure that it is correct and that the embedding bias shape is as expected.
* **Review the triton server configuration:** Review the triton server configuration to ensure that it is correct and that the embedding bias shape is as expected.

## **Q: What are the potential consequences of not resolving this issue?**

A: The potential consequences of not resolving this issue are:

* **Failed tests:** The tests will fail, and the issue will not be resolved.
* **Incorrect model behavior:** The model may behave incorrectly, leading to incorrect results.
* **System instability:** The system may become unstable, leading to crashes or other issues.

## **Q: What are the best practices to prevent this issue in the future?**

A: The best practices to prevent this issue in the future are:

* **Regularly review the model configuration:** Regularly review the model configuration to ensure that it is correct and that the embedding bias shape is as expected.
* **Regularly review the tensorrt_llm model:** Regularly review the tensorrt_llm model to ensure that it is correct and that the embedding bias shape is as expected.
* **Regularly review the triton server configuration:** Regularly review the triton server configuration to ensure that it is correct and that the embedding bias shape is as expected.