Use a Different Parallelization Technique

Oct 13, 2025 by wbrum.org 36 views

Introduction

In this article, we will explore a bug that occurs when running batch inference with DeepSpeed's Distributed Data Parallel (DDP) and Model Parallelism (TP) enabled. The bug causes the inference code to hang indefinitely when the zero stage is set to 3. We will provide a minimal code snippet to reproduce the issue and discuss possible solutions.

Background

DeepSpeed is a library that provides optimized and scalable deep learning training and inference. It supports various parallelization techniques, including Distributed Data Parallel (DDP) and Model Parallelism (TP). DDP allows for data parallelism across multiple GPUs, while TP enables model parallelism by splitting the model into smaller parts and processing them in parallel.

Minimal Code Snippet

The following code snippet demonstrates the bug:

import os

import torch
import torch.distributed as dist
from transformers import AutoModelForCausalLM, AutoTokenizer

import deepspeed

# Initialize distributed environment
def setup_distributed():
    dist.init_process_group(backend="nccl", init_method="env://")
    local_rank = int(os.getenv("LOCAL_RANK", 0))
    torch.cuda.set_device(local_rank)
    return local_rank


def load_model(model_name="facebook/opt-1.3b", local_rank=0):
    # Ensure distributed environment is set up
    if not dist.is_initialized():
        dist.init_process_group(backend="nccl", init_method="env://")

    world_size = dist.get_world_size()  # Number of GPUs available
    torch.cuda.set_device(local_rank)  # Assign each process to a GPU

    print(
        f"Loading model {model_name} on rank {local_rank}, using {world_size} GPUs for model parallelism"
    )

    # Load model and tokenizer
    model = AutoModelForCausalLM.from_pretrained(model_name)
    tokenizer = AutoTokenizer.from_pretrained(model_name)

    # ✅ DeepSpeed Inference config for Model Parallelism
    ds_config = {
        # "replace_with_kernel_inject": False,  # Enables optimized inference kernels
        "tensor_parallel": {"tp_size": 2},  # Enables Model Parallelism
        "dtype": "bf16"
        if torch.cuda.is_bf16_supported()
        else "fp16",  # Automatic dtype selection
    }

    # ✅ Initialize DeepSpeed for Model Parallel Inference
    model = deepspeed.init_inference(model, config=ds_config)

    return model, tokenizer


# Perform inference with data parallelism
def batch_inference(model, tokenizer, prompts, local_rank):
    inputs = tokenizer(prompts, return_tensors="pt", padding=True, truncation=True).to(
        f"cuda:{local_rank}"
    )
    with torch.no_grad():
        outputs = model.generate(**inputs, max_length=150, synced_gpus=True)
    return tokenizer.batch_decode(outputs, skip_special_tokens=True)


def main():
    local_rank = setup_distributed()
    model, tokenizer = load_model(local_rank=local_rank)

    # Each GPU gets a different batch
    global_batch = [
        [
            "What is AI?",
            "Explain deep learning.",
        ],  # Batch for GPU 0
        [
            "Tell me a joke.",
            "What is reinforcement learning? Tell me all the details",
        ],  # Batch for GPU 1
    ]
    prompts = global_batch[local_rank] if local_rank < len(global_batch) else []

    print(f"GPU {local_rank} prompts:", prompts)
    # Perform batch inference
    results = batch_inference(model, tokenizer, prompts, local_rank)
    print(f"GPU {local_rank} results:", results)

    dist.barrier()  # Ensure all GPUs finish


if __name__ == "__main__":
    main()

Running the Code

To reproduce the bug, run the code with the following command:

NCCL_DEBUG=INFO NCCL_BLOCKING_WAIT=1 NCCL_ASYNC_ERROR_HANDLING=1 deepspeed --num_gpus 2 test_deepspeed.py

The Bug

The bug occurs when the zero stage is set to 3 and DDP + TP are enabled. The code hangs indefinitely, and no error message is printed.

Possible Solutions

To fix the bug, we can try the following solutions:

Disable Model Parallelism: Set tensor_parallel to {"tp_size": 1} to disable model parallelism.
Increase the Zero Stage: Increase the zero stage to a higher value, such as 4 or 5, to see if the bug is resolved.
Use a Different Parallelization Technique: Try using a different parallelization technique, such as DDP only or TP only, to see if the bug is resolved.

Conclusion

Q&A

Q: What is the bug that occurs when running batch inference with DeepSpeed's DDP and TP enabled? A: The bug causes the inference code to hang indefinitely when the zero stage is set to 3.

Q: What is the cause of the bug? A: The bug is caused by the combination of DDP and TP enabled with the zero stage set to 3.

Q: How can I reproduce the bug? A: To reproduce the bug, run the code with the following command:

NCCL_DEBUG=INFO NCCL_BLOCKING_WAIT=1 NCCL_ASYNC_ERROR_HANDLING=1 deepspeed --num_gpus 2 test_deepspeed.py

Q: What are the possible solutions to fix the bug? A: There are several possible solutions to fix the bug:

Disable Model Parallelism: Set tensor_parallel to {"tp_size": 1} to disable model parallelism.
Increase the Zero Stage: Increase the zero stage to a higher value, such as 4 or 5, to see if the bug is resolved.
Use a Different Parallelization Technique: Try using a different parallelization technique, such as DDP only or TP only, to see if the bug is resolved.

Q: What are the benefits of using DeepSpeed's DDP and TP? A: DeepSpeed's DDP and TP provide several benefits, including:

Scalability: DDP and TP enable scalable training and inference on large models and datasets.
Performance: DDP and TP can improve performance by reducing communication overhead and increasing parallelism.
Flexibility: DDP and TP provide flexibility in terms of model parallelism and data parallelism.

Q: How can I troubleshoot the bug? A: To troubleshoot the bug, you can try the following steps:

Check the code: Review the code to ensure that it is correct and that there are no syntax errors.
Check the environment: Ensure that the environment is set up correctly, including the number of GPUs and the zero stage.
Check the logs: Check the logs to see if there are any error messages or warnings that may indicate the cause of the bug.

Q: Where can I find more information about DeepSpeed's DDP and TP? A: You can find more information about DeepSpeed's DDP and TP in the following resources:

DeepSpeed documentation: The DeepSpeed documentation provides detailed information about DDP and TP, including configuration options and usage examples.
DeepSpeed community: The DeepSpeed community provides a forum for discussing DeepSpeed-related topics, including DDP and TP.
DeepSpeed GitHub repository: The DeepSpeed GitHub repository provides access to the source code and issue tracker for DeepSpeed.

Conclusion

In this article, we explored a bug that occurs when running batch inference with DeepSpeed's DDP and TP enabled. We provided a minimal code snippet to reproduce the issue and discussed possible solutions. By disabling model parallelism, increasing the zero stage, or using a different parallelization technique, we can resolve the bug and run the code successfully. We also provided a Q&A section to answer common questions about the bug and DeepSpeed's DDP and TP.