Cosine similarity calculation

Oct 13, 2025 by wbrum.org 105 views

Bug Report: Unit Test Failure in tests/models/embedding/vision_language/test_phi3v.py

Environment Information

We are using the following environment to run the unit tests:

Operating System: Ubuntu 24.04.1 LTS (x86_64)
Python Version: 3.12.9 (packaged by Anaconda, Inc.)
PyTorch Version: 2.5.1+cu124
CUDA Version: 12.4
Nvidia Driver Version: 550.127.05
cuDNN Version: 9.1.0.70
Transformers Version: 4.49.0
Triton Version: 3.1.0
Vector Quantize PyTorch Version: 1.21.2

Bug Description

The unit test tests/models/embedding/vision_language/test_phi3v.py is failing when running the test case test_models_image[half-TIGER-Lab/VLM2Vec-Full]. The test case is checking the embeddings generated by the Phi-3.5-vision-instruct model using the vllm library.

Error Message

The error message is as follows:

tests/models/embedding/vision_language/test_phi3v.py:119: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/models/embedding/vision_language/test_phi3v.py:69: in _run_test                                                            
    check_embeddings_close(                                                                                                      
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                                                                                                                                 
    def check_embeddings_close(                                                                                                  
        *,                                                                                                                       
        embeddings_0_lst: Sequence[list[float]],                                                                                 
        embeddings_1_lst: Sequence[list[float]],                                                                                 
        name_0: str,                                                                                                             
        name_1: str,                                                                                                             
        tol: float = 1e-3,                                                                                                       
    ) -> None:                                                                                                                   
        assert len(embeddings_0_lst) == len(embeddings_1_lst)                                                                    
                                                                                                                                 
        for prompt_idx, (embeddings_0, embeddings_1) in enumerate(                                                               
                zip(embeddings_0_lst, embeddings_1_lst)):                                                                        
            assert len(embeddings_0) == len(embeddings_1), (                                                                     
                f"Length mismatch: {len(embeddings_0)} vs. {len(embeddings_1)}")                                                 
                                                                                                                                 
            sim = F.cosine_similarity(torch.tensor(embeddings_0),                                                                
                                      torch.tensor(embeddings_1),                                                                
                                      dim=0)                                                                                     
                                                                                                                                 
            fail_msg = (f"Test{prompt_idx}:"                                                                                     
                        f"\n{name_0}:\t{embeddings_0[:16]!r}"                                                                    
                        f"\n{name_1}:\t{embeddings_1[:16]!r}")   
>           assert sim >= 1 - tol, fail_msg
E           AssertionError: Test0:
E           hf: [0.0005359649658203125, 0.0028438568115234375, -0.033050537109375, 0.009765625, 0.003566741943359375, 0.004444122314453125, -0.0088958740234375, -0.032867431640625, -0.006267547607421875, -0.021881103515625, 0.021087646484375, 0.0010557174682617188, 0.0147247314453125, -0.004642486572265625, 0.0032863616943359375, 0.006305694580078125]
E           vllm:       [0.0029010772705078125, 0.005504608154296875, -0.030792236328125, 0.00962066650390625, 0.0016222000122070312, 0.0036258697509765625, -0.00925445556640625, -0.03411865234375, -0.005634307861328125, -0.0216522216796875, 0.022430419921875, 0.0002880096435546875, 0.016082763671875, -0.00467681884765625, 0.004161834716796875, 0.004535675048828125]

tests/models/embedding/utils.py:32: AssertionError
------------------------------------------------------- Captured log call -------------------------------------------------------
WARNING  transformers_modules.microsoft.Phi-3.5-vision-instruct.4a0d683eba9f1d0cbfb6151705d1ee73c25a80ca.modeling_phi3_v:logging.py:329 You are not running the flash-attention implementation, expect numerical differences.
======================================================= warnings summary ========================================================
tests/models/embedding/vision_language/test_phi3v.py::test_models_image[half-TIGER-Lab/VLM2Vec-Full]
  /home/tan/miniconda3/envs/vllm073/lib/python3.12/site-packages/transformers/models/auto/image_processing_auto.py:594: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================================== short test summary info ====================================================
FAILED tests/models/embedding/vision_language/test_phi3v.py::test_models_image[half-TIGER-Lab/VLM2Vec-Full] - AssertionError: Test0:
=========================================== 1 failed, 1 warning in 311.72s (0:05:11) ============================================

**Troubleshooting Steps**

To troubleshoot this issue, we can try the following steps:

1. **Check the model weights**: Make sure that the model weights are correctly loaded and that there are no issues with the model architecture.
2. **Verify the input data**: Check that the input data is correctly formatted and that there are no issues with the data preprocessing.
3. **Check the cosine similarity calculation**: Verify that the cosine similarity calculation is correct and that there are no issues with the implementation.
4. **Check the tolerance value**: Verify that the tolerance value is correctly set and that there are no issues with the comparison.

**Additional Information**

* **vllm version**: 0.1.dev5072+g63d635d
* **Transformers version**: 4.49.0
* **Triton version**: 3.1.0
* **Vector Quantize PyTorch version**: 1.21.2

**Before Submitting a New Issue**

* Make sure you have searched for relevant issues and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.<br/>
**Q&A: Troubleshooting Unit Test Failure in `tests/models/embedding/vision_language/test_phi3v.py`**

**Q: What is the issue with the unit test `tests/models/embedding/vision_language/test_phi3v.py`?**

A: The unit test `tests/models/embedding/vision_language/test_phi3v.py` is failing when running the test case `test_models_image[half-TIGER-Lab/VLM2Vec-Full]`. The test case is checking the embeddings generated by the `Phi-3.5-vision-instruct` model using the `vllm` library.

**Q: What is the error message?**

A: The error message is as follows:

tests/models/embedding/vision_language/test_phi3v.py:119:

tests/models/embedding/vision_language/test_phi3v.py:69: in _run_test
check_embeddings_close(

def check_embeddings_close(                                                                                                  
    *,                                                                                                                       
    embeddings_0_lst: Sequence[list[float]],                                                                                 
    embeddings_1_lst: Sequence[list[float]],                                                                                 
    name_0: str,                                                                                                             
    name_1: str,                                                                                                             
    tol: float = 1e-3,                                                                                                       
) -> None:                                                                                                                   
    assert len(embeddings_0_lst) == len(embeddings_1_lst)                                                                    
                                                                                                                             
    for prompt_idx, (embeddings_0, embeddings_1) in enumerate(                                                               
            zip(embeddings_0_lst, embeddings_1_lst)):                                                                        
        assert len(embeddings_0) == len(embeddings_1), (                                                                     
            f"Length mismatch: {len(embeddings_0)} vs. {len(embeddings_1)}")                                                 
                                                                                                                             
        sim = F.cosine_similarity(torch.tensor(embeddings_0),                                                                
                                  torch.tensor(embeddings_1),                                                                
                                  dim=0)                                                                                     
                                                                                                                             
        fail_msg = (f"Test{prompt_idx}:"                                                                                     
                    f"\n{name_0}:\t{embeddings_0[:16]!r}"                                                                    
                    f"\n{name_1}:\t{embeddings_1[:16]!r}")

      assert sim >= 1 - tol, fail_msg

E AssertionError: Test0: E hf: [0.0005359649658203125, 0.0028438568115234375, -0.033050537109375, 0.009765625, 0.003566741943359375, 0.004444122314453125, -0.0088958740234375, -0.032867431640625, -0.006267547607421875, -0.021881103515625, 0.021087646484375, 0.0010557174682617188, 0.0147247314453125, -0.004642486572265625, 0.0032863616943359375, 0.006305694580078125] E vllm: [0.0029010772705078125, 0.005504608154296875, -0.030792236328125, 0.00962066650390625, 0.0016222000122070312, 0.0036258697509765625, -0.00925445556640625, -0.03411865234375, -0.005634307861328125, -0.0216522216796875, 0.022430419921875, 0.0002880096435546875, 0.016082763671875, -0.00467681884765625, 0.004161834716796875, 0.004535675048828125]

tests/models/embedding/utils.py:32: AssertionError ------------------------------------------------------- Captured log call ------------------------------------------------------- WARNING transformers_modules.microsoft.Phi-3.5-vision-instruct.4a0d683eba9f1d0cbfb6151705d1ee73c25a80ca.modeling_phi3_v:logging.py:329 You are not running the flash-attention implementation, expect numerical differences. ======================================================= warnings summary ======================================================== tests/models/embedding/vision_language/test_phi3v.py::test_models_image[half-TIGER-Lab/VLM2Vec-Full] /home/tan/miniconda3/envs/vllm073/lib/python3.12/site-packages/transformers/models/auto/image_processing_auto.py:594: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use slow_image_processor_class, or fast_image_processor_class instead warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ==================================================== short test summary info ==================================================== FAILED tests/models/embedding/vision_language/test_phi3v.py::test_models_image[half-TIGER-Lab/VLM2Vec-Full] - AssertionError: Test0: =========================================== 1 failed, 1 warning in 311.72s (0:05:11) ============================================

Q: What are the possible causes of this issue?

A: The possible causes of this issue are:

Model weights: The model weights may not be correctly loaded or there may be issues with the model architecture.
Input data: The input data may not be correctly formatted or there may be issues with the data preprocessing.
Cosine similarity calculation: The cosine similarity calculation may not be correct or there may be issues with the implementation.
Tolerance value: The tolerance value may not be correctly set or there may be issues with the comparison.

Q: How can I troubleshoot this issue?

A: To troubleshoot this issue, you can try the following steps:

Check the model weights: Make sure that the model weights are correctly loaded and that there are no issues with the model architecture.
Verify the input data: Check that the input data is correctly formatted and that there are no issues with the data preprocessing.
Check the cosine similarity calculation: Verify that the cosine similarity calculation is correct and that there are no issues with the implementation.
Check the tolerance value: Verify that the tolerance value is correctly set and that there are no issues with the comparison.

Q: What are the next steps to resolve this issue?

A: The next steps to resolve this issue are:

Investigate the model weights: Investigate the model weights to ensure that they are correctly loaded and that there are no issues with the model architecture.
Verify the input data: Verify the input data to ensure that it is correctly formatted and that there are no issues with the data preprocessing.
Check the cosine similarity calculation: Check the cosine similarity calculation to ensure that it is correct and that there are no issues with the implementation.
Check the tolerance value: Check the tolerance value to ensure that it is correctly set and that there are no issues with the comparison.

Q: What are the best practices to avoid this issue in the future?

A: The best practices to avoid this issue in the future are:

Thoroughly test the model: Thoroughly test the model to ensure that it is working correctly and that there are no issues with the model architecture.
Verify the input data: Verify the input data to ensure that it is correctly formatted and that there are no issues with the data preprocessing.
Check the cosine similarity calculation: Check the cosine similarity calculation to ensure that it is correct and that there are no issues with the implementation.
Check the tolerance value: Check the tolerance value to ensure that it is correctly set and that there are no issues with the comparison.