Investigate the cause of the issue

Oct 13, 2025 by wbrum.org 33 views

Introduction

K-Nearest Neighbors (KNN) is a widely used algorithm in machine learning for classification and regression tasks. However, when dealing with KNN, a common issue arises when two or more neighbors have identical distances but different labels. In this case, the result depends on the ordering of the training data. This article explores a similar issue with KNN tie breakers changing based on the subset of the train.

Describe the Bug

According to the scikit-learn documentation, when two neighbors k and k+1 have identical distances but different labels, the result will depend on the ordering of the training data. This is also true for KNN without going into classification, where the scope is only to find the NN without voting the class. However, the code provided in this article shows a different ordering for NN based on the selection of the train set.

Steps/Code to Reproduce

import pandas as pd
from sklearn.datasets import fetch_file
from sklearn.model_selection import train_test_split
from sklearn.neighbors import NearestNeighbors

url = 'https://archive.ics.uci.edu/static/public/891/data.csv'
filepath = fetch_file(url)

df = pd.read_csv(filepath)

y = df['Diabetes_binary'].to_numpy()
x = df.drop(['ID', 'Diabetes_binary'], axis=1).to_numpy()
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

x_train = x_train[0:100]
x_test = x_test[0:1]

ml = NearestNeighbors(n_neighbors=6, algorithm='brute').fit(x_train)
d,n =  ml.kneighbors(x_test, return_distance=True)

x_train = x_train[0:98]

ml = NearestNeighbors(n_neighbors=6, algorithm='brute').fit(x_train)
d2,n2 =  ml.kneighbors(x_test, return_distance=True)

print(n)
print(n2)

print(d)
print(d2)

Expected Results

[[33 58 2 97 46  5]]
[[33 58  2 97 46  5]]
[[7.61577311 7.93725393 8.30662386 8.30662386 8.60232527 8.60232527]]
[[7.61577311 7.93725393 8.30662386 8.30662386 8.60232527 8.60232527]]

Actual Results

[[33 58 97  2 46  5]]
[[33 58  2 97 46  5]]
[[7.61577311 7.93725393 8.30662386 8.30662386 8.60232527 8.60232527]]
[[7.61577311 7.93725393 8.30662386 8.30662386 8.60232527 8.60232527]]

Versions

System:
    python: 3.12.3 (tags/v3.12.3:f6650f9, Apr  9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)]
executable: C:\Users\Corrado\Desktop\Corrado\Politobox\Research\repository\scalabaleLearning\scalabaleLearning\Scripts\python.exe
   machine: Windows-10-10.0.19045-SP0
Python dependencies:
      sklearn: 1.6.1
          pip: None
   setuptools: 75.3.0
        numpy: 1.26.4
        scipy: 1.14.1
       Cython: None
       pandas: 2.2.0
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\Corrado\OneDrive - Politecnico di Torino\Corrado\Politobox\Research\repository\scalabaleLearning\scalabaleLearning\Lib\site-packages\numpy.libs\libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libscipy_openblas
       filepath: C:\Users\Corrado\OneDrive - Politecnico di Torino\Corrado\Politobox\Research\repository\scalabaleLearning\scalabaleLearning\Lib\site-packages\scipy.libs\libscipy_openblas-5b1ec8b915dfb81d11cebc0788069d2d.dll
        version: 0.3.27.dev
threading_layer: pthreads
   architecture: SkylakeX
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\Corrado\OneDrive - Politecnico di Torino\Corrado\Politobox\Research\repository\scalabaleLearning\scalabaleLearning\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

Conclusion

In conclusion, the KNN tie breakers changing based on the subset of the train is a common issue that can arise when dealing with KNN. The code provided in this article shows a different ordering for NN based on the selection of the train set. This issue can be resolved by using a different algorithm or by modifying the code to handle tie breakers. The expected results and actual results are provided in the article to demonstrate the issue.

Recommendations

Use a different algorithm: Consider using a different algorithm that does not have this issue, such as K-Means or Hierarchical Clustering.
Modify the code: Modify the code to handle tie breakers by using a different method, such as using a random tie breaker or by using a custom function to handle tie breakers.
Use a different dataset: Consider using a different dataset that does not have this issue.
Check the documentation: Check the documentation of the library or framework being used to see if there are any known issues or workarounds for this problem.

Future Work

Investigate the cause of the issue: Investigate the cause of the issue to determine why the KNN tie breakers are changing based on the subset of the train.
Develop a solution: Develop a solution to the issue, such as a custom function to handle tie breakers or a different algorithm that does not have this issue.
Test the solution: Test the solution to ensure that it works correctly and does not introduce any new issues.

References

Scikit-learn documentation: Scikit-learn documentation on K-Nearest Neighbors.
K-Nearest Neighbors algorithm: K-Nearest Neighbors algorithm documentation.
Tie breakers: Tie breakers documentation.

Appendix

Code: The code used to reproduce the issue is provided in the article.
Expected results: The expected results are provided in the article.
Actual results: The actual results are provided in the article.
KNN Tie Breakers Changing Based on the Subset of the Train: Q&A =================================================================

Q: What is the KNN tie breaker issue?

A: The KNN tie breaker issue is a problem that arises when two or more neighbors have identical distances but different labels. In this case, the result depends on the ordering of the training data.

Q: Why does the KNN tie breaker issue occur?

A: The KNN tie breaker issue occurs because the KNN algorithm uses a brute force approach to find the nearest neighbors. When two or more neighbors have identical distances, the algorithm will return the neighbors in the order they appear in the training data.

Q: How can I resolve the KNN tie breaker issue?

A: There are several ways to resolve the KNN tie breaker issue:

Use a different algorithm: Consider using a different algorithm that does not have this issue, such as K-Means or Hierarchical Clustering.
Modify the code: Modify the code to handle tie breakers by using a different method, such as using a random tie breaker or by using a custom function to handle tie breakers.
Use a different dataset: Consider using a different dataset that does not have this issue.
Check the documentation: Check the documentation of the library or framework being used to see if there are any known issues or workarounds for this problem.

Q: What are some common causes of the KNN tie breaker issue?

A: Some common causes of the KNN tie breaker issue include:

Insufficient data: If the training data is insufficient, the KNN algorithm may not be able to find a unique solution.
Overfitting: If the model is overfitting, it may be sensitive to small changes in the training data, leading to tie breakers.
Noise in the data: If the data contains noise, it may lead to tie breakers.

Q: How can I prevent the KNN tie breaker issue?

A: To prevent the KNN tie breaker issue, you can:

Use a larger dataset: Using a larger dataset can help to reduce the likelihood of tie breakers.
Use a different algorithm: Consider using a different algorithm that does not have this issue.
Use a different distance metric: Using a different distance metric, such as the Euclidean distance, can help to reduce the likelihood of tie breakers.
Use a tie breaker method: Consider using a tie breaker method, such as a random tie breaker or a custom function to handle tie breakers.

Q: What are some best practices for handling tie breakers in KNN?

A: Some best practices for handling tie breakers in KNN include:

Use a consistent tie breaker method: Use a consistent tie breaker method throughout the model.
Use a random tie breaker: Consider using a random tie breaker to avoid bias.
Use a custom function to handle tie breakers: Consider using a custom function to handle tie breakers, especially if the tie breakers are complex.
Document the tie breaker method: Document the tie breaker method used in the model, so that others can understand and replicate the results.

Q: What are some common mistakes to avoid when handling tie breakers in KNN?

A: Some common mistakes to avoid when handling tie breakers in KNN include:

Not handling tie breakers at all: Failing to handle tie breakers can lead to inconsistent results.
Using a different tie breaker method for each model: Using a different tie breaker method for each model can lead to inconsistent results.
Not documenting the tie breaker method: Failing to document the tie breaker method used in the model can lead to confusion and inconsistent results.
Not testing the model with tie breakers: Failing to test the model with tie breakers can lead to unexpected results.