Improve Health Check Endpoint

Oct 13, 2025 by wbrum.org 2 views

Introduction

Dataprep is a crucial component of the GenAIComps system, responsible for ingesting and processing large amounts of data. However, a bug has been identified where dataprep becomes unresponsive when ingesting files, causing the health check endpoint to fail and resulting in Kubernetes readiness check failures. In this article, we will delve into the details of this bug, provide reproduce steps, and discuss potential solutions.

Background

Dataprep is a microservice infrastructure that uses Redis as its backend. It is designed to handle large amounts of data and provide real-time processing capabilities. However, when ingesting files, dataprep becomes unresponsive, causing the health check endpoint to fail. This failure is critical, as it affects the overall performance and reliability of the GenAIComps system.

Reproduce Steps

To reproduce this bug, follow these steps:

Launch Dataprep Service

First, launch the dataprep service using the following command:

ip_address=$(hostname -I | awk 'print $1}') DATAPREP_PORT="11108" TEI_EMBEDDER_PORT="10221" export host_ip=$ {ip_address} export REDIS_HOST=$ip_address export REDIS_PORT=6379 export DATAPREP_PORT="11108" export TEI_EMBEDDER_PORT="10221" export REDIS_URL="redis//${ip_address: ${REDIS_PORT}" export EMBEDDING_MODEL_ID="BAAI/bge-base-en-v1.5" export TEI_EMBEDDING_ENDPOINT="http://$ ip_address}${TEI_EMBEDDER_PORT" export INDEX_NAME="rag_redis" service_name="redis-vector-db tei-embedding-serving dataprep-redis"

cd GenAIComps/comps/dataprep/deployment/docker_compose/ docker compose up ${service_name} -d


2. **Launch Health Check Script**

   Next, launch a new terminal to run the health check script:
   ```bash
while true; do sleep 1; echo "check dataprep health..."; curl --connect-timeout 1 --max-time 1 http://localhost:11108/v1/health_check; echo "";done

Ingest PDF Files

Finally, open a new terminal to ingest PDF files to dataprep:

cd GenAIComps/tests/dataprep curl http://localhost:11108/v1/dataprep/ingest -X POST -H "Content-Type: multipart/form-data" -F "files=@./ingest_dataprep.pdf"


4. **Check Log**

   After ingesting the PDF files, check the log of the health check script. You will find that the health check has failed.

**Raw Log**
------------

The raw log of the health check script is as follows:
```shell
check dataprep health...
{"Service Title":"opea_service@dataprep","Service Description":"OPEA Microservice Infrastructure"}
check dataprep health...
{"Service Title":"opea_service@dataprep","Service Description":"OPEA Microservice Infrastructure"}
check dataprep health...
{"Service Title":"opea_service@dataprep","Service Description":"OPEA Microservice Infrastructure"}
check dataprep health...
{"Service Title":"opea_service@dataprep","Service Description":"OPEA Microservice Infrastructure"}
check dataprep health...
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

check dataprep health...
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

check dataprep health...
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

check dataprep health...
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

check dataprep health...
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

check dataprep health...
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

Conclusion

In conclusion, the bug where dataprep becomes unresponsive when ingesting files is a critical issue that affects the overall performance and reliability of the GenAIComps system. By following the reproduce steps outlined in this article, developers can easily reproduce this bug and work towards resolving it. Potential solutions include optimizing the dataprep service, improving the health check endpoint, and enhancing the Kubernetes readiness check mechanism.

Recommendations

To resolve this bug, we recommend the following:

Optimize Dataprep Service
- Review the dataprep service code and optimize it for better performance.
- Consider using a more efficient data processing algorithm.
Improve Health Check Endpoint
- Review the health check endpoint code and improve it for better reliability.
- Consider using a more robust health check mechanism.
Enhance Kubernetes Readiness Check
- Review the Kubernetes readiness check mechanism and enhance it for better reliability.
- Consider using a more robust readiness check mechanism.

Introduction

In our previous article, we discussed the bug where dataprep becomes unresponsive when ingesting files, causing the health check endpoint to fail and resulting in Kubernetes readiness check failures. In this article, we will provide a Q&A section to address common questions and concerns related to this bug.

Q: What is the root cause of this bug?

A: The root cause of this bug is the dataprep service becoming unresponsive when ingesting files. This is likely due to a combination of factors, including inefficient data processing algorithms, inadequate health check mechanisms, and suboptimal Kubernetes readiness checks.

Q: How can I reproduce this bug?

A: To reproduce this bug, follow the steps outlined in our previous article:

Launch Dataprep Service
- Launch the dataprep service using the following command:

cd GenAIComps/comps/dataprep/deployment/docker_compose/ docker compose up ${service_name} -d


2.  **Launch Health Check Script**

    *   Launch a new terminal to run the health check script:
        ```bash
while true; do sleep 1; echo "check dataprep health..."; curl --connect-timeout 1 --max-time 1 http://localhost:11108/v1/health_check; echo "";done

Ingest PDF Files
- Open a new terminal to ingest PDF files to dataprep:

cd GenAIComps/tests/dataprep curl http://localhost:11108/v1/dataprep/ingest -X POST -H "Content-Type: multipart/form-data" -F "files=@./ingest_dataprep.pdf"


4.  **Check Log**

    *   After ingesting the PDF files, check the log of the health check script. You will find that the health check has failed.

**Q: What are the potential solutions to this bug?**
---------------------------------------------------

A: Potential solutions to this bug include:

1.  **Optimize Dataprep Service**

    *   Review the dataprep service code and optimize it for better performance.
    *   Consider using a more efficient data processing algorithm.
2.  **Improve Health Check Endpoint**

    *   Review the health check endpoint code and improve it for better reliability.
    *   Consider using a more robust health check mechanism.
3.  **Enhance Kubernetes Readiness Check**

    *   Review the Kubernetes readiness check mechanism and enhance it for better reliability.
    *   Consider using a more robust readiness check mechanism.

**Q: How can I prevent this bug from occurring in the future?**
---------------------------------------------------------

A: To prevent this bug from occurring in the future, follow these best practices:

1.  **Regularly Review and Optimize Code**

    *   Regularly review and optimize the dataprep service code for better performance.
    *   Consider using more efficient data processing algorithms.
2.  **Implement Robust Health Check Mechanisms**

    *   Implement robust health check mechanisms to ensure the dataprep service is functioning correctly.
    *   Consider using more robust health check mechanisms.
3.  **Enhance Kubernetes Readiness Check**

    *   Enhance the Kubernetes readiness check mechanism to ensure the dataprep service is functioning correctly.
    *   Consider using more robust readiness check mechanisms.

**Conclusion**
----------

In conclusion, the bug where dataprep becomes unresponsive when ingesting files is a critical issue that affects the overall performance and reliability of the GenAIComps system. By following the reproduce steps and potential solutions outlined in this article, developers can work towards resolving this bug and ensuring the reliability and performance of the GenAIComps system.