automounting of API credentials

Oct 13, 2025 by wbrum.org 5 views

Feature Request: Allow Prometheus to Scrape Metrics Even with Disabled Automounting of API Credentials

Introduction

Prometheus is a popular monitoring and alerting tool used in Kubernetes environments to gather critical metrics such as pod status, resource usage, and node metrics. However, when Kubernetes automounts API credentials for service accounts, Prometheus can easily scrape Kubernetes metrics without issues. But, in some cases, for security reasons, automounting of API credentials needs to be disabled. This presents a problem for Prometheus, as it is unable to access the Kubernetes API and scrape metrics when the automounting is turned off. In this article, we will discuss the need for a feature that allows Prometheus to scrape Kubernetes metrics even when the automounting of API credentials is disabled.

Problem Statement

When Kubernetes automounts API credentials for service accounts, Prometheus can easily scrape Kubernetes metrics without issues. However, in some cases, for security reasons, automounting of API credentials needs to be disabled. This presents a problem for Prometheus, as it is unable to access the Kubernetes API and scrape metrics when the automounting is turned off. The current behavior of Prometheus requires Kubernetes credentials to be automatically mounted for accessing the Kubernetes API for service discovery. Disabling the automounting of the service account token causes Prometheus to fail in scraping metrics, which limits security flexibility in Kubernetes environments.

Desired Behavior

Allow Prometheus to scrape Kubernetes metrics even when the automounting of API credentials is disabled. This could be achieved in one of the following ways:

Option 1: Manual API Token Management

Allow the use of manually provided API credentials (such as a Kubernetes token or certificate) via Kubernetes secrets or config files. This would enable users to provide credentials manually, which would allow Prometheus to function even when automounting is disabled.

Option 2: Improved Service Discovery

Provide an option in the Prometheus configuration to use manual tokens for service discovery when automounting is disabled. This would enable users to configure Prometheus to use manual tokens for service discovery, which would allow it to function even when automounting is disabled.

Option 3: Fallback Mechanism

Introduce a fallback mechanism that allows Prometheus to function without the service account token mounted by default. This would enable Prometheus to function even when automounting is disabled, by providing a fallback mechanism that allows it to access the Kubernetes API without the service account token.

Why This is Needed

This feature is necessary for environments where security policies require stricter control over service account tokens and API credentials. Disabling the automatic mounting of API credentials is a common security best practice, especially in regulated or highly secure environments. Without this feature, users are forced to choose between compromising their security practices (by enabling automounting) or losing critical monitoring capabilities (since Prometheus can't scrape Kubernetes metrics without the credentials).

Use Case

For example, in highly secure environments, an organization might disable automounting of service account tokens to restrict unnecessary access to the Kubernetes API. However, they still need Prometheus to gather Kubernetes metrics such as pod status, resource usage, and node metrics. Currently, Prometheus fails to operate in such a configuration, making it difficult to maintain both security and observability.

Benefits

Allowing Prometheus to function in a more secure and flexible manner, without the need for automounting credentials, will benefit many organizations facing similar security challenges. This feature will enable users to maintain both security and observability, which is critical in highly secure environments.

Conclusion

In conclusion, the feature request to allow Prometheus to scrape metrics even with disabled automounting of API credentials is necessary for environments where security policies require stricter control over service account tokens and API credentials. This feature will enable users to maintain both security and observability, which is critical in highly secure environments. We believe that this feature will benefit many organizations facing similar security challenges and will enable them to maintain both security and observability.

Future Work

Future work on this feature could include:

Implementing the suggested solutions (Option 1 and Option 2)
Testing and validating the feature to ensure it works as expected
Providing documentation and support for the feature
Continuously monitoring and improving the feature to ensure it meets the needs of users

References

Prometheus Documentation
Kubernetes Documentation
Kubernetes Security Best Practices
Q&A: Feature Request - Allow Prometheus to Scrape Metrics Even with Disabled Automounting of API Credentials

Introduction

In our previous article, we discussed the need for a feature that allows Prometheus to scrape Kubernetes metrics even when the automounting of API credentials is disabled. In this article, we will answer some frequently asked questions (FAQs) related to this feature request.

Q: Why is automounting of API credentials disabled in some environments?

A: Automounting of API credentials is disabled in some environments for security reasons. Disabling the automatic mounting of API credentials is a common security best practice, especially in regulated or highly secure environments.

Q: What is the current behavior of Prometheus when automounting of API credentials is disabled?

A: When automounting of API credentials is disabled, Prometheus is unable to access the Kubernetes API and scrape metrics. This is because Prometheus requires Kubernetes credentials to be automatically mounted for accessing the Kubernetes API for service discovery.

Q: What are the benefits of allowing Prometheus to scrape metrics even with disabled automounting of API credentials?

A: Allowing Prometheus to scrape metrics even with disabled automounting of API credentials will benefit many organizations facing similar security challenges. This feature will enable users to maintain both security and observability, which is critical in highly secure environments.

Q: How can users provide credentials manually via a Kubernetes Secret and mount it into the Prometheus pod?

A: Users can provide credentials manually via a Kubernetes Secret and mount it into the Prometheus pod by following these steps:

Create a Kubernetes Secret containing the API credentials.
Mount the Secret into the Prometheus pod using a ConfigMap or a Volume.
Configure Prometheus to use the mounted Secret for authentication.

Q: What is the fallback mechanism that allows Prometheus to function without the service account token mounted by default?

A: The fallback mechanism is a feature that allows Prometheus to function without the service account token mounted by default. This feature would enable Prometheus to access the Kubernetes API without the service account token, allowing it to scrape metrics even when automounting is disabled.

Q: How can users implement an explicit Prometheus configuration option to bypass the need for automounting API credentials when scraping Kubernetes metrics?

A: Users can implement an explicit Prometheus configuration option to bypass the need for automounting API credentials when scraping Kubernetes metrics by following these steps:

Configure Prometheus to use a manual token for service discovery.
Set the service_discovery option to manual in the Prometheus configuration file.
Provide the manual token in the Prometheus configuration file.

Q: What are the potential risks of allowing Prometheus to scrape metrics even with disabled automounting of API credentials?

A: The potential risks of allowing Prometheus to scrape metrics even with disabled automounting of API credentials include:

Increased security risk due to the exposure of API credentials.
Potential for unauthorized access to the Kubernetes API.
Increased complexity in managing API credentials.

Q: How can users mitigate the potential risks of allowing Prometheus to scrape metrics even with disabled automounting of API credentials?

A: Users can mitigate the potential risks of allowing Prometheus to scrape metrics even with disabled automounting of API credentials by:

Implementing strict access controls and authentication mechanisms.
Using secure storage for API credentials.
Regularly reviewing and updating API credentials.

Conclusion

In conclusion, the feature request to allow Prometheus to scrape metrics even with disabled automounting of API credentials is necessary for environments where security policies require stricter control over service account tokens and API credentials. We hope that this Q&A article has provided valuable information and insights to users who are interested in this feature request.