Update Pydantic to the latest version

Oct 13, 2025 by wbrum.org 119 views

Introduction

In recent versions of Pandera, specifically v0.23.0 and above, a peculiar issue has been observed when using Pydantic's core schema logic with Pandera's Category type. The Category type is resolving as 'any' in the Pydantic core schema, which does not exist as a key in the type_map. This results in a KeyError exception being thrown. In this article, we will delve into the underlying logic and explore possible solutions to this issue.

Understanding the Issue

The issue arises from the way Pydantic's core schema logic handles the Category type from Pandera. When the Category type is used in a Pydantic model, it is resolved as 'any' in the Pydantic core schema. However, this type does not exist as a key in the type_map, which is used to map types to their corresponding schema definitions. As a result, when Pydantic tries to access the schema for the 'any' type, it throws a KeyError exception.

Reproducible Example

To demonstrate this issue, let's consider a reproducible example using Pandera v0.23.1:

import pydantic
import pandera as pa
from pandera.typing.pandas import DataFrame

class MyPanderaModel(pa.DataFrameModel):
    x: pa.Category = pa.Field(dtype_kwargs={'categories': ['a','b','c'], 'ordered': True})

class MyPydanticModel(pydantic.BaseModel):
    df: DataFrame[MyPanderaModel]

When we try to create an instance of the MyPydanticModel class, we get the following error:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    class MyPydanticModel(pydantic.BaseModel):
  File ".../python3.12/site-packages/pydantic/_internal/_model_construction.py", line 224, in __new__
    complete_model_class(
  File ".../python3.12/site-packages/pydantic/_internal/_model_construction.py", line 602, in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/main.py", line 702, in __get_pydantic_core_schema__
    return handler(source)
           ^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 610, in generate_schema
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 879, in _generate_schema_inner
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "...python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 691, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1071, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1263, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2056, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2037, in inner_handler
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 884, in _generate_schema_inner
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 986, in match_type
    return self._match_generic_type(obj, origin)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1007, in _match_generic_type
    from_property = self._generate_schema_from_property(origin, obj)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 759, in _generate_schema_from_property
    schema = get_schema(
             ^^^^^^^^^^^
  File ".../python3.12/site-packages/pandera/typing/pandas.py", line 209, in __get_pydantic_core_schema__
    type_map[schema_json_columns[key]["items"]["type"]]
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'any'

Possible Solutions

While the underlying logic of Pydantic's core schema is complex, there are a few possible solutions to this issue:

Update Pydantic to the latest version: The issue may be resolved in a later version of Pydantic. We can try updating Pydantic to the latest version and see if the issue persists.
Use a different type: Instead of using the Category type from Pandera, we can use a different type that is supported by Pydantic. For example, we can use the str type instead of Category.
Implement a custom type handler: We can implement a custom type handler that maps the Category type to a supported type. This would require modifying the Pydantic code, which may not be feasible.
Raise a custom exception: We can raise a custom exception when the Category type is encountered, instead of throwing a KeyError exception. This would require modifying the Pydantic code, which may not be feasible.

Conclusion

In conclusion, the issue of Pydantic's core schema resolving the Category type from Pandera as 'any' is a complex problem that requires a deep understanding of the underlying logic. While there are possible solutions to this issue, they may require modifying the Pydantic code, which may not be feasible. Further investigation and collaboration with the Pydantic developers may be necessary to resolve this issue.

Future Work

To resolve this issue, we can consider the following future work:

Collaborate with Pydantic developers: We can collaborate with the Pydantic developers to understand the underlying logic and implement a solution that works for both Pydantic and Pandera.
Implement a custom type handler: We can implement a custom type handler that maps the Category type to a supported type, which would require modifying the Pydantic code.
Raise a custom exception: We can raise a custom exception when the Category type is encountered, instead of throwing a KeyError exception, which would require modifying the Pydantic code.

Q: What is the issue with Pydantic's core schema resolving the Category type from Pandera as 'any'?

A: The issue arises from the way Pydantic's core schema logic handles the Category type from Pandera. When the Category type is used in a Pydantic model, it is resolved as 'any' in the Pydantic core schema. However, this type does not exist as a key in the type_map, which is used to map types to their corresponding schema definitions. As a result, when Pydantic tries to access the schema for the 'any' type, it throws a KeyError exception.

Q: What are the possible solutions to this issue?

A: There are several possible solutions to this issue:

Update Pydantic to the latest version: The issue may be resolved in a later version of Pydantic. We can try updating Pydantic to the latest version and see if the issue persists.
Use a different type: Instead of using the Category type from Pandera, we can use a different type that is supported by Pydantic. For example, we can use the str type instead of Category.
Implement a custom type handler: We can implement a custom type handler that maps the Category type to a supported type. This would require modifying the Pydantic code, which may not be feasible.
Raise a custom exception: We can raise a custom exception when the Category type is encountered, instead of throwing a KeyError exception. This would require modifying the Pydantic code, which may not be feasible.

Q: How can I implement a custom type handler?

A: Implementing a custom type handler requires modifying the Pydantic code. This can be a complex process and may require significant changes to the Pydantic codebase. However, if you are familiar with the Pydantic codebase and have the necessary expertise, you can implement a custom type handler that maps the Category type to a supported type.

Q: What are the benefits of using a custom type handler?

A: Using a custom type handler can provide several benefits, including:

Improved compatibility: By implementing a custom type handler, you can ensure that the Category type is handled correctly in Pydantic, even if it is not a supported type.
Increased flexibility: A custom type handler can provide more flexibility in terms of how the Category type is handled, allowing you to customize the behavior to meet your specific needs.
Better error handling: A custom type handler can provide better error handling, allowing you to catch and handle errors related to the Category type in a more robust and reliable way.

Q: What are the potential drawbacks of using a custom type handler?

A: Using a custom type handler can also have some potential drawbacks, including:

Increased complexity: Implementing a custom type handler can add complexity to the Pydantic codebase, making it more difficult to maintain and update.
Incompatibility issues: A custom type handler may not be compatible with all versions of Pydantic, which can lead to incompatibility issues and errors.
Security risks: A custom type handler may introduce security risks if it is not implemented correctly, allowing attackers to exploit vulnerabilities in the Pydantic codebase.

Q: How can I raise a custom exception when the Category type is encountered?

A: Raising a custom exception when the Category type is encountered requires modifying the Pydantic code. This can be a complex process and may require significant changes to the Pydantic codebase. However, if you are familiar with the Pydantic codebase and have the necessary expertise, you can raise a custom exception when the Category type is encountered.

Q: What are the benefits of raising a custom exception?

A: Raising a custom exception can provide several benefits, including:

Improved error handling: By raising a custom exception, you can catch and handle errors related to the Category type in a more robust and reliable way.
Increased flexibility: Raising a custom exception can provide more flexibility in terms of how the Category type is handled, allowing you to customize the behavior to meet your specific needs.
Better error messages: A custom exception can provide better error messages, allowing you to provide more informative and helpful error messages to users.

Q: What are the potential drawbacks of raising a custom exception?

A: Raising a custom exception can also have some potential drawbacks, including:

Increased complexity: Implementing a custom exception can add complexity to the Pydantic codebase, making it more difficult to maintain and update.
Incompatibility issues: A custom exception may not be compatible with all versions of Pydantic, which can lead to incompatibility issues and errors.
Security risks: A custom exception may introduce security risks if it is not implemented correctly, allowing attackers to exploit vulnerabilities in the Pydantic codebase.