26 Data validation: Python (Pandera)

26.1 Introduction to Python’s Pandera

Python’s Pandera is a package designed to make data validation/checking of dataframes and other dataframe-like objects easy, readable and robust. Key features of Pandera that we will discuss include:

The ability to define a data schema and use it to validate dataframes and other dataframe-like objects
Check the types and properties of columns
Perform statistical validation of data (we do not cover this here; see Hypothesis Testing in the Pandera documentation)
Execute all validation in a lazy manner, so that validation rules are executed before raising an error
Handle invalid data in a number of ways, including throwing errors, writing data validation logs, and dropping observations that are invalid

The Pandera documentation is the reference for everything in this chapter, and the place to look for the features we do not cover.

26.1.1 Validating data with Pandera

In the simplest use case, Pandera can be use to validate data by first defining an instance of the DataFrameSchema class. This object specifies the properties we expect (and thus would like to check) for our dataframe index and columns. After the DataFrameSchema instance has been created and defined, the DataFrameSchema.validate method can be applied to a pandas.DataFrame instance to validate, or check, all of the properties we specified that we expect for our dataframe index and columns in the DataFrameSchema instance.

26.1.2 Dataframe Schema

When we create an instance of the DataFrameSchema class, we can specify the the properties we expect (and thus would like to check) for our dataframe index and columns.

Creating `pa.DataFrameSchema` and setting required columns

To create an instance of the DataFrameSchema class we first import the Pandera package using the alias pa, and then the function pa.DataFrameSchema. Below we demonstrate creating an instance of the DataFrameSchema class for the first two columns of the Wisconsin Breast Cancer data set from the UCI Machine Learning Repository (Dua and Graff 2017).

import pandas as pd
import pandera.pandas as pa


schema = pa.DataFrameSchema(
    {
        "class": pa.Column(),
        "mean_radius": pa.Column()
    }
)

Note

By default all columns listed are required to be in the dataframe for it to pass validation. If we wanted to make a column optional, we would set required=False in the column constructor.

Specifying column types

We can specify the type we expect each column to be, by writing the type as the first argument to pa.Column. Possible values include:

a built-in Python type, such as int, float, str or bool
a numpy data type, such as numpy.int64
a pandas data type, such as pd.StringDtype or pd.CategoricalDtype(["a", "b"])
a string alias recognized by pandas, such as "int64"
a Pandera DataType

The Pandera documentation recommends the built-in Python types for common data types, which is what we use in this chapter.

See the Pandera Data Type Validation docs for details beyond what we present here.

If we continue our example from above, we can specify that we expect the class column to be a string and the mean_radius column to be a float as shown below:

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str),
        "mean_radius": pa.Column(float)
    }
)

Missingness/null values

By default Column objects assume there should be no null/missing values. If you want to allow missing values, you need to set nullable=True in the column constructor. We demonstrate that below for the mean_radius column of our working example. Note that we do not set this to be true for our class column as we likely do not want to be working with observations where the target/response variable is missing.

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str),
        "mean_radius": pa.Column(float, nullable=True)
    }
)

If you wanted to allow a percentage of the values for a particular column to be allowed to be missing, then you could do this by writing a lambda function in a call to pa.Check in the column constructor. We show an example of that below where allow up to 5% of the mean_radius column values to be missing.

Note

This is putting the cart a bit before the horse here, as we have not yet introduced pa.Check. We will do that in the next section, so please fell free to skip this and come back to this example after you have read that.

By default, Pandera removes missing values before it runs a check, so a check that counts missing values would never see any. Setting ignore_na=False makes Pandera give the check every value, missing or not (see Handling null values in the Pandera documentation).

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str),
        "mean_radius": pa.Column(float,
                                pa.Check(lambda s: s.isna().mean() <= 0.05,
                                    element_wise=False,
                                    ignore_na=False,
                                    error="Too many null values in 'mean_radius' column."),
                                nullable=True)
    }
)

Note

Above we have created our custom check on-the-fly using a lambda function. We could do this here because the check was fairly simple. If we needed a custom check that was more complex (e.g., needs to generate data as part of the check) then we would be better to register our custom check. For situations like this, we direct the reader to the Pandera Extension docs.

Checking values in columns

Pandera’s pa.Check class is useful for checking values within columns. For any type of data, there is usually some reasonable range of values that we would expect. These usually come from domain knowledge about the data. For example, a column named age for a data set about adult human patients age in years should probably be an integer and have a range of values between 18 and 122 (the oldest person whose age has ever been independently verified). To specify a check for a range like this, we can use the pa.Check.between method. We demonstrate how to do this below with our working example to check that the mean_radius values are between 5 and 45, inclusive.

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str),
        "mean_radius": pa.Column(float, pa.Check.between(5, 45), nullable=True)
    }
)

In our working example, we might also want to check that the class column only contains the strings we think are acceptable for our category label, which would be "Benign" and "Malignant". We can do this using the pa.Check.isin method, which we demonstrate below:

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str, pa.Check.isin(["Benign", "Malignant"])),
        "mean_radius": pa.Column(float, pa.Check.between(5, 45), nullable=True)
    }
)

There are many more built-in pa.Check methods. A list can be found in the Pandera Check API docs: - https://pandera.readthedocs.io/en/stable/reference/generated/pandera.api.checks.Check.html#pandera.api.checks.Check

If there is a check you wish to do that is not part of the Pandera Check API you have two options:

Use a lambda function with boolean logic inside of pa.Check (good for simple checks, similar to the percentage of missingness in the section above), or
Register our custom check (see how to in Pandera Extension docs)

Duplicates

To check for duplicate rows, we list the columns that should be unique together in the unique argument of pa.DataFrameSchema (see Validating the joint uniqueness of columns in the Pandera documentation). Listing every column means that no two rows can be identical. We also set report_duplicates="exclude_first", so that Pandera treats the first copy of a row as valid and only reports the rows that repeat it:

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str, pa.Check.isin(["Benign", "Malignant"])),
        "mean_radius": pa.Column(float, pa.Check.between(5, 45), nullable=True)
    },
    unique=["class", "mean_radius"],
    report_duplicates="exclude_first"
)

Empty observations

Pandera has no built-in check for rows where every value is missing, so we write our own check with pa.Check. A check given to the checks argument of pa.DataFrameSchema receives the whole data frame, and should return one True or False for each row, so that Pandera can report which rows failed (see Wide checks in the Pandera documentation). df.isna().all(axis="columns") is True for each row where the value in every column is missing, so ~df.isna().all(axis="columns") is True for every row that is not empty.

By default, Pandera removes every row that contains a missing value before it runs a check on the whole data frame, which would hide exactly the rows we are looking for. Setting ignore_na=False makes Pandera run the check on every row (see Handling null values):

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str, pa.Check.isin(["Benign", "Malignant"])),
        "mean_radius": pa.Column(float, pa.Check.between(5, 45), nullable=True)
    },
    unique=["class", "mean_radius"],
    report_duplicates="exclude_first",
    checks=[
        pa.Check(
            lambda df: ~df.isna().all(axis="columns"),
            ignore_na=False,
            error="Empty rows found."
        )
    ]
)

26.1.3 Data validation

Once we have specified our the properties we expect (and thus would like to check) for our dataframe index and columns by creating an instance of pa.DataFrameSchema, we can use the pa.DataFrameSchema.validate method on a dataframe to check if the dataframe is valid considering the schema we specified.

To demonstrate this, below we create two very simple versions of the Wisconsin Breast Cancer data set. One which we expect to pass our validation checks, and one where we introduce four data anomalies that should cause some checks to fail.

First we create two data frames:

import numpy as np

valid_data = pd.DataFrame({
    "class": ["Benign", "Benign", "Malignant"],
    "mean_radius": [6.0, 31.2, 22.8]
})

invalid_data = pd.DataFrame({
    "class": ["Benign", "Benign", "benign", "Malignant", np.nan],
    "mean_radius": [6.0, 6.0, 31.2, -9999, np.nan]
})

Let’s see what happens when we apply pa.DataFrameSchema.validate to our valid data:

schema.validate(valid_data)

	class	mean_radius
0	Benign	6.0
1	Benign	31.2
2	Malignant	22.8

It returns a dataframe and does not throw an error. Excellent! What happens when we pass clearly invalid data?

schema.validate(invalid_data)

---------------------------------------------------------------------------
SchemaError                               Traceback (most recent call last)
Cell In[11], line 1
----> 1 schema.validate(invalid_data)

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:232, in DataFrameSchema.validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
    220     check_obj = check_obj.map_partitions(  # type: ignore [operator]
    221         self._validate,
    222         head=head,
   (...)    228         meta=check_obj,
    229     )
    230     return check_obj.pandera.add_schema(self)
--> 232 return self._validate(
    233     check_obj=check_obj,
    234     head=head,
    235     tail=tail,
    236     sample=sample,
    237     random_state=random_state,
    238     lazy=lazy,
    239     inplace=inplace,
    240 )

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:252, in DataFrameSchema._validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
    242 def _validate(
    243     self,
    244     check_obj: pd.DataFrame,
   (...)    250     inplace: bool = False,
    251 ) -> pd.DataFrame:
--> 252     return self.get_backend(check_obj).validate(
    253         check_obj,
    254         schema=self,
    255         head=head,
    256         tail=tail,
    257         sample=sample,
    258         random_state=random_state,
    259         lazy=lazy,
    260         inplace=inplace,
    261     )

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:133, in DataFrameSchemaBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
    128 components = self.collect_schema_components(
    129     check_obj, schema, column_info
    130 )
    132 # run the checks
--> 133 error_handler = self.run_checks_and_handle_errors(
    134     error_handler,
    135     schema,
    136     check_obj,
    137     column_info,
    138     sample,
    139     components,
    140     lazy,
    141     head,
    142     tail,
    143     random_state,
    144 )
    146 if error_handler.collected_errors:
    147     if getattr(schema, "drop_invalid_rows", False):
    148         # if the failure cases are a string, it means the error is
    149         # a schema-level error.

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:227, in DataFrameSchemaBackend.run_checks_and_handle_errors(self, error_handler, schema, check_obj, column_info, sample, components, lazy, head, tail, random_state)
    216         else:
    217             error = SchemaError(
    218                 schema,
    219                 data=check_obj,
   (...)    225                 reason_code=result.reason_code,
    226             )
--> 227         error_handler.collect_error(
    228             get_error_category(result.reason_code),
    229             result.reason_code,
    230             error,
    231             result.original_exc,
    232         )
    234 return error_handler

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/api/base/error_handler.py:117, in ErrorHandler.collect_error(self, error_type, reason_code, schema_error, original_exc)
    110 """Collect schema error, raising exception if lazy is False.
    111 
    112 :param error_type: type of error
    113 :param reason_code: string representing reason for error
    114 :param schema_error: ``SchemaError`` object.
    115 """
    116 if not self._lazy:
--> 117     raise schema_error from original_exc
    119 # delete data of validated object from SchemaError object to prevent
    120 # storing copies of the validated DataFrame/Series for every
    121 # SchemaError collected.
    122 if hasattr(schema_error, "data"):

SchemaError: columns '('class', 'mean_radius')' not unique:
    class  mean_radius
1  Benign          6.0

Wow, that’s a lot, but what is clear is that an error was thrown. If we read through the error message to the end we see the important, and useful piece of the error message:

SchemaError: columns '('class', 'mean_radius')' not unique:
    class  mean_radius
1  Benign          6.0

The error arose because row 1 of our invalid_data is a copy of row 0, and we specified in our pa.DataFrameSchema instance that the class and mean_radius columns must be unique together.

What about the other errors we expect from our invalid data? For example, the class column contains the string "benign", but we only accept "Benign" and "Malignant", there’s a value of -9999 in the mean_radius column that is well outside of the range we said was valid in the schema (5, 45), and we have an empty row as well. Why are these validation errors not reported? Pandera’s default is to throw an error after the first instance of non-valid data. To change this behaviour, we can set lazy=True. When we do this we see that all errors get reported.

schema.validate(invalid_data, lazy=True)

---------------------------------------------------------------------------
SchemaErrors                              Traceback (most recent call last)
Cell In[12], line 1
----> 1 schema.validate(invalid_data, lazy=True)

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:232, in DataFrameSchema.validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
    220     check_obj = check_obj.map_partitions(  # type: ignore [operator]
    221         self._validate,
    222         head=head,
   (...)    228         meta=check_obj,
    229     )
    230     return check_obj.pandera.add_schema(self)
--> 232 return self._validate(
    233     check_obj=check_obj,
    234     head=head,
    235     tail=tail,
    236     sample=sample,
    237     random_state=random_state,
    238     lazy=lazy,
    239     inplace=inplace,
    240 )

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:252, in DataFrameSchema._validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
    242 def _validate(
    243     self,
    244     check_obj: pd.DataFrame,
   (...)    250     inplace: bool = False,
    251 ) -> pd.DataFrame:
--> 252     return self.get_backend(check_obj).validate(
    253         check_obj,
    254         schema=self,
    255         head=head,
    256         tail=tail,
    257         sample=sample,
    258         random_state=random_state,
    259         lazy=lazy,
    260         inplace=inplace,
    261     )

File ~/work/reproducible-and-trustworthy-workflows-for-data-science/reproducible-and-trustworthy-workflows-for-data-science/.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:161, in DataFrameSchemaBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
    159         check_obj = self.drop_invalid_rows(check_obj, error_handler)
    160     else:
--> 161         raise SchemaErrors(
    162             schema=schema,
    163             schema_errors=error_handler.schema_errors,
    164             data=check_obj,
    165         )
    167 # Propagate nested Column drop_invalid_rows to the container output
    168 # (non-mutating, see _apply_nested_drop_invalid_rows).
    169 if any(getattr(c, "drop_invalid_rows", False) for c in components):

SchemaErrors: {
    "DATA": {
        "DUPLICATES": [
            {
                "schema": null,
                "column": null,
                "check": "multiple_fields_uniqueness",
                "error": "columns '('class', 'mean_radius')' not unique:    class  mean_radius1  Benign          6.0"
            }
        ],
        "DATAFRAME_CHECK": [
            {
                "schema": null,
                "column": "class",
                "check": "isin(['Benign', 'Malignant'])",
                "error": "Column 'class' failed element-wise validator number 0: isin(['Benign', 'Malignant']) failure cases: benign"
            },
            {
                "schema": null,
                "column": "mean_radius",
                "check": "in_range(5, 45)",
                "error": "Column 'mean_radius' failed element-wise validator number 0: in_range(5, 45) failure cases: -9999.0"
            },
            {
                "schema": null,
                "column": null,
                "check": "Empty rows found.",
                "error": "DataFrameSchema 'None' failed element-wise validator number 0: <Check <lambda>: Empty rows found.> failure cases: nan, nan"
            }
        ]
    },
    "SCHEMA": {
        "SERIES_CONTAINS_NULLS": [
            {
                "schema": null,
                "column": "class",
                "check": "not_nullable",
                "error": "non-nullable series 'class' contains null values:4    NaNName: class, dtype: str"
            }
        ]
    }
}

26.1.4 Handling invalid data

By default Pandera will throw an error when a check is not passed. Depending on your situation, this can be a desired expected behaviour (e.g., a static data analysis published in a report) or a very undesired behaviour that could potentially be dangerous (e.g., autonomous driving application). In the latter case, we would want to do something different than throw an error. Possibilities we will cover here include dropping invalid observations and writing log files that report the errors.

26.1.5 Dropping invalid observations

In an in-production system, dropping non-valid data could be a reasonable path forward instead of throwing an error. Another situation where this might be a reasonable thing to do is when training a machine learning model with a million observations. You don’t want to throw an error in the middle of training if only one observation is invalid!

To change the behaviour of pa.DataFrameSchema.validate to instead return a dataframe with the invalid rows dropped we need to do two things:

add drop_invalid_rows=True to our pa.DataFrameSchema instance
add lazy=True to our call to the pa.DataFrameSchema.validate method

Below we demonstrate this with our working example.

schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str, pa.Check.isin(["Benign", "Malignant"]), nullable=True),
        "mean_radius": pa.Column(float, pa.Check.between(5, 45), nullable=True)
    },
    unique=["class", "mean_radius"],
    report_duplicates="exclude_first",
    checks=[
        pa.Check(
            lambda df: ~df.isna().all(axis="columns"),
            ignore_na=False,
            error="Empty rows found."
        )
    ],
    drop_invalid_rows=True
)

schema.validate(invalid_data, lazy=True)

	class	mean_radius
0	Benign	6.0

Notice that Pandera also dropped the duplicate row and the empty row. It can do that because both unique and our empty row check tell Pandera exactly which rows failed. A check that returns a single True or False for the whole data frame, such as lambda df: ~df.duplicated().any(), cannot tell Pandera which rows to drop, and recent versions of Pandera stop with an error if you combine one with drop_invalid_rows=True.

26.1.6 Writing data validation logs

Is removing the rows sufficient? Not at all! A human should be told that there was invalid data so that upstream data collection, cleaning and transformation processes can be reviewed to minimize the chances of future invalid data. One way to do this is to again specify lazy=True so that all errors can be observed and reported. Then we can get the SchemaErrors and write them to a log file. We show below how to do this for our working example so that the valid rows are returned as a dataframe named validated_data and the errors are logged as a file called validation_errors.log:

import json
import logging
import pandas as pd
import pandera.pandas as pa

# Configure logging
logging.basicConfig(
    filename="validation_errors.log",
    filemode="w",
    format="%(asctime)s - %(message)s",
    level=logging.INFO,
)

# Define the schema
schema = pa.DataFrameSchema(
    {
        "class": pa.Column(str, pa.Check.isin(["Benign", "Malignant"]), nullable=True),
        "mean_radius": pa.Column(float, pa.Check.between(5, 45), nullable=True),
    },
    unique=["class", "mean_radius"],
    report_duplicates="exclude_first",
    checks=[
        pa.Check(
            lambda df: ~df.isna().all(axis="columns"),
            ignore_na=False,
            error="Empty rows found."
        )
    ],
    drop_invalid_rows=False,
)

# Initialize error cases DataFrame
error_cases = pd.DataFrame()
data = invalid_data.copy()

# Validate data and handle errors
try:
    validated_data = schema.validate(data, lazy=True)
except pa.errors.SchemaErrors as e:
    error_cases = e.failure_cases

    # Convert the error message to a JSON string
    error_message = json.dumps(e.message, indent=2)
    logging.error("\n" + error_message)

# Filter out invalid rows based on the error cases
if not error_cases.empty:
    invalid_indices = error_cases["index"].dropna().unique()
    validated_data = data.drop(index=invalid_indices).reset_index(drop=True)
else:
    validated_data = data

26.1 Introduction to Python’s Pandera

26.1.1 Validating data with Pandera

26.1.2 Dataframe Schema

Creating pa.DataFrameSchema and setting required columns

Specifying column types

Missingness/null values

Checking values in columns

Duplicates

Empty observations

26.1.3 Data validation

26.1.4 Handling invalid data

26.1.5 Dropping invalid observations

26.1.6 Writing data validation logs

Creating `pa.DataFrameSchema` and setting required columns