How to convert a list of Pydantic BaseModels to Pandas Dataframe

A quick and dirty profiling yield the following values:

from pydantic import BaseModel
import pandas as pd
from fastapi.encoders import jsonable_encoder
class SomeModel(BaseModel):
    col1: int
    col2: str

data = [SomeModel(col1=1,col2="foo"),SomeModel(col1=2,col2="bar")]*4*10**5

import cProfile

cProfile.run( 'pd.DataFrame([s.dict() for s in data])' ) # around 3.4s
cProfile.run( 'pd.DataFrame(jsonable_encoder(data))' ) # around 20.6s
cProfile.run( 'pd.DataFrame([s.__dict__ for s in data])' ) # around 0.59s
cProfile.run( 'pd.DataFrame([dict(s) for s in data])' ) # around 1.1s
cProfile.run( 'pd.DataFrame([vars(s) for s in data])' ) # 0.62s

Test run with pydantic 1.10.11 – there exists later pydantic versions

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)