Note: As @ritchie46’s answer states, this solution may be redundant since pandas version 0.25 per the new argument cache_dates that defaults to True Try using this function for parsing dates: def lookup(date_pd_series, format=None): “”” This is an extremely fast approach to datetime parsing. For large data, the same dates are often repeated. Rather than re-parse … Read more
Answer recommended by NLP Collective
You need a bit modify solution, because sometimes it return 2 and sometimes only one column: df2 = pd.DataFrame({‘STATUS’:[‘Estimated 3:17 PM’,’Delayed 3:00 PM’]}) df3 = df2[‘STATUS’].str.split(n=1, expand=True) df3.columns = [‘STATUS_ID{}’.format(x+1) for x in df3.columns] print (df3) STATUS_ID1 STATUS_ID2 0 Estimated 3:17 PM 1 Delayed 3:00 PM df2 = df2.join(df3) print (df2) STATUS STATUS_ID1 STATUS_ID2 0 … Read more
There is a built in lookup function that can handle this type of situation (looks up by row/column). I don’t know how optimized it is, but may be faster than the apply solution. In [9]: df[‘value’] = df.lookup(df.index, df[‘best’]) In [10]: df Out[10]: Date best a b c d value 0 1990 a 5 4 … Read more
A quick and dirty profiling yield the following values: from pydantic import BaseModel import pandas as pd from fastapi.encoders import jsonable_encoder class SomeModel(BaseModel): col1: int col2: str data = [SomeModel(col1=1,col2=”foo”),SomeModel(col1=2,col2=”bar”)]*4*10**5 import cProfile cProfile.run( ‘pd.DataFrame([s.dict() for s in data])’ ) # around 3.4s cProfile.run( ‘pd.DataFrame(jsonable_encoder(data))’ ) # around 20.6s cProfile.run( ‘pd.DataFrame([s.__dict__ for s in data])’ ) … Read more
Use rename with parameter index pass a dictionary to the index parameter df.rename(index={‘alpha’: ‘mu’})
It’s just the way you think it would be, apply accepts args and kwargs and passes them directly to some_func. df.apply(some_func, var1=’DOG’, axis=1) Or, df.apply(some_func, args=(‘DOG’, ), axis=1) 0 foo-x-DOG 1 bar-y-DOG dtype: object If for any reason that won’t work for your use case, then you can always fallback to using a lambda: df.apply(lambda … Read more
You can write the table straight into a .docx file using the python-docx library. If you are using the Conda or installed Python using Anaconda, you can run the command from the command line: conda install python-docx –channel conda-forge Or to pip install from the command line: pip install python-docx After that is installed, we … Read more
I don’t think pandas offers a way to do this in read_csv. Perhaps the neatest (in one pass) is to use collections.deque: from collections import deque from StringIO import StringIO with open(fname, ‘r’) as f: q = deque(f, 2) # replace 2 with n (lines read at the end) In [12]: q Out[12]: deque([‘7,8,9\n’, ‘10,11,12’], … Read more
Define a function: def unique_non_null(s): return s.dropna().unique() Then use it in the aggregation: df.groupby(‘b’).agg({ ‘a’: [‘min’, ‘max’, unique_non_null], ‘c’: [‘first’, ‘last’, unique_non_null] })