drop-duplicates – Tarik Billa

Pandas drop_duplicates method not working on dataframe containing lists

May 7, 2023 by Tarik

drop_duplicates won’t work with lists in your dataframe as the error message implies. However, you can drop duplicates on the dataframe casted as str and then extract the rows from original df using the index from the results. Setup df = pd.DataFrame({‘Keyword’: {0: ‘apply’, 1: ‘apply’, 2: ‘apply’, 3: ‘terms’, 4: ‘terms’}, ‘X’: {0: [1, … Read more

Drop all duplicate rows across multiple columns in Python Pandas

October 10, 2022 by Tarik

This is much easier in pandas now with drop_duplicates and the keep parameter. import pandas as pd df = pd.DataFrame({“A”:[“foo”, “foo”, “foo”, “bar”], “B”:[0,1,1,1], “C”:[“A”,”A”,”B”,”A”]}) df.drop_duplicates(subset=[‘A’, ‘C’], keep=False)