Use this code and don’t waste your time: Q1 = df.quantile(0.25) Q3 = df.quantile(0.75) IQR = Q3 – Q1 df = df[~((df < (Q1 – 1.5 * IQR)) |(df > (Q3 + 1.5 * IQR))).any(axis=1)] in case you want specific columns: cols = [‘col_1’, ‘col_2’] # one or more Q1 = df[cols].quantile(0.25) Q3 = df[cols].quantile(0.75) … Read more
There’s no single “best” test for an outlier. Ideally, you should incorporate a-priori information (e.g. “This parameter shouldn’t be over x because of blah…”). Most tests for outliers use the median absolute deviation, rather than the 95th percentile or some other variance-based measurement. Otherwise, the variance/stddev that is calculated will be heavily skewed by the … Read more
If you have multiple columns in your dataframe and would like to remove all rows that have outliers in at least one column, the following expression would do that in one shot. df = pd.DataFrame(np.random.randn(100, 3)) import numpy as np from scipy import stats df[(np.abs(stats.zscore(df)) < 3).all(axis=1)] description: For each column, it first computes the … Read more