See pandas: IO tools for all of the available .read_ methods. Try the following code if all of the CSV files have the same columns. I have added header=0, so that after reading the CSV file’s first row, it can be assigned as the column names. import pandas as pd import glob import os path … Read more
RukTech’s answer, df.set_value(‘C’, ‘x’, 10), is far and away faster than the options I’ve suggested below. However, it has been slated for deprecation. Going forward, the recommended method is .iat/.at. Why df.xs(‘C’)[‘x’]=10 does not work: df.xs(‘C’) by default, returns a new dataframe with a copy of the data, so df.xs(‘C’)[‘x’]=10 modifies this new dataframe only. … Read more
Quick Answer: The simplest way to get row counts per group is by calling .size(), which returns a Series: df.groupby([‘col1′,’col2’]).size() Usually you want this result as a DataFrame (instead of a Series) so you can do: df.groupby([‘col1’, ‘col2’]).size().reset_index(name=”counts”) If you want to find out how to calculate the row counts and other statistics for each … Read more
Use the isna() method (or it’s alias isnull() which is also compatible with older pandas versions < 0.21.0) and then sum to count the NaN values. For one column: >>> s = pd.Series([1,2,3, np.nan, np.nan]) >>> s.isna().sum() # or s.isnull().sum() for older pandas versions 2 For several columns, this also works: >>> df = pd.DataFrame({‘a’:[1,2,np.nan], … Read more
You can use pd.Series.isin. For “IN” use: something.isin(somewhere) Or for “NOT IN”: ~something.isin(somewhere) As a worked example: import pandas as pd >>> df country 0 US 1 UK 2 Germany 3 China >>> countries_to_keep [‘UK’, ‘China’] >>> df.country.isin(countries_to_keep) 0 False 1 True 2 False 3 True Name: country, dtype: bool >>> df[df.country.isin(countries_to_keep)] country 1 UK … Read more
The or and and python statements require truth-values. For pandas, these are considered ambiguous so you should use “bitwise” | (or) or & (and) operations: df = df[(df[‘col’] < -0.25) | (df[‘col’] > 0.25)] These are overloaded for these kinds of data structures to yield the element-wise or or and. Just to add some more … Read more
either: df[‘index1’] = df.index or, .reset_index: df = df.reset_index(level=0) so, if you have a multi-index frame with 3 levels of index, like: >>> df val tick tag obs 2016-02-26 C 2 0.0139 2016-02-27 A 2 0.5577 2016-02-28 C 6 0.0303 and you want to convert the 1st (tick) and 3rd (obs) levels in the index … Read more
The idiomatic way to do this with Pandas is to use the .sample method of your data frame to sample all rows without replacement: df.sample(frac=1) The frac keyword argument specifies the fraction of rows to return in the random sample, so frac=1 means to return all rows (in random order). Note: If you wish to … Read more
Based on github issue #620, it looks like you’ll soon be able to do the following: df[df[‘A’].str.contains(“hello”)] Update: vectorized string methods (i.e., Series.str) are available in pandas 0.8.1 and up.
NEVER grow a DataFrame row-wise! TLDR; (just read the bold text) Most answers here will tell you how to create an empty DataFrame and fill it out, but no one will tell you that it is a bad thing to do. Here is my advice: Accumulate data in a list, not a DataFrame. Use a … Read more