pandas – Page 241 – Tarik Billa

How do I expand the output display to see more columns of a Pandas DataFrame?

September 6, 2022 by Tarik

Update: Pandas 0.23.4 onwards This is not necessary. Pandas autodetects the size of your terminal window if you set pd.options.display.width = 0. (For older versions see at bottom.) pandas.set_printoptions(…) is deprecated. Instead, use pandas.set_option(optname, val), or equivalently pd.options.<opt.hierarchical.name> = val. Like: import pandas as pd pd.set_option(‘display.max_rows’, 500) pd.set_option(‘display.max_columns’, 500) pd.set_option(‘display.width’, 1000) Here is the help … Read more

Writing a pandas DataFrame to CSV file

September 5, 2022 by Tarik

To delimit by a tab you can use the sep argument of to_csv: df.to_csv(file_name, sep=’\t’) To use a specific encoding (e.g. ‘utf-8’) use the encoding argument: df.to_csv(file_name, sep=’\t’, encoding=’utf-8′)

Convert list of dictionaries to a pandas DataFrame

September 5, 2022 by Tarik

If ds is a list of dicts: df = pd.DataFrame(ds) Note: this does not work with nested data.

Pretty-print an entire Pandas Series / DataFrame

September 5, 2022 by Tarik

You can also use the option_context, with one or more options: with pd.option_context(‘display.max_rows’, None, ‘display.max_columns’, None): # more options can be specified also print(df) This will automatically return the options to their previous values. If you are working on jupyter-notebook, using display(df) instead of print(df) will use jupyter rich display logic (like so).

Use a list of values to select rows from a Pandas dataframe

September 1, 2022 by Tarik

You can use the isin method: In [1]: df = pd.DataFrame({‘A’: [5,6,3,4], ‘B’: [1,2,3,5]}) In [2]: df Out[2]: A B 0 5 1 1 6 2 2 3 3 3 4 5 In [3]: df[df[‘A’].isin([3, 6])] Out[3]: A B 1 6 2 2 3 3 And to get the opposite use ~: In [4]: df[~df[‘A’].isin([3, … Read more

“Large data” workflows using pandas [closed]

September 1, 2022 by Tarik

I routinely use tens of gigabytes of data in just this fashion e.g. I have tables on disk that I read via queries, create data and append back. It’s worth reading the docs and late in this thread for several suggestions for how to store your data. Details which will affect how you store your … Read more

How to deal with SettingWithCopyWarning in Pandas

August 30, 2022 by Tarik

The SettingWithCopyWarning was created to flag potentially confusing “chained” assignments, such as the following, which does not always work as expected, particularly when the first selection returns a copy. [see GH5390 and GH5597 for background discussion.] df[df[‘A’] > 2][‘B’] = new_val # new_val not set in df The warning offers a suggestion to rewrite as … Read more

How to drop rows of Pandas DataFrame whose value in a certain column is NaN

August 30, 2022 by Tarik

Don’t drop, just take the rows where EPS is not NA: df = df[df[‘EPS’].notna()]

How to add a new column to an existing DataFrame?

August 30, 2022 by Tarik

Edit 2017 As indicated in the comments and by @Alexander, currently the best method to add the values of a Series as a new column of a DataFrame could be using assign: df1 = df1.assign(e=pd.Series(np.random.randn(sLength)).values) Edit 2015 Some reported getting the SettingWithCopyWarning with this code. However, the code still runs perfectly with the current pandas … Read more

Create a Pandas Dataframe by appending one row at a time

August 30, 2022 by Tarik

You can use df.loc[i], where the row with index i will be what you specify it to be in the dataframe. >>> import pandas as pd >>> from numpy.random import randint >>> df = pd.DataFrame(columns=[‘lib’, ‘qty1’, ‘qty2’]) >>> for i in range(5): >>> df.loc[i] = [‘name’ + str(i)] + list(randint(10, size=2)) >>> df lib qty1 … Read more