transform scipy sparse csr to pandas?

If A is csr_matrix, you can use .toarray() (there’s also .todense() that produces a numpy matrix, which is also works for the DataFrame constructor):

df = pd.DataFrame(A.toarray())

You can then use this with pd.concat().

A = csr_matrix([[1, 0, 2], [0, 3, 0]])
    
  (0, 0)    1
  (0, 2)    2
  (1, 1)    3

<class 'scipy.sparse.csr.csr_matrix'>

pd.DataFrame(A.todense())

   0  1  2
0  1  0  2
1  0  3  0

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
0    2 non-null int64
1    2 non-null int64
2    2 non-null int64

In version 0.20, pandas introduced sparse data structures, including the SparseDataFrame.

In pandas 1.0, SparseDataFrame was removed:

In older versions of pandas, the SparseSeries and SparseDataFrame classes were the preferred way to work with sparse data. With the advent of extension arrays, these subclasses are no longer needed. Their purpose is better served by using a regular Series or DataFrame with sparse values instead.

The migration guide shows how to use these new data structures.

For instance, to create a DataFrame from a sparse matrix:

from scipy.sparse import csr_matrix

A = csr_matrix([[1, 0, 2], [0, 3, 0]])

df = pd.DataFrame.sparse.from_spmatrix(A, columns=['A', 'B', 'C'])

df

   A  B  C
0  1  0  2
1  0  3  0

df.dtypes
A    Sparse[float64, 0]
B    Sparse[float64, 0]
C    Sparse[float64, 0]
dtype: object

Alternatively, you can pass sparse matrices to sklearn to avoid running out of memory when converting back to pandas. Just convert your other data to sparse format by passing a numpy array to the scipy.sparse.csr_matrix constructor and use scipy.sparse.hstack to combine (see docs).

Leave a Comment