My approach is similar to @hellpanderrr’s, but instead tests for list-ness rather than using isnan:
df['ids'] = df['ids'].apply(lambda d: d if isinstance(d, list) else [])
I originally tried using pd.isnull (or pd.notnull) but, when given a list, that returns the null-ness of each element.