Give it a regex capture group:
df.A.str.extract('(\d+)')
Gives you:
0 1
1 NaN
2 10
3 100
4 0
Name: A, dtype: object
(\d+) is a regex capturing group, and \d+ specifies a regex pattern that matches only digits. Note that this will only work for whole numbers and not floats.