Your problem here is that to_datetime silently failed so the dtype remained as str/object, if you set param errors="coerce" then if the conversion fails for any particular string then those rows are set to NaT.
df['Date'] = pd.to_datetime(df['Date'], errors="coerce")
So you need to find out what is wrong with those specific row values.
See the docs