CSS transitions work by defining two states for the object using CSS. In your case, you define how the object looks when it has the class "saved" and you define how it looks when it doesn’t have the class "saved" (it’s normal look). When you remove the class "saved", it will transition to the other state according to the transition settings in place for the object without the "saved" class.
If the CSS transition settings apply to the object (without the "saved" class), then they will apply to both transitions.
We could help more specifically if you included all relevant CSS you’re using to with the HTML you’ve provided.
My guess from looking at your HTML is that your transition CSS settings only apply to .saved and thus when you remove it, there are no controls to specify a CSS setting. You may want to add another class ".fade" that you leave on the object all the time and you can specify your CSS transition settings on that class so they are always in effect.