The instance declaration you made is the correct way to go. It seems you forgot to remove that faulty deriving clause from the original data declaration.
data Operace = Op (Int->Int->Int) String (Int->Int->Int)
instance Show Operace where
show (Op op str inv) = show str