You should look at the problem from a different perspective. Testing whether a function uses a transaction is useless from a behavioral viewpoint. It does not give you any information on whether the function BEHAVES as expected.
What you should test is the behavior, i.e. expected outcome is correct. For clarity, lets say you execute operation A and operation B within the function (executed within one transaction). Operation A credits a user 100 USD in your app. Operation B debits the users credit card with 100 USD.
You should now provide invalid input information for the test, so that debiting the users credit card fails. Wrap the whole function call in an expect { ... }.not_to change(User, :balance).
This way, you test the expected BEHAVIOR – if credit card debit fails, do not credit the user with the amount. Also, if you just refactor your code (e.g. you stop using transactions and rollback things manually), then the result of your test case should not be affected.
That being said, you should still test both operations in isolation as @luacassus mentioned. Also, it is exactly right that your test case should fail in case you made an “incompatible” change (i.e. you change the behavior) to the sourcecode as @rb512 mentioned.