"This is my string with date in specified format $($theDate.ToString('u'))"
or
"This is my string with date in specified format $(Get-Date -format 'u')"
The sub-expression ($(...)) can include arbitrary expressions calls.
Microsoft Documents both standard and custom DateTime format strings.