Try the following test:
\documentclass{article}
\usepackage{xifthen}% provides \isempty test
\newcommand{\optarg}[1][]{%
\ifthenelse{\isempty{#1}}%
{}% if #1 is empty
{(((#1)))}% if #1 is not empty
}
\begin{document}
Testing \verb|\optarg|: \optarg% prints nothing
Testing \verb|\optarg[]|: \optarg[]% prints nothing
Testing \verb|\optarg[test]|: \optarg[test]% prints (((test)))
\end{document}
The xifthen package provides the \ifthenelse construct and the \isempty test.
Another option is to use the ifmtarg package (see the ifmtarg.sty file for the documentation).