I remember reading something once from Microsoft that stated:
“.NET 4.0 adds in-process side-by-side
compatibility with earlier versions of
the Framework. This feature allows an
application process to simultaneously
run part of its code using the .NET
Common Language Runtime (CLR) 4 and
other parts using older versions of
the CLR.”
So to answer your question you only need both if your application uses features from both or altenatively you have separate web applications that use .NET 3.5 and .NET 4.0.
.NET 4.0 should be fine by itself, as long as you can upgrade your 3.5 applications to 4.0. This page has some useful infrmation.
Heres where I read it.