A simple solution is to guard the add_subdirectory calls in both the client and the server CMake list file with an if using a TARGET conditional, i.e.:
if (NOT TARGET plugin)
add_subdirectory("${CMAKE_SOURCE_DIR}/common/libplugin")
endif()
This prevents the libplugin sub-directory from being added more than once.