If you do decide to use C++, then The Synthesis Toolkit is worth noting. I use it for a schoolproject and it is very usable, developed at stanford university, crossplatform (win, mac and linux), free and opensource. An extract from the wikipedia page on it:
Versions of the STK instrument classes
have been integrated into ChucK,
Csound, Real-Time Cmix, Max/MSP (as
part of PeRColate) and SuperCollider
They have a lot of testcode included + on the site are some tutorials to get started with their library. (But you do have to know some C++ of course)
STK has some classed to read/write audio files (and matlab files), realtime audio/midi io, some basic algorithms (usage based on similar functions out of matlab), …
Another obvious option and probably THE most used environment to learn/test dsp stuff is Matlab (or free octave, but i havent used it before). I sometimes test stuff out in Matlab before implementing it in C++.
EDIT: In the past year I’ve been using JUCE which is a pretty cool C++ library centered around C++ audio programming, though its more leaning towards something like QT. One of their developers (Timur Doumler) has a nice talk about real time audio with C++ and JUCE on CppCon 2015, definitely worth checking out, even if you don’t end up using JUCE: https://youtu.be/boPEO2auJj4
Another resource that might be interesting is the audio section of the awesome cpp github readme which contains a number of libraries related to audio