Looks like you already assumed correct. The ability to parse query strings was taken out of V4 because there have been requests over the years to support different implementation. With that, the team decided it would be best for users to decide what that implementation looks like. We recommend importing a query string lib. The one you mentioned has worked great for me so far.
const queryString = require('query-string');
const parsed = queryString.parse(props.location.search);
You can also use new URLSearchParams if you want something native and it works for your needs
const search = props.location.search; // could be '?foo=bar'
const params = new URLSearchParams(search);
const foo = params.get('foo'); // bar
You can read more about the decision here