Docs for create-react-app explains how to serve same build from different relative paths.
If you put homepage as
"homepage": ".",
assets will be served relative to index.html. You will then be able to move your app from http://mywebsite.example to http://mywebsite.example/relativepath or even http://mywebsite.example/relative/path without having to rebuild it.
For development purposes, serving using yarn start or npm start is good enough. App will be available in localhost