Yes you should cache your manifest.json file, because if you a building a PWA, then it must have a functionality to Add to home screen. In your manifest file, it contains a start_url that needs to be cached by service worker and should return a 200 response when offline.
Let’s assume anyone is offline browsing your app and hasn’t added the web app on home screen, then it will ask user if he/she wants to add to home screen or not provided you have cached your manifest file.