The referrer whitelisting works for MPD and HLS URLs. The mp4 files are directly served by CDN, so we do not have control over the referrer protection.
If you do not intend to use the MP4 URL directly, I would recommend disabling the MP4 processing in the Processing Settings. With this change, your videos will be referrer-protected and will work only on authorized domain(s).