I am seeing this error message when I run Firefox headless through selenium using xvfb. It turns out that the message was a red herring for me. The message is only a warning, not an error. It is not why Firefox was not starting correctly.
The reason that Firefox was not starting for me was that it had been updated to a version that was no longer compatible with the Selenium drivers that I was using. I upgraded the selenium drivers to the latest and Firefox starts up fine again (even with this warning message about RANDR).
New releases of Firefox are often only compatible with one or two versions of Selenium. Occasionally Firefox is released with NO compatible version of Selenium. When that happens, it may take a week or two for a new version of Selenium to get released. Because of this, I now keep a version of Firefox that is known to work with the version of Selenium that I have installed. In addition to the version of Firefox that is kept up to date by my package manager, I have a version installed in /opt/ (eg /opt/firefox31/). The Selenium Java API takes an argument for the location of the Firefox binary to be used. The downside is that older versions of Firefox have known security vulnerabilities and shouldn’t be used with untrusted content.