You could start with the Ace editor (formerly Bespin and Skywriter). It’s aimed at code editing, so it’s missing formatting and other features, but you may find a useful core of functionality to base a rich text editor on.
In action: http://ajaxorg.github.com/ace/build/editor.html
Code: https://github.com/ajaxorg/ace
Update: As @theazureshadow points out, the current editor doesn’t use canvas as I originally reported. Bespin used canvas before it merged with Ace, which uses the DOM. Peeking briefly under the hood, it doesn’t appear they are using contentEditable or designMode, though. (There’s a <textarea /> that follows the cursor around and spans and divs to show the text – a bunch of custom JS to wire it all together, methinks.)
From ace.ajax.org > History:
Bespin started as part of Mozilla Labs and was based on the
<canvas>tag, while Ace is the Editor component of the Cloud9 IDE
and is using the DOM for rendering.