Text rendering
An overview of our options. Latest update 16-11-2021.
Goals
I think our goal should be:
- Aim for high quality text rendering (though not necessarily perfect).
- Make sure that there is a default font that looks good and is complete and always available.
- Provide support (one way or another) for all languages, including CJK.
- Preferably support user-provided fonts, so graphs in papers can be publication-worthy quality with a uniform style.
- Preferably an approach where we can start simple and improve on with further steps.
Methods to render text
Leveraging a GUI toolkit
We could leverage the text support of e.g. Qt to create text overlays. We have an example for this in pygfx.
Bitmaps
Historically, fonts are rendered using glyph bitmaps that match the screen resolution. All glyphs in use are put into an atlas: a large image/texture that has all glyphs packed together.
A downside is that the bitmaps should match the screen pixels, so the bitmaps are font-size specific, and rendering glyphs at an angle is not possible without degrading quality.
SDF
In 2007 Chris Green from Valve published an approach that uses scalar distance fields (SDF) to render glyphs. This approach makes it possible to create a special kind if bitmap (which encodes distances to the edge instead of a mask), that can then be used to render that glyph at any scale and rotation. This was a big step in games, but also for generic visualization.
There are variations on the way to calculate an SDF. The current industy standard seems to be anti-aliased euclidian distance transform.
MSDF
Around 2015, Viktor Chlumsky published his Master thesis, in which he proposes a method for a Multi-channel Signed Distance Field (MSDF). The key contribution is that directional information is encoded too, making is possible to produce sharp corners. It produces better results than normal SDF, at a smaller scale. The fragment shader is changed only slightly, causing just a minor performance penalty.
Glyphy
The Glyphy C library converts bezier curves to arcs and then calculates the signed distance to those arcs in the shader, producing better results than normal SDF. It's not well documented, but is by the author of Harfbuzz ...
Vector rendering
It's also possible to render the Bezier curves from the glyphs directly on the GPU. See e.g. http://wdobbie.com/. This approach seems to require some preprocessing to divide the bezier curves into cells. If this can be changed to create an atlas (i.e. each cell is one glyph) then this may be a feasible approach for a dynamic context like ours. Also see Slug, which looks pretty neat, but is not open source and covered by a patent.
Also see this fairly simple approach: https://medium.com/@evanwallace/easy-scalable-text-rendering-on-the-gpu-c3f4d782c5ac It uses a prepass to calculate the winding number for the rough shape, then composes the final shape and applies aa in a fragment shader. A disadvantage (for us) is that it needs multiple passes, making it harder to fit into our renderer, I think.
How do other libs do text rendering?
How visvis does text
https://github.com/almarklein/visvis/tree/master/text
Visvis produces glyphs on the fly using FreeType. Freetype is available by default on Linux, on Windows it must be installed. If FreeType is not available, visvis falls back on system that uses pre-rendered fonts. Visvis also ships with FreeSans, so there is always a good sans font available. Further, it can make use of other fonts installed on the system.
The rendered glyps are on a fixed resolution and put in an atlas texture. When rendered, the glyph is sampled using a smoothing kernel, giving reasonable results at multiple scales.
Further, visvis includes a mini-SDL to create bold, italic, and math symbols using escaping with backslash.
How vispy does text
https://github.com/vispy/vispy/tree/main/vispy/util/fonts and https://github.com/vispy/vispy/tree/main/vispy/visuals/text
Vispy produces SDF glyphs on the fly. It first gets a glyph bitmap using FreeType (via Rougier's FreeType wrapper) on Windows and Linux, and Quartz on MacOS. The bitmap is then converted to an SDF using either the GPU or Cython. It can use most fonts installed on the system. The code to list and load fonts is platform specific.
The SDF glyphs are packed into an atlas texture, and rendered using regular SDF rendering.
How Matplotlib does text
Unless I'm mistaken, they use the GUI backend to render text.
Discussion
SDF rendering already provides pretty solid results and is widely used. MSDF seems like a very interesting extension that may make it possible to create better results with smaller SDF sizes. Whether this is true depends also on the complexity of the glyphs - it has been reported that it does not provide much advantage for e.g. Chinese glyphs. We'd need tests to visually judge what glyph sizes are needed in both approaches.
In terms of dependencies, to create SDF we can get away with just FreeType to create the bitmaps, plus steal some code from vispy to generate the SDF. For MSDF there is that one project, but its not that actively maintained, so at this point it may be a liability. It may be worth investigating if its possible to generate an MSDF from a bitmap.
Vector rendering seems to be popular lately. It produces better results, but is more complex and less performant than SDFs. Further you may need a bitmap fallback for high scales. There is also the question of how to generate the vector-data from the font files.
If you want to do render text perfectly, you should take into account kerning, ligatures, combinatory glyphs, left-to-right languages, etc. We can do kerning, but the rest is way more complex. Tools that handle these correctly are e.g. Slug and Harfbuzz. The latter might be feasible to use - there are multiple Python wrappers.
More details obtaining a glyph
I imagine that somewhere in our code we have a method create_glyph(font, char)
that returns an SDF/MSDF/bezier-curve-set and some metadata. We can place this info in an atlas texture together with other used glyphs. Now, how does this function work?
The below more or less assumes SDF / MSDF, but similar questions apply to the Bezier-based rendering.
Using prerendered glyph maps
We could produce prerendered glyph atlas, so create_glyph()
samples one glyph from that (offline) atlas and to put it in our GPU atlas. That way, the dependencies for rendering the glyphs are only needed by devs, and not the users.
See e.g. https://github.com/Chlumsky/msdf-atlas-gen A disadvantages is that the font and available characters are limited by what atlasses we chose to ship. If we'd ship CJK fonts, that would grow the package size a lot, so maybe we'd need a pygfx-cjk-fonts
or something.
Some measurements for rendering all chars from a font:
- OpenSans -> 1011 glyphs -> 1.5MB @ 32 0.5MB @ 16
- FreeSans -> 4372 glyphs -> 7MB @ 32 2.3MB @ 16
- NotoSans -> 2300 glyps -> 3.5MB @ 32 1.1MB @ 16
- NotoSansJP -> 16735 glyps -> ?? 15.1MB @ 16
This shows that with 16x16 MSDF glyphs, the atlas is about 3x the size of the .ttf file. Or 9x for 32x32. Not bad, but I don't now yet whether 16x16 is enough, especially for the more complex glyphs in CJK fonts.
Using a tool to render glyph on the fly
If we can produce the SDF glyph from the fontfile at the user, then support for all languages, and custom fonts is obtained. The cost is that we need a dependency for this.
One option is FreeType. There is Rougiers https://github.com/rougier/freetype-py which provides binaries for multiple platforms (but not MacOS M1 or Linux aarch64 yet). IIUC freetype can render a glyph bitmap at any scale, but to create an SDF we need to do some extra work ourselves. I am not sure if a MSDF is feasible, because you'd miss the vectors, or can we do edge detection on a high-rez glyph to find the orientation?
Another option is https://github.com/Chlumsky/msdfgen. We'd need to build and distribute the code for all platforms that we support. Since this code is not that actively maintained, this feels a bit tricky.
Typesetting
Typesetting is the process of positioning the quads that contain the glyphs. This includes kerning (some combination of glyhs have a custom distance). Also inclded are justifcation, alignment, text wrapping etc.
I think its not too hard to do this in pure Python, though existing solutions may make things easier. Examples are Slug and Harfbuzz, but these are not feasible dependencies. I don't know of any Python libraries for typesetting.
For the kerning we would need the kerning maps. FreeType can provide this info. I think that msdfgen also provides it in the CSV/JSON metadata.
Formatting
We should consider a formatting approach so users can create bold/italic text etc. Visvis uses escaping, but maybe an micro-HTML-subset would be nicer. Or something using braces. Or maybe its out of scope. This is definitely something that we can do later.
Fonts of interest
- OpenSans is a pretty nice and open font.
- FreeSans is a GNU project that aims to cover almost all Unicode, except CJK.
- Noto is a family of over 100 fonts that aims to have full Unicode coverage. It basically has a font for each script. The plain NotoSans has over 3k glyphs and with support for Latin, Cyrillic and Greek scripts it covers 855 languages.
discussion