Matplotlibs SVG text rendering can be configured either in the matplotlibrc or in code.
From Customizing Matplotlib with style sheets and rcParams:
#svg.fonttype : 'path' # How to handle SVG fonts:
# 'none': Assume fonts are installed on the machine where the SVG will be viewed.
# 'path': Embed characters as paths -- supported by most SVG renderers
# 'svgfont': Embed characters as SVG fonts -- supported only by Chrome,
# Opera and Safari
This translates to the following code for neither embedding the font nor rendering the text as path:
import matplotlib.pyplot as plt
plt.rcParams['svg.fonttype'] = 'none'