Using object-fit on a with child elements, including a

I don’t know if it’s ok for you, but I think it can be done mostly in css if you allow a little bit more HTML.

Consider the following html

<div class="outerContainer">
  <div class="canvasContainer">  
    <canvas width="640" height="360"></canvas>
  </div>
  <div class="beside">
  </div>
</div>

I just added a little div around the canvas. This way we let the canvas handle it’s things and we use the div for the flex things.

Using the following css:

* {
  box-sizing: border-box;
}

.outerContainer {
  display: flex;
  border: 0.5em solid #444;
  margin-bottom: 2em;
}

.canvasContainer canvas {
  width: 100%;
  background: #777;
  margin-bottom: -4px
}

.canvasContainer {
  flex-grow: 1;
  background: #77a;
}

/* This element has a fixed width, and should be whatever height the <canvas> is */
.outerContainer .beside {
  flex-basis: 3em;
  flex-grow: 0;
  flex-shrink: 0;
  background: #7a7;
}

We have canvas container taking all the available space. And then the canvas adapt accordingly with the image scaling in it.
However I don’t know why, there was a bit of a margin on the bottom of the canvas, hence the negative margin on it.

fiddle: https://jsfiddle.net/L8p6xghb/3/

As a side note, if you’re looking to use that for captions, there are html5 element for it, like <figure> and <figcaption>!