C-LARA

I have been testing the new image generation functionality, and this has thrown up some interesting issues. I thought people following the picture book work might be curious to see the details, so here’s a nice example from La Fontaine’s classic story of the Crow and the Fox, where C-LARA is currently getting it wrong.

First, for the benefit of those who can’t remember the story, it’s very simple: the Fox sees the Crow with a piece of cheese in its mouth and craftily says it’s sure the bird has a lovely voice. The Crow falls for the trick, opens its mouth to sing, and drops the cheese. The Fox triumphantly says that learning not to trust flattery is such a useful lesson that it’s surely worth a piece of cheese. With that introduction, on the second page of the text, i.e. the page directly after the title page, we have this text:

Maître corbeau, sur un arbre perché,
Tenoit en son bec un fromage.

[Master crow, perched on a branch,
Held in his beak a cheese.}

The image specification generated by o1-mini and passed to DALL-E-3 starts like this:

Create an illustration for page 2 of "Le Corbeau et le Renard" using the traditional early 19th-century French lithographic style. The scene should depict the corbeau perched on an oak tree holding a wheel of cheese in its beak, consistent with the detailed specifications of previous elements and pages.

One of the images produced looks like this:

and when GPT-4o is asked to produce a detailed description of the image, it includes the following section:

### Cheese:
- **Appearance:** A large wheel of cheese is suspended from one of the tree branches. It is stylized and round with holes, reminiscent of Swiss cheese.
- **Color and Texture:** The cheese is depicted in a light color that contrasts with the darker elements of the scene, showing detailed shading to indicate texture.

In the next step, when GPT-4o is asked to compare the specification against the description and note discrepancies, it gives us this:

The size and position of the cheese do not match the specification; the description mentions it suspended from a branch rather than in the raven's beak. Additionally, the raven is described as large, contrasting with the specified medium size. Overall, the details are mostly consistent, capturing the style and pastoral setting, but these discrepancies are noteworthy.

Now here’s the interesting thing: the AI rates the match as 3 on a scale of 0 to 4, i.e. “Good”, and accepts the image! It’s spotted the critical difference, that the cheese is hanging from the tree branch rather than being in the crow’s beak, but it doesn’t have enough context to realise that this is important. We showed it the prompt and the image description, but we never made clear which aspects of the prompt were part of the story and which were just decoration.

There are several ways to try to address this issue; the one I’ll try first is to add an extra step before creating the image specification, where we ask the AI to write down the essential elements of the image. Then we’ll include the “essential elements” text in the instructions passed to the comparison-and-rating step, so that the AI instance doing that task is better informed.

It remains to be seen if this will work, but I hope I’ve shown you how interesting the picture book image generation task is, and how much intelligence it requires.