The reversal: a huge gap between the web app and direct API calls

But testing changed everything. I could not tell whether the model failed to understand my prompt or the images I sent. This time, both accuracy and consistency were very low—accuracy was around 50%, and it often contradicted itself. My batteries had only two labels, so I thought: wow, it is just making things up.

But when I sent a single image to a VLM for description, the gap from ChatGPT on the web was not visibly that large. For both single-image and multi-image requests, I put the images into this structure:

1
OpenAIMessage(
2
  role="user",
3
  content=[
4
    {"type": "text", "text": "user_prompt"},
5
    {"type": "image_url", "image_url": "data:image/png;base64,iVBORw..."},
6
    ...(images)
7
  ]
8
)

At that point, I thought the model difference might really be the issue, so I tested API calls with models such as gemini-flash-2.5 and chatgpt-5.1-chat. But even with the same model, its accuracy differed greatly from what I got on the web. Once the number of images exceeded four, consistency also began to decline.

After a little research, I found that this looked very similar to the long-context retrieval and positional-bias problem described by Lost in the Middle. The large difference between web and API performance may be because:

• The web app may perform per-image pre-summarization, reranking, or selective feeding.
• The web app may have stronger system prompts and output constraints, such as forcing JSON or a response for each image.
• API-side parameters—temperature, max output, tool choice, and concurrency order—may also affect consistency.

A base64 image is not reasoned over directly

Using base64 for an image_url is merely convenient for HTTP transport. During inference, the model decodes the base64 into an image. In other words, a VLM actually sees the user prompt plus the image.

How images are resized and counted as tokens

See OpenAI’s Calculating costs documentation.

How pixels become vision tokens

OpenAI does not publish this. The following are approaches used by some open-source VLMs:

How vision tokens are concatenated with or isolated from prompt tokens

Concatenation is the simplest, end-to-end approach: let the LLM decide for itself. But visual tokens consume context length, and attention is more easily diluted with multiple images. It faces the same problem as long context: attention is not only diluted, but unevenly so; it also accelerates context growth.

Cross-attention treats vision as external memory that text queries on demand. It is easier to control the “visual-information budget” in engineering terms and better suited to long sequences and multiple images, but it adds modules and training complexity. Query uncertainty makes it hard to know whether the model actually saw a particular image. Its essential problem is therefore similar to the previous approach: there, a model may selectively ignore an image; here, the image may never be queried.

Query bottlenecks: I do not understand them well enough yet…

Uncertain quantity

A user might send more than ten images and blow up the token budget, assuming the request does not exceed the input/output limit. With more images, less attention goes to each one, and we do not know how it is allocated. Some images may not be “attended to” at all.

Multiple images split attention, and in a real application we do not know how many images will compete for it. The design should therefore accommodate any acceptable input count.

CAUTION

Of course, VLM reasoning alone cannot achieve this today. As the battery-detection retest below shows, images beyond a single request’s token limit are simply truncated.

Uncertain relationships

For example, an indecisive user may submit many character images—two images of each character—and ask us to compare them and choose the one they prefer. The image pairs are highly coupled. A model may realize that these pairwise relationships exist, but not know which images form the pairs. With more images, it is unlikely to match every similar character at once.

Sometimes images are coupled. We want to guide the model to notice that coupling instead of dropping everything in a pile and asking it to figure it out.

Uncertain specific tasks

Often, the task is not written directly in the current message; it must be inferred from the entire context. For example, while role-playing:

1
U: "Don't you dislike mushrooms?"
2
A: "Yes. If you dare put any in the soup, I will show you what regret means."
3
U: "Then look at this: [an image of food being cooked, with many ingredients including mushrooms]"

Often the puzzle is not stated outright, such as “find the mushrooms in the image.” More commonly, the models I encounter do not connect their reply to the whole context very much. They put nearly all their attention weight on the latest user instruction—something like “describe this” or “look at this”—then start describing the entire scene at length. That may be a learned preference. It is not the response we ideally want: “Hm? I think I see mushrooms. You put mushrooms in it, didn’t you? You did, right?”

Advantages

TIP

• It does not care how many images arrive. In theory, even twenty or thirty can be processed. The JSON can directly guide the model’s visual understanding.
• We can customize JSON keys through prompts to create coupling or separation and clarify image relationships.
• It turns a seemingly heavier and harder multi-image analysis task into a text-understanding task that can rely on text for its answer. Images occupy less of the context and text more, allowing the model to work with language and making it more likely to attend to the entire context.

Disadvantages

CAUTION

• Expensive—not just a little expensive. Every image needs its own extraction pass, then all images may be sent again for the multi-image pass. The improvement becomes more obvious with more images; for one or two images, there is little reason to do this. It should not be the default method, but an option users can enable.
• Slow—even with concurrency, it adds at least one response round compared with direct conversation.

Send image content directly to the model

Test code:

1
from __future__ import annotations
2
import base64
3
from openai import OpenAI
4

5

6
def image_to_base64(path: str) -> str:
7
    with open(path, "rb") as f:
8
        return base64.b64encode(f.read()).decode("utf-8")
9

10

11
image_paths = ["pic/1.jpg", "pic/2.jpg", ..., "pic/10.jpg"]
12

13
for i in range(10):
14
    image_contents = []
15
    for path in image_paths[: i + 1]:
16
        image_contents.append(
17
            {
18
                "type": "image_url",
19
                "image_url": {"url": f"data:image/jpeg;base64,{image_to_base64(path)}"},
20
            }
21
        )
22
    response = client.chat.completions.create(
23
        model=model_name,
24
        messages=[{"role": "user", "content": [{"type": "text", "text": prompt_text}, *image_contents]}],
25
        temperature=0,
26
    )
27
    print(f"Number of input images: {i + 1}")
28
    print(response.choices[0].message.content)

Output:

1
Ground truth: p1 intact p2 intact p3 damaged p4 intact p5 damaged p6 intact p7 intact p8 damaged p9 intact p10 damaged
2
Input images: 1  p1 intact
3
Input images: 2  p1 intact p2 intact
4
Input images: 3  p1 intact p2 intact p3 damaged
5
Input images: 4  p1 intact p2 intact p3 damaged p4 damaged
6
Input images: 5  p1 intact p2 intact p3 damaged p4 intact p5 damaged
7
Input images: 6  p1 intact p2 intact p3 damaged p4 damaged p5 damaged
8
Input images: 7  p1 intact p2 intact
9
Input images: 8  p1 intact p2 intact p3 damaged
10
Input images: 9  p1 intact p2 intact p3 damaged p4 damaged
11
Input images: 10 p1 intact p2 intact p3 damaged p4 damaged p5 damaged

As you can see, consistency started becoming a problem after more than three images. With five inputs, it happened to guess all of them correctly. But beyond five, the model did not reply about the other images at all. I suspect they were truncated because the single-request input-token limit was exceeded.

Apply a map-reduce approach

For it, this task consists of independent items, so the number of items and image truncation do not have much impact. Of course, truncation remains painful for tasks that require relationships between images.

There is also a substantial issue with token consumption and overly fast context growth. Concurrent single-image processing is already expensive. Combining the outputs into the original context makes that context grow very quickly, bringing the long-context attention bottleneck earlier.

The prompt I use

1
You are a "Vision Extractor" responsible for extracting facts and evidence relevant to the user's question from input images.
2
You do not need to write the final natural-language answer; another chat model will produce the final conversational response.
3
Your goal is to use concise, reusable, machine-consumable structured output to accurately describe the information in the images that is relevant to the question, and to state uncertainty explicitly.

The complete prompt includes an explanation of the inputs, five key principles—describe only what is visible, support downstream reasoning, focus on what is relevant to the question, control length and density, and handle private information—and strict requirements for JSON output.

Performance on relational tasks

It looks as though I did a poor job with send_text, or perhaps truncating by character count made the layout look awful. Another point: gpt-5.1-2025-11-13 replies rather stiffly. Version 5.2 costs far more. I generally use vision fallback: the vision model is called only when looking at images, generates summaries, and passes them to a chat model. That lets me choose a chat model with more human warmth, while also deciding whether to send it the summaries together with the images.