Gemini 4 Neon fails scanned document test while GPT-6.2 Astra succeeds
A new benchmark reveals that Gemini 4 Neon hit its output limits on complex visual tasks, while GPT-6.2 Astra handled high-reasoning document analysis with high accuracy.
The short version
- Gemini 4 Neon hit a 10,000-token completion cap on all three test calls and returned no answers.
- GPT-6.2 Astra scored 52 out of 54 questions correct in the same scanned-document test.
- The test involved three synthetic vendor-audit packets containing 24,000 words and three scanned tables.
- Astra's only two misses were due to selecting a lookalike row instead of the correct one in a complex dataset.
- Both models scored perfectly in a previous text-only pilot, showing the difficulty of visual data processing.
Why it matters
This test highlights a significant gap in how different AI models handle 'multimodal' reasoning, where the AI must combine visual information with complex logic. For businesses using AI to automate document auditing, these results show that model choice is critical for reliability in visual tasks.
The Scanned Document Challenge
Kingy AI recently conducted a rigorous test to see how two leading AI models handle complex, real-world document processing. The test used three synthetic vendor-audit packets that included about 24,000 words of text and three scanned tables. These tables contained specific data like billed rates, receiving counts, and posted adjustments.
The documents were intentionally difficult to process because they included mild skew, blur, and copy speckle. The models were required to match identifiers across different documents and apply specific controlling amendments. They also had to calculate invoices or decide to hold them based on the evidence provided in the scans.
To ensure a high standard of accuracy, a point was only awarded if the model provided the exact expected value and the complete set of record citations. If a model included a missing, extra, or duplicated citation, it lost the point for that question. This setup tested the models' ability to perform precise data extraction and logical reasoning simultaneously.
Gemini 4 Neon vs. GPT-6.2 Astra
The results showed a stark contrast between the performance of the two models. Gemini 4 Neon failed to provide any answers at all, scoring 0 out of 54 questions. Each of the three calls for Neon reported hitting a 10,000-token reasoning cap, meaning the model reached its limit before it could finish the task.
In contrast, GPT-6.2 Astra performed very well, scoring 52 out of 54 questions correctly. It demonstrated high proficiency in handling rate and amendment precedence, scoring 18 out of 18 in those categories. It also performed well on quantity and scanned records, scoring 17 out of 18.
Astra's two misses were specific and technical. In one instance, it selected a lookalike receiving row instead of the correct one, which led to an incorrect invoice calculation. While its internal arithmetic was consistent based on the row it chose, the initial selection error caused the final answer to be wrong. As Kingy AI reported, the model's ability to navigate these complex identifiers was the primary differentiator.
Comparing Text and Visual Tasks
The performance gap in the scanned document test is particularly notable because both models performed perfectly in an earlier pilot. That pilot used text-only data from synthetic facilities casebooks, where both models answered all 48 questions correctly. This suggests that the difficulty lies specifically in the visual and multimodal nature of the scanned images.
In the text-only test, the input sizes were roughly 32,000 tokens for Neon and 29,000 tokens for Astra. The models were asked to join records, apply corrections, and calculate invoices without the added complexity of interpreting scanned tables. Both models achieved a perfect 16 out of 16 score on each packet in that earlier trial.
The current test used much larger inputs, ranging from 37,000 to 41,000 input tokens per request. By adding scanned PNG images without using OCR (optical character recognition) or external tools, the test pushed the models to rely entirely on their internal reasoning. This highlighted how Gemini 4 Neon struggled with the increased complexity of visual data compared to its text-only performance.
Costs and Technical Settings
The researchers used specific settings to ensure a fair comparison between the two systems. They requested High reasoning effort for both models through OpenRouter and disabled provider fallback. They also set a 10,000-token cap per request for the combined total of reasoning and visible output.
The total cost for the six requests in the scanned document test was $2.207585. Gemini 4 Neon cost $0.786483 for its three requests, even though it returned no answers. GPT-6.2 Astra cost $1.421102 for its three requests, which included a $0.244902 cache-write premium.
The test was conducted on November 21, 2026, using specific snapshots of the models. Neon used the google/gemini-4-neon-20261118 snapshot via Google Vertex global, while Astra used the openai/gpt-6.2-astra-20261117 snapshot via OpenAI. These settings ensured that the models were tested under similar constraints regarding reasoning depth and output limits.
Limitations and Next Steps
While the test provides a clear look at how these models handle complex documents, it has some specific limitations. The benchmark used a narrow view of accuracy, latency, and cost because it only used three related packets. It did not test larger scan collections or the impact of search functions.
The results also note that provider routes and preprocessing can affect how a model performs. Because compute and image preprocessing can vary by provider even at the same effort setting, these results are specific to the configurations used by Kingy AI. The test was independent, as neither Google nor OpenAI sponsored the study.
For users who need to perform consequential work, the researchers suggest a cautious approach. They recommend checking the cited rows and calculations against the original documents manually. Future testing could explore how these models handle larger datasets or more varied types of visual distortions in scanned files.
What to watch
- Whether Gemini 4 Neon receives a higher token cap to handle complex visual tasks.
- How GPT-6.2 Astra handles larger batches of scanned documents beyond three packets.
Try it
If you use AI for data entry, try uploading a blurry scan of a table and ask the model to 'cite the specific row' for a calculation to see if it can handle visual noise.
Sources and links
- OfficialOpenRouter’s image-input documentation openrouter.ai
- ReportingKingy AI: the article we sourced
First reported by Kingy AI. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.