RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work
In briefRealityBench: Simulated environments for testing whether vision-language models can document physical work
SummaryThe paper builds simulated field-work scenes in Blender across eleven environments, such as rooftop solar installation and vehicle repair, to test 26 vision-language models. It evaluates whether an image is acceptable as proof of work and whether a model can understand the task-relevant state shown in the image. Top models handle text in frame well but struggle with measurements and small parts or spatial relations, and the best models match a single non-author rater on acceptability, which is itself hard to judge due to rater disagreement.