Vision–language models have made zero-shot visual control an appealing proposition for networked autonomous systems. A single pretrained model, prompted in natural language, promises to replace a hand-engineered perception and planning stack, and the same model can be served from an on-board accelerator or from a hosted endpoint reached over the network. Both deployments are now practical, and the choice between them is a systems question involving latency, energy, and recurring cost.
That systems question presupposes a prior one that is rarely asked: does the controller use the camera at all?
The concern is concrete. Simulated driving benchmarks reward forward progress and penalize collisions. In such environments, a policy that always decelerates is close to optimal, and it requires no perception. If a VLM controller emits a near-constant action, aggregate metrics will report competence that the model does not have, and the resulting deployment decision—edge or cloud, which model, and at what cost—will be made on evidence that measures the simulator rather than the model.
We make both the failure and its useful boundary measurable through:
- An input-ablation battery with blind controls, a noise floor, lane-axis reflection, baseline policies, and four pipeline-integrity guards.
- Evidence that monolithic scores and downstream sophistication do not certify perception: constants can outperform scripts, and visual intent reduces an MPC hierarchy from a higher-dimensional control objective to a smaller set of goals.
- A graded capability structure in which longitudinal hazard recognition can be recovered despite failed lateral geometry, while prompt formulation strongly changes measured grounding.
Background and Related Work
On-board inference bounds latency and keeps data local but consumes limited memory and energy. Hosted inference offers larger models at the cost of network delay, recurring charges, and less control over the stack. We compare both on identical stimuli.
Local energy is directly observable; hosted energy is not derivable from the API.
Models
The structured local arm uses Qwen3-VL-8B, Gemma 4-12B, Qwen3.5-9B, Qwen2.5-VL-7B, Ministral 3-8B, and MiniCPM-V-4.5. Recorded Q4_K_M Ollama 0.32.5 builds run sequentially on one A100-80GB.
Preflight pins one request mode per model. Exact digests and templates are released.
Constructive Renderer-Calibrated Controller
The structured controller operates on a calibrated top-down representation with an explicit ego reference and lane convention. In addition to the released simulator frame, we provide a legend-augmented view and evaluate symmetry consistency by reflecting the scene across the lane axis.
Symmetry-Consensus Longitudinal Guardian
We reuse the completed local calls in a post-hoc but leakage-controlled analysis. The pre-existing 16-frame admission set selects two of six models and a threshold over four hazard votes. Two models assess the original and reflected views, and both selections are frozen before the other 272 frames are read.
A 2–2 split permits abstention. Episode-cluster bootstrap and nested leave-one-episode-out tests measure uncertainty and selection stability.
In offline modular replay, the VLMs receive only longitudinal authority while a deterministic RGB parser retains collision and lateral-clearance authority.
Discussion and Conclusion
The MPC experiments clarify the boundary of this modular strategy. A competent low-level optimizer can accurately execute a valid reference and exploit known obstacle geometry, but it cannot reconstruct geometry that is absent from the high-level visual intent.
Adding MPC downstream therefore improves tracking, not grounding: collapsed or one-sided VLM directives remain collapsed after optimization.
The appropriate architectural conclusion is not to discard VLMs, but to restrict their authority to capabilities that can be independently validated. In the present setting, VLMs are most credible as selective longitudinal hazard monitors, while metric geometry, constraint enforcement, and actuation remain under deterministic or otherwise verifiable components.
Code Availability
All code, data, and experimental artifacts are publicly available in the VLControl repository. The repository contains the complete evaluation pipeline, including the input-ablation battery, structured local controllers, MPC experiments, and the symmetry-consensus guardian.
Frequently Asked Questions
What central question does the study ask about vision-language controllers? It asks whether the controller uses the camera at all, rather than merely achieving favorable aggregate control metrics.
Why can constant-action policies appear competent in simulated driving benchmarks? Because forward progress is rewarded and collisions are penalized, a policy that always decelerates can be close to optimal without using perception.
What role does the MPC component play? MPC improves tracking of valid visual intent but cannot reconstruct geometry absent from the high-level VLM directive.
Which capabilities are VLMs considered most credible for in this setting? They are considered most credible as selective longitudinal hazard monitors, while geometry, constraints, and actuation remain under deterministic or verifiable components.