Assessing VQA Model Reliability Through Systematic Evaluation of Corrupted Question Handling

Assessing the ability of Visual Question Answering Models (VQAs) to handle question answering (QA) in multi-page documents requires addressing a key limitation: their vulnerability to distorted input. When questions contain typographical errors or incorrect references, VQAs often fail to recognize that these seemingly valid queries are actually unanswerable. This challenge is amplified in visually rich documents, where multimodal elements like figures, tables, and complex layouts introduce additional layers of ambiguity. In this thesis project, a new framework is introduced, specifically designed to test VQAs’ robustness against corrupted questions. Unlike traditional VQA benchmarks, which provide little consideration for distorted inputs, the framework systematically alters questions at various levels—manipulating linguistic entities, document structures, and visual layouts. A preliminary verification step ensures that these modifications produce genuinely unanswerable questions before they are used for evaluation. The framework employs targeted performance metrics to measure how accurately models respond with "Unable to Determine", analyze the extent to which document layout and multimodal elements impact performance, and explore correlations between incorrect responses and their original, uncorrupted counterparts.