Detection and Mitigation of Issues in Video Summarization Using Video Large Language Models

Video Large Language Models (VLLMs) have recently demonstrated strong capabilities in multimodal understanding. Yet, their effectiveness in structured video-to-text summarization remains insufficiently studied, as most evaluations focus solely on plain-text outputs. This thesis investigates the zero-shot capabilities of six state-of-the-art VLLMs across five distinct summary formats: plain text, event-based, spatially contextualized, timeline, and spatio-temporal. The analysis reveals persistent limitations in structured summaries, particularly in providing valid temporal annotations and adhering to required output formats. To address these challenges, four categories of mitigation strategies are introduced. The first employs Chain-of-Thought prompting, where external video knowledge extracted from lightweight models is injected into prompts to guide temporal and spatial reasoning. The second applies LLM-based post-hoc refinement, which identifies and corrects structural errors through tailored instructions. To further exploit external knowledge, two novel pipelines are designed to integrate auxiliary cues into structured summarization. Multimodal co-Summarization strategy preprocesses external video knowledge into a provisional summary using a lightweight TextLLM, which is then combined with the video input by a VLLM to generate the final structured output. QA-assisted hierarchical summarization leverages VLLMs’ video question answering capabilities to extract event-level details from pre-segmented clips, based on timestamps obtained from external models, and merges them into coherent summaries. Experiments conducted on a new mixed-type benchmark of 100 videos sampled from five datasets demonstrate that these strategies yield substantial improvements. QA-assisted hierarchical summarization eliminates formatting issues and enhances spatio-temporal alignment, while multimodal co-summarization achieves statistically significant gains in timeline evaluation. Object-based cues contribute to improved temporal grounding in both strategies, though they frequently introduce event fragmentation. In contrast, action-based cues consistently enhance event detection and temporal coherence, while scene-based cues strengthen spatial grounding. The findings confirm that combining VLLMs with lightweight external models and TextLLMs enables more reliable, cost-effective, and contextually grounded video summarization without the need for fine-tuning. The proposed benchmark, issue taxonomy, and mitigation frameworks provide a foundation for future research for Video-to-Text summarization.