视频语义描述即理解视频内容并使用自然语言进行描述，在人机交互、自动驾驶和智能安防等方面具有巨大的应用潜力。近年来，随着深度学习技术在计算机视觉和自然语言处理领域的突破，许多研究开始关注以图像语义描述和视频语义描述为代表的跨模态任务。同时，由于视频内容的复杂性以及视频帧的时序特性，视频语义描述比图像语义描述具有更大的挑战。视频内容涉及视觉、语音等多模态特征，描述任务要求生成文本与视频内容对齐，而现有的主流视频语义描述数据集的描述标注往往只考虑了视频的视觉信息，对语音、视频文字等视频中的其他模态信息考虑较少。因此，本文围绕如何融合视频中的多模态特征进行视频语义描述这个课题，开展了如下研究。首先，本文共同构建了一个基于新闻内容的多模态视频语义描述数据集。该数据集采集了来自YouTube的1,200个完整视频以及裁剪后的11,000个新闻视频片段，同时从粗粒度、中粒度和细粒度三个角度对视频内容进行描述标注，以提供对视频内容的不同粒度理解。本文也统计分析了数据集内容，表明了数据集的应用前景。其次，本文围绕视频语义描述模型的轻量化展开了研究，本文设计了一种结合注意力机制的单层LSTM模型，以达到轻量化的效果。本文也设计了一种基于transformer的轻量化零样本视频语义描述模型，该模型能达到与现有方法相当的零样本视频语义描述效果。最后，本文围绕融合语音以及优化生成文本特征的视频语义描述模型展开了研究，本文首先提出了一种融合语音与视觉特征的视频语义描述方法，该模型设计了融合上下文信息的新闻视频语义特征感知模块和语音引导模块，分别用来融合视觉信息和上下文的音频信息以及使用语音来引导描述语句的生成。之后，本文提出了一种优化生成文本语义信息提取的视频语义描述模型。该模型设计了文本注意力模块与帧注意力模块，采用注意力机制充分提取和融合生成单词的文本特征以及视频帧的视觉特征。综上所述，本文提出的视频语义描述数据集包含了比现有数据集更广泛的模态信息，为将来的多模态视频语义描述任务提供了技术支撑。本文在公开数据集以及本文构建的数据集上验证了所提出视频语义描述方法有效性。
Video captioning refers to understanding video content and using natural language to describe it, which has great application potential in human-computer interaction, autonomous driving and intelligent security. In recent years, with the breakthroughs in deep learning within the field of computer vision and natural language processing, many researches have begun to focus on cross-modal tasks, such as image captioning and video captioning. Due to the temporal sequence of video as well as the diversity and complexity of video content, video captioning presents greater challenges than image captioning. Video content involves multi-modal features, such as visual and audio features, and the captioning task requires generating text that aligns with the video content. However, the caption annotations of existing mainstream datasets usually consider only visual features, often neglecting other modal features in video, such as audio and textual features. Therefore, this paper focuses on the topic of how to integrate multi-modal features in video captioning, and carries out the following research.Firstly, this paper proposes a multi-modal video captioning dataset based on news videos. The dataset contains 1,200 full videos from YouTube and 11,000 cropped news video clips. The dataset crops each 2-3 minute news video into approximately 10-20 second clips based on event timestamps. Captions of video are annotated from three perspectives of coarse-grained, medium-grained and fine-grained to provide different granularity understanding of the video content. This paper also statistically analyzes the content of the dataset and demonstrates its potential applications.Secondly, this paper focuses on the lightweight design of video captioning models. This paper designs a single-layer LSTM model combined with the attention mechanism to achieve the lightweight effect. This paper also designs a lightweight zero-shot video captioning model based on transformer, which can achieve comparable performance to the current methods.Finally, this paper focuses on the fusion of audio feature and the optimization on generated text feature extraction in video captioning model. This paper first proposes a video captioning model that integrates audio and visual features. In this model, we design the context-aware module which integrates visual information with contextual audio information and the audio-guided module that uses audio to guide the generation of captions. Then, this paper proposes a video captioning model that optimizes on the semantic extraction of generated text features. In this model, we design the text attention module and the frame attention module. The attention mechanism is used in these two modules to fully extract and integrate the textual features of generated words and the visual features of video frames.In this paper, the proposed video captioning dataset contains more extensive modal information than existing datasets, which provides more possibilities for future multi-modal video captioning tasks. The effectiveness of the proposed video captioning methods is validated on the public and proposed datasets.
