登录 EN

添加临时用户

多模态情感分析中的鲁棒性评估研究

Robustness Evaluation in Multimodal Sentiment Analysis

作者:毛惠生
  • 学号
    2020******
  • 学位
    硕士
  • 电子邮箱
    mhs******.cn
  • 答辩日期
    2023.05.22
  • 导师
    徐华
  • 学科名
    计算机科学与技术
  • 页码
    51
  • 保密级别
    公开
  • 培养单位
    024 计算机系
  • 中文关键词
    多模态情感分析,鲁棒性,评估方法,特征表示,噪声
  • 英文关键词
    Multimodal Sentiment Analysis, Robustness, Evaluation Methodology, Feature Representation, Noise

摘要

多模态情感分析任务旨在通过综合利用视频中的人脸表情信息、音频中的语音语调信息以及文本语义信息这三种模态的信息来判断说话人的情感极性。在现实应用场景中,原始数据一般都包含噪声。如何在有噪声干扰的条件下准确地进行情感分析,是该领域近期的一个关注热点。现有工作虽然在对抗噪声干扰的鲁棒性上取得了一定提升,但是各项研究工作之间由于任务定义和评价方法的不同,无法进行统一的鲁棒性对比。此外,现有研究多数聚焦于表示学习和特征融合等模型层面的鲁棒性提升,而忽略了多模态特征对于鲁棒性的影响。针对上述问题,本文从模型和特征两个层面对多模态情感分析任务中的鲁棒性进行了评估。为此,本文分三个阶段进行了递进式的研究。首先,在没有噪声干扰的情况下,对不同特征在多模态情感分析任务中的有效性进行了评估,为后续进行特征层面的鲁棒性评估奠定了基础。其次,根据多模态情感分析中常见噪声的分类,提出了一套标准化的鲁棒性评估方法,包含两个评价指标和一个评测数据集,能够针对全部噪声种类开展鲁棒性评估实验。利用该评估方法,对多种针对无噪声条件下多模态情感分析任务设计的模型进行了鲁棒性评估,并以此来验证该评估方法的可行性。最后,在前两个阶段的研究方法基础和数据支撑之上,面向近些年有噪声条件下的多模态情感分析任务的相关工作,从模型和特征两个层面进行了鲁棒性评估。结合三个阶段的研究,实现了对多模态情感分析领域中鲁棒性的系统化评估研究,为后续研究工作提供了可靠的基线实验结果和鲁棒性评估方法参照。在三个阶段的研究过程中,本文分别构建了多模态情感分析平台2.0版本、多模态情感分析鲁棒性演示平台和多模态情感分析鲁棒性评估系统,用以辅助开展研究和实验。其中,多模态情感分析平台2.0版本支持自定义多模态特征提取功能,能够使用自定义特征对多种模型进行训练调参和测试。多模态情感分析鲁棒性演示系统既可以作为辅助研究工具,可视化地分析噪声对多模态情感分析任务的影响,又可以作为演示平台,为该研究领域吸引更多的关注和热度。多模态情感分析鲁棒性评估系统能够便捷地对多模态情感分析模型进行标准化的鲁棒性评估,并生成可视化的分析对比结果。三个系统均采用模块化设计,能够方便地对新模型进行集成,具有良好的可扩展性。

Multimodal sentiment analysis aims to determine the speaker‘s sentiment by integrating information of three modalities: facial expressions from video modality, prosodic information from audio modality, and semantic information from text modality. In real-world scenarios, the raw data typically contains noise. Therefore, accurately performing sentiment analysis under noisy conditions is a recent research focus in this field. Although existing works have made certain improvements in robustness against noise interference, a unified robustness performance comparison cannot be conducted among different research works due to the differences in task definition and evaluation methods. Furthermore, most existing studies have focused on improving the robustness of the model at the level of representation learning and feature fusion methods, while ignoring the influence of multimodal features on the model‘s robustness performance.To address the aforementioned issues, this thesis evaluates the robustness of multimodal sentiment analysis at both the model and feature levels. For this purpose, the research is carried out in three stages. Firstly, under noise-free conditions, the effectiveness of different features in multimodal sentiment analysis is evaluated, laying the foundation for subsequent feature-level robustness evaluation. Secondly, considering common noise types in multimodal sentiment analysis, a standardized robustness evaluation method is proposed, which includes two evaluation metrics and a benchmark dataset, enabling robustness evaluation across all noise categories. Using this evaluation method, robustness evaluations are performed on various multimodal sentiment analysis models proposed under noise-free conditions, thereby validating the feasibility of the evaluation method. Finally, building upon the methods and data support from the first two stages, robustness evaluations are conducted on multimodal sentiment analysis models proposed in recent years under noisy conditions, considering both the model and feature levels. By combining the research from the three stages, a systematic evaluation study on the robustness of multimodal sentiment analysis is achieved, providing reliable baseline experimental results and robustness evaluation methods for future research. Throughout the three stages of research, this thesis constructs a multimodal sentiment analysis platform version 2.0, a robustness demonstration platform for multimodal sentiment analysis, and a robustness evaluation system for multimodal sentiment analysis to facilitate research and experimentation. The multimodal sentiment analysis platform version 2.0 supports custom multimodal feature extraction, enabling the use of custom features to train, tune, and test various models. The robustness demonstration system for multimodal sentiment analysis serves as both an auxiliary research tool, providing visual analysis of the impact of noise on multimodal sentiment analysis tasks, and a demonstration platform to attract more attention and interest in this research field. The robustness evaluation system for multimodal sentiment analysis enables convenient standardized robustness evaluation of multimodal sentiment analysis models and generates visual comparative analysis results. All three systems are modularly designed for easy integration of new models and possess good scalability.