Optimization Pathways for Transformer-Based Multimodal Pre-training Models in Open-Vocabulary Tasks

Abstract

The open-vocabulary tasks require models to identify the categories that do not exist in the training set, transcending the traditional
closed-set assumption constraints, serving as an important research topic in the multimodality sector. The Transformer architecture possesses
excellent feature modeling capabilities and serves as the main support for multimodal pre-training models. However, it still faces open-vocabulary issues such as insufficient modal alignment and deficient generalization capabilities. This study summarizes the main optimization directions of Transformer-based multimodal pre-training models in open-vocabulary tasks, systematically analyzes the fundamental frameworks
and application logic of various optimization pathways from three aspects: pre-training strategies, modal fusion methodologies, and finetuning methodologies, and generalizes current research dilemmas and development trends, providing reference for relevant research.