Some Modalities are More Equal Than Others:
Decoding and Architecting Multimodal Integration in MLLMs

Tianle Chen^1*, Chaitanya Chakka^1*, Arjun Reddy Akula², Xavier Thomas¹, Deepti Ghadiyaram¹

¹Boston University, ²Google DeepMind
Equal Contribution

We propose MMA-Bench to expose how MLLMs behave when sight, sound, and language conflict. Each example presents a controlled modality conflict (e.g., audio, video, or text) and asks two modality-specific questions. Correct answers differ across modalities, forcing the model to attend to the reliable modality.

---

📢 Updates

• [2025-12] 🚀 MMA-Bench dataset and code are released!
• [2025-12] 📄 Paper is now available on arXiv.

---

📖 Abstract

Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning.

---

🏆 Leaderboard

We evaluate models on MMA-Bench under two key settings: Aligned (standard AV) and Misaligned (Visual/Audio conflict).

Semantic Misalignment (Main Benchmark)

Accuracy (%) on Visual-focused and Audio-focused prompts when modalities conflict.

Rank	Model	Size	Visual (Align)	Visual (Misalign)	Audio (Align)	Audio (Misalign)
🥇	Qwen2.5-Omni-7B + Ours	7B	94.68	94.37	88.14	79.79
🥈	Gemini-2.5-Pro	-	97.90	95.28	60.37	24.95
🥉	Gemini-2.0-Flash	-	96.71	91.91	57.21	9.42
4	Qwen3-Omni-30B	30B	92.88	76.68	57.39	14.58
5	Qwen2.5-Omni-7B (Base)	7B	76.68	58.72	46.60	25.16
6	ChatBridge	13B	51.64	54.71	41.61	7.07
7	VideoLLaMA2	7B	56.35	36.11	36.12	18.46
8	PandaGPT	13B	28.75	29.79	13.12	1.18

---

🛠️ Installation

Environment Setup

git clone [https://github.com/cskyl/MMA-Bench.git](https://github.com/cskyl/MMA-Bench.git)
cd MMA-Bench

LLAMA-Factory Integration

Our training pipeline leverages LLAMA-Factory for efficient fine-tuning.

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation

📂 Data Preparation

We curate MMA-Bench from AudioSet using a rigorous 2-stage filtering pipeline to ensure semantic alignment.

1. Download Metadata: Get the JSON metadata containing YouTube IDs and timestamps.
2. Generate Perturbations: Create the benchmark variants, including Audio-Visual Swaps (Misaligned), Audio Removed, and Frames Zeroed according to the video clips.
3. Process Videos: During the fine-tuning step, using our script to download and preprocess samples. We strictly enforce a center crop to 504x504 and a standardized 10s duration.

# 1. Download metadata from Hugging Face
hf download dghadiya/MMA-Bench --repo-type=dataset

# Run processing script (requires ffmpeg)
python scripts/prepare_data.py 

🚀 Training & Evaluation

Modality-Aware Fine-Tuning

We use LoRA(Low-Rank Adaptation) to fine-tune to handle conflicting modalities.

# Setup training with LLAMA-Factory
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation

Key Configurations:

• learning_rate: 1e-4
• batch_size: 4
• lora_rank: 8
• fp16: true
• gradient_accumulation_steps: 4

📚 Model Zoo & Baselines

We evaluate our method against the following state-of-the-art MLLMs utilized in our study:

Model	Type	Link
Qwen2.5-Omni	Open Source	GitHub
Qwen3-Omni	Open Source	GitHub
VideoLLaMA2	Open Source	GitHub
ChatBridge	Open Source	GitHub
PandaGPT	Open Source	GitHub
Gemini-2.5-Pro	Close Source	Google DeepMind
Gemini-2.0-Flash	Close Source	Google DeepMind
Gemini-2.0-Flash-Lite	Close Source	Google DeepMind

---

🖊️ Citation

If you find this work helpful, please cite our paper:

@article{chen2025some,
  title={Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs},
  author={Chen, Tianle and Chakka, Chaitanya and Akula, Arjun Reddy and Thomas, Xavier and Ghadiyaram, Deepti},
  journal={arXiv preprint arXiv:2511.22826},
  year={2025}
}

Acknowledgements

We thank the open-source community for Qwen2.5-Omni and LLAMA-Factory for their excellent codebases.

We thank our collaborators and colleagues for their valuable feedback and support throughout this project. We also respectfully acknowledge that Arjun Reddy Akula participated in an advisory capacity only.