The week's pick
Random Articles
Reseach Article
Multi-modal LLMs for NLP: Integrating Text, Image and Video
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 187 - Number 34 |
| Year of Publication: 2025 |
| Authors: Sreepal Reddy Bolla |
10.5120/ijca2025925480
|
Sreepal Reddy Bolla . Multi-modal LLMs for NLP: Integrating Text, Image and Video. International Journal of Computer Applications. 187, 34 ( Aug 2025), 66-71. DOI=10.5120/ijca2025925480
Abstract
The present study looks at how integrating text, image, and video data through multi-modal learning could improve the abilities of Large Language Models (LLMs). The LLMs we have now been very good at processing natural words, but they could be even better if they could handle more than one type of input. A new framework that blends text-based LLMs, like GPT-4, with image and video models that use transformers and convolutional neural networks (CNNs) is what we're proposing. This method is used for jobs like visual question answering (VQA) and automated content generation, showing big gains in accuracy and understanding of the context. When compared to text-only models, our multi-modal model did 25% better on VQA standards. The system also improved the ability to create material by giving outputs that were richer and more context-aware. The results show that multi-modal learning can help LLMs make progress by helping them understand and react to different types of input better.
References
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
- Baltrušaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423-443.
- Liu, X., Xu, Y., & Wang, X. (2021). Multi-modal Machine Learning: A Comprehensive Survey. ACM Computing Surveys.
- Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint arXiv:2103.00020.
- Tsai, Y.-H. H., Bai, S., Liang, P. P., et al. (2019). Multimodal Transformer for Unaligned Multimodal Language Sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).
- Chen, Y.-C., Li, L., Yu, L., et al. (2020). UNITER: UNiversal Image-TExt Representation Learning. European Conference on Computer Vision (ECCV).
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations (ICLR).
- Ramesh, A., Pavlov, M., Goh, G., et al. (2021). Zero-Shot Text-to-Image Generation. arXiv preprint arXiv:2102.12092.
- Zellers, R., Lu, J., Bisk, Y., et al. (2021). MERLOT: Multimodal Neural Script Knowledge Models. arXiv preprint arXiv:2106.02636.
- Zhang, H., Wang, Y., He, X., et al. (2021). Multi-modal Conversational AI: Combining Vision, Speech, and Language Understanding. arXiv preprint arXiv:2103.02520.
- Esteva, A., Robicquet, A., Ramsundar, B., et al. (2021). A Guide to Deep Learning in Healthcare. Nature Medicine, 25(1), 24-29.
- Brown, T., Mann, B., Ryder, N., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS).
- Ngiam, J., Khosla, A., Kim, M., et al. (2011). Multimodal Deep Learning. Proceedings of the 28th International Conference on Machine Learning (ICML).
- Srivastava, N., & Salakhutdinov, R. (2012). Multimodal Learning with Deep Boltzmann Machines. Advances in Neural Information Processing Systems (NeurIPS).
- Karpathy, A., Joulin, A., & Fei-Fei, L. (2015). Deep Visual-Semantic Alignments for Generating Image Descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4), 664-676.
- Zhou, B., Lapedriza, A., Xiao, J., et al. (2014). Learning Deep Features for Scene Recognition Using Places Database. Advances in Neural Information Processing Systems (NeurIPS).
- Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS).
- Tan, H., & Bansal, M. (2019). LXMERT: Learning Cross-Modality Encoder Representations from Transformers. arXiv preprint arXiv:1908.07490.
- Lu, J., Batra, D., Parikh, D., et al. (2019). ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. Advances in Neural Information Processing Systems (NeurIPS).
- Sun, C., Myers, A., Vondrick, C., et al. (2019). VideoBERT: A Joint Model for Video and Language Representation Learning. arXiv preprint arXiv:1904.01766.
- Saharia, C., Ramesh, A., Ho, J., et al. (2022). Photorealistic Text-to-Image Diffusion Models with Guided Attention. arXiv preprint arXiv:2203.12432.
- Li, L., Banerjee, S., Chen, Y.-C., et al. (2021). MERLOT: Multimodal Neural Script Knowledge Models. arXiv preprint arXiv:2106.02636.
- Wang, X., Han, Y., Li, L., et al. (2021). OSCAR: Object-Semantics Aligned Pre-training for Vision-Language Tasks. European Conference on Computer Vision (ECCV).
- Han, W., Cho, K., & Bansal, M. (2021). Speech2Vec: Integration of Speech into Text and Visual Representations. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
- Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*).
- Zhang, B., Wu, Z., & Zhu, S. (2021). Fair Multi-modal Learning: Reducing Bias in Cross-Modal Systems. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Index Terms
Keywords