CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning

by   Hongyu Hu, et al.
Shanghai Jiao Tong University
ByteDance Inc.
FUDAN University

Nowadays, the research on Large Vision-Language Models (LVLMs) has been significantly promoted thanks to the success of Large Language Models (LLM). Nevertheless, these Vision-Language Models (VLMs) are suffering from the drawback of hallucination – due to insufficient understanding of vision and language modalities, VLMs may generate incorrect perception information when doing downstream applications, for example, captioning a non-existent entity. To address the hallucination phenomenon, on the one hand, we introduce a Contrastive Instruction Evaluation Method (CIEM), which is an automatic pipeline that leverages an annotated image-text dataset coupled with an LLM to generate factual/contrastive question-answer pairs for the evaluation of the hallucination of VLMs. On the other hand, based on CIEM, we further propose a new instruction tuning method called CIT (the abbreviation of Contrastive Instruction Tuning) to alleviate the hallucination of VLMs by automatically producing high-quality factual/contrastive question-answer pairs and corresponding justifications for model tuning. Through extensive experiments on CIEM and CIT, we pinpoint the hallucination issues commonly present in existing VLMs, the disability of the current instruction-tuning dataset to handle the hallucination phenomenon and the superiority of CIT-tuned VLMs over both CIEM and public datasets.


page 3

page 5

page 6


Instruction Tuning for Large Language Models: A Survey

This paper surveys research works in the quickly advancing field of inst...

CiT: Curation in Training for Effective Vision-Language Data

Large vision-language models are generally applicable to many downstream...

MIMIC-IT: Multi-Modal In-Context Instruction Tuning

High-quality instructions and responses are essential for the zero-shot ...

Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets?

Instruction-tuning has become an integral part of training pipelines for...

ClusterLLM: Large Language Models as a Guide for Text Clustering

We introduce ClusterLLM, a novel text clustering framework that leverage...

On the Limits of Minimal Pairs in Contrastive Evaluation

Minimal sentence pairs are frequently used to analyze the behavior of la...

Enhancing PLM Performance on Labour Market Tasks via Instruction-based Finetuning and Prompt-tuning with Rules

The increased digitization of the labour market has given researchers, e...

Please sign up or login with your details

Forgot password? Click here to reset