From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models

by   Masahiro Suzuki, et al.
The University of Tokyo

Instruction tuning is essential for large language models (LLMs) to become interactive. While many instruction tuning datasets exist in English, there is a noticeable lack in other languages. Also, their effectiveness has not been well verified in non-English languages. We construct a Japanese instruction dataset by expanding and filtering existing datasets and apply the dataset to a Japanese pre-trained base model. We performed Low-Rank Adaptation (LoRA) tuning on both Japanese and English existing models using our instruction dataset. We evaluated these models from both quantitative and qualitative perspectives. As a result, the effectiveness of Japanese instruction datasets is confirmed. The results also indicate that even with relatively small LLMs, performances in downstream tasks would be improved through instruction tuning. Our instruction dataset, tuned models, and implementation are publicly available online.


page 1

page 2

page 3

page 4


Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation

Instruction tuning has shown great promise in the field of natural langu...

Maybe Only 0.5 Training Data Instruction Tuning

Instruction tuning for large language models (LLMs) has gained attention...

Instruction Tuning for Large Language Models: A Survey

This paper surveys research works in the quickly advancing field of inst...

llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology

This study constructed a Japanese chat dataset for tuning large language...

Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets?

Instruction-tuning has become an integral part of training pipelines for...

Enhancing PLM Performance on Labour Market Tasks via Instruction-based Finetuning and Prompt-tuning with Rules

The increased digitization of the labour market has given researchers, e...

Time Travel in LLMs: Tracing Data Contamination in Large Language Models

Data contamination, i.e., the presence of test data from downstream task...

Please sign up or login with your details

Forgot password? Click here to reset