Auditing Preference Biases and Fine-Tuning Language Models with Direct Preference Optimization

Summary

This tutorial presents an end-to-end preference-learning workflow using the Anthropic HH-RLHF dataset and Direct Preference Optimization (DPO). It covers preparing a Colab environment, loading and parsing chosen-rejected response pairs, auditing the dataset for structural and length-based biases, and performing lexical shortcut diagnostics. The process includes tokenizer-aware length filtering and a version-robust DPO training pipeline with TRL and optional LoRA. It fine-tunes a Qwen2.5-0.5B-Instruct model, evaluates reward accuracy and training behavior, analyzes performance across HH-RLHF subsets, inspects potential length bias, generates sample responses, and saves the policy. The workflow emphasizes robustness and compatibility, including handling dependency issues in Colab.