Pitfalls in Evaluating Generative AI — Hung-yi Lee, Lecture 4

Summary

This public lesson extends AI literacy to evaluation: metrics, human preferences, model judges and benchmark contamination.

Examples show how exact matching, attractive presentation and averages can distort judgments of capability. Evaluation begins with deciding what the task actually requires.

The slides recommend validating model judgments on a small scale before expanding use. They offer material for teaching how to scrutinize AI evidence, rather than evidence that AI improves student learning.

Uploaded on 20 October; the class was on 17 October. Slides were read, without a full spoken transcript.