Guest Lecture: "Large Language Models, Model Collapse, and the Conservation of Information" with George Montañez, Ph.D.

Location: Cashion 311

"Large Language Models, Model Collapse, and the Conservation of Information".

Abstract: Do Large Language Models (LLMs) think and reason? Are they perpetual information machines, producing endless coherent and correct text from finite training data? We explore how LLMs work and whether they produce rational thought and endless information. We show how theoretical considerations and experimental results from philosophy, statistics, information theory, and machine learning argue against the thesis that LLMs are rational, information-generating entities.

George D. Montañez is an associate professor in the Department of Computer Science at Harvey Mudd College, in Claremont, California. He is a Visiting Fellow at the University of Cambridge (Clare Hall) during the 2025-2026 academic year. Montañez is a former data scientist with Microsoft AI+Research. He obtained a PhD in machine learning from Carnegie Mellon University, with additional degrees in computer science (BS, University of California Riverside; MS, Baylor University) and machine learning (MS, Carnegie Mellon University). His work sits at the intersection of machine learning and information theory. He is a former NSF Graduate Research Fellow and Ford Foundation Predoctoral Fellow. 

His work has been presented at conferences such as AAAI, IEEE CEC, and ICAART, including a best paper award at CIKM 2014, best student paper award at IEEE SMC 2017, best student paper award at IJCNN 2017, and best paper award at ICAART 2020. He has interned at Microsoft Research, Yahoo!, and Bing, and given invited talks at Meta, Microsoft, General Electric and research institutions across the US. He has written on topics including applied machine learning, biological complexity, and the ethics of AI.