“Language models are trained to predict the next word,” said Yilun Du, a researcher at MIT who was previously a research fellow at OpenAI, and one of the paper’s authors. “They are not trained to tell people they don’t know what they’re doing.” The result is bots that act like precocious people-pleasers, making up answers instead of admitting they simply don’t know.
That’s why I pity the students who rely on ChatGPT to write their homework assignments …