Prediction Requires a Model of the World

To predict the next word in "She dropped the glass and it ___", the model needs something like knowledge of gravity, glass, and floors. To continue a story it needs a sense of how people react to events. Predicting language well seems to require an implicit model of the things language is about: people, objects, physics, culture, cause and effect.

Whether this counts as a world model in any deep sense is debated. What is not debated is the effect: models trained only on text answer questions about physics, etiquette, and consequences at a level nobody predicted from the training objective alone.