Media

Episode 4 covers neural language modeling and sequence-to-sequence learning, moving from Bengio et al.'s 2003 paper "A Neural Probabilistic Language Model" to Sutskever et al.'s 2014 paper "Sequence to Sequence Learning with Neural Networks." Episodes 1-3 covered representation and memory: PDP showed how concepts can be learned as distributed representations, while LSTMs showed how these representations can persist through time. Bengio et al.'s work applies these ideas to language through neural next-token prediction, helping establish a central paradigm in modern AI. Sutskever et al.'s seq2seq model extends this paradigm to more complex sequence tasks such as translation and summarization.

Episode 3 covers the 1997 paper "Long Short-Term Memory" by Sepp Hochreiter and Jurgen Schmidhuber. The episode follows PDP because it addresses a limitation of distributed representations alone: learned representations are more useful when they can be preserved and related over time. LSTMs introduce a mechanism for maintaining and updating information across sequences.

Episode 2 covers the PDP research group's 1986 work "Parallel Distributed Processing: Explorations in the Microstructure of Cognition." Following Hopfield, PDP represents a major chronological and conceptual development in AI architecture. It introduces distributed representations, which became an important conceptual foundation for later developments in deep learning.

Episode 1 covers John Hopfield's 1982 paper "Neural Networks and Physical Systems with Emergent Collective Computational Abilities." The series begins with Hopfield, rather than earlier neural models such as the perceptron, because its focus is AI architecture rather than learning algorithms. Hopfield presents a rigorous account of neural networks as dynamical systems, framing neural computation as an emergent property of a network's structure and dynamics.