Research on reward-driven learning has produced and substantiated theories of model-free and model-based reinforcement learning (RL), which respectively explain how humans and animals learn reflexive habits and build prospective plans. A highly developed line of work has unearthed the role of striatal dopamine in model-free learning, while the prefrontal cortex (PFC) appears to critically subserve model-based learning. The recent theory of meta-reinforcement learning (meta-RL) explained a wide array of findings by positing that the model-free dopaminergic reward prediction error trains the recurrent prefrontal network to execute arbitrary RL algorithms—including model-based RL—in its activations.
In parallel, a nascent understanding of a third reinforcement learning system is emerging: a non-parametric system that stores memory traces of individual experiences rather than aggregate statistics. Research on such episodic learning has revealed its unmistakeable traces in human behavior, developed theory to articulate algorithms underlying that behavior, and pursued the contention that the hippocampus is centrally involved. These developments lead to a set of open questions about (1) how the neural mechanisms of episodic learning relate to those underlying incremental model-free and model-based learning and (2) how the brain arbitrates among the contributions of this abundance of valuation strategies.
This thesis extends meta-RL to provide an account for episodic learning, incremental learning, and the coordination between them. In this theory of episodic meta-RL (EMRL), episodic memory reinstates activations in the prefrontal network based on contextual similarity, after passing them through a learned gating mechanism (Chapters 1 and 2). In simulation, EMRL can solve episodic contextual water maze navigation problems and episodic contextual bandit problems, including those with Omniglot class contexts and others with compositional structure (Chapter 3). Further, EMRL reproduces episodic model-based RL and its coordination with incremental model-based RL on the episodic two-step task (Vikbladh et al., 2017; Chapter 4). Chapter 5 discusses more biologically detailed extensions to EMRL, and Chapter 6 analyzes EMRL with respect to a set of recent empirical findings. Chapter 7 discusses EMRL in the context of various topics in neuroscience.
|Commitee:||Cohen, Jon, Daw, Nathaniel, Norman, Ken, Witten, Ilana|
|School Location:||United States -- New Jersey|
|Source:||DAI-B 80/06(E), Dissertation Abstracts International|
|Subjects:||Neurosciences, Cognitive psychology, Computer science|
|Keywords:||Deep learning, Episodic memory, Model-based learning, Model-free learning, Reinforcement learning, Working memory|
Copyright in each Dissertation and Thesis is retained by the author. All Rights Reserved
The supplemental file or files you are about to download were provided to ProQuest by the author as part of a
dissertation or thesis. The supplemental files are provided "AS IS" without warranty. ProQuest is not responsible for the
content, format or impact on the supplemental file(s) on our system. in some cases, the file type may be unknown or
may be a .exe file. We recommend caution as you open such files.
Copyright of the original materials contained in the supplemental file is retained by the author and your access to the
supplemental files is subject to the ProQuest Terms and Conditions of use.
Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be