The ability to learn low-dimensional representations of high-dimensional data is foundational to general cognitive functions such as comparison, abstraction, prediction, and planning. Additionally, systems that can learn in an unsupervised continuous fashion should be able to automatically adapt to changes in the target distribution and reach levels of accuracy beyond what can be achieved using static datasets. To achieve these capabilities, we propose two approaches: Attention-Dependent Autoencoders (ADA), and Autoencoder Leader-follower Clustering (ALC).
ADA produces vector embeddings of images, broadly analogous to the declarative memory engrams formed by the entorhinal-cortex hippocampal system. In order for these embedded representations to be useful within a cognitive architecture, they must be compact, their relative vector distances should reflect semantic distance, and the encoding process should allow modulation via attentional mechanisms. It must also be possible to decode the embeddings back to the input space, and critically, the reconstructions must preserve the semantics of the original data with respect to the cognitive system as a whole. To address these goals, we use “conservational loss” to train an autoencoder that generates reconstructions which conserve the activations of a single-class semantic segmenter, which we treat as a visual attention model. The resulting autoencoder preserves class-specific regions of images and can be modulated using the segmentation masks as attention vectors. The semantic embeddings produced by the encoder are shown to be amenable to distance metrics, and the reconstructions of the decoder shown to preserve the target-class better than a generic autoencoder, even appearing competitive with JPEG at lower bit-rates. We also suggest the use of autoencoder conservational loss as a post-2 deployment error metric for the attention-model and discuss the broader implications of conservational loss in general.
ALC addresses the catastrophic interference problem in continuous learning by using a gateless mixture-of-experts generated through autoencoder cloning, such that clones model different portions of the sample distribution in a shifting landscape built through a leader-follower clustering algorithm, with reconstruction error serving as the distance metric. To address scalability issues, ALC employs shared pseudo-rehearsal so the autoencoders of the ensemble can gradually consolidate their learned patterns into fewer autoencoders.
|Commitee:||Esterline, Albert, Xu, Jinsheng, Stephens, Joseph D, Severa, William M|
|School:||North Carolina Agricultural and Technical State University|
|School Location:||United States -- North Carolina|
|Source:||DAI-B 81/3(E), Dissertation Abstracts International|
|Subjects:||Computer science, Cognitive psychology, Artificial intelligence|
|Keywords:||Attention, Autoencoder, Continuous learning, Deep learning, Embeddings, Representation learning|
Copyright in each Dissertation and Thesis is retained by the author. All Rights Reserved
The supplemental file or files you are about to download were provided to ProQuest by the author as part of a
dissertation or thesis. The supplemental files are provided "AS IS" without warranty. ProQuest is not responsible for the
content, format or impact on the supplemental file(s) on our system. in some cases, the file type may be unknown or
may be a .exe file. We recommend caution as you open such files.
Copyright of the original materials contained in the supplemental file is retained by the author and your access to the
supplemental files is subject to the ProQuest Terms and Conditions of use.
Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be