Function approximation can be used to improve the performance of reinforcement learners. Traditional techniques, including Tile Coding and Kanerva Coding, can give poor performance when applied to large-scale problems. In our preliminary work, we show that this poor performance is caused by prototype collisions and uneven prototype visit frequency distributions. We describe our adaptive Kanerva-based function approximation algorithm, based on dynamic prototype allocation and adaptation. We show that probabilistic proto-type deletion with prototype splitting can make the distribution of visit frequencies more uniform, and that dynamic prototype allocation and adaptation can reduce prototoype collsisions. This approach can significantly improve the performance of a reinforcement learner.
We then show that fuzzy Kanerva-based function approximation can reduce the similarity between the membership vectors of state-action pairs, giving even better results. We use Maximum Likelihood Estimation to adjust the variances of basis functions and tune the receptive fields of prototypes. This approach completely eliminates prototype collisions, and greatly improve the ability of a Kanerva-based reinforcement learner to solve large-scale problems.
Since the number of prototypes remains hard to select, we describe a more effective approach for adaptively selecting the number of prototypes. Our new rough sets-based Kanerva-based function approximation uses rough sets theory to explain how prototype collisions occur. Our algorithm eliminates unnecessary prototypes by replacing the original prototype set with its reduct, and reduces prototype collisions by splitting equivalence classes with two or more state-action pairs. The approach can adaptively select an effective number of prototypes and greatly improve a Kanerva-based reinforcement learners ability.
Finally, we apply function approximation techniques to scale up the ability of reinforcement learners to solve a real-world application: spectrum management in cognitive radio networks. We use multi-agent reinforcement learning approach with decentralized control can be used to select transmission parameters and enable efficient assignment of spectrum and transmit powers. However, the requirement of RL-based approaches that an estimated value be stored for every state greatly limits the size and complexity of CR networks that can be solved. We show that function approximation can reduce the memory used for large networks with little loss of performance. We conclude that our spectrum management approach based on reinforcement learning with Kanerva-based function approximation can significantly reduce interference to licensed users, while maintaining a high probability of successful transmissions in a cognitive radio ad hoc network.
|Commitee:||Aslam, Javed A., Dy, Jennifer|
|Department:||Electrical and Computer Engineering|
|School Location:||United States -- Massachusetts|
|Source:||DAI-B 72/04, Dissertation Abstracts International|
|Keywords:||Cognitive radio network, Function approximation, Fuzzy logic, Reinforcement learning, Rough set|
Copyright in each Dissertation and Thesis is retained by the author. All Rights Reserved
The supplemental file or files you are about to download were provided to ProQuest by the author as part of a
dissertation or thesis. The supplemental files are provided "AS IS" without warranty. ProQuest is not responsible for the
content, format or impact on the supplemental file(s) on our system. in some cases, the file type may be unknown or
may be a .exe file. We recommend caution as you open such files.
Copyright of the original materials contained in the supplemental file is retained by the author and your access to the
supplemental files is subject to the ProQuest Terms and Conditions of use.
Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be