A human’s remarkable ability to manipulate unfamiliar objects with little prior knowledge of them is a constant inspiration for robotics research. Despite the interest of the research community, and despite its practical value, robust manipulation of novel objects in cluttered environments still remains a largely unsolved problem. Classic solutions (e.g. involving 6D object pose estimation) typically require prior knowledge of the objects (e.g. class categories or 3D CAD models), which may not be available outside of highly constrained settings. More recent deep learning methods using end-to-end convolutional networks (e.g. raw pixels to motor torques) have the potential to model complex skills that generalize, but they remain highly data inefficient—and robot data (e.g. trial and error) is expensive.
In this thesis, we consider an approach to learning manipulation called visual affordances. The idea is to use classic controllers to design motion primitives, then use convolutional networks to map from visual observations (e.g. images) to the perceived affordances (e.g. confidence scores or action-values) of the primitives for every pixel of the input. By leveraging dense equivariant state and action representations, this formulation can be used to acquire complex vision-based manipulation skills (e.g. pushing, grasping, throwing) on real robot platforms that generalize to novel objects, while using orders of magnitude less data. While visual affordances may not be directly compatible with classic planning frameworks that involve explicit forward simulation or propagation, in this thesis we show that it is possible to workaround this limitation by extending it with model-free reinforcement learning to sequence primitive picking motions for more complex manipulation policies. We also study how it can be combined with residual physics (learning to predict residual values on top of control parameter estimates from an initial analytical controller) to enable learning end-to-end visuomotor policies that leverage the benefits of analytical models while still maintaining the capacity (via data-driven residuals) to account for real-world dynamics that are not explicitly modeled. Finally, we conclude by discussing the limitations of learning visual affordances, which suggest directions for future work.
|Commitee:||Rodriguez, Alberto, Rusinkiewicz, Szymon, Russakovsky, Olga, Finkelstein, Adam|
|School Location:||United States -- New Jersey|
|Source:||DAI-B 81/5(E), Dissertation Abstracts International|
|Subjects:||Robotics, Artificial intelligence, Computer science|
|Keywords:||Artificial intelligence, Computer vision, Deep learning, Machine learning, Robotics, Visual affordances|
Copyright in each Dissertation and Thesis is retained by the author. All Rights Reserved
The supplemental file or files you are about to download were provided to ProQuest by the author as part of a
dissertation or thesis. The supplemental files are provided "AS IS" without warranty. ProQuest is not responsible for the
content, format or impact on the supplemental file(s) on our system. in some cases, the file type may be unknown or
may be a .exe file. We recommend caution as you open such files.
Copyright of the original materials contained in the supplemental file is retained by the author and your access to the
supplemental files is subject to the ProQuest Terms and Conditions of use.
Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be