This dissertation presents a design-driven thesis, aimed specifically at: (1) the adaptive control of the human musculo-skeletal system using reinforcement learning; and (2) a detailed study of reinforcement learning as a useful paradigm for use real-world complex control problems (by studying inverted pendulum systems as sub-problems of the human balance problem).; Established software engineering principles are used specifically to develop a Q learning based reinforcement controller, as well as various problem plants to control. These plants include double link pendulum systems, triple link pendulum systems, and a detailed bipedal human musculo-skeletal system. In order to address structural limitations of reinforcement learning, viz. the “curse of dimensionality” in which representational space requirements grow geometrically, a distributed architecture is proposed and used in the triple link and human model systems.; A single Q learning controller was able to learn to keep an inverted double link pendulum balanced for at least one hour within a day of computer time. The controller would apply a single point torque (from a small set of choices) to each joint. Based on subsequent experiments, a set of optimal parameter values for the pendulum control system was determined. These values were used to specify a single controller for a triple link inverted pendulum. The geometrically increased Q space size, however, predicted that such a controller would require at least a year of computer time to balance. Therefore, a novel architecture was proposed in which each link of the pendulum was controlled by its own Q learning controller. These reduced controllers received sensory information from a tuned subset of the original sensory space, as well as some globally shared sensory data. The distributed systems were able to learn to keep the inverted triple link system balanced for about half an hour after learning for about five days of computer time.; Given these results, this dissertation successfully demonstrates a design for human balance control using Q learning. Structural limitations of Q learning are addressed in the pendulum systems that are built as test-beds for the final human system. Although the final human controller was not able to remain upright for as long as the pendulum systems, rates of performance improvement, particularly when seen as function of underlying problem space size, improved in the human controller. (Abstract shortened by UMI.)
展开▼
