Health-aware hierarchical control for smart manufacturing using reinforcement learning

摘要

Manufacturing facilities are laid out in a natural hierarchy of assembly lines, work cells, machines, and components. Currently, prognostics and health management (PHM) information is confined to the lowest levels of this hierarchy and finds primary use in decisions and control policies for machine maintenance and replacement. For the smart manufacturing systems of the future, however, PHM information should be passed to all levels of the hierarchy and incorporated into high level decision making about production quantities, rates, and locations. This paper proposes a hierarchical control methodology that passes PHM health estimates up the hierarchy and optimization objectives down the hierarchy. Individual nodes in the hierarchy are modeled as Markov decision processes (MDPs) and reinforcement learning is used to estimate optimal policies. This work makes several contributions to the PHM community. First, we propose a novel model of a control system that makes uses of health information throughout the manufacturing hierarchy. Second, we define a reinforcement learning based approach to solving the MDPs for optimal or near-optimal policies. Third, we illustrate the method on a numerical example based on a simulation of a real-world manufacturing environment.