Scalable network analytics for characterization of outbreak influence in voluminous epidemiology datasets

摘要

Planning for large-scale epidemiological outbreaks in livestock populations often involves executingcompute-intensive disease spread simulations. To capture the probabilities of variousoutcomes, these simulations are executed several times over a collection of representative inputscenarios, producing voluminous data. The resulting datasets contain valuable insights, includingsequences of events that lead to extreme outbreaks. However, discovering and leveraging suchinformation is also computationally expensive. In this study, we set out to achieve two goals, ie,(1) providing a distributed framework for modeling disease transmission at scale using Spark,including improvements to the default GraphX partitioning strategy, and (2) giving planners andepidemiologists a means to analyze interactions between entities (herds) during simulated diseaseoutbreaks. Using our disease transmission network (DTN), planners or analysts can isolateherds that have a disproportionate effect on epidemiological outcomes, enabling effective allocationoflimited resources such as vaccinationsandfield personnel.We use a representativedatasetto verify our approach and optimized the underlying graph partitioning algorithm to ensure thesystem will scalewith increases in the dataset size or number of participatingmachines. Our analysisincludes identification of influential herds as well as the creation ofmachine learning modelsfor accurate classifications that generalize to other datasets.