System Self-Awareness and Related Methods for Improving the Use and Understanding of Data within DoD

摘要

Data sources for DoD applications often include disparate realtimesensor and archival sources with multiple dimensions, as wellas very high rates and very large volumes. The data can includestructured data that are of traditional forms (for example, storedin relational databases, Excel, or XML files with well-definedlabels with meta-data). Unstructured data can also be included,such as free text, Word, PDF, PowerPoint, and emails. A largepercentage of data are unstructured. It remains a daunting task toretain logical integrity of the separate data sources, and supportmultiple parallel and asynchronous functions of storage, analysis, search, and retrieval of these data sources. Analysts, atpresent, must manually comb through immense volumesof multisource, multiclassification-level intelligence datato find previously unknown and undiscovered patterns,associations, relationships, trends, and anomalies. Thiseffort can be tedious and slow, and can depend largelyupon the individual analyst's experience. Withoutaccurate and timely front-end analysis, link analysis andinferences about future trends are not possible. Theauthors' methodology cross-examines and considersall of the data to create a full picture, and thus gainimproved system self-awareness. Analysts and theintelligence community could benefit from automated,scalable, and robust tools and methods to analyze suchlarge data sets quickly and thoroughly, to create andsustain situational awareness in real time.