Cleaning safety records using text mining algorithms.

摘要

This Project aims to de-identify the safety records using Natural Language Processing. By removing identifying information, near miss records can be shared across industry participants. The strategy for removing identifying information was to remove proper nouns. Proper nouns were identified by part of speech tagging using NLTK and a list of proper nouns developed by the project. This project achieved high accuracy (98%) and reasonable precision (45%) on records in upper and lower case. The performance on records written in all upper case was significantly worse. The majority of errors were due to capitalization, spelling, uncommon words, maritime specific words, and titles. This document presents twelve approaches to improve algorithm performance.