Data anonymisation by replacement of sensitive information in a log

摘要

Transforming information in an accumulated log, e.g. a log of internet or messaging activity facilitated by a server, into an anonymized secure log by replace confidential information such as user names, locations, internet addresses etc. Messages in the log are classified into clusters according to similarities in the message, for example similar data formats and positions, then variable and static portions of the messages in each cluster are identified (e.g. variable portions will contain dynamic information such as user name, static portions will contain static data such as type identifiers). Variable portions are first compared to a blacklist of known sensitive confidential data, then unmatched variable portions are compared to matched ones, for example to see if they are in the same position in a message as known confidential data, to determine their confidentiality. Sensitive data is replaced or masked, ideally using data with similar attributes so that semantic content is retained in the secure log which can then be used by third parties for marketing analysis or analysis of malicious activity etc.