The anatomy of a search and mining system for digital humanities : Search And Mining Tools for Language Archives (SAMTLA)

摘要

Humanities researchers are faced with an overwhelming volume of digitisedudprimary source material, and "born digital" information, of relevance to theirudresearch as a result of large-scale digitisation projects. The current digital toolsuddo not provide consistent support for analysing the content of digital archivesudthat are potentially large in scale, multilingual, and come in a range of dataudformats. The current language-dependent, or project specific, approach to tooluddevelopment often puts the tools out of reach for many research disciplines inudthe humanities. In addition, the tools can be incompatible with the wayudresearchers locate and compare the relevant sources. For instance, researchersudare interested in shared structural text patterns, known as parallel passages"udthat describe a specific cultural, social, or historical context relevant to theirudresearch topic. Identifying these shared structural text patterns is challenginguddue to their repeated yet highly variable nature, as a result of differences inudthe domain, author, language, time period, and orthography.udThe contribution of the thesis is a novel infrastructure that directly addressesudthe need for generic, udflexible, extendable, and sustainable digital toolsudthat are applicable to a wide range of digital archives and research in theudhumanities. The infrastructure adopts a character-level n-gram StatisticaludLanguage Model (SLM), stored in a space-optimised k-truncated suffix treeuddata structure as its underlying data model. A character-level n-gram modeludis a relatively new approach that is competitive with word-level n-gram models,udbut has the added advantage that it is domain and language-independent,udrequiring little or no preprocessing of the document text unlike word-leveludmodels that require some form of language-dependent tokenisation and stemming.udCharacter-level n-grams capture word internal features that are ignoredudby word-level n-gram models, which provides greater udexibility in addressingudthe information need of the user through tolerant search, and compensationudfor erroneous query specification or spelling errors in the document text. Furthermore,udthe SLM provides a unified approach to information retrieval andudtext mining, where traditional approaches have tended to adopt separate dataudmodels that are often ad-hoc or based on heuristic assumptions. In addition,udthe performance of the character-level n-gram SLM was formally evaluatedudthrough crowdsourcing, which demonstrates that the retrieval performance ofudthe SLM is close to that of the human level performance.udThe proposed infrastructure, supports the development of the Samtla (SearchudAnd Mining Tools for Language Archives), which provides humanities researchersuddigital tools for search, browsing, and text mining of digital archivesudin any domain or language, within a single system. Samtla supersedes many ofudthe existing tools for humanities researchers, by supporting the same or similarudfunctionality of the systems, but with a domain-independent and languageindependentudapproach. The functionality includes a browsing tool constructedudfrom the metadata and named entities extracted from the document text, audhybrid-recommendation system for recommending related queries and documents.udHowever, some tools are novel tools and developed in response toudthe specific needs of the researchers, such as the document comparison tooludfor visualising shared sequences between groups of related documents. Furthermore,udSamtla is the first practical example of a system with a SLM asudits primary data model that supports the real research needs of several caseudstudies covering different areas of research in the humanities.