Kvasir: Scalable Provision of Semantically Relevant Web Content on Big Data Framework

摘要

The Internet is overloading its users with excessive information flows, so that effective content-based filtering becomes crucial in improving user experience and work efficiency. Latent semantic analysis has long been demonstrated as a promising information retrieval technique to search for relevant articles from large text corpora. We build Kvasir, a semantic recommendation system, on top of latent semantic analysis and other state-of-the-art technologies to seamlessly integrate an automated and proactive content provision service into web browsing. We utilize the processing power of Apache Spark to scale up Kvasir into a practical Internet service. In addition, we improve the classic randomized partition tree to support efficient indexing and searching of millions of documents. Herein we present the architectural design of Kvasir, the core algorithms, along with our solutions to the technical challenges in the actual system implementation.