In this thesis we tried to analyse different methodologies of access to unstructured data on websites. Our main focus was on different techniques of gathering information from presentation layer (HTML parsing) with the use of specific tools that we can find in the open source community as well as downsides of commercial data scrapers and scraping services.udBecause of experience in PHP programming language and a plethora of tools, libraries and products implemented in it, we focused on techniques of web scraping with Curl library in combination with Xpath. Other techniques were also the use of ''headless'' browsers for advanced scraping of data on websites where AJAX requests are used extensively and a tool for automatization of website functionality testing Mink.udWith the rise and demand of webcrawlers many content providers try to disable access for them by tracking access of the bots. There are different uses of anonymization tools and user identification techniques being used on websites that we analyzed, as well as tackled the legislation concerning webscraping and most widely known legal cases in this industry.udLastly, we mentioned positive and negative aspects of the implemented scraper, as well as upgrading and extending the implementation in terms of request parallelization and distributed control on different servers.ud
展开▼
机译：在本文中，我们试图分析访问网站上非结构化数据的不同方法。我们的主要重点是通过使用我们可以在开源社区中找到的特定工具，以及从商业数据抓取工具和抓取服务的不利方面，使用表示工具从表示层（HTML解析）收集信息的不同技术。 ud由于有PHP的经验编程语言以及用其实现的大量工具，库和产品，我们专注于结合使用Curl库和Xpath进行Web抓取的技术。其他技术还包括使用“无头”浏览器在广泛使用AJAX请求的网站上进行高级数据抓取，以及用于自动进行网站功能测试的工具Mink。 ud随着网络抓取工具的兴起和需求，许多内容提供商试图通过跟踪机器人的访问权限来禁止他们访问。我们分析的网站上使用了匿名化工具和用户标识技术，并解决了有关网络抓取的法律和该行业中最广为人知的法律案件。 ud最后，我们提到了已实现的抓取工具的正面和负面方面，以及在不同服务器上的请求并行化和分布式控制方面升级和扩展实现。 ud
展开▼
