Developing a data harvester in the Amazon cloud for the automated assimilation of Florida's healthy beaches reports into the GCOOS data portal

摘要

The Florida Department of Health funds beach water sampling for 34 of Florida's coastal counties. Mote Marine Laboratory, in conjunction with the Gulf of Mexico Coastal Ocean Observing System (GCOOS) developed an automated data harvester that used Web scraping technology to capture the reported data from each monitored county and store the data in a MySQL database. The database was queried nightly and the results were used to build XML files that were ingested by GCOOS and published through the GCOOS Data Portal. In early 2013 the Florida Department of Health outsourced their Web site to a commercial service provider. The new Healthy Beaches Web site used a mash-up of Google Maps and JavaScript code and no longer returned a standard HTML document. The conversion from traditional Web page to dynamic mash-up rendered our data harvester inoperable. Our entire data pipeline had to be completely rebuilt.