Screen scraping can be effective at getting free data very quickly. When attempting to screen scrape large amounts of data, I often use Google Chrome’s “Developer Tools” to obtain the steps necessary to recreate a web request. Here is an example process I used to screen scrape data from pricescope.com which contains a database about diamonds for sale online. I will use this data in an upcoming post on how to build statistical models.
Learning to screen scrape data using Google Chrome and curl
