This application was built using spring boot and selenium webdriver for the web crawling.
The aim of the application was to crawl over a particular set of pages, extract some key bits of information and then return that information back to the client in JSON form.
The rationale was to separate out the concerns and give each entity a single responsibilty principle, hence I have one service that's focussed on extracting product information and another service for building the json.
To run the application you will need to have maven installed locally and Java 7.
You will also need the download chrome driver from this url https://sites.google.com/a/chromium.org/chromedriver/ and update the location specified in the string DRIVER_LOCATION in PageConsumerService, I tried to run it with firefox but was encountering javascript errors.
Go into the root directory and run mvn:clean install - this will download all of your dependencies.
Once the application has been built you can run it with the following command: