Many companies have a requirement to take some data from the Internet and load it into a database or spreadsheet whether this be financial data or other data.
SMR can offer a bespoke solution to this problem making data gathering a lot more efficient and automated. Meaning a computer does the grunt work rather than you freeing your time for more productive endeavours.
The data gathering process is typically:-
- Fetch the page or pages from the Internet and store them locally.
- Parse the local web pages to extract the data.
- Load the extracted data into a database or convert it to a spread sheet format.
By splitting the process into three separate stages the best tools for the job can be chosen. The hardest part is usually step 2 and the natural choice here is Javascript as this is the language of the Web Browser. Other technologies are used as necessary for the other parts and are often dependent on the chosen platform, that is Linux / OSX or Windows.
Recent projects in this category have been extracting stock market prices and loading them into MySQL database and extracting CV data from a website and adding the data to a spreadsheet.