71% finance organisations use web scraping to gather external data

Internal data (74%) remains the most popular source for businesses, but there are clear signs web scraping is set to overtake it soon. Over two-thirds of respondents (68%) plan to increase their data budgets in the coming year, while 80 per cent believe they will focus more strongly on web scraping in the same period. Other methods, such as third-party data aggregators (used by 56%) remain an important part of the data-gathering ecosystem but are now a long way behind web scraping and internal data.

Key highlights:

• 71% of respondents use external data gathered using web scraping to guide their business decisions;
• Web scraping follows closely behind internal data (74%), with the former’s adoption on an upward trajectory;
• Web scraping and internal data rank well ahead of other data-gathering methods, with third-party aggregators being the third-most-popular option (56%);
• 80% believe their focus will shift more heavily towards web scraping in the next 12 months, while 68% plan to increase their data budgets during the same period;
• 38% say data risk and legal complications are their main concerns around adopting web scraping, while 36% cite complicated tech and budget constraints, and 34% a lack of technical expertise.
• Almost three-quarters (71%) of financial services organisations are now using web scraping to gather external data to guide their decision-making.

Gediminas Rickevičius, VP of Global Partnerships at Oxylabs said, “internal data is relatively cheap, efficient, and company-unique, so it’s not a surprise that it’s the dominant acquisition method. Alternative data and web scraping being so close, however, is something completely new. There’s a clear interest and even necessity to uncover new ways of unearthing data-driven insights from untapped sources. It’s clear – financial services companies are expecting to get their edge through web-scraped alternative data.”

Despite web scraping’s rise, there remain some concerns among decision-makers about how to implement it in a safe, cost-effective, and compliant manner. Almost four in ten (38%) state risk and legal complications as the main factors discouraging them from using web scraping. Alongside this, 36 per cent are concerned about budget constraints or the need to introduce complicated new technology, and 34 per cent believe they lack the technical know-how to make web scraping a success.

Vaidotas Šedys, Head of Risk Management at Oxylabs added, “risk is inherent to all new technologies, as pioneering a new field means there are no footsteps to follow. Any web scraping risks, however, can be greatly mitigated by applying data collection and management best practices and employing legal and risk professionals to ensure web scraping is done in a compliant, safe, and ethical way.

“These practices may take some time to get used to, but done in the right way, there’s every chance that financial services businesses can gain valuable new insights, improve customer experiences and achieve strong ROI through web scraping.”