Turning terabytes of Big Data into insights

Turning terabytes of Big Data into insights
7 minute read
Jerome Hancock

A real-world market research case study with Big Data

The market research industry is no stranger to consumer data. However, the scale of “Big Data” generated through online behavior brings a host of challenges and opportunities for insights professionals and marketers alike. As consumers leave an endless supply of digital breadcrumbs online, how can we most effectively analyze and act on this behavior at the individual level?

Such was the challenge ArchDaily.com faced after amassing 20 terabytes of Big Data over the past three years. As the leading architectural website worldwide, interpreting this data was much more complicated than it had anticipated. With 150-200 million-page views a month, the company could see behavior volumes. However, ArchDaily.com didn’t know what was driving user behavior. It wanted to better predict architectural trends for and identify key drivers to optimize its online content strategy.

Insights extraction isn’t an easy process, especially on the scale of Big Data, but here at SKIM, we’re data experts and we dove deep into tackling this analytical challenge.

Read on to learn more about the insights approach we used for ArchDaily.com, and business outcomes realized from turning terabytes of data into action.

Big Data analytical challenge

ArchDaily.com was created by two architectural students in 2006. Originally designed as a blog, it offered an alternative to expensive architectural magazine subscriptions, ArchDaily.com is now the leading architectural web portal providing inspiration and knowledge to help build better cities globally. The site has 14 million architects accessing it on a monthly basis. Architects use ArchDaily.com as their main source of inspiration and information, and to access lists of curated tools and suppliers.

With tremendous monthly website traffic, like most online platforms, ArchDaily actively seeks new ways to further monetize their data and in 2017 started to track their data in earnest. At that time, they had a very healthy viewership indeed. In addition to tracking page views and other basic metrics, they had the foresight to start tracking many more ‘events’ on the site – everything from all clicks to bookmarking behavior. While they may not have considered it a ‘Big Data strategy’ at the time, they knew tracking user behavior on the site was important and this began their journey into Big Data extraction.

SKIM Big Date and Market Research whitepaper

While the ArchDaily data science team was able to identify popular projects based on page views and high levels of interaction, they didn’t know if viewership was high because users were interested in the article or because, being the leading provider of architectural content and publishing items on the topic, they were helping to fuel this popularity. They realized they needed a way to take the publication bias out of the data.

Combining insights and data science to better predict online behavior

ArchDaily.com’s data scientists used machine learning to create models, but accessing, processing, and extracting value from this semi-structured Big Data, while ensuring analytical and methodological rigor, was a challenging task. They realized the value that applying advanced analytics methods to extract richer insights from their data could help them better predict and understand user’s online behavior.

“For this Big Data project, we needed a methodology expert to add analytical rigor to our data science team and that’s what brought us to SKIM,” said ArchDaily.com’s COO, Guillermo Zedan.

So ArchDaily.com teamed up with us and leveraged our Big Data research framework to address their two most pressing business challenges:

  1. Identify, quantify, and forecast the popular architectural trends, while accounting for their own publication bias.
  2. Segment and cluster all the articles on the site based on key performance metrics to identify and contrast key drivers of blockbuster articles versus other articles.

SKIM Big Data Market Research Framework

Download our paper “From Terabytes to Insights: Implications of Big Data on market research” for more information on SKIM’s approach to Big Data

Big Data research solution

We assembled a team of methodologists, data science and advanced analytics experts to create a customized research solution for ArchDaily.com. The solution was implemented in a cloud environment using Google Cloud Platform; we combined machine learning algorithms and robust statistical modeling to extract value from ArchDaily’s internal data as well as external benchmarks. More specifically, based on ArchDaily’s three years of data, we segmented over ten-thousand articles featuring architecture projects and narrowed down approximately 1,200 variables in order to profile and identify useful segments that added tangible value to their business strategy.

For example, we learned that if ArchDaily.com published a competition, although it would drive activity, it was short-lived behavior, which did not add much value to their business. Conversely, the ability to accurately quantify the popularity of the rising trend in sustainable materials (e.g. bamboo) added tremendous credibility to their forecasts and creates immediate value-add for their advertisers, making their future reports very attractive to their customers.

Putting Big Data insights into action

New user acquisition and the engagement of existing users is the cornerstone of any successful online platform, so we developed search forecasts for more than 200 key architecture terms in order to better understand what the sites visitors were looking for. By integrating the forecasting dashboard into ArchDaily.com’s planning and content curation process, they can now foresee architectural trends that will be increasingly in demand by their target users in the future, independent of their past publication schedule.

As a result, ArchDaily.com is now able to incorporate learnings of what matters most to ArchDaily.com’s users into their curation process, in order to optimize their editorial planning and content strategy. The online platform also now has the tools needed to increase the likelihood of publishing a blockbuster article. More importantly, through this Big Data analysis, ArchDaily.com can now identify and deliver more relevant content to architects and architecture enthusiasts, so that they can in turn design better cities and living spaces across the globe.

SKIM Big Data Insights Dashboard

Want to learn more about the technical research techniques and tools used to process the 20 terabytes of Big Data? Download our paper now

“There is a time lag between behavior the manufacturers can see through purchasing behavior and what we can see on the platform – between the exploration phase of the project and the construction. Things that are trending on our platform now will impact the market three years from now.

You can imagine how valuable that would be for business forecasting,” said Zedan.

Starting your Big Data insights journey

As more data sources become available, we see more opportunities for insights extraction and data-driven marketing activation across all industries. Aside from the general trends or observations ascertained from the data, it’s the actionable insights that are most valuable. As ArchDaily.com experienced, this is the real competitive advantage a Big Data strategy can offer.

At SKIM, we believe in this increasing world of Big Data, insights professionals need to adapt to today’s reality; they need to look into, and integrate, multiple data sets to provide a more holistic assessment. We need to become data source “agnostic.”

Download our Big Data whitepaper for more best practices we recommend in this area.

However, working with Big Data presents multiple challenges and issues that you may not encounter in traditional market research projects. Specifically, you’ll need to consider the frequency and degree of interaction between the client and insights agency, the software and hardware architecture required to handle big unstructured data, and the combination of machine learning algorithms and multivariate statistical analysis employed. You’ll quickly realize that Big Data insights projects like the one we undertook with ArchDaily.com, are a unique journey of discovery of both insights and process best practices.

From Big Data analysis, to data-fusion, machine learning and automation techniques, SKIM continues to be at the forefront of advanced analytics and innovation. We’re constantly innovating new ways to help leading companies better understand and predict decision behavior.

Do you want to extract richer insights from your Big Data? Contact us and our enthusiastic analytical minds can help think along on how you can get started today.

Topics
Advanced Modeling Cases Market Research Trends Technology Telecom
Jerome Hancock

Written by

Jerome Hancock

Jerome Hancock is a Senior Director at SKIM, leading the European Methodology and Analytics team. He has worked in marketing analytics for over 15 years, using predictive models to help clients optimize their marketing activities. He has consulted across a range of industries including CPG, Travel and Tech, using statistical methodologies such as conjoint, segmentation, key drivers, marketing mix and forecasting. He holds a Business degree from Nottingham and an MBA from Henley Business School.

More about Jerome Hancock