Technology needs in the age of Big Data

Data science revolves around a 4-stage process: business understanding, data gathering/processing, modeling, and deployment. In recent years, each of these stages they have gone through fundamental changes, with data gathering being the most notable due to the big data boom.

When it comes to the modeling and the deployment stages, the most common practice up until today is to perform data analysis locally in the researcher’s machine. The resulting model is then later deployed and applied on batches of new data. Next the output is generated – a report containing insights resulting from the analysis. Usually, every 6-12 months, the model is re-estimated with new data, to keep it up to date.

Just like the data gathering stage, the later modeling and deployment stages have also entered a new era – one that opens up tremendous opportunities for agile decision making. The fundamental shift in new ways of analyzing and reporting data is possible thanks to cloud computing and online dashboarding. The “Cloud” refers to a shared pool of computing resources that exist in a physical location called ‘data centers’ which are spread around the world. Users of cloud computing can run models in real time and instantaneously update them whenever new batches of data arrive. As of today, the 3 main cloud players are: Amazon Web Services, Microsoft Azure, and Google Cloud Platform.

Furthermore, the direct connection between the cloud and online dashboards allows for reports to be automatically updated by the minute. In the 2021 GreenBook Research and Trends (GRIT) Report online dashboards are becoming a staple and top priority among suppliers as buyers continue to demand speed, real-time access to data/insights and interactive visualizations. Many of our clients who are innovators in the insights industry are increasingly opting for interactive dashboards over traditional static reporting methods.

The future started yesterday

Organizations that take advantage of cloud computing, real-time reporting and predictive advanced analytics are clearly farther along in their ‘big data’ journey. These companies typically have insights or data science teams with high analytics competence. There is transparency and harmonization across data sets (and departments). They no longer rely on single data sets, but instead use data-fusion, or data integration, analytics techniques to build more accurate and reliable predictions.

In fact, every company now has moved or wants to move towards this environment, and away from the siloed one. The reason is because this environment allows companies to quickly recognize market trends, predict the consequences and adapt strategies accordingly. It is agile decision making in action.

We have a front-row view to see how some of our clients are leveraging technology and data-fusion analytics to capitalize on opportunities and accelerate growth. As an analogy, if we were in a car race, the car with a cloud computing engine would be cruising easily at speed 200km/h whereas the rest would struggle to even reach half its speed. In a world where if you snooze you lose, those who cannot keep up will certainly get their engines burned.

Authors:

• 

• 

•