Big data is ancient history in retailing. In the 80’s and 90’s logistics and stock-control were put under the Just In Time-spotlight and in the noughties we saw the introduction of electronic loyalty cards gathering data on consumer behaviour. At the same time, we have also witnessed retailing extending from brick and mortar to the web. As a result of these trends, vast amounts of data, structured and unstructured, are collected throughout the day by retailers on their operations, customers, and wider environments. Technology has been in the forefront in looking to find solutions to the big data challenge, and maybe finally, the technology capabilities are catching up to offer a solution.
With technology capabilities catching up, there is a promise to merge the disparate data-sources and stores in search of “small data”, revealing the hidden trends that give you that extra %-point advantage in the face of shorter and faster product life-cycles. There is also a promise of un-covering those hidden nuggets in our on-line behaviour that can lead to a new opportunity or help predict a disruption.
Investing in the right technologies is never easy and, for retailing, there is a need to invest wisely for the longer term. There has been, and still is, a lot of scepticism around data and analytics technologies, including AI. However, the technology underpinning data management and analytics is now really coming of age, and there are many opportunities to improve the insights into data and the end-to-end process in retailing.
With massive parallel processing (MPP) technologies such as Hadoop, Google Big Data and Hydra being available through most cloud vendors and utilised by the analytics platform providers, it does look like the barrier to take advantage of all that available data is being removed. Machine learning and AI is speeding up the automation of data collection and analysis, compared to human effort, has further lowered this barrier. However, there are challenges to taking advantage of this new technology, which many in the industry is acutely aware of. In my opinion there are three significant challenges that the retail sector faces as it looks to modernise its technology estates.
Firstly, enabling the heritage platforms that underpin all operations to converse with the new technologies and services. Most new technologies rely on the cloud and are designed very differently from traditional ERP solutions. Making the decision to move your heritage ERP to the cloud is not one to be taken lightly. Clouding, wholly or partially, is the first step to enabling new technologies to interface with the existing estate. The trend of creating micro services, , to take advantage of the rising cloud-based API-economy in advanced data management is gaining momentum. This is further aided by the growing desire to create a partner ecosystem, helping retailing to gain access to insights using partner technologies that interface with the retailers’ core systems. Combining MMP database technologies with cloud-based analytics tools and enabling interfacing with the retailers’ ERP solutions, to drive value from the data is a rapidly growing trend.
Creating system integration points brings us to the second challenge. How to interface and manage the data that will give you that hidden trend and the potential competitive advantage? Each retailer and partner will have created their own data classification and hierarchy with varying meta-data dimensions to gain insights into that data. These meta-data structures tend to have grown organically from one or two key data points. The challenge for most data professionals has always been two-fold: how to create a meta-data structure that can evolve with the business; and how to combine data from different sources in a meaningful way? MPP technologies are useful in processing large volumes of data, however, they need to be augmented with data management technologies that address the two pain points mentioned above.
Data management technology is an exciting and growing field, with semantic technologies such as Apache Cassandra and Neo4j, becoming more commonly used. Semantic technologies uncouple meta-data from the data itself and crucially, from the application, creating system-independent data, is taking master data management to new heights. Having system-independent data improves the process of combining data-sets for analysis and increases the opportunities available to retailers to expand their data analytics ecosystem with partners. Commercial ontologies, effectively providing a “frame work of dictionaries”, is a further trend in data management improving the opportunities for retailers and their partners to make full use of the data. Both the emergence of ontology tools and semantic technologies are removing the need to create and manage shared data structures between the retailer and their ecosystem. It also removes the need for the retailer to invest in building the capability in-house.
The final challenge I see on the journey to modernisation is the decision-making process itself. Traditionally data is treated and “forced” into reporting structures that often mimic the hierarchy of the organisation, with small decisions being made at the lower level. However, lots of wrong, or conflicting, small decisions can be detrimental over time. Many technologists and data professionals believe this is where AI becomes valuable, analysing the small decisions and the data the decision was based on, to capture trends and promote course corrections earlier than the human reporting structure can.
Where to focus when technologies are constantly evolving? Where should the investment be directed to maximise the benefit? We can safely make the assumption that cloud and the API-economy is here to stay. As is the increasing volume of data generated, shared and analysed. By focusing on creating the capability to adapt and adopt to technology and data changes, IT professionals can pave the way for future profitability. So too will thoughtfully structuring your cloud and re-architecting key services and functionalities to maximise cloud-based partnerships. Identifying data-sets to be decoupled from the relevant systems, and given the semantic treatment of separating the data from the meta-data and application, will allow data professionals to write better analysis programmes to find that much-sought, game-changing “small data”.
Retailing connects the world, and despite the advances in the way we use the big data it generates, it still has many secrets to be revealed.
