Ninety percent of all the data that currently exists today was created in the last two years alone. Let that sink in.
Sounds crazy, but less so when considering that each day, around 2.5 quintillion bytes of data are created, according to a recent report by Domo.
As the scale of data is growing so quickly, the tools and trends used to make sense of all that information need to advance at a spectacular pace. And companies are clamoring to make that happen. By staying ahead of trends like open-source data warehouses and Bayesian optimization, local tech companies like Chewy and SessionM are placing all those bytes where they belong.
As big data tools like AI and machine learning grow more complex, Chewy’s VP of Software Engineering Karthikeyan Janakiraman wants processes to stay simple for his team. The leader at the pet supply e-commerce company said he hopes evolutions in big data tech will allow his engineers to work more efficiently with fewer hassles.
What are the top big data trends you’re watching that are significantly impacting the industry?
We’re seeing impactful changes when it comes to data management and use cases emerging from relating disparate systems. Machine learning and AI are now the primary drivers of data consumption across users. We’re seeing the availability of tools that account for relational and dimensional modeling, transactional integrity and data governance in a way that’s easy to use. We’re also seeing companies in this space start to consider the user experience and help companies tap into the benefits of the cloud.
But the biggest shift has been moving to a scalable, open-source data warehouse model using public cloud. We appreciate when our tools offer solutions to the friction points of elasticity, scalability, cost efficiency and agility.
We want to work with tools and software that make life easier, not more complicated.”
What under-the-radar big data trends are you watching that the industry isn’t talking about?
The tensions we feel are probably similar to the ones felt throughout the industry. What we look for, and what we appreciate, is when our service providers adopt the same attention to customer service that we would take in our own business. We’ve seen some of our vendors and service providers take that approach and we hope to see more of it.
How are these trends affecting the future of your company?
We’re really excited about the ability of big data to help us keep delivering on our mission as a company. The best tools and services emerging in this space are the ones that create a more seamless work environment for our employees. We hope that the industry continues to evolve in a way that makes the work of our engineers less time-consuming. We want to work with tools and software that make life easier, not more complicated. We want to make sure that we’re organizing our teams in a way that allows every member to work on projects that directly impact the customer experience. In order to do that, we have to find simpler ways to explore the data and build on our findings faster.
Good decisions are at the heart of any business' success. And SessionM’s Head of Data Science and Engineering Amelio Vázquez-Reina said companies can make even better decisions in the future with techniques like reinforced learning. As the next wave of big data trends continues to unfold, the company plans to take advantage of evolving practices by applying them to their marketing services.
What are the top big data trends you’re watching that are significantly impacting the industry?
Delta data-lakes for extract, transform and load operations; automated machine learning; data governance and the advent of graph databases.
Many important industry areas like robotics could benefit from AIs that can be efficiently trained in simulated environments.”
What under-the-radar big data trends are you watching that the industry isn’t talking about?
One is off-line and off-policy reinforcement learning. Most companies are sitting on vast volumes of user data that they didn’t gather through active experimentation or by executing a well-defined policy. This data has enormous potential for optimization, but doing this well is hard.
High-dimensional Bayesian optimization is another trend. Most companies and systems need to optimize choices and decisions in the face of uncertainty. Doing this well and efficiently is still a challenging problem.
A third trend is transferring the learning of agents trained in efficient physics environments and engines to other spaces. Many important industry areas like robotics could benefit from AIs that can be efficiently trained in simulated environments, and later transferred to and calibrated on real-world scenarios.
How are these trends affecting the future of your company?
Reinforced learning and the Markov decision processes have the potential to power decisions most companies and software systems face. However, leveraging them adequately and safely in a complex system can be hard. At SessionM, we have identified many product opportunities for next-best-action marketing that would immensely benefit from thoughtful reinforced learning, and we are working hard to bring these products to market.
