The Big Data Trends These Boston Companies Are Watching

by Alton Zenon III
November 7, 2019

Ninety percent of all the data that currently exists today was created in the last two years alone. Let that sink in.

Sounds crazy, but less so when considering that each day, around 2.5 quintillion bytes of data are created, according to a recent report by Domo. 

As the scale of data is growing so quickly, the tools and trends used to make sense of all that information need to advance at a spectacular pace. And companies are clamoring to make that happen. By staying ahead of trends like open-source data warehouses and Bayesian optimization, local tech companies like Chewy and SessionM are placing all those bytes where they belong.

 

Chewy team members eating and chating
chewy

As big data tools like AI and machine learning grow more complex, Chewy’s VP of Software Engineering Karthikeyan Janakiraman wants processes to stay simple for his team. The leader at the pet supply e-commerce company said he hopes evolutions in big data tech will allow his engineers to work more efficiently with fewer hassles. 

 

What are the top big data trends you’re watching that are significantly impacting the industry?

We’re seeing impactful changes when it comes to data management and use cases emerging from relating disparate systems. Machine learning and AI are now the primary drivers of data consumption across users. We’re seeing the availability of tools that account for relational and dimensional modeling, transactional integrity and data governance in a way that’s easy to use. We’re also seeing companies in this space start to consider the user experience and help companies tap into the benefits of the cloud.

But the biggest shift has been moving to a scalable, open-source data warehouse model using public cloud. We appreciate when our tools offer solutions to the friction points of elasticity, scalability, cost efficiency and agility.

We want to work with tools and software that make life easier, not more complicated.”

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

The tensions we feel are probably similar to the ones felt throughout the industry. What we look for, and what we appreciate, is when our service providers adopt the same attention to customer service that we would take in our own business. We’ve seen some of our vendors and service providers take that approach and we hope to see more of it.

 

How are these trends affecting the future of your company?

We’re really excited about the ability of big data to help us keep delivering on our mission as a company. The best tools and services emerging in this space are the ones that create a more seamless work environment for our employees. We hope that the industry continues to evolve in a way that makes the work of our engineers less time-consuming. We want to work with tools and software that make life easier, not more complicated. We want to make sure that we’re organizing our teams in a way that allows every member to work on projects that directly impact the customer experience. In order to do that, we have to find simpler ways to explore the data and build on our findings faster.

 

SessionM team members standing in group photo
sessionm

Good decisions are at the heart of any business' success. And SessionM’s Head of Data Science and Engineering Amelio Vázquez-Reina said companies can make even better decisions in the future with techniques like reinforced learning. As the next wave of big data trends continues to unfold, the company plans to take advantage of evolving practices by applying them to their marketing services.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

Delta data-lakes for extract, transform and load operations; automated machine learning; data governance and the advent of graph databases.

Many important industry areas like robotics could benefit from AIs that can be efficiently trained in simulated environments.”

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

One is off-line and off-policy reinforcement learning. Most companies are sitting on vast volumes of user data that they didn’t gather through active experimentation or by executing a well-defined policy. This data has enormous potential for optimization, but doing this well is hard.

High-dimensional Bayesian optimization is another trend. Most companies and systems need to optimize choices and decisions in the face of uncertainty. Doing this well and efficiently is still a challenging problem.

A third trend is transferring the learning of agents trained in efficient physics environments and engines to other spaces. Many important industry areas like robotics could benefit from AIs that can be efficiently trained in simulated environments, and later transferred to and calibrated on real-world scenarios.

 

How are these trends affecting the future of your company?

Reinforced learning and the Markov decision processes have the potential to power decisions most companies and software systems face. However, leveraging them adequately and safely in a complex system can be hard. At SessionM, we have identified many product opportunities for next-best-action marketing that would immensely benefit from thoughtful reinforced learning, and we are working hard to bring these products to market.

 

Jobs from companies in this blog58 open jobs
All Jobs
Data + Analytics
Design + UX
Dev + Engineer
HR
Internships
Marketing
Operations
Product
Project Mgmt
Sales
Developer
new
Chewy
Boston
Developer
new
Chewy
Boston
HR
new
Chewy
Boston
Developer
new
Chewy
Boston
Operations
new
SessionM
Boston
Operations
new
Chewy
Boston
Project Mgmt
new
Chewy
Boston
Internships
new
SessionM
Boston
Developer
new
Chewy
Boston
Product
new
Chewy
Boston
Marketing
new
Chewy
Boston
Product
new
SessionM
Boston
Data + Analytics
new
SessionM
Boston
Product
new
Chewy
Boston
Product
new
Chewy
Boston
Marketing
new
Chewy
Boston
Operations
new
SessionM
Boston
Developer
new
SessionM
Boston
Sales
new
SessionM
Boston
Developer
new
SessionM
Boston
Developer
new
SessionM
Boston
Developer
new
SessionM
Boston
Data + Analytics
new
SessionM
Boston
HR
new
Chewy
Boston
Data + Analytics
new
Chewy
Boston
Developer
new
Chewy
Boston
Design + UX
new
Chewy
Boston
Data + Analytics
new
Chewy
Boston
Product
new
Chewy
Boston
HR
new
Chewy
Boston
HR
new
Chewy
Boston
Data + Analytics
new
Chewy
Boston
Developer
new
Chewy
Boston
Operations
new
Chewy
Boston
Developer
new
Chewy
Boston
Developer
new
Chewy
Boston
Data + Analytics
new
Chewy
Boston
Design + UX
new
Chewy
Boston
Developer
new
Chewy
Boston
Developer
new
Chewy
Boston
Developer
new
Chewy
Boston
Data + Analytics
new
Chewy
Boston
Developer
new
Chewy
Boston
Developer
new
Chewy
Boston
Product
new
Chewy
Boston
Data + Analytics
new
Chewy
Boston
Data + Analytics
new
Chewy
Boston
Marketing
new
Chewy
Boston

Boston startup guides

LOCAL GUIDE
Best Companies to Work for in Boston
LOCAL GUIDE
Best Software Engineer Jobs in Boston
LOCAL GUIDE
Coolest Tech Offices in Boston
LOCAL GUIDE
Best Sales Jobs in Boston
LOCAL GUIDE
Best Perks at Boston Tech Companies
LOCAL GUIDE
Best Marketing Jobs in Boston
LOCAL GUIDE
Best Data Scientist Jobs in Boston
LOCAL GUIDE
Best Design Jobs in Boston