Being a data engineer isn’t confined to one set of responsibilities.
For example, Dennis Hume, a data engineer at alcohol e-commerce platform Drizly, is currently responsible for transitioning the company’s data orchestration over to Dagster. That project ladders up what Hume believes to be the purpose of a data engineer.
“Ultimately, the role of data engineering is to support data stakeholders across the organization and not be the bottleneck for data work,” Hume said.
Cross-team collaboration is similarly practiced at software development company MORSE Corp. Lead Data Scientist Lena Bartell is responsible for managing a team of data scientists, software engineers and data engineers — all of whom are constantly working together to create algorithms and data processors.
Built In Boston caught up with Hume and Bartell to get a better pulse on their day to day and how they are enabled to be successful in their roles.
Data Engineer Dennis Hume said that in his role at Drizly, he works closely with the analytics and DevOps teams to ensure they’re all on the same page and are able to leverage each other’s work on any given project.
Typically, what does your role as a data engineer entail?
I work very closely with the analytics team, which includes data scientists and analysts. Within our data stack, we have a lot of shared spaces (dbt, Snowflake, etc.) and we want to make sure that everything is flexible enough for the range of jobs on the team, but also that there are enough guardrails that no single process can prevent other people from doing their job.
Outside of analytics, I work a lot with our DevOps team. The infrastructure requirements for analytics are usually different from the core application, so we always want to make sure we’re all on the same page and can support each other. I want to be able to leverage all the work they do but think of how it applies to analytics use cases.
Ultimately, the role of data engineering is to support data stakeholders across the organization.”
What’s an interesting project you’re currently working on?
We are transitioning our data orchestration over to Dagster. As we grew the data science team, Dagster has been something the team can coalesce around. We treat the work done by analytics like a core application and Dagster helps ensure that when we build something it has predefined environments, tests and workflows that align with all the roles on the team. We want as little bespoke work as possible. If we have done something successfully in the past, it should be trivial for someone else to spin up a similar process.
What’s an important skill a data engineer needs to be successful in their role?
The most important thing is trying to stay informed about the data landscape and know how you can evolve your data stack with your organization. Many of the technologies we use did not exist a few years ago and now I can’t imagine life without them. When you evaluate a new piece of technology, you want to think about how it complements what exists within your stack and what it would potentially replace.
Ultimately, the role of data engineering is to support data stakeholders across the organization and not be the bottleneck for data work. That’s why it’s important to see what other people are doing in the data world and how other organizations are solving problems. Building a solution from scratch should rarely be your first thought — even if you have a clever pun for your project name.
MORSE Corp. is an artificial intelligence company that provides customer-focused algorithm and software development services. Lead Data Scientist Lena Bartell said that the company’s engineering teams are designed to be multidisciplinary in order to create diverse teams where they can continuously learn something new from one another.
Typically, what does your role as a data engineer entail?
As the lead of a data science and data engineering project, I have a team of about 10 people who have expertise in data science, software engineering and data engineering. Together, we primarily build Python-based algorithms and data processors that are wrapped and deployed into Linux-based systems and AWS cloud instances, and run in real-time environments using Docker containers.
My typical day-to-day tasks including the following:
- Go through email and reply to messages.
- Check-in with the team for 15-30 minutes to review progress and issues.
- Review work completed by other folks on the team to make sure it meets best practices and works as expected.
- Meet with individuals ad-hoc to work through any bugs or blockers.
- Write/test/run code and algorithms on the data to make sure they run and work as expected.
- And plan tasks for the team for the upcoming days and weeks and review decisions with management.
Teams are built with engineers who have a mixture of skills, so you’re always working with people from both similar and different backgrounds.”
What’s one thing that might surprise people about your role as a data engineer at your company?
One thing that might surprise people about the roles at MORSE is that data engineers (or any particular discipline) are not separated off into their own group. Teams are built with engineers who have a mixture of skills, so you’re always working with people from both similar and different backgrounds. This is great for building a diverse and talented project team as well as allows all of us to gain knowledge on new topics.
