Designs, builds, and maintains scalable data pipelines on Databricks, focusing on data quality, integrity, and governance, while collaborating with teams to deliver data solutions.
Description
At CDW, we make it happen, together. Trust, connection, and commitment are at the heart of how we work together to deliver for our customers. It's why we're coworkers, not just employees. Coworkers who genuinely believe in supporting our customers and one another. We collectively forge our path forward with a level of commitment that speaks to who we are and where we're headed. We're proud to share our story and Make Amazing Happen at CDW. The Senior Data Engineer designs, builds, and maintains scalable data pipelines and Lakehouse solutions on the Databricks platform to support enterprise data and AI initiatives. This role partners closely with architects, analytics, and data science teams to deliver high-quality, reliable data products. The engineer operates with significant autonomy and contributes deep technical expertise across the full data lifecycle.
* What you'll do:
* Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform.
* Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies.
* Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.
* Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers).
* Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog.
* Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies.
* Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues.
* Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
* Support documentation of data processes, standards, and data flows.
What we expect of you:
* 5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
* Proficiency in Python, including PySpark, and SQL.
* Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
* Experience implementing data quality, governance, and security practices across data pipelines.
* Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is a plus.
* Experience configuring and integrating external AI models and working with AI governance and monitoring tools is a plus.
* Experience with asynchronous programming patterns in Python for building scalable data or AI workloads is a plus.
* Strong problem-solving, collaboration, and communication skills.
* CDW is committed to being an AI-fluent organization
* We're looking for people who bring curiosity, a learner's mindset, and a willingness to engage with ever-evolving technology and tools. We value adopting AI as a partner, openness to experimentation, and a shared interest in learning together on AI. Our goal is to create a culture where AI enhances- not replaces- human creativity and decision-making. You don't need to be an expert today; what matters is your readiness to explore, adapt, and grow with us as we integrate AI responsibly and effectively into our work.Additionally, CDW is committed to fostering an equitable, transparent, and respectful hiring process for all applicants. During our application process, our goal is to understand your experience, strengths, skills, and qualifications. As an AI forward company, we see AI not just as a tool, but as a catalyst for new ways of thinking, creating, and communicating. We encourage candidates to embrace an AI mindset, one that's curious, adaptive, and ready to explore what's possible. We welcome thoughtful use of AI to expand your perspective and elevate how you share your story, while ensuring your application remains rooted in your own background, judgment, and voice.
* About Us
* CDW is a Fortune 500 technology solutions provider that helps businesses, government, education, and healthcare organizations achieve what's possible through technology. What makes CDW different isn't just what we do- it's how we do it. At CDW we act as one- building trust, speaking candidly, and working together to achieve more. We play to win- focusing on what matters most and delivering for our customers. And we think forward- staying curious, moving fast, and continuously learning. We believe meaningful work happens when people feel supported, heard, and empowered to contribute. That's why we think of ourselves as coworkers, not just employees- working together to solve complex challenges and deliver real impact for our customers and communities. As a full-stack, full-lifecycle technology partner, CDW brings deep expertise, strong relationships, and broad industry knowledge to help turn ideas into outcomes. When you join CDW, you become part of a collaborative environment where your work matters, your growth is supported, and your contributions help shape what's next. Together, we deliver the full promise of what technology can do.
* Together, we Make Amazing Happen.
* CDW is an equal opportunity employer. All qualified applicants will receive consideration for employment without regards to race, color, religion, sex, sexual orientation, gender identity, national origin, disability status, protected veteran status or any other basis prohibited by state and local law.
At CDW, we make it happen, together. Trust, connection, and commitment are at the heart of how we work together to deliver for our customers. It's why we're coworkers, not just employees. Coworkers who genuinely believe in supporting our customers and one another. We collectively forge our path forward with a level of commitment that speaks to who we are and where we're headed. We're proud to share our story and Make Amazing Happen at CDW. The Senior Data Engineer designs, builds, and maintains scalable data pipelines and Lakehouse solutions on the Databricks platform to support enterprise data and AI initiatives. This role partners closely with architects, analytics, and data science teams to deliver high-quality, reliable data products. The engineer operates with significant autonomy and contributes deep technical expertise across the full data lifecycle.
* What you'll do:
* Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform.
* Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies.
* Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.
* Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers).
* Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog.
* Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies.
* Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues.
* Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
* Support documentation of data processes, standards, and data flows.
What we expect of you:
* 5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
* Proficiency in Python, including PySpark, and SQL.
* Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
* Experience implementing data quality, governance, and security practices across data pipelines.
* Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is a plus.
* Experience configuring and integrating external AI models and working with AI governance and monitoring tools is a plus.
* Experience with asynchronous programming patterns in Python for building scalable data or AI workloads is a plus.
* Strong problem-solving, collaboration, and communication skills.
* CDW is committed to being an AI-fluent organization
* We're looking for people who bring curiosity, a learner's mindset, and a willingness to engage with ever-evolving technology and tools. We value adopting AI as a partner, openness to experimentation, and a shared interest in learning together on AI. Our goal is to create a culture where AI enhances- not replaces- human creativity and decision-making. You don't need to be an expert today; what matters is your readiness to explore, adapt, and grow with us as we integrate AI responsibly and effectively into our work.Additionally, CDW is committed to fostering an equitable, transparent, and respectful hiring process for all applicants. During our application process, our goal is to understand your experience, strengths, skills, and qualifications. As an AI forward company, we see AI not just as a tool, but as a catalyst for new ways of thinking, creating, and communicating. We encourage candidates to embrace an AI mindset, one that's curious, adaptive, and ready to explore what's possible. We welcome thoughtful use of AI to expand your perspective and elevate how you share your story, while ensuring your application remains rooted in your own background, judgment, and voice.
* About Us
* CDW is a Fortune 500 technology solutions provider that helps businesses, government, education, and healthcare organizations achieve what's possible through technology. What makes CDW different isn't just what we do- it's how we do it. At CDW we act as one- building trust, speaking candidly, and working together to achieve more. We play to win- focusing on what matters most and delivering for our customers. And we think forward- staying curious, moving fast, and continuously learning. We believe meaningful work happens when people feel supported, heard, and empowered to contribute. That's why we think of ourselves as coworkers, not just employees- working together to solve complex challenges and deliver real impact for our customers and communities. As a full-stack, full-lifecycle technology partner, CDW brings deep expertise, strong relationships, and broad industry knowledge to help turn ideas into outcomes. When you join CDW, you become part of a collaborative environment where your work matters, your growth is supported, and your contributions help shape what's next. Together, we deliver the full promise of what technology can do.
* Together, we Make Amazing Happen.
* CDW is an equal opportunity employer. All qualified applicants will receive consideration for employment without regards to race, color, religion, sex, sexual orientation, gender identity, national origin, disability status, protected veteran status or any other basis prohibited by state and local law.
Similar Jobs at CDW
Information Technology
The Senior Platform/DevOps Engineer designs, automates, and operates Databricks platforms, ensuring scalability and reliability while collaborating with data and engineering teams.
Top Skills:
AWSAzureAzure DevopsDatabricksDelta LakeGCPGithub ActionsJavaPysparkPythonScalaSQLTerraform
Information Technology
The Senior Analytics Engineer transforms curated data into analytical assets, develops BI-ready models, and collaborates with data stakeholders to deliver analytics solutions.
Top Skills:
AWSAzureDatabricksGCPGitPysparkPythonSQL
Information Technology
The Senior ML/MLOps Engineer designs and manages scalable machine learning solutions on Databricks, collaborating with teams to productionize models and ensure reliable operations.
Top Skills:
Azure DevopsAzure Machine LearningDatabricksMlflowPythonPyTorchScikit-LearnSQLTensorFlow
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
