Principal Data Architect

Who are we?

A strategic and trusted insurance partner, Berkshire Hathaway Specialty Insurance (BHSI), provides a broad range of commercial property, casualty and specialty insurance coverages and outstanding service to customers and brokers around the world. Part of Berkshire Hathaway’s insurance operations, we bring our solutions to market with our stellar brand name, top-rated balance sheet, and the expertise of our global team of professionals, who exude excellent capabilities and strong character. 

We are a values-based organization where respect, integrity, excellence, collaboration, and passion define who we are and how we do business. We value diversity of backgrounds, experience, and perspectives and strive to foster an inclusive environment that enables all our team members to bring their best selves to work. We are one team committed to building a culture where every teammate has the opportunity to contribute and be recognized. Want to be part of the team building the finest property, casualty, and specialty lines insurance company in the world?

Learn more about our unique culture and history.

Job Opportunity: 

BHSI is modernizing its data ecosystem, moving from a monolithic, reporting-centric infrastructure to a Lakehouse platform that supports insight-driven decisions and AI-enabled outcomes at the speed of business. We are seeking a Principal Data Architect to set the technical direction for that platform. 

You will own the architecture of our Databricks-based Lakehouse across data engineering, analytics, and AI/ML, ensuring it is secure, performant, cost-efficient, and ready for Generative AI. This is a hands-on, senior individual-contributor role for a technologist and strategic thinker, not a people-management position. You will report to the Global Head of Data Analytics Solutions and partner with Enterprise Technology portfolio leaders and business stakeholders.

Responsibilities:

• Platform Architecture & Strategy: Define and evolve the enterprise Lakehouse reference architecture on Databricks, supporting batch, streaming, and AI workloads at scale across AWS or Azure. 
• AI & Generative AI Enablement: Lead the design and rollout of Databricks AI capabilities, including vector search, model serving, and LLM orchestration, so teams can build, fine-tune, and deploy models securely against enterprise data. 
• Governance & Security by Design: Implement lineage, access controls, and auditing with Unity Catalog to meet regulatory requirements and ensure data quality across domains. 
• MLOps & Engineering Excellence: Establish CI/CD and infrastructure-as-code standards for data pipelines and ML models. Optimize Spark performance, cluster configuration, and compute spend. 
• Technical Leadership & Influence: Set architecture standards, run design reviews, and mentor engineers and data scientists. Translate business strategy into a phased technical roadmap and build alignment with engineering and business leaders. 

Qualifications, Skills and Experience:

• 15+ years in data engineering, data architecture, or software architecture, including 5+ years architecting on the Databricks Lakehouse Platform 
• Deep expertise in Apache Spark, Delta Lake, Unity Catalog, and MLflow
• Proven experience designing infrastructure for both traditional ML and Generative AI workloads (RAG, fine-tuning, vector search, agentic frameworks) 
• Strong proficiency in Python and SQL, infrastructure-as-code (e.g., Terraform), and software design patterns 
• Hands-on architecture experience on AWS or Microsoft Azure 
• Demonstrated ability to influence senior technical and business stakeholders without direct authority 

Preferred Qualifications:

• Databricks Professional or Expert-level certification 
• Experience with advanced agentic architectures, DSPy, or graph databases 
• Background in insurance, financial services, fintech, or other regulated environments 
• Track record of optimizing cloud compute costs while improving pipeline throughput 

BHSI Offers:

The base salary range for this position in Boston is $200,000 to $240,000, along with annual bonus eligibility; a candidate’s salary is determined by their relevant skills and experience. We value our teammates – both their capabilities and character – as demonstrated by our amazing culture.

NOTE: Compensation will be commensurate with experience. This job description is not intended to be all-inclusive. Team Member may perform other related duties as negotiated to meet the ongoing needs of the organization.