Role: Senior Solution Architect– Spark Expert
We are seeking a Senior Data Solutions Consultant with deep expertise in Apache Spark and Databricks. This role requires a decisive, agentic-forward approach to problem-solving. You will independently lead complex customer engagements, bridging hands-on data engineering excellence with strategic technical leadership. As a trusted advisor, you will proactively build and implement solutions that unlock significant value for our customers.
Customer Leadership: Lead engagements from discovery to delivery with full ownership and accountability.
Architecture & Optimization: Architect, implement, and optimize large-scale Spark/Databricks workloads focusing on scalability and cost-efficiency.
Spark Expertise: Apply expert-level Spark fundamentals to troubleshoot and tune distributed data jobs and SQL workloads.
Proactive Delivery: Design and execute PoCs and workshops to demonstrate solution vision and inspire customer confidence.
Strategic Advocacy: Act as a technical advocate for customers, anticipating issues and applying creative, data-driven solutions.
Cross-functional Partnership: Collaborate with product and engineering teams to improve customer experience and platform features.
Mentorship: Champion best practices in PySpark, Databricks, and Lakehouse design while mentoring junior engineers.
Experience: 5–10 years in data engineering or analytics solution delivery.
Databricks Mastery: 3+ years of hands-on expertise, including job optimization, debugging, and workload governance.
Technical Depth: Deep understanding of Spark Internals, PySpark, and Databricks SQL (DBSQL) performance tuning.
Modern Lakehouse: Strong knowledge of Delta Lake, Structured Streaming, and cloud infrastructure.
Autonomy: Proven ability to identify and solve complex technical problems with minimal oversight.
Communication: Exceptional skills to simplify technical complexity for business leaders.
Certifications: Databricks Certified Data Engineer Professional.
Advanced Skills: Experience with Delta Live Tables (DLT) and Unity Catalog (UC).
Consulting Background: Experience in large-scale data migration or AI-assisted analytics in enterprise environments.
Similar Jobs
What you need to know about the Boston Tech Scene
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
