Design and optimize Databricks ETL/ELT pipelines on Azure using PySpark, SQL, Delta Lake, and Kafka. Build CDC and near-real-time processing solutions, ingest data from diverse sources, implement data quality controls, and apply SCD Type 1 and Type 2 modeling. Automate workflows with Databricks Workflows and GitHub-based CI/CD while collaborating in an Agile environment.
About This Opportunity
CTI Staffing is partnering with a leading retail organization to find a Senior Databricks Data Engineer for their enterprise data team based in the Greater Boston, MA area.
The team is building out a modern Azure data platform that brings together data from many retail systems. This role matters because reliable, incremental pipelines are what let the business trust its data and act on it faster.
What You’ll Do
- Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
- Build Change Data Capture pipelines for incremental and near real-time processing.
- Develop scalable data solutions on Azure using Delta Lake and Kafka streaming.
- Improve existing pipelines by quickly learning current codebases and delivering targeted upgrades.
- Ingest and process data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, message queues, and Kafka topics.
- Implement data quality rules and quarantine error records.
- Apply data modeling techniques including SCD Type 1 and Type 2.
- Automate workflows with Databricks Workflows and GitHub-based CI/CD, and work with developers, architects, and product owners in an Agile team.
Requirements
What You Bring
Must-Have:
- Extensive hands-on Databricks experience, able to contribute from day one.
- Proven experience building end-to-end data pipelines with CDC.
- Strong PySpark and SQL skills.
- Experience working in Microsoft Azure.
- Working knowledge of Delta Lake and distributed data processing.
- Experience with Kafka or comparable streaming technologies.
- Hands-on experience with SCD Type 1 and Type 2 data modeling.
- Familiarity with GitHub and CI/CD for version control and deployment.
Nice-to-Have:
- Python experience beyond PySpark.
- Databricks Workflows and Unity Catalog experience.
- Experience with message queues and API-based ingestion.
- Retail, CPG, or supply chain data background.
- Databricks certification.
Technical Environment: Azure, Databricks, PySpark, SQL, Delta Lake, Kafka, Unity Catalog, GitHub, CI/CD, SQL Server, JSON, XML, REST APIs.
Similar Jobs
3D Printing • Fintech • Payments • Manufacturing
The Senior Data Engineer will build data pipelines and infrastructure, modernize legacy systems, and ensure data reliability for analytics at ABCorp.
Top Skills:
Azure Data FactoryAzure Data Lake StorageAzure DatabricksAzure Synapse AnalyticsDbtPythonSQL
.png)
Mobile • Retail • Software • Business Intelligence
Manages SaaS revenue recognition, journal entries, accruals, deferred revenue, account reconciliations, and month-end and quarter-end close activities. Supports financial statement preparation, audits, contract validation, and compliance with accounting standards. The role also improves revenue reporting, analyzes SaaS metrics, and develops scalable accounting processes while collaborating with sales, operations, and finance teams.
Top Skills:
Asc 606Crm DataErp/Accounting SystemsExcel
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Provides statistical expertise for clinical development projects in inflammation and immunology, including trial design, protocol development, statistical analysis plans, analyses, regulatory submissions, results interpretation, reports, publications, and presentations. Collaborates with cross-functional study teams, ensures regulatory-compliant statistical deliverables, and contributes to statistical methodology research.
Top Skills:
RSAS
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
