Design, build, and maintain cloud-native ETL/data pipelines using Databricks and AWS Glue. Ingest and transform structured and semi-structured data with PySpark, Python, and SQL. Integrate sources including MongoDB, implement data quality checks, optimize pipeline performance, and support production deployments, monitoring, and troubleshooting while collaborating with architects and analysts.
About the Role
We are looking for a Data Pipeline Engineer with strong expertise in Databricks, AWS Glue, Python, SQL, and MongoDB to design, develop, and optimize scalable data pipelines. The ideal candidate should have experience building cloud-native ETL solutions, transforming large datasets, and delivering high-quality data for analytics and business applications.
Key Responsibilities
- Develop and maintain scalable ETL/data pipelines using Databricks and AWS Glue.
- Build data ingestion and transformation workflows for structured and semi-structured data.
- Write efficient PySpark, Python, and SQL code for large-scale data processing.
- Integrate data from multiple sources, including MongoDB and relational databases.
- Implement data quality, validation, and reconciliation checks.
- Optimize pipeline performance, reliability, and scalability.
- Collaborate with architects, analysts, and application teams to deliver data solutions.
- Support production deployments, monitoring, and troubleshooting.
Required Skills
- 5+ years of experience in Data Engineering or ETL development.
- Strong hands-on experience with Databricks and PySpark.
- Experience with AWS Glue and AWS data services.
- Proficiency in Python and SQL.
- Experience working with MongoDB.
- Knowledge of ETL, data modeling, and data warehousing concepts.
- Familiarity with Delta Lake/Lakehouse architecture is preferred.
- Experience with Git and Agile development practices.
Good to Have
- Delta Lake
- Apache Spark optimization
- AWS S3
- CI/CD pipelines
- Healthcare or Benefits domain experience
Similar Jobs
.png)
Cloud • Insurance • Payments • Software • Business Intelligence • App development • Big Data Analytics
Build, model, and maintain scalable BigQuery-based data solutions on GCP. Implement performant data models, storage partitioning/clustering, ETL improvements, data validation, and documentation. Collaborate with architects, data scientists, and engineers to deliver governed, high-quality data for reporting and AI initiatives.
Top Skills:
Ansi SqlBigQueryBigquery SqlConfluenceGCPJIRAPythonSQL
Security • Cybersecurity
Lead design and modernization of scalable, secure data systems and real-time pipelines for xOT cybersecurity. Create data contracts, observability, lineage, and cross-functional solutions supporting cloud and on-prem deployments, Kubernetes, and stream processing to enable threat detection and intelligence.
Top Skills:
CloudData Observability FrameworksDockerGoJvm (Java/Scala/Kotlin)KubernetesMessage QueuesNode.jsOn-PremisesPythonRustStream Processing
Security • Cybersecurity
Design and maintain data pipelines, models, and transformation layers for analytics. Build data contracts, observability and lineage tooling, collaborate with product engineering, and optimize data structures for performance and scalability.
Top Skills:
Cloud-Native ArchitecturesData ContractsData Observability FrameworksDimensional ModelingDockerGoIncremental Data ModelsJvm (Java/Scala/Kotlin)KubernetesLineage TrackingMessage QueuingNode.jsPythonStream Processing
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
