Design and develop a big-data processing engine and heavy ETL pipelines (including geo-location analysis) across cloud and on-prem. Build distributed, partitioned pipeline architectures using Spark/PySpark, Scala, Airflow, Kubernetes, and cloud platforms; follow Agile, clean-code, and software design best practices while supporting deployments and client-facing interactions.
Description
We are looking for a Big Data / Data Engineer to join our growing team. In this role, you will be responsible for the design and development of our big data processing engine a heavy ETL pipeline handling complex logic including geo-location analysis, running across both cloud and on-prem environments.
Requirements:
- 4+ years of experience in Data Engineering
- 2+ years of hands-on experience with Scala (must) or Java (optional)
- 2+ years of experience with SQL and NoSQL DBs
- Hands-on experience with Spark & PySpark (must)
- Experience with Airflow (must)
- Experience with AWS and/or GCP
- Experience with Kubernetes (k8s) and on-prem deployments
- Solid understanding of distributed systems design, pipeline architecture, and partitioning strategies
- Deep understanding of Agile principles, practices and values
- Solid understanding of design patterns, software development techniques, and clean code practices
Advantage:
- Experience with Apache Iceberg, Helm Charts, or data catalog tools
- Experience with Spring Framework
- Experience with microservice architecture
- Experience working with DevOps and CI/CD practices
- Customer-facing experience or comfort working directly with clients
- Bachelor's, BSc or Master's degree in Computer Science or Engineering
Similar Jobs
Edtech • Social Impact
Manage engineering projects and QA for a small, remote engineering team: track work, run bug pipeline, write and automate tests (Capybara when useful), coordinate engineers with product and internal stakeholders, and report status to leadership while using AI to streamline repetitive testing tasks.
Top Skills:
Ai ToolsAnthropicCapybaraClaudeDjangoRspecRuby On Rails
Edtech • Social Impact
Own end-to-end experience design for CodePath’s learning platform: map flows, navigation, and IA; build and maintain a design system; run product discovery with users and stakeholders; produce engineer-ready wireframes, prototypes, and specs; partner with PMs and engineers; and direct a UI contractor to deliver high-quality, consistent experiences.
Top Skills:
AIAnthropicClaudeCSSFigmaHTMLLlmTailwind
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Lead post-launch lifecycle, adoption, stabilization and continuous optimization of the CFC CRM platform across global markets. Own BAU roadmap, operational governance, integrations, release readiness, value measurement, and cross-functional orchestration with Data, Digital, Compliance, and Markets to ensure CRM delivers sustained commercial value and scalability.
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
