Design and develop big data processing engines with a focus on ETL pipeline, including complex logic and geo-location analysis.
Description
We are looking for a Big Data / Data Engineer to join our growing team. In this role, you will be responsible for the design and development of our big data processing engine a heavy ETL pipeline handling complex logic including geo-location analysis, running across both cloud and on-prem environments.
Requirements:
- 4+ years of experience in Data Engineering
- 2+ years of hands-on experience with Scala (must) or Java (optional)
- 2+ years of experience with SQL and NoSQL DBs
- Hands-on experience with Spark & PySpark (must)
- Experience with Airflow (must)
- Experience with AWS and/or GCP
- Experience with Kubernetes (k8s) and on-prem deployments
- Solid understanding of distributed systems design, pipeline architecture, and partitioning strategies
- Deep understanding of Agile principles, practices and values
- Solid understanding of design patterns, software development techniques, and clean code practices
Advantage:
- Experience with Apache Iceberg, Helm Charts, or data catalog tools
- Experience with Spring Framework
- Experience with microservice architecture
- Experience working with DevOps and CI/CD practices
- Customer-facing experience or comfort working directly with clients
- Bachelor's, BSc or Master's degree in Computer Science or Engineering
Similar Jobs
Aerospace • Information Technology • Software • Cybersecurity • Design • Defense • Manufacturing
The Staff Analyst is responsible for managing employee lifecycle processes, supporting employee engagement activities, and providing operational support for budget planning and reporting.
Digital Media • Gaming • Information Technology • Software • Sports • Esports • Big Data Analytics
Lead the design and execution of performance testing strategies, develop scalable testing frameworks, and mentor engineering teams while ensuring system resilience and reliability.
Top Skills:
C#GatlingGoJavaJmeterK6LocustPython
Aerospace • Information Technology • Software • Cybersecurity • Design • Defense • Manufacturing
Responsible for developing avionics systems, creating analytical models, performing validation and verification, and collaborating with cross-functional teams, while ensuring compliance with system requirements.
Top Skills:
Arp4754AIbm DoorsMatlabMbseSimulinkSysml
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
