Build and maintain large-scale data processing pipelines using Apache Spark (PySpark/Scala) and Python. Design ETL workflows, implement distributed data processing, use version control (Git), and work with cloud platforms (GCP/AWS/Azure).
Job Title: Data Engineer (PySpark / Scala / Python)
Location: Remote
Job Description:We are hiring a Data Engineer with strong hands-on experience in PySpark, Scala, and Python. You must have solid expertise in Apache Spark, as it will be the core technology used for building and managing large-scale data processing pipelines.
Experience with cloud platforms like Google Cloud Platform (GCP), Microsoft Azure, or AWS is a plus.
Required Skills:Strong hands-on experience with Apache Spark
Proficient in PySpark
Experience in Scala and Python
Knowledge of ETL processes and data pipeline design
Understanding of distributed data processing
Familiarity with version control tools like Git
Basic knowledge of cloud platforms (GCP, AWS, or Azure)
Experience with cloud-native data tools (e.g., Dataproc, Glue, EMR, BigQuery)
Familiarity with workflow/orchestration tools like Airflow or Cloud Composer
Experience with CI/CD for data engineering
Exposure to both structured and unstructured data
Similar Jobs
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Build and operate production data pipelines supporting analytics, AI, and agentic workflows. Responsibilities include implementing canonical data models, maintaining Databricks or Snowflake platforms, monitoring reliability, responding to incidents, validating healthcare data mappings, improving performance and cost, and documenting architecture. Requires strong SQL and Python skills, cloud data platform experience, ETL/ELT orchestration expertise, and healthcare or pharmaceutical data experience.
Top Skills:
Ai/Ml WorkflowsDatabricksEltETLHedisOmopPythonSnowflakeSQL
AdTech • Automotive • Big Data • Consumer Web
Administer and enhance Edmunds’ Databricks data platform and AWS infrastructure. Build and maintain ETL pipelines, infrastructure-as-code tooling using Terraform or CDK, and operational dashboards, alerts, and reports. Collaborate with business, engineering, analytics, security, and infrastructure teams to support data platform users and AI solutions. Evaluate new technologies, troubleshoot platform issues, and improve operational, cost, and security visibility.
Top Skills:
SparkAWSAws CdkDatabricksInfrastructure As Code (Iac)PythonScalaSQLTerraform
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Architects, deploys, and operates CrowdStrike’s data platform infrastructure and services. Responsibilities include administering Airflow and Superset, managing Terraform infrastructure, building CI/CD pipelines, implementing security and access controls, developing DBT data models, cataloging assets in OpenMetadata, writing Python automation, and mentoring engineers. The role requires cloud, container orchestration, SQL, data modeling, compliance, and AI technology expertise, along with eligibility for CJIS clearance.
Top Skills:
AIApache AirflowApache SupersetAWSAzureCi/CdDbtDockerGCPIamKubernetesMachine Learning PipelinesOciOpenmetadataPythonSQLTerraform
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
