Design, build, and maintain ETL/ELT data pipelines and orchestration using Airflow, Python, and PySpark. Develop custom connectors, implement DataOps and CI/CD for data workflows, monitor and troubleshoot pipelines, validate data quality, and document processes while collaborating with cross-functional teams and supporting large-scale distributed data systems.
Greetings Everyone
Who are we?
For the past 20 years, we have powered many Digital Experiences for the Fortune 500. Since 1999, we have grown from a few people to more than 4000 team members across the globe that are engaged in various Digital Modernization. For a brief 1 minute video about us, you can check https://youtu.be/uJWBWQZEA6o.
What will you do? What are we looking for?
Job Description :
Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.
Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.
Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
Stay up-to-date with emerging technologies and best practices in data engineering.
Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.
Experience:
Bachelor’s degree in computer science, Engineering, or a related field
8 + years’ experience in data engineering, ELT development, and data modeling.
Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
Demonstrated experience in developing custom connectors for data ingestion from various sources.
Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
Experience implementing DataOps principles and practices, including data CI/CD pipelines.
Excellent problem-solving and troubleshooting skills, with a strong attention to detail.
Effective communication and collaboration abilities, with a proven track record of working in cross-functional teams.
Familiarity with data visualization tools Apache Superset and dashboard development.
Understanding of distributed systems and working with large-scale datasets.
Familiarity with data governance frameworks and practices.
Knowledge of data streaming and real-time data processing technologies (e.g., Apache Kafka).
Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes.
Experience with Agile development methodologies and working in cross-functional Agile teams.
Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.
Excellent analytical and problem-solving skills, with a keen attention to detail.
Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.
Required Skills –
Python, Pyspark, Sql,Airflow, Trino, Hive, Snowflake, Agile Scrum
Good to have–
Linux,Openshift, Kubernentes, Superset
Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.
Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.
Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
Stay up-to-date with emerging technologies and best practices in data engineering.
Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.
Experience:
Bachelor’s degree in computer science, Engineering, or a related field
8 + years’ experience in data engineering, ELT development, and data modeling.
Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
Demonstrated experience in developing custom connectors for data ingestion from various sources.
Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
Experience implementing DataOps principles and practices, including data CI/CD pipelines.
Excellent problem-solving and troubleshooting skills, with a strong attention to detail.
Effective communication and collaboration abilities, with a proven track record of working in cross-functional teams.
Familiarity with data visualization tools Apache Superset and dashboard development.
Understanding of distributed systems and working with large-scale datasets.
Familiarity with data governance frameworks and practices.
Knowledge of data streaming and real-time data processing technologies (e.g., Apache Kafka).
Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes.
Experience with Agile development methodologies and working in cross-functional Agile teams.
Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.
Excellent analytical and problem-solving skills, with a keen attention to detail.
Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.
Required Skills –
Python, Pyspark, Sql,Airflow, Trino, Hive, Snowflake, Agile Scrum
Good to have–
Linux,Openshift, Kubernentes, Superset
Compensation, Benefits and Duration
Minimum Compensation: USD 33,000
Maximum Compensation: USD 118,000
Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
This position is available for independent contractors
No applications will be considered if received more than 120 days after the date of this post
Similar Jobs
Big Data • Cloud • Productivity • Software • Database • Analytics • Automation
As a Senior Data Engineer at Jellyfish, you'll build and maintain data pipelines, optimize orchestration, automate CI/CD processes, and enhance data integration while ensuring high performance and reliability.
Top Skills:
AirflowBigQueryDagsterDatabricksDbtPrefectPysparkPythonRedisSnowflakeSQLTerraform
Automotive • Healthtech • Financial Services
The Senior Data Engineer will design and build scalable data platforms, implement data pipelines, and improve data operations using Azure and various modern tools.
Top Skills:
.Net Core.Net FrameworkAirflowAzure Cloud ServicesAzure Data Lake Storage Gen2Azure Devops PipelinesAzure SynapseC#CloudFormationDbtPower BIPythonSnowflakeSQLTerraform
Fintech • Payments • Financial Services
Lead Data Engineer responsible for defining architecture, engineering standards, and best practices for the enterprise data platform, collaborating with distributed teams for modernization and scalability.
Top Skills:
AirflowAurora PostgresqlAWSKafkaRedshiftSQL Server
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
