Hexaware Technologies Logo

Hexaware Technologies

Big Data Engineer

Posted One Month Ago
Remote
Hiring Remotely in United States
Senior level
Remote
Hiring Remotely in United States
Senior level
Build and maintain ETL and data pipelines using Python, PySpark, and AWS services. Orchestrate workflows, develop event-driven integrations, optimize data storage and queries, process API and JSON data, and implement data quality monitoring. Support CI/CD, production operations, cloud migration, data lake and lakehouse initiatives, and near-real-time systems while collaborating with stakeholders to deliver technical solutions.
The summary above was generated by AI

Build and maintain ETL pipelines using Python and PySpark on AWS Glue and related platforms. 

Orchestrate workflows using AWS Step Functions and Lambda. - Implement messaging and event-driven integrations using SNS and SQS. Design and optimize storage and querying solutions in Amazon Redshift, RDS, Oracle and S3-based architectures. 

Write efficient SQL for transformations, validation, and reporting. - Integrate data from APIs and process structured and semi-structured JSON data. - Implement data quality checks, monitoring, and operational support processes. 

Participate in CI/CD and version control practices for deployment and release management. 

Collaborate with cross-functional teams to translate business requirements into technical solutions. 

8 years of software development experience across the appropriate platform. 

Strong hands-on experience with Python, PySpark, API’s and SQL. 

Experience with ETL/data pipeline development and Orchestration using Step functions / AirFlow .

 Working knowledge of AWS services including Glue, Lambda, Step Functions, Redshift, S3, SNS, and SQS. 

Experience with Athena, EMR, Kinesis, DynamoDB, or RDS. - Good Knowledge on CloudWatch, logging, and production support. 

Understanding of data warehousing, data lakes, Lake House and query optimization. 

Experience with GitLab/Terraform or similar and CI/CD workflows. 

Good understanding of using AI tools like Github Copilot or similar for code productivity .

Exposure to enterprise data lake or cloud migration initiatives. 

Have an eye to solving complex problems, great communication with stakeholders .

Have a good understanding of performance engineering of code pipelines and near real time systems - Good understanding on Agents and MCP".

A Bachelor’s Degree in Information Technology, Computer Science, Business Administration, or a related field is required.

Hexaware Technologies Boston, Massachusetts, USA Office

Boston, United States

Similar Jobs

6 Days Ago
In-Office or Remote
90K-110K Annually
Senior level
90K-110K Annually
Senior level
Artificial Intelligence • Information Technology • Software • Consulting
Designs, builds, and operates large-scale Hadoop-based data pipelines and analytics platforms. Responsibilities include ingesting, transforming, and analyzing structured and unstructured data; developing Spark applications; managing streaming systems; working with relational and NoSQL stores; orchestrating workflows; troubleshooting distributed systems; and supporting reliable, scalable production data platforms. Preferred experience includes cloud Hadoop platforms, lakehouse formats, data governance, Kubernetes, CI/CD, and infrastructure as code.
Top Skills: AirflowApache AtlasApache HudiApache IcebergSparkAws EmrAzure HdinsightCi/CdCollibraDatabricksDelta LakeFlinkHadoopHbaseHdfsHiveInfrastructure As CodeJavaKafkaKubernetesNosql DatabasesOoziePythonRelational DatabasesScalaShellSpark StreamingSQLSqoopTrino
One Month Ago
In-Office or Remote
132K-170K Annually
Senior level
132K-170K Annually
Senior level
Insurance
Designs, develops, and maintains large-scale data processing systems, including ETL processes, data models, scalable pipelines, databases, data warehouses, and cloud-based big data solutions. Uses Snowflake, Fivetran, dbt, and Airflow while applying DevOps and security practices. Optimizes distributed processing and storage systems, collaborates with data scientists and analysts, and mentors junior engineers.
Top Skills: Apache AirflowDbtFivetranNoSQLSnowflakeSQL
13 Days Ago
Remote
Georgia, USA
Senior level
Senior level
Travel
Own and lead the big data infrastructure platform, including deploying, upgrading, and operating HDFS, object storage, Iceberg, monitoring, and alerting. Drive infrastructure modernization, migrate Iceberg catalogs, implement multi-tenant quotas, improve observability, and support large-scale data architecture changes. Collaborate with a parallel infrastructure engineer, share solutions, provide coverage, and influence platform architecture.
Top Skills: Apache IcebergApache OzoneFlinkGravitinoHadoopHdfsIceberg Rest CatalogsKafkaLakekeeperMinioS3Trino

What you need to know about the Boston Tech Scene

Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.

Key Facts About Boston Tech

  • Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
  • Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
  • Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
  • Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account