Design, develop, and optimize scalable batch and real-time data pipelines on GCP. Implement Medallion Architecture, build Python and PySpark transformations, optimize BigQuery and SQL workloads, deploy Dataflow pipelines, and orchestrate workflows with Cloud Composer and Airflow. Manage GCS storage, CI/CD, version control, IAM, data governance, security, reliability, scalability, and cost efficiency across enterprise data platforms.
This is a remote position.
KANINI is seeking a highly skilled Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and modern data architecture. The ideal candidate will have hands-on experience designing scalable data pipelines, implementing Medallion Architecture, and building robust enterprise-grade data solutions.
This role requires strong technical proficiency in BigQuery, PySpark, Dataflow, and Airflow, along with a solid understanding of cloud data governance, performance optimization, and CI/CD practices.
Key Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines on GCP
- Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data processing
- Build high-performance data transformations using Python and PySpark
- Develop and optimize complex SQL queries for analytical workloads
- Work extensively with BigQuery for large-scale data processing and performance tuning
- Develop and deploy pipelines using Cloud Dataflow
- Orchestrate workflows using Cloud Composer (Apache Airflow)
- Manage data storage and lifecycle using Google Cloud Storage (GCS)
- Implement version control and CI/CD pipelines using Git-based tools
- Ensure data security, governance, and access control using GCP IAM
- Optimize data solutions for performance, scalability, reliability, and cost-efficiency
Required Skills & Experience
- Strong hands-on experience with Google Cloud Platform (GCP)
- Expertise in BigQuery (partitioning, clustering, query optimization)
- Proven experience implementing Medallion Data Architecture
- Strong programming skills in Python and PySpark
- Hands-on exposure on Java
- Advanced proficiency in SQL (complex joins, window functions, performance tuning)
- Hands-on experience with Cloud Dataflow
- Experience with Cloud Composer (Airflow) for orchestration
- Experience working with Google Cloud Storage (GCS)
- Knowledge of version control systems (Git) and CI/CD practices
- Strong understanding of GCP IAM and cloud security best practices
Preferred Qualifications
- Experience working with large-scale enterprise data platforms
- Knowledge of data warehousing and data lake concepts
- Familiarity with real-time streaming frameworks
- Experience in data governance and data quality frameworks
- Exposure to
Similar Jobs
Artificial Intelligence • Cloud • Software • Infrastructure as a Service (IaaS)
Lead Runpod’s security organization and own the security posture of its cloud platform, infrastructure, and corporate environments. Responsibilities include security strategy, cloud and application security, SecOps, GRC, compliance certifications, secure software development, threat modeling, vulnerability management, incident response, vendor risk, security automation, and infrastructure security. Build and mentor a distributed security team while partnering with Engineering, Infrastructure, Product, GTM, and Legal to protect customer AI workloads and support scalable growth.
Top Skills:
Ci/CdCloud InfrastructureGpu ClustersHipaaIamIso 27001KubernetesLinuxNetworkingSoc 2VirtualizationZero-Trust Architecture
Consumer Web • Healthtech • Professional Services • Social Impact • Software
Build and lead product and application security efforts, partnering with Product and Engineering on secure design, development, code reviews, vulnerability discovery, and security guardrails. Develop AI-native security tooling and support incident response, vulnerability management, penetration testing, and security operations. Help establish secure development practices and scalable security systems across Headway’s platform.
Top Skills:
AWSDatadogEcsFargateFastapiGitKafkaLaceworkPagerdutyPostgresPython 3ReactRedisS3SemgrepSnykSparkSqlalchemyTypescript
Artificial Intelligence • Cloud • Consumer Web • Productivity • Software • App development • Data Privacy
Owns company-critical, long-term technical problems across multiple teams and organizations. Defines technical strategy, architectures, roadmaps, and engineering standards for large-scale customer-facing systems. Remains hands-on in software development while driving AI adoption in products and engineering workflows. Partners with senior product, engineering, and executive leaders to resolve complex problems, improve reliability and performance, and shape Dropbox’s technical direction.
Top Skills:
Agentic FrameworksAIConcurrencyDatabasesDistributed SystemsFrontend SystemsLlm ApisMlMobile SystemsSearch SystemsSoftware DevelopmentStorage Systems
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
