Railroad19
Jobs
Senior Data Engineer (GCP • Python • Iceberg • Delta Lake • Kafka • Snowflake • Databricks)
Railroad19
Senior Data Engineer (GCP • Python • Iceberg • Delta Lake • Kafka • Snowflake • Databricks)
Design and build modern GCP lakehouse architectures, including Python and Spark pipelines, BigQuery ingestion, Kafka CDC, Delta Lake and Iceberg UniForm integrations, Delta Sharing endpoints, Snowflake and Databricks connectivity, data lineage, governed access controls, and semantic-layer alignment. Collaborate cross-functionally, troubleshoot full-stack issues, and deliver reusable data-sharing adapters and end-to-end data features.
Railroad19, Inc is seeking a Senior Data Engineer with deep, hands‑on experience in building modern lakehouse architectures on GCP. This role focuses on designing, developing in Python & Spark, and delivering reusable data‑sharing adapters that connect BigQuery‑backed data products to Snowflake and Databricks using Iceberg UniForm and Delta Sharing.
About Railroad19:
At Railroad19, Inc, we develop customized software solutions and provide software development services. We’re a specialized team of developers and architects. As such, we only bring an “A” team to the table, through hard work and a desire to lead the industry — this is our company culture — this is what sets Railroad19 apart.
As a Railroad19 employee, you will be part of a company that values your work and gives you the tools you need to succeed. Our headquarters is in Saratoga Springs, New York, but this position is 100% remote. Railroad19 provides competitive compensation and excellent benefits, including Medical/Dental/Vision/Pet Insurance, Paid Time Off, and 401 (k).
NO 1099, C2C, Corp-to-Corp; only full-time employment.
NO Agencies.
Core Responsibilities:
- Design and implement the UniForm write layer (Delta + Iceberg dual metadata).
- Build GCS → BigQuery ingestion pipelines for structured operational datasets.
- Develop and implement in Python and Spark.
- Implement Kafka-based CDC patterns for real-time and near-real-time ingestion.
- Develop data lineage, dependency tracking, and modular adapter code.
- Configure Snowflake Horizon external tables for zero-copy reads.
- All data hub tables are to be written once using Delta Lake with Iceberg UniForm enabled… readable by all target consumers without conversion.
- Implement and certify Delta Sharing endpoints for Databricks consumers.
- Build governed access layer components: RBAC, connector registry entries, tenant-scoped authorization.
- Align semantic layer models with LookML and KPI catalog definitions.
- Collaborate with cross‑functional teams to deliver end‑to‑end features.
- Troubleshoot issues across the full stack and contribute to code quality.
Skills/Experience:
- 6+ years of proven enterprise-level experience in Python & Spark
- Advanced experience in GCP BigQuery
- Strong working knowledge of Apache Iceberg, Delta Lake, Iceberg UniForm
- Experience with Delta Sharing; Kafka / CDC pipelines
- Specific work experience in Snowflake Horizon Catalog; Databricks Unity Catalog
- Solid experience with Data lake architecture & ingestion pipeline design
- Excellent Communication skills and the ability to work cohesively with multiple teams.
Preferred Experience – Nice to Have
- Prior delivery in enterprise SaaS, media, or advertising technology.
- Active daily use of AI-assisted development tools (Claude Code preferred).
Similar Jobs
Artificial Intelligence • Cloud • Software • Infrastructure as a Service (IaaS)
Lead Runpod’s security organization and own the security posture of its cloud platform, infrastructure, and corporate environments. Responsibilities include security strategy, cloud and application security, SecOps, GRC, compliance certifications, secure software development, threat modeling, vulnerability management, incident response, vendor risk, security automation, and infrastructure security. Build and mentor a distributed security team while partnering with Engineering, Infrastructure, Product, GTM, and Legal to protect customer AI workloads and support scalable growth.
Top Skills:
Ci/CdCloud InfrastructureGpu ClustersHipaaIamIso 27001KubernetesLinuxNetworkingSoc 2VirtualizationZero-Trust Architecture
Consumer Web • Healthtech • Professional Services • Social Impact • Software
Build and lead product and application security efforts, partnering with Product and Engineering on secure design, development, code reviews, vulnerability discovery, and security guardrails. Develop AI-native security tooling and support incident response, vulnerability management, penetration testing, and security operations. Help establish secure development practices and scalable security systems across Headway’s platform.
Top Skills:
AWSDatadogEcsFargateFastapiGitKafkaLaceworkPagerdutyPostgresPython 3ReactRedisS3SemgrepSnykSparkSqlalchemyTypescript
Artificial Intelligence • Cloud • Consumer Web • Productivity • Software • App development • Data Privacy
Owns company-critical, long-term technical problems across multiple teams and organizations. Defines technical strategy, architectures, roadmaps, and engineering standards for large-scale customer-facing systems. Remains hands-on in software development while driving AI adoption in products and engineering workflows. Partners with senior product, engineering, and executive leaders to resolve complex problems, improve reliability and performance, and shape Dropbox’s technical direction.
Top Skills:
Agentic FrameworksAIConcurrencyDatabasesDistributed SystemsFrontend SystemsLlm ApisMlMobile SystemsSearch SystemsSoftware DevelopmentStorage Systems
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
