Lead enterprise data engineering efforts supporting defense missions. Responsibilities include prioritizing technical tasks, mentoring engineers, conducting code reviews, translating government requirements into scalable data architectures, developing Databricks ETL/ELT pipelines with Python, PySpark, and SQL, troubleshooting pipeline performance and reliability, and documenting systems. The role collaborates with government stakeholders, product owners, data scientists, analysts, and technical teams to deliver secure, high-throughput data platforms and actionable insights.
About the Organization
Now is a great time to join Redhorse Corporation. We are a solution-driven company delivering data insights and technology solutions to customers with missions critical to U.S. national interests. We’re looking for thoughtful, skilled professionals who thrive as trusted partners building technology-agnostic solutions and want to apply their talents supporting customers with difficult and important mission sets.
About the Role
Redhorse transforms the way government uses data and technology. To support this mission, we are seeking a Senior Data Engineer / Lead to join our team supporting the US National Defense Strategy. In this role, you will play a vital role in translating complex defense requirements into scalable data pipelines and actionable insights, directly contributing to the modernization of critical defense systems.
Key Responsibilities
- Technical Leadership & Task Oversight: Provide day-to-day task prioritization and technical direction to team members; lead code reviews, ensure design best practices, and mentor engineers in modular Python development and SQL optimization.
- Requirements Development & Architecture: Work directly with government stakeholders to elicit, refine, and translate complex mission requirements into robust technical specifications, interface designs, and scalable data models.
- Pipeline & Platform Development: Architect, develop, and optimize high-throughput ETL/ELT workflows using Databricks; maintain, refactor, and enhance existing Python and SQL codebases for performance, stability, and security.
- Pipeline Diagnostics & Troubleshooting: Identify pipeline bottlenecks, debug complex data workflows, and implement automated solutions to enhance pipeline reliability and data integrity.
- Cross-Functional Collaboration: Partner closely with government product owners, data scientists, Data Analysts, and technical peers to document architecture, coordinate technical sprints, and communicate engineering trade-offs to technical and non-technical audiences.
- Technical Documentation: Develop new and maintain existing pipeline architecture documentation relating to pipeline code architecture and configurations to ensure high system reliability and seamless team onboarding.
Required Experience/Clearance
- Security Clearance: Must be a U.S. Citizen with a current, active U.S. Government Secret clearance. (Applicants without an active Secret clearance cannot be considered).
- Experience Level: Minimum of 8+ years of professional experience in data engineering, building enterprise-scale data platforms and pipelines.
- Technical Guidance: Demonstrated experience providing technical mentorship, performing code reviews, and providing day-to-day technical task oversight to other engineers.
- Databricks: Proven, hands-on experience designing and optimizing data transformation workflows in Databricks.
- Python/PySpark & SQL: Advanced skills in modular, object-oriented Python/PySpark scripting and extensive experience writing, tuning, and optimizing complex SQL queries.
- Data Visualization: Proven experience building functional, user-centric reports and dashboards using modern data visualization tools.
Desired Experience
- Hands-on experience with Apache Spark and distributed computing frameworks.
- Familiarity with modern agile and DevOps tools such as GitLab Jira.
- Experience working with DoD, defense, or intelligence community data environments and compliance standards.
- Knowledge of data governance, metadata management, and data cataloging frameworks.
- Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related technical discipline.
We encourage all candidates who meet the basic requirements to apply, even if you do not have any of the following experience:
The salary range provided for this position represents the anticipated base salary for successful candidates. Actual compensation will be determined based on a variety of factors, including relevant experience, education, certifications, skills, security clearance level, geographic location, market conditions, and internal equity. In addition to base salary, eligible employees may participate in Redhorse's comprehensive benefits programs and may be eligible for performance-based or other incentive compensation, where applicable.
Redhorse Corporation is an equal opportunity employer. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability, or any other protected class.
If you are a qualified individual with a disability or a disabled veteran, you may request a reasonable accommodation if you are unable or limited in your ability to access job openings or apply for a job on this site as a result of your disability. You can request reasonable accommodations by contacting Talent Acquisition at [email protected]
Redhorse Corporation shall, in its discretion, modify or adjust the position to meet Redhorse’s changing needs. This job description is not a contract and may be adjusted as deemed appropriate in Redhorse’s sole discretion.
Similar Jobs
Solar • Renewable Energy
Lead the rebuild and modernization of the company’s data platform by designing and implementing an Azure and Databricks lakehouse. Responsibilities include managing Azure Data Factory pipelines, developing Delta Lake architectures, creating data models and transformations with SQL and Python, establishing development standards, monitoring data quality, troubleshooting performance issues, leading technical projects, and mentoring or hiring data engineering staff.
Top Skills:
AirbyteAzureAzure Data FactoryCi/CdDatabricksDelta LakeEstuaryFivetranGitInformaticaMicrosoft DynamicsNetSuiteOraclePower BIPythonSAPSQLSsisUnity Catalog
Financial Services
Leads data engineering within an agile team, designing secure and scalable data platforms, pipelines, architectures, storage, access, analytics, governance, backup, and recovery strategies. Develops advanced data models, evaluates security controls, documents best practices, and drives responsible AI-assisted engineering adoption. Provides technical leadership through architecture decisions, mentoring, cross-team influence, and operational support using Python, Databricks, Spark, AWS, and modern lakehouse patterns.
Top Skills:
SparkAWSDatabricksLakehouse ArchitecturesNosql DatabasesPythonRelational DatabasesStreaming Architectures
Database • Analytics
Leads the design, development, optimization, and migration of modern data platforms in Snowflake. Builds pipelines with dbt, Python, and cloud services; gathers client requirements; creates solution designs and project plans; advises on Snowflake performance, cost, security, and workload management; integrates BI, governance, and AI/ML tools; contributes to accelerators and mentors junior engineers. The role is remote, client-facing, consultative, and focused primarily on technical delivery rather than sales.
Top Skills:
Apache AirflowAWSAzureCloverdxDbtFivetranGoogle Cloud PlatformInformaticaLookerPower BIPythonSnowflakeSnowparkSnowpipeSQL
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
