The Databricks Data Engineer role involves building ETL/ELT pipelines, working with data transformations, and ensuring efficient data handling across cloud services, primarily using Databricks and AWS.
Databricks Data EngineerWho we are
Why you will love working with us/ Perks
Horizon Industries Limited (Horizon) is a dynamic IT and Management Consulting firm based in the Washington, DC area. Founded in 1996, Horizon Industries Limited (Horizon) has evolved into a company of highly qualified professionals that provides full-cycle IT Consultant and Management support in both the private and public sectors.
Horizon’s culture is based on our core values, and we PRIDE ourselves on a diverse, employee, and family-centric approach.
Who we are looking for
We are seeking a Databricks Data Engineer for a full-time position supporting our various clients
We are committed to your growth and success in the IT software development industry. We are looking for a Databricks Data Engineer ready to assume leadership responsibilities in a supportive team environment focused on your success as a function of client success.
What your day-to-day will look like / Responsibilities:- Build end-to-end implementation of multiple ETL/ELT pipelines, demonstrating efficient data transformation and ingest patterns to move raw data from data producers to an enterprise data ecosystem, with a focus on performance and reliability
- Assess and understand the ETL jobs, workflows, BI tools, and reports
- Address technical inquiries concerning customization, integration, enterprise architecture and general feature / functionality of data products
- Experience in crafting database / data warehouse solutions in the cloud (Preferably AWS, Azure, Alternatively GCP)
- Key must have skill sets – Python, SQL, Databricks, AWS Data Services
- Experience with message queuing, stream processing, and highly scalable ‘big data’ data stores
- Experience manipulating, processing, and extracting value from large, disconnected datasets
- Experience manipulating structured and unstructured data for analysis
- Experience with data modeling tools and processes
- Experience aggregating and transforming data from multiple datasets to create data products
- Support an Agile software development lifecycle
- Ability to hold a position of public trust with the US government.
- S. in Computer Science or equivalent
- 4+ years' experience in data engineering and big data. At least 2 years' of professional services experience interacting directly with clients.
- Big data tools: Hadoop, Spark, Kafka, etc.
- Relational SQL and NoSQL databases and experience working with relational databases
- AWS cloud services: EC2, S3, RDS, Glue, Step Functions, Lamda, EMR, DynamoDB, DocumentDB, Redshift, Aurora, Athena
- Data Platforms: Databricks
- Data streaming systems: Batch,Kafka, Storm, Spark-Streaming, etc.
- Languages: Python, R, Scala, Go
- Ability to inspect existing data pipelines, discern their purpose and functionality, and re-implement them efficiently in Databricks.
- Extensive knowledge of data warehousing concepts and hands-on experience deploying pipelines using Databricks **a must
- Data modeling and database design skills and knowledge of version control
- Excellent verbal and written communication skills
- Experience architecting scalable and fault-tolerant data solutions across Azure, AWS, and Databricks
- Databricks Data Engineer Professional certification a plus.
- Preference for candidates with Databricks Professional certifications
Education: Bachelor’s degree in a technical field such as software engineering, engineering, computer science, etc.
Location : Horizon is headquartered in the heart of Tyson’s Corner, VA. Occasional travel may be required.
Why you will love working with us/ Perks
- A comprehensive benefits package including healthcare (medical, dental, vision and disability)
- a 401k program where you are 100% vested from day one with an employer match after 90 days.
- an Educational Assistance program.
- a Student Loan Repayment Program
- Gym Reimbursement Program.
- Paid Time off
- Dynamics, passionate, multi-disciplinary team of creative minds to work with, and many more.
Horizon is an Equal Employment Opportunity employer, and it is our policy to consider all applicants for employment. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, or national origin.
EOE/Vet/Disabled
Similar Jobs
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Senior Data Engineer on PwC's Managed Data, Analytics & Insights team to design, build and manage advanced data ecosystems. Responsibilities include designing data solutions and scalable pipelines, solving complex problems, mentoring junior staff, maintaining high delivery standards, and building client relationships while aligning solutions to business context.
Top Skills:
DatabricksKafka
Information Technology • Software
Design, build, and optimize cloud-native Databricks Lakehouse data pipelines on AWS. Implement governance (Unity Catalog, lineage, retention), medallion architecture, data quality frameworks, and performance tuning. Enable secure analytics, semantic layers, and self-service reporting. Collaborate with architects, cloud and cybersecurity teams, and produce documentation, operational procedures, and platform modernization deliverables.
Top Skills:
Auto LoaderAws AthenaAws CloudwatchAws GlueAws IamAws LambdaAws RedshiftAws S3CdcDatabricksDatabricks SqlDelta Live TablesDelta SharingPythonSQLStreamingUnity Catalog
Information Technology
Lead design and delivery of Databricks-based Lakehouse solutions. Architect, develop, test, and deploy scalable ETL/ELT pipelines using Python, PySpark, and SQL. Mentor engineers, enforce Databricks best practices, integrate with cloud platforms (Azure/AWS), support CI/CD, perform code reviews, and collaborate with data scientists and stakeholders to meet analytics and AI requirements.
Top Skills:
AdfAdlsAWSAws LambdaAzureAzure DevopsCi/CdData LakehouseDatabricksDatabricks Asset BundlesEltETLEvent HubGitGitGlueJIRAMedallion ArchitectureNotebook RepositoriesPostgresPysparkPythonRdbmsS3SQLUnit Testing Frameworks
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
