Senior Databricks Data Engineer

You could be the one who changes everything for our 28 million members by using technology to improve health outcomes around the world.  As a diversified, national organization, Centene's technology professionals have access to competitive benefits including a fresh perspective on workplace flexibility.
 

Position Purpose:

Develops and operationalizes data pipelines to make data available for consumption (reports and advanced analytics), including data ingestion, data transformation, data validation / quality, data pipeline optimization, and orchestration. Engages with the DevSecOps Engineer during continuous integration and continuous deployment.

• Helps guide the design and implementation of complex data management procedures around data staging, data ingestion, data preparation, data provisioning, and data destruction (scripts, programs, automation, assisted by automation, etc.)

• Provides guidance to data engineers in the design, development, implementation, testing, documenting, and operating of large-scale, high-volume, high-performance data structures for business intelligence analytics

• Designs, develops, and maintains real-time processing applications and real-time data pipelines

• Ensure quality of technical solutions as data moves across Centene’s environments

• Provides senior level knowledge and insight into the changing data environment, data processing, data storage, and utilization requirements for the company and drives the team toward viable solutions

• Develops, constructs, tests, and maintains architectures using advanced programming language and tools

• Drives ways to improve data reliability, efficiency, and quality and deploys a solution; use data to discover tasks that can be automated

• Performs other duties as assigned

• Complies with all policies and standards

Education/Experience:

A Bachelor's degree in a quantitative or business field (e.g., statistics, mathematics, engineering, computer science) and Requires 4 – 6 years of related experience.
Or equivalent experience acquired through accomplishments of applicable knowledge, duties, scope and skill reflective of the level of this position.
Technical Skills:

• One or more of the following skills are desired.

• Experienced in implementations using Databricks Lakehouse Medallion architecture and Unity Catalog (for data governance, lineage tracking, and access control).

• Skilled in building and optimizing complex data pipelines using PySpark,  SQL and/or Python on Databricks (or other comparable modern cloud DBMS preferably on AWS).

• Proficient in handling relational, flat files and hierarchical (JSON, XML) data formats on cloud platforms (AWS, Azure, GCP).

• Skilled in deploying production-grade pipelines with Databricks workflows (leveraging CI/CD – Gitlab, DAB etc.),

• Experienced in data pipeline optimization strategies, development of automated monitoring.

• Familiarity with Databricks API integrations and Connector tools for BI platforms.

Soft Skills:

• Problem Solving: Demonstrated ability to analyze and resolve intermediate to complex data engineering challenges using a data driven approach; seeks guidance as needed.

• Self-starter: Self-motivated to learn and apply new technologies. Demonstrates curiosity and handles ambiguity well.

• Communication: Capable of translating technical concepts into appropriate specs/insights for technical and non-technical stakeholders..

• Project Management: Demonstrated track record of accountability in producing own deliverables. Expected to show resilience even under pressure

• Collaboration: Cross-functional collaboration across technical and business teams to  produce successful outcomes.

• Accuracy & Governance: Attention to detail in ensuring high quality results.

Centene offers a comprehensive benefits package including: competitive pay, health insurance, 401K and stock purchase plans, tuition reimbursement, paid time off plus holidays, and a flexible approach to work with remote, hybrid, field or office work schedules.  Actual pay will be adjusted based on an individual's skills, experience, education, and other job-related factors permitted by law, including full-time or part-time status.  Total compensation may also include additional forms of incentives. Benefits may be subject to program eligibility.

Centene is an equal opportunity employer that is committed to diversity, and values the ways in which we are different. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or other characteristic protected by applicable law.

Qualified applicants with arrest or conviction records will be considered in accordance with the LA County Ordinance and the California Fair Chance Act