Senior Cloud & ML Ops Engineer - Databricks

Please note that this position is not open to anyone on a H-1B or F-1 student visa including those eligible for CPT/OPT or the Stem OPT extension.

The Senior Cloud & ML Ops Engineer builds upon the expertise of Engineer II by leading high-impact automation and cloud optimization projects, ensuring scalability, security, and efficiency in AI/ML workbenches. This role focuses on enhancing data accessibility, integrating advanced AI-driven automation, and mentoring junior engineers.

This senior role requires a deep understanding of cloud infrastructure, workflow automation, and AI/ML platform governance. The Senior Engineer is expected to own critical production pipelines, improve CI/CD processes for ML models, and implement best-in-class security frameworks for data science environments.

Why This Role Matters:

The Senior Cloud & ML Ops Engineer plays a critical role in enabling enterprise-scale AI/ML innovation by ensuring that data science teams have a secure, scalable, and high-performing infrastructure. By leading automation efforts, optimizing cloud environments, and mentoring the next generation of engineers, this role directly impacts the efficiency and success of AI-driven business strategies.

Position Responsibilities:

• Lead automation initiatives to improve the scalability and efficiency of AI/ML workflows.
• Architect and maintain highly available cloud-based data science environments on platforms like Databricks and Snowflake.
• Enhance monitoring and observability for AI/ML infrastructure, ensuring performance optimization and cost efficiency.
• Improve and enforce security and compliance measures across data science environments.
• Develop and refine CI/CD pipelines to streamline model deployment and management in production.
• Collaborate with data scientists and engineers to drive innovation and operational excellence.
• Mentor junior engineers by providing guidance on infrastructure best practices, cloud security, and automation.
• Optimize cloud cost management strategies to ensure efficient resource utilization.

Minimum Qualifications Required:

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or related field.
• 5+ years of experience in cloud-based infrastructure management and AI/ML workbench administration.
• Expertise in Databricks, and cloud-based data platforms (AWS, Azure, GCP).
• Strong programming skills in Python, SQL, and automation scripting (Terraform, Bash, or similar).
• Experience with workflow orchestration tools such as Apache Airflow or Prefect.
• Deep understanding of cloud security, IAM roles, and governance best practices.
• Proven ability to lead projects and mentor junior engineers.

Nice to Have (Preferred Qualifications):

• Certifications: Azure/AWS/GCP Data Architect, Databricks Advanced Developer, with additional credentials in cloud-based data engineering, scalable pipeline design, and end-to-end data integration.
• Experience integrating machine learning models into production pipelines.
• Proficiency in Kubernetes, Docker, and containerized AI/ML workloads.
• Experience working with real-time data streaming technologies such as Kafka or Kinesis.
• Strong knowledge of FinOps for cloud cost optimization.
• Strong regulatory modeling experience.

#LI-JB3