Senior Associate - Security Data Lake Engineer

Location Designation: Hybrid - 3 days per quarter
As part of Technology, you'll have the opportunity to contribute to groundbreaking initiatives that shape New York Life's digital landscape. Leverage cutting-edge technologies like Generative AI to increase productivity, streamline processes, and create seamless experiences for clients, agents, and employees. Your expertise fuels innovation, agility, and growth - driving the company's success.
Role Overview
New York Life is seeking a Security Data Lake Engineer to design, implement, and optimize the enterprise-scale security data lake and advanced analytics capabilities. This role will serve as a technical specialist responsible for building scalable data pipelines, enabling data science-driven use cases, and ensuring the long-term health and cost-effectiveness of the data lake.
The Security Data Lake Engineer will collaborate with SOC Analysts, Threat Intelligence, Data Scientists, and other Cybersecurity teams to unlock new detection and analytics capabilities, including AI-driven anomaly detection, semantic search, and retrieval-augmented generation (RAG). This role is ideal for an engineer who thrives at the intersection of big data, cloud services, and applied data science, with a focus on enabling cybersecurity outcomes.
What You'll Do:
Data Lake Engineering & Stewardship
- Architect, build, and manage data pipelines across AWS services (S3, Kinesis, Lambda), APIs, and Cribl
- Ensure data lake performance and usability through indexing, query optimization, and aggregation pipeline management for 3.5TB+ daily volumes.
- Act as the dedicated owner to maintain data quality and prevent "data swamp" degradation.
- Establish a Data Lake Center of Excellence (C4E) to empower decentralized analytics development while maintaining governance and standards.
Security Analytics & Data Science Enablement
- Collaborate with security teams to define, develop, and deploy data-driven use cases.
- Build ML pipelines for anomaly detection, vector search, semantic search, and other advanced security analytics.
- Integrate and optimize Elasticsearch, vector databases, and KNN/percolator search capabilities.
- Develop and maintain Jupyter Notebook workflows for data wrangling, modeling, and analysis.
Innovation & AI Integration
- Apply machine learning and natural language processing techniques, including retrieval-augmented generation (RAG).
- Partner with AI adoption initiatives to ensure secure and effective use of enterprise data for AI/ML.
- Stay current with emerging technologies and techniques in data science, big data, and cloud analytics.
Collaboration & Knowledge Retention
- Work closely with SOC Engineers, Threat Hunters, and data platform teams to integrate log sources and telemetry into the data lake.
- Provide guidance and mentorship across Cyber teams on building analytics use cases.
- Serve as the in-house expert to reduce reliance on external consulting spend, ensuring institutional knowledge is retained within NYL.
What You'll Bring:
- 5+ years of hands-on experience in data engineering, big data platforms, or applied data science.
- Proficiency with AWS services (S3, Kinesis, Lambda, Bedrock) and enterprise-scale analytics design.
- Strong programming skills in Python and/or Scala; experience with Jupyter Notebooks.
- Experience with indexing, query optimization, and data pipeline optimization at scale (3.5TB+ daily).
- Advanced knowledge of Elasticsearch including advanced search methods (KNN vector search, percolator reverse search), and performance tuning, platform maintenance, etc.
- Experience applying ML techniques for anomaly detection, semantic search, embeddings, and NLP.
Bonus Qualifications
- Familiarity with Cribl for data routing and transformation.
- Advanced degree in mathematics, computer science, or related empirical sciences.
- Previous experience in cybersecurity environments or willingness to learn security context quickly.
Education & Certifications
- Bachelor's degree in Computer Science, Data Science, Cybersecurity, or related field; advanced degree preferred.
- Certifications in AWS Data Engineering, Machine Learning, or Big Data technologies a plus.
Pay Transparency
Salary Range: $121,000-$172,500
Overtime eligible: Exempt
Discretionary bonus eligible: Yes
Sales bonus eligible: No
Actual base salary will be determined based on several factors but not limited to individual's experience, skills, qualifications, and job location. Additionally, employees are eligible for an annual discretionary bonus. In addition to base salary, employees may also be eligible to participate in an incentive program.
Company Overview
At New York Life, our 180-year legacy of purpose and integrity fuels our future. As we evolve into a more technology-, data-, and AI-enabled organization, we remain grounded in the values that drive lasting impact.
Our diverse business portfolio creates opportunities to make a difference across industries and communities-inviting bold thinking, collaborative problem-solving, and purpose-driven innovation. Here, you'll find the rare balance of long-standing stability and forward momentum, supported by an inclusive team that honors tradition while embracing progress.
As a Fortune 100 mutual company, we offer a place to grow your skills, contribute to meaningful work, and deliver solutions that matter. Your ideas drive what's next, and your growth powers it.
Our Benefits
We provide a full package of benefits for employees - and have unique offerings for a modern workforce, including leave programs, adoption assistance, and student loan repayment programs. Based on feedback from our employees, we continue to refine and add benefits to our offering, so that you can flourish both inside and outside of work.Click hereto discover more about our comprehensive benefit options or visit our NYL Benefits Site.
Our Commitment to Inclusion
At New York Life, fostering an inclusive workplace is fundamental to who we are and how we serve our communities. We have a longstanding commitment to creating an environment where individuals can contribute their best and succeed together. This foundation is rooted in our core values of humanity and integrity, ensuring that every employee feels valued and supported. By embracing a broad range of perspectives and experiences, we achieve greater success and fulfill our promise of providing financial security and peace of mind to families across all communities. Click here to learn more about New York Life's leadership in this space.
Recognized as one of Fortune's World's Most Admired Companies, New York Life is committed to improving local communities through a culture of employee giving and volunteerism, supported by the Foundation. We're proud that due to our mutuality, we operate in the best interests of our policy owners. To learn more about career opportunities at New York Life, please visit the Careers page of www.NewYorkLife.com.
Visit our LinkedIn to see how our employees and agents are leading the industry and impacting communities.
Visit our Newsroom to learn more about how our company is constantly evolving to meet our clients' and employees' needs.
Job Requisition ID: 92845