The AI Data Engineer evaluates and validates AI agent output, creates datasets for testing, implements evaluation automation, and communicates technical findings to stakeholders.
Veeva Systems is a mission-driven organization and pioneer in industry cloud, helping life sciences companies bring therapies to patients faster. As one of the fastest-growing SaaS companies in history, we surpassed $2B in revenue in our last fiscal year with extensive growth potential ahead.
At the heart of Veeva are our values: Do the Right Thing, Customer Success, Employee Success, and Speed. We're not just any public company – we made history in 2021 by becoming a public benefit corporation (PBC), legally bound to balancing the interests of customers, employees, society, and investors.
As a Work Anywhere company, we support your flexibility to work from home or in the office, so you can thrive in your ideal environment.
Join us in transforming the life sciences industry, committed to making a positive impact on its customers, employees, and communities.
The Role
This role is responsible for ensuring the reliability, accuracy, and safety of our Veeva AI Agents through rigorous evaluation and systematic validation methodologies. We're looking for experienced candidates with:
1. A meticulous, critical, and curious mindset with a dedication to product quality in a rapidly evolving technological domain
2. Exceptional analytical and systematic problem-solving capabilities
3. Excellent ability to communicate technical findings to both engineering and product management audiences
4. Ability to learn application areas quickly
Thrive in our Work Anywhere environment: We support your flexibility to work remotely or in the office within Canada or the US, ensuring seamless collaboration within your product team's time zone.
Join us and be part of a mission-driven organization transforming the life sciences industry.
What You'll Do
- Evaluation Strategy & Planning: Define and establish comprehensive evaluation strategies for new AI Agents. Prioritize the integrity and coverage of test data sets to reflect real-world usage and potential failure modes
- LLM Output Integrity Assessment: Programmatically and manually evaluate the quality of LLM-generated content against predefined metrics (e.g., factual accuracy, contextual relevance, coherence, and safety standards)
- Creating High-Fidelity Datasets: Design, curate, and generate diverse, high-quality test data sets, including challenging prompts and scenarios. Evaluate LLM outputs to proactively identify system biases, unsafe content, hallucinations, and critical edge cases
- Automation of Evaluation Pipelines: Develop, implement, and maintain scalable automated evaluations to ensure efficient, continuous validation of agent behavior and prevent regressions with new features and model updates
- Root Cause Analysis: Understand model behaviors and assist in the trace and root-cause analysis of identified defects or performance degradations
- Reporting & Performance Metrics: Clearly document, track, and communicate performance metrics, validation results, and bug status to the broader development and product teams
Requirements
- Data Integrity & Validation: A strong, specialized understanding of data quality principles, including methods for validating datasets against bias, integrity concerns, and quality standards. Ability to craft diverse and adversarial test data to uncover AI edge cases
- Prompt Engineering & Model Expertise: Demonstrated skill in advanced prompt engineering techniques to create evaluation scenarios that test the AI's reasoning, action planning, and adherence to system instructions. Deep knowledge of LLM common failure modes (hallucination, incoherence, jailbreaking)
- Automated Evaluation Implementation: 5+ years of experience designing and deploying automated evaluation pipelines to assess complex, agentic AI behaviors. Familiarity with quality metrics such as task success rate, semantic similarity, and sentiment analysis for output measurement
- Debugging Agentic Systems: Must be comfortable with the specific challenges of debugging agentic systems, including tracing and interpreting an agent's internal reasoning, tool use, and action sequence to pinpoint failure points
- Programming & Frameworks: 5+ years of experience using Python to develop custom evaluation frameworks, writing scripts, and integrating pipelines with CI/CD systems. Familiarity with standard test automation tools (e.g., Pytest, modern web automation tools)
- Bachelor's degree in Data Science, Machine Learning, Computer Science, or a related field, with experience in Gen AI / LLMs
- High work ethic. Veeva is a hard-working company
- High integrity and honesty. Veeva is a PBC and a “do the right thing” company. We expect that from all employees
- Applicants must have the unrestricted right to work in the United States or Canada. Veeva will not provide sponsorship at this time
Learn More
- Engineer Perspective: 3 Reasons to Consider Veeva
- Engineering at Veeva
Perks & Benefits
- Medical, dental, vision, and basic life insurance
- Flexible PTO and company paid holidays
- Retirement programs
- 1% charitable giving program
Compensation
- Base pay: $85,000 - $225,000
- The salary range listed here has been provided to comply with local regulations and represents a potential base salary range for this role. Please note that actual salaries may vary within the range above or below, depending on experience and location. We look at compensation for each individual and base our offer on your unique qualifications, experience, and expected contributions. This position may also be eligible for other types of compensation in addition to base salary, such as variable bonus and/or stock bonus.
#LI-Remote
#LI-MidSenior
Veeva’s headquarters is located in the San Francisco Bay Area with offices in more than 15 countries around the world.
Veeva is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, sex, sexual orientation, gender identity or expression, religion, national origin or ancestry, age, disability, marital status, pregnancy, protected veteran status, protected genetic information, political affiliation, or any other characteristics protected by local laws, regulations, or ordinances. If you need assistance or accommodation due to a disability or special need when applying for a role or in our recruitment process, please contact us at [email protected].
Top Skills
Ci/Cd
Pytest
Python
Web Automation Tools
Veeva Boston, Massachusetts, USA Office
Our main office is located in downtown Boston and is within walking distance to great restaurants and cafes. Few minute walk to three T Stations.
Similar Jobs
Fintech • Mobile • Software • Financial Services
The Senior Data Engineer will design, build, and maintain data solutions for risk management, focusing on data warehousing and pipeline development. They will mentor other engineers and drive data strategy.
Top Skills:
AirtableAnsibleApache AirflowApache KafkaCloudFormationDbtGithub ActionsGitlab Ci/CdPythonSnowflakeSQLTerraform
Software
As a Senior AI/Data Engineer, you'll build scalable data systems for decision-making products, enhance AI/ML capabilities, and lead cross-functional collaborations to deliver customer-centric data solutions.
Top Skills:
AirflowAWSAzureBigQueryDatabricksDbtGCPPythonRedshiftScikit-LearnSnowflakeSQL
Healthtech • Insurance
Responsible for designing and deploying AI solutions using low-code tools, integrating AI models into applications, and collaborating with teams to enhance workflows.
Top Skills:
AWSAzureCopilot StudioDatabricks AppsDjangoFastapiFlaskGCPMicrosoft Power AppsPythonSQLStreamlitUipath
What you need to know about the Boston Tech Scene
Boston is a powerhouse for technology innovation thanks to world-class research universities like MIT and Harvard and a robust pipeline of venture capital investment. Host to the first telephone call and one of the first general-purpose computers ever put into use, Boston is now a hub for biotechnology, robotics and artificial intelligence — though it’s also home to several B2B software giants. So it’s no surprise that the city consistently ranks among the greatest startup ecosystems in the world.
Key Facts About Boston Tech
- Number of Tech Workers: 269,000; 9.4% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Thermo Fisher Scientific, Toast, Klaviyo, HubSpot, DraftKings
- Key Industries: Artificial intelligence, biotechnology, robotics, software, aerospace
- Funding Landscape: $15.7 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Summit Partners, Volition Capital, Bain Capital Ventures, MassVentures, Highland Capital Partners
- Research Centers and Universities: MIT, Harvard University, Boston College, Tufts University, Boston University, Northeastern University, Smithsonian Astrophysical Observatory, National Bureau of Economic Research, Broad Institute, Lowell Center for Space Science & Technology, National Emerging Infectious Diseases Laboratories
