ML QA Analyst

ML QA Analyst

Boston (Hybrid)

About OnCorps
OnCorps LLC is a trailblazer in the financial services sector, harnessing the power of advanced AI and machine learning technologies to revolutionize the industry. Our innovative solutions empower financial institutions to enhance efficiency, security, and customer satisfaction – all in support of our mission to reshape financial services through technology. 

The Opportunity

This role is central to how our AI systems learn, improve, and perform. As an ML QA Analyst, you’ll work under the Data Science umbrella and own key tasks across model evaluation, training, and ongoing maintenance for the pipelines and work products built by our DS team. You’ll validate agent behavior, strengthen prompts, ensure high-quality labeled data, and keep our models running reliably in production.

It’s an ideal fit for someone with a STEM background, strong analytical instincts, and a passion for precision who wants hands-on experience in a cutting-edge AI environment. This role also offers a clear growth path—successful team members often progress into Data Science roles or transition into Product as Business/Product Analysts.

Key Responsibilities

• Labeling: Owning data labeling for PDF / Excel-based ML ingestion engines across the company. This includes both training-based labeling and live-run corrective labeling.
• Agent Evaluation Review: Review evaluation jobs where agents are underperforming, digging into individual logs and traces and providing documentation of possible prompt weaknesses.
• Agent Regression Testing: Interacting with customer facing “chat” agents. Validating the integrity of an agent’s response to both “happy-path” and adversarial prompts pre-deployment & when model updates are in testing.
• Dashboarding: Owning client-specific dashboard deployments. Working with product/CS stakeholders to deploy client-specific visualizations. Testing dashboards for errors / logical gaps.
• Maintenance-Mode Model Upkeep: Initiating and monitoring routine model retraining for models in a steady-state & fixing job failures in production. Working with Data Science to determine appropriate thresholds for escalation. 
• AI Table Annotation
  
  

Requirements

• STEM BA/BS
• High attention to detail
• Ability to think critically and analytically
• Ability to follow and understand a complex ruleset
• Familiarity with LLMs/Agents is a plus
• Knowledge of SQL is a plus
• Knowledge of Python is a plus
• Comfortable working in fast-paced environments and meeting tight deadlines
• U.S. based with work authorization
  
  

Work Environment

• Primarily desk-based role during standard hours, utilizing PC, video, and phone
• Requires flexibility to work in-office and remotely based on business needs
  

Why You Should Join Us

Innovative Environment: Work with cutting-edge AI and machine learning technologies in a company that is at the forefront of financial services innovation.
Growth Opportunities: As a rapidly growing startup, we offer significant opportunities for professional development and career advancement.
Collaborative Culture: Be part of a collaborative and inclusive team where your ideas are valued, and your contributions make a real impact.
Work-Life Balance: Enjoy the flexibility of a hybrid working environment, allowing you to balance your professional and personal life effectively.
Competitive Compensation: We offer a competitive salary and a comprehensive benefits package.

Apply today and be a part of the future of financial services technology!

If you are passionate about solutions engineering and eager to work in a cutting-edge AI-driven company, we would love to hear from you!

OnCorps LLC is an equal opportunity employer committed to diversity and inclusion. We do not discriminate based on race, religion, color, national origin, gender, sexual orientation, pregnancy, age, marital status, veteran status, or disability status. We provide reasonable accommodations for persons with disabilities. Applicants with equal employment opportunity questions should contact OnCorps at [email protected] or (617) 286-4448. Retaliation against applicants raising EEO issues is prohibited.