AI/ML Engineer

About the Role

We are hiring AI/ML Engineers to build, optimize, and advance the quality and safety systems at the core of the Mentiora platform. You will develop evaluation methodologies, fine-tune models and judges, and create tools that quantify and continuously improve AI agent behavior at production scale, across providers.

You'll work directly with a founding team of senior engineers from Google DeepMind, Google, Zalando, and IBM Research. As an early hire, you'll have high autonomy and visibility, and play a central role in product, research, and engineering strategy.

What We Offer

• Competitive compensation
• Access to compute and tooling resources from top-tier partners
• Opportunity to shape core product strategy
• A collaborative, impact-driven team culture

Responsibilities

• Implement and optimize Mentiora's evaluation, safety, and autorating systems
• Build ML pipelines for data collection, processing, and model benchmarking
• Develop tools for prompt engineering, fine-tuning, and quality assessment
• Design and implement novel evaluation and improvement methodologies for AI agents in production
• Develop new approaches for multi-model routing, complexity prediction, and quality estimation
• Monitor and improve model performance, latency, and cost metrics
• Collaborate with engineering teams to productionize and validate research findings
• Publish or present impactful findings to advance the field
• Example tasks
  • Optimize meta-prompts for LLM judge prompt generation to fit the labeled data.
  • Build a robust asynchronous evaluation pipeline using LLM judges.
  • Generate synthetic data for fine-tuning using client-provided evaluation criteria and labeled datasets.
  • Optimize client agent prompts using client-provided metric definitions.
  • Fine-tune judge models to increase performance and decrease costs.

Minimum Qualifications

• Bachelor's, Master's, or PhD in Computer Science, Machine Learning, Applied Mathematics, or a related field
• 3+ years of professional experience in AI/ML engineering, data science, or research
• Strong programming skills in Python and ML frameworks (PyTorch, TensorFlow, or similar)
• Experience with NLP and deep learning systems
• Knowledge of cloud platforms and ML infrastructure
• Understanding of ML model deployment, monitoring, and evaluation practices, including LLM-as-a-judge approaches
• Strong analytical and problem-solving skills, with excellent communication abilities

Preferred Qualifications

• Experience with prompt engineering, or model evaluation
• Background in building production ML systems at scale
• Knowledge of transformer architectures and modern NLP techniques
• Experience with MLOps tools and practices
• Familiarity with distributed computing and large-scale experiments
• Publications or significant contributions in AI/ML research (NeurIPS, ICML, ACL, EMNLP, etc.)
• Previous experience at an AI/ML-focused startup or research lab