About the Role

We are hiring AI/ML Engineers to build, optimize, and advance our Mentiora AI inference and evaluation systems. You will work on implementing machine learning models, developing evaluation pipelines, and creating tools that quantify and improve Mentiora AI quality across multiple providers.

You'll work directly with the founding team. As an early hire, you'll have high autonomy and visibility, and play a central role in product, research, and engineering strategy.

What We Offer

• Competitive compensation

• Access to compute and tooling resources from top-tier partners

• Opportunity to shape core product strategy

• A collaborative, impact-driven team culture

Responsibilities

• Implement and optimize Mentiora AI evaluation frameworks and autorating systems

• Build ML pipelines for data collection, processing, and model benchmarking

• Develop tools for prompt engineering, fine-tuning, and quality assessment

• Design and implement novel evaluation methodologies and metrics

• Develop new approaches for multi-model inference routing and quality prediction

• Monitor and improve model performance, latency, and cost metrics

• Collaborate with engineering teams to productionize and validate research findings

• Publish or present impactful findings to advance the field

• Example tasks

  • Optimize meta-prompts for LLM judge prompt generation to fit the labeled data.

  • Build a robust asynchronous evaluation pipeline using LLM judges.

  • Generate synthetic data for fine-tuning using client-provided evaluation criteria and labeled datasets.

  • Optimize client agent prompts using client-provided metric definitions.

  • Fine-tune judge models to increase performance and decrease costs.

Minimum Qualifications

• Bachelor's, Master’s, or PhD in Computer Science, Machine Learning, Applied Mathematics, or a related field

• 3+ years of professional experience in AI/ML engineering, data science, or research

• Strong programming skills in Python and ML frameworks (PyTorch, TensorFlow, or similar)

• Experience with NLP and deep learning systems

• Knowledge of cloud platforms and ML infrastructure

• Understanding of ML model deployment, monitoring, and evaluation practices, including LLM-as-a-judge approaches

• Strong analytical and problem-solving skills, with excellent communication abilities

Preferred Qualifications

• Experience with prompt engineering, or model evaluation

• Background in building production ML systems at scale

• Knowledge of transformer architectures and modern NLP techniques

• Experience with MLOps tools and practices

• Familiarity with distributed computing and large-scale experiments

• Publications or significant contributions in AI/ML research (NeurIPS, ICML, ACL, EMNLP, etc.)

• Previous experience at an AI/ML-focused startup or research lab