Prometheus: Rubric-based LLM Evaluation

[
  {
    "id": "prom_001",
    "text": "Explain the difference between machine learning and deep learning to someone with no technical background.",
    "response": "Machine learning is like teaching a computer to learn from examples. Imagine showing a computer thousands of photos of cats and dogs - it learns patterns to tell them apart. Deep learning is a specific type of machine learning that uses layers of processing, like how your brain has layers of neurons. Think of machine learning as the big umbrella, and deep learning as one powerful tool under that umbrella. Deep learning is especially good at complex tasks like understanding images and language because it can automatically discover the important features to look for.",
    "rubric": "Score 1: The explanation uses heavy jargon and technical terms without simplification, making it inaccessible to non-technical readers.\nScore 2: The explanation attempts to simplify but still relies on some technical terms without adequate explanation.\nScore 3: The explanation uses some analogies but may be inconsistent in accessibility or miss key differences.\nScore 4: The explanation uses clear analogies and simple language to convey both concepts and their relationship, with only minor unclear points.\nScore 5: The explanation uses excellent analogies, is fully accessible to non-technical readers, clearly distinguishes the two concepts, and explains their relationship intuitively.",
    "reference_answer": "Think of machine learning as teaching a computer to make decisions by showing it lots of examples, similar to how a child learns. You show it many pictures of apples and oranges, and eventually it can tell them apart on its own. Deep learning is a more advanced version of this - it's inspired by how our brains work, using layers of artificial 'neurons.' It's particularly good at complex tasks like recognizing faces in photos or understanding spoken words. The key difference: regular machine learning often needs humans to point out what features to look for (like 'look at the color and shape'), while deep learning figures out the important features on its own."
  },
  {
    "id": "prom_002",
    "text": "Write a professional summary for a software engineer's resume with 5 years of experience in backend development.",
    "response": "Experienced software engineer with 5 years of professional experience specializing in backend development. Proficient in Java, Python, and Node.js with expertise in designing and implementing RESTful APIs, microservices architectures, and database optimization. Demonstrated track record of improving system performance by up to 40% through code optimization and infrastructure improvements. Strong collaborator with experience leading cross-functional teams in Agile environments. Passionate about writing clean, maintainable code and implementing best practices in CI/CD pipelines.",
    "rubric": "Score 1: The summary is generic, lacks specificity, uses first person, or reads like a job description rather than a personal summary.\nScore 2: The summary mentions relevant skills but lacks quantifiable achievements and uses vague language.\nScore 3: The summary includes relevant technologies and some achievements but could be more specific or impactful.\nScore 4: The summary is well-structured, includes specific technologies, quantifiable achievements, and conveys professional value clearly.\nScore 5: The summary is concise yet comprehensive, includes specific technologies and metrics, demonstrates clear value proposition, uses strong action-oriented language, and is tailored for a backend role.",
    "reference_answer": "Results-driven backend engineer with 5+ years building scalable, high-performance systems serving millions of users. Expert in Java and Python microservices, with hands-on experience designing distributed systems using AWS, Kubernetes, and PostgreSQL. Led migration of monolithic application to microservices architecture, reducing deployment time by 60% and improving system reliability to 99.9% uptime. Skilled in mentoring junior developers and driving engineering best practices across Agile teams."
  }
]

// ... and 8 more items