Automated Essay Scoring

Holistic and analytic scoring of student essays using a deep-neural approach to automated essay scoring (Uto, arXiv 2022). Annotators provide overall quality ratings, holistic scores on a 1-6 scale, and detailed feedback comments for educational assessment.

Configuration Fileconfig.yaml

# Automated Essay Scoring
# Based on Uto, arXiv 2022
# Paper: https://arxiv.org/abs/2012.13422
# Dataset: https://www.kaggle.com/c/asap-aes
#
# Score student essays on overall quality using a Likert scale,
# provide a holistic numeric score via slider, and give written feedback.
#
# Scoring Rubric (1-6):
# 1 - Very Poor: Off-topic, incoherent, or extremely limited
# 2 - Poor: Minimal development, significant errors
# 3 - Below Average: Some development but lacks clarity or organization
# 4 - Average: Adequate development with some errors
# 5 - Good: Well-developed, clear, minor errors
# 6 - Excellent: Exceptional development, sophisticated, virtually error-free
#
# Guidelines:
# - Read the essay prompt topic first, then the full essay
# - Consider organization, development, language use, and mechanics
# - Use the full range of the scoring scale
# - Provide constructive feedback explaining your score

annotation_task_name: "Automated Essay Scoring"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: likert
    name: essay_quality
    description: "Rate the overall quality of this essay (1 = Very Poor, 6 = Excellent)"
    size: 6
    min_label: "Very Poor"
    max_label: "Excellent"

  - annotation_type: slider
    name: holistic_score
    description: "Provide a holistic score for the essay on a 1-6 scale"
    min_value: 1
    max_value: 6
    starting_value: 3

  - annotation_type: text
    name: feedback
    description: "Provide constructive feedback on the essay. Comment on strengths, weaknesses, and suggestions for improvement."

annotation_instructions: |
  You will read student essays written in response to a given prompt topic.

  For each essay:
  1. Read the prompt topic to understand what the student was asked to write about.
  2. Read the full essay carefully.
  3. Rate the overall quality on a 1-6 Likert scale (Very Poor to Excellent).
  4. Provide a holistic score using the slider (1-6).
  5. Write constructive feedback explaining your assessment.

  Consider these dimensions:
  - Organization and structure
  - Development of ideas and arguments
  - Language use and vocabulary
  - Grammar, spelling, and mechanics
  - Adherence to the prompt topic

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 14px; margin-bottom: 16px;">
      <strong style="color: #a16207;">Prompt Topic:</strong>
      <p style="font-size: 15px; line-height: 1.6; margin: 6px 0 0 0;">{{prompt_topic}}</p>
    </div>
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px;">
      <strong style="color: #0369a1;">Student Essay:</strong>
      <p style="font-size: 16px; line-height: 1.8; margin: 8px 0 0 0; white-space: pre-wrap;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

[
  {
    "id": "essay_001",
    "text": "Computers have changed the way we live in many ways. First, we can communicate with people all over the world through email and social media. Second, computers help us learn new things by providing access to information on the internet. We can research any topic and find answers quickly. Third, computers are used in businesses to keep track of important data and make work easier. However, some people think computers are bad because we spend too much time on them. I think the benefits outweigh the negatives because computers help us in education, work, and staying connected.",
    "prompt_topic": "Write an essay explaining how computers have changed the way people live. Use specific details and examples to support your response."
  },
  {
    "id": "essay_002",
    "text": "I think that patience is really important and here is why. When your patient you dont get mad at little things. Like when your in traffic and everybody is honking there horns, a patient person just waits calmly. Also in school when you dont understand something, being patient helps you keep trying until you get it right. My grandma always says patience is a virtue and I think shes right because everything good takes time.",
    "prompt_topic": "Write about patience. Being patient means that you are understanding and tolerant. Describe a situation in which patience is important."
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/text/education/automated-essay-scoring
potato start config.yaml

Details

Annotation Types

likertslidertext

Domain

NLPEducation

Use Cases

Essay ScoringWriting AssessmentEducational NLP

Related Designs

Coreference Resolution (OntoNotes)

Link pronouns and noun phrases to the entities they refer to in text. Based on the OntoNotes coreference annotation guidelines and CoNLL shared tasks. Identify mention spans and cluster coreferent mentions together.

likertradio

FinBERT - Financial Headline Sentiment Analysis

Classify sentiment of financial news headlines as positive, negative, or neutral, based on the FinBERT model (Araci, arXiv 2019). Annotators also rate market outlook on a bearish-to-bullish scale and provide reasoning for their sentiment judgment.

radiotext

FLORES - Machine Translation Quality Estimation

Machine translation quality assessment using the FLORES-101 benchmark (Goyal et al., TACL 2022). Annotators rate translation quality on a Likert scale, identify error categories, and provide detailed error notes.

likerttext