Skip to content

09 / Research & calibration

An evidence program still to be earned.

No result has been earned yet. CoAcademic has published no agreement studies, reports no accuracy figures, and will not publish numbers before they exist. This is the methods sheet for the program intended with founding pilot course teams.

01Status

What has and has not been done

Being explicit here matters more than looking established. Nothing below has produced a result.

Status: no calibration study has been completed. No agreement, accuracy, bias, or time-savings figure exists, and none is claimed anywhere on this site.

02Methods

The studies to be run

Planned product

  1. 01

    Agreement against faculty marks

    Compare proposed bands to educator-approved outcomes across real rubrics and disciplines, reporting distribution and disagreement patterns rather than a single headline figure.

  2. 02

    Rubric-fidelity testing

    Check that proposals cite criteria and passages actually present in the submission, and flag any proposal that reasons outside the confirmed rubric.

  3. 03

    Inter-rater baseline

    Establish how much human markers already disagree on the same scripts, so any claim about the system is measured against a real baseline rather than an assumed one.

  4. 04

    Fairness review

    Examine whether proposals vary systematically across writing styles, language backgrounds, and disciplinary conventions.

  5. 05

    Failure analysis

    Catalogue where proposals are wrong and why, and publish the categories alongside any positive result.

  6. 06

    Open method notes

    Publish methodology, sample sizes, and limitations with every reported result. A result without its limitations is marketing.

03Calibration in practice

Calibration is a faculty activity

Founding pilot scope

The calibration CoAcademic supports is not a statistical adjustment applied behind the scenes. It is the conversation in which faculty agree what a band means, recorded so it can be applied consistently and revisited later.

The Founding Assessment Assurance Pilot

One course team, one assignment, one confirmed rubric — audited, calibrated, reviewed against evidence, and decided by faculty. Enrolment is not open; the page explains the intended scope and what a pilot would require from both sides.

Read the pilot scope