Inter-Rater Reliability in Psychology: Definition & Formula

An error occurred trying to load this video.

Try refreshing the page, or contact customer support.

Coming up next: The Affordable Care Act's Impact on Mental Health Services

You're on a roll. Keep up the good work!

Take Quiz Watch Next Lesson
Your next lesson will play in 10 seconds
  • 0:00 What Is Inter-Rater…
  • 1:23 Calculating…
  • 1:54 Cohen's Kappa
  • 3:58 Spearman's Rho
  • 4:49 Lesson Summary
Save Save Save

Want to watch this again later?

Log in or sign up to add this lesson to a Custom Course.

Log in or Sign up

Speed Speed

Recommended Lessons and Courses for You

Lesson Transcript
Instructor: Orin Davis
We use inter-rater reliability to ensure that people making subjective assessments are all in tune with one another. Generally measured by Spearman's Rho or Cohen's Kappa, the inter-rater reliability helps create a degree of objectivity.

What Is Inter-Rater Reliability

How, exactly, would you recommend judging an art competition? After all, evaluating art is highly subjective, and I am sure that you have encountered so-called 'great' pieces that you thought were utter trash. So, how can a pair of judges possibly determine which piece of art is the best one? Especially if each judge has a different opinion, bias, et cetera, it may seem at first blush that there is no fair way to evaluate the pieces.

That's where inter-rater reliability (IRR) comes in. Inter-rater reliability is a level of consensus among raters. In the case of our art competition, the judges are the raters. Even though there is no way to describe 'best,' we can give the judges some outside pieces that they can use to calibrate their judgments so that they are all in tune with each other's performances.

For example, we can ask them to rate the pieces on aspects like 'originality,' 'caliber of technique,' and one or two other aspects that contribute to whether a piece of art is good. We can then determine the extent to which the judges agree on their ratings on the calibration pieces, and compute the IRR. Based on that measure, we will know if the judges are more or less on the same page when they make their determinations and as a result, we can at least arrive upon a convention for how we define 'good art' this competition, anyway.

Calculating Inter-Rater Reliability (IRR)

While there are many ways to compute IRR, the two most common methods are to use Cohen's Kappa and Spearman's Rho. Cohen's Kappa is used when the rating is nominal and discrete (e.g., yes/no; note that order doesn't matter), and essentially assesses the extent to which judges agree relative to how much they would agree if they just rated things at random. Spearman's Rho is used for more continuous, ordinal measures (e.g., scale of 1-10), and reflects the correlation between the ratings of judges.

Cohen's Kappa

Suppose we asked two art judges to rate 100 pieces on their originality on a yes/no basis. For each piece, there will be four possible outcomes: two in which they agree (yes-yes; no-no), and two in which they disagree (yes-no; no-yes). Let's say that they both called 40 pieces 'original' (yes-yes), and 30 pieces 'not original' (no-no). For another 10 pieces, Judge A said 'original' while Judge B disagreed, and for the other 20 pieces, Judge B said 'original' while Judge A disagreed.

Based on this, the judges agree on 70/100 paintings, or 70% of the time. But what are the odds of the judges agreeing by chance?

From the results, we also see that Judge A said 'original' for 50/100 pieces, or 50% of the time, and said 'not original' the other 50% of the time. Judge B however, declared 60 pieces 'original' (60%), and 40 pieces 'not original' (40%).

To unlock this lesson you must be a Member.
Create your account

Register to view this lesson

Are you a student or a teacher?

Unlock Your Education

See for yourself why 30 million people use

Become a member and start learning now.
Become a Member  Back
What teachers are saying about
Try it risk-free for 30 days

Earning College Credit

Did you know… We have over 200 college courses that prepare you to earn credit by exam that is accepted by over 1,500 colleges and universities. You can test out of the first two years of college and save thousands off your degree. Anyone can earn credit-by-exam regardless of age or education level.

To learn more, visit our Earning Credit Page

Transferring credit to the school of your choice

Not sure what college you want to attend yet? has thousands of articles about every imaginable degree, area of study and career path that can help you find the school that's right for you.

Create an account to start this course today
Try it risk-free for 30 days!
Create an account