methodological question about difference scores

Bob Green (
Wed, 31 Mar 1999 20:48:46 +1000

I would appreciate advice on calculating the difference between two grids.
I am not familar with the relevant literature though would appreciate
references or comments:

I am aware of two methods (though there may be other methods) to calculate
the level of agreement between two persons who rated the same elements on
the same constructs. These methods which do not give the same results are:

(1) A difference grid (the absolute difference for each cell, summed by row
and column)

(2) Transforming ratings into a "disagreement score", e.g., a 0 if there was
agreement, 1 if person
used 4 and the other 1 or 2, and 2 if they were on opposite ends of the scale.

I also considered:

Bannister's consistency score
the possibility of joining the two grids and performing a cluster analysis etc
comparison on mean ratings for certain elements/constructs

Is there a methodologically preferable way of calculating difference (or
alternatively sameness).

Any advice is appreciated,

Bob Green