Thank you for your response to my request for more details of Generalized
Procrustes Analysis.
I must say. I'm stumped for a sensible response. You say that what the
technique does is to provide, inter al, a content analysis; but I don't
see how that's possible. I can see how the rotation of the element
relationship in search for similarities of ratings on the constructs used
will discover that my trivial example:
>> Bland 3 5 7 2 1 5 6 4 Flavoursome
>> Cheap 3 5 7 2 1 5 6 4 Expensive
shares variance and, if you want to define it in these terms, "meaning".
But it isn't capable of discovering that
>> Mythical 3 5 7 2 1 5 6 4 Historical
has a different meaning, as you say yourself:
>Highly correlated constructs were also carefully discussed to check for
>"spurious" correlations. So, your Mythical/Historical construct wouldn't be
included in the >grid in the first place!
What you're doing is to make a personal decision to exclude any construct
and ratings which would pose problems to your analytic procedure; and the
procedure itself is purely statistical, being incapable of measuring the
meanings being expressed in the content of the constructs themselves.
Okay. Let's start with the observation that you, as a researcher, are
using a technique which IMHO is incapable of assessing meaning (no wonder
you put inverted commas round words such as "spurious" correlation and
"true" correlation; dammit, my construct of "mythical - historical" is
certainly not spurious _to me_, and your technique isn't capable of
addressing that issue!) But, but...
... nevertheless, something very subtle and interesting is happening when
you use the technique.
_You yourself_ have some very well developed _personal_ standards which
you use to decide that a construct is unsuited to analysis by your
statistical technique: _you_ are carrying out a content analysis
yourself, before going on to use your technique on just those constructs
which your judgement tells you are amenable to the technique.
Now, my purpose is _not_ to decry the technique as being incomplete
(though if I were being PCP-purist I'd do so). But rather, it's to invite
you to consider what's going on when you carry out this personal content
analysis. What are the minimum things we might say about it?
a) It ought to be possible for you to articulate the personal standards
according to which you carry out this judgement.
b) These then become part of your analysis, and should be checked for
reliability against someone else's attempt to do the same thing. Do you
both agree on the rules you're using?
c) Your Generalized Procrustes Analysis will work very well on those
constructs on which you both agree and which are "true" in the way you've
defined it, but will be incapable of handling those constructs which
either one of you, or both, labels as "spurious" until you've carried out
this more qualitative analysis; and then it might be applicable or not:
you can tell far better than I!).
d) If not, then some other technique is required to handle the "spurious"
constructs, (assuming you accept that your approach ought to be capable
of handling these). No reason why you should, though speaking personally
I'm uncomfortable if I can't address these, since my own view is that
_no_ construct is "spurious" and I personally would be cheating to
characterise a construct as such. If you _do_ see it worth your while to
examine the way in which you're making these judgements, perhaps you'll
see it as worth your while to grapple with the rather more qualitative
analysis of content and meaning, and build these into your research
design.
e) The obvious measure of reliability to use in this situation would be
the Perrault and Leigh (1989) statistic.
Phew1 Sorry to be so long-winded about it!
Actually, if you're uncomfortable with the qualitative approach I'm
advocating above, there's another, partial solution to the measurement of
meanings which is rather more quantitiative. It has its artificialities,
but let's be pragmatic about it!
Drawing inter al. on Osgood, Borman has suggested that it is possible to
characterise the content or meaning of a construct, for
statistical-analytic purposes, in terms of the way in which that
_construct_ (not element, construct!) is rated on a standard set of
comparator traits which are assumed to be commonly understood by all
respondents. This allows him to recognise the similarity between the
meanings different people ascribe to a particular construct by
correlating the ratings different people give to that particular
construct, on the standard set of comparator traits. His approach does,
of course, simply drive the search for meaning into the background (since
he has to _assume_ that the comparator traits are all understood
identically by respondents!), but his approach has proved useful in
situations in which different people's construing of, as it happens,
subordinates' job performance, was involved. A pragmatic solution to a
practical problem; and it occured to me that, if you prefer a completely
quantitative approach, you might find it helpful and relevant.
The references are as follows:
Borman, W.C. "Implications of personality theory and research for the
rating of work performance in organizations" in Landy, F., Zedeck, S. &
Cleveland, J., (eds.) _Peformance Measurement and Theory_ Hilsdale, NJ:
Lawrence Erlbaum 1983.
Borman, W.C. "Personal constructs, performance schemata, and 'folk
theories' of subordinate effectiveness: explorations in an army officer
sample" _Organizational Behavior and Human Decision Processes_ 1987, 40,
307-322.
Perrault, W.D. Jnr. & Leigh, L.E. "Reliability of nominal data based on
qualitative judgements" _Journal of Marketing Research_ 1989, XXVI, May,
135-148.
NB Laurie Thomas has done some work on tea-tasting which, on the face of
it, may have some similarities with your own research. Laurie, are you
out there and looking in on this conversation?
Kind regards,
Devi Jankowicz
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%