Issues for Knowledge Management from
Experiences in Supporting Group Knowledge Elicitation & Creation
in Ill-defined, Emerging Situations
John T. Nosek
Computer & Information Sciences, Temple University
Philadelphia, PA 19122
Michael D. McNeese
Armstrong Laboratory, Fitts Human Engineering Division
AL/CFHI, Bldg 248, 2255 H Street
Wright Patterson AFB, OH 45433-7022
Knowledge management involves both the elicitation/creation of knowledge, and the storage/retrieval of meaningful knowledge in future states. In ill-defined, emerging situations, team members grope to make sense of the situation, react to stimuli from the external environment, and interact with each other and human artifacts to develop an interpretation of the environment. A general model of the group knowledge elicitation and creation process is presented and lessons derived from experiences in supporting this process are offered.
While organizational environments become more complex and dynamic, there exists limited time to exploit an opportunity or solve a problem. Real time decision makers under stress, for example, combat fighter pilots, come across a fast unfolding situation and must quickly make an acceptable decision. They focus on assessing the situation and taking acceptable actions that present themselves - optimization is not really possible nor sought. Because of increasing time pressures to act quickly, organizational decision makers are beginning to find themselves in a similar position to real time decision makers -- situation assessment is critical and acceptable actions are sought. We are not trying to say that fighter pilots and organizational decision makers work within the same split second time frames, however, the changes for fighter warriors and organizational decision makers can inform our understanding of group sensemaking. For example, with the increasing use of ìreal time information in the cockpitî, pilots will be able to view advanced imagery and communicate/collaborate using highly sophisticated technologies. As these advancements become prevalent, the view of the pilot as individual warfighter becomes obsolete, and barriers between real time and organization decision making breakdown. Within this complex, broad bandwidth decision space, there are many possible actor-to-actor or actor-to-agent couplings that underlie group sensemaking.
Complex, ill-structured, situation domains span such areas as command and control, business planning, new-product design, process reengineering, and information systems development. However, in all cases, the growing importance of group sensemaking, i.e., the elicitation and creation of group knowledge relevant to an emerging situation, is becoming clear -- those that do a better job of it will have a better chance of survival and increased competitive advantage.
Participants engaged in group knowledge elicitation and creation in these ill-structured domains, whether distributed or not, face the existence of multiple and conflicting interpretations about an emerging situation. They are not certain about what questions to ask, and if questions are posed, no clear answer is forthcoming (Daft and Lengel, 1986). They grope through a recursive, discontinuous process of many difficult steps subjected to interference, feedback loops, and dead ends that more closely resembles fermentation than an assembly line (Mintzberg et al. 1976). The participants' experience-based intuitive understanding may depend on insufficient or no-longer-relevant experience(Dreyfus & Dreyfus, 1986). All those involved must effectively and efficiently scan for and filter relevant information to create and maintain a shared mental model. Shared mental models have the problem of knowledge or truth maintenance in that the information that was true for yesterday (or even an hour ago) may have decayed, have subtle changes, or may have demonstrably changed. These changes occurring over the entire decision space can play havoc with meaning, interpretations, and choice of actions, and highlight the need for conflict resolution, multi-source sensemaking, and the social construction of knowledge.
Knowledge management involves both the elicitation/creation of knowledge and the subsequent retrieval of meaningful knowledge in future situations. Therefore, group knowledge that is elicited and created must be stored in a manner that it can be accessed in a meaningful way in future situations requiring group knowledge elicitation and creation. This strong, cyclical relationship between the elicitation/creation of group knowledge and the storage/retrieval of relevant knowledge in future situations demands a better understanding of how to support group knowledge elicitation and creation. This paper discusses some issues that have emerged in our efforts to support this process.
Figure 1 depicts the basic components of a general
model of group knowledge elicitation and creation. Each of the
following subsections briefly discuss these components:
Group. There are
myriad group characteristics which affect the process and subsequent
outcomes, and not all will be discussed. For example, individual
member characteristics may complicate the process when personality,
job status or political agenda introduce conflict (Dennis et al.
1988), and greater numbers of participants multiply these individual
effects. Group size affects productivity both positively and
negatively (Applegate, 1991). Larger groups increase process
complexity with more inputs, viewpoints and expertise to be processed
(Nunamaker et al. 1990). At the same time, the greater expertise of
larger groups should improve task-related outcomes. Some group
attributes can reduce process complexity, such as cohesiveness and
positive previous work experience as a group (Nunamaker et al.
associated with ill-defined, emerging situations can be highly
complex, and task complexity is a key component of group process
complexity. One can relate the cognitive effort of information
processing tasks to the nature of the task, rather than to the
capabilities of the problem solvers, the number of problem solvers,
or process characteristics. Complex tasks are non-routine with more
uncertainty and which require new knowledge and unique solutions.
Wood (1987) rates complex tasks according to the number of acts or
information cues, and according to relationships among acts and
information cues, as well as changes among them.
Process support includes tools, techniques, and methods which can be
used to support the group knowledge elicitation and creation
Interactions of Group, Task, and Process
Support Within a Context. The role of
group process loss/gain (Steiner, 1972), as a function of the
collective induction (Laughlin, 1989) that ensues in group
experiences, impacts the level and quality of the knowledge elicited
and created. It is important to consider the extent that these
factors are interrelated with task and process support considerations
while keeping in mind that the environment determines what is
possible and what is constrained when process models are invoked.
Outcomes. Outcomes of the group knowledge elicitation and creation process are categorized as task-related; process-related; and group-related.
The authors have attempted to employ various technologies in a variety of situations over a number of years to support group knowledge elicitation and creation. One author has employed leading edge mobile (notebooks, wireless LANS, and WAN connectivity), group support technology (cognitive mapping, document sharing, and desktop conferencing) to provide any-time, any-place support for information systems development (Nosek & Mandviwalla, 1996). The other author has actively employed the AKADAM cognitive engineering methods to find better ways to support future fighter pilots in air warfare (McNeese, Zaff, Citera, Brown, & Whitaker, 1995). The development and use of the AKADAM method has evolved and been iterated many times for different case studies that require individual and/or groups to elicit and create knowledge. In these iterations, depending on the situations encountered, concept mapping, design storyboarding, and IDEF functional decomposition (in various combinations and in decreasing order of frequency and success) were used. While these experiences are diverse, the following sections summarize the lessons learned, and discuss issues to improve knowledge management.
As reported in McNeese, Zaff, Citera, Brown, & Whitaker (1995) there are three principles for user-centered / group centered approaches to knowledge elicitation: a) actively employ a means of shared communication, b) facilitate the unconstrained expression of knowledge, c) generate knowledge representations compatible with the capabilities, limitations, and needs of the stakeholders. A basic tenet underlying all of these principles is that individual and group knowledge building is highly participatory and evolutionary often feeding off successive loops of recursive understanding. In some instances, ìtoo many cooks have spoiled the broth; while in other situations ìtwo heads have amounted to be more than oneî. Part of the wisdom associated with knowledge elicitation practices is knowing which of these adages applies to the situation under consideration. There have been many lessons learned especially regarding the constraints of these techniques that can and have been used to continuously improve application effectivity. These lessons successfully show the interdependence of group, task, and process variables with varying constraints on stimulus and environmental conditions as defined by the actual context of knowledge elicitation activities. In this sense, one might say they are derived from "knowledge elicitation in the wildî considerations. Many of the following lessons learned have been selected from the case studies reported in the McNeese et al. (1995) review and are geared toward use of the concept mapping typology as a knowledge management tool:
* We have found that many of our sessions start off with questions asked of experts that are too broad in scope (as defined by the top level node of the map). This ends up creating fractionated knowledge, only attending to surface-level knowledge while never progressing to deep-seated knowledge, and typically results in excessive time expenditures to complete the map. This becomes problematic if the session is time-limited - which has a net effect of just skimming off the surface of the map. Too, if the participants come from varied multi-disciplinary perspectives, then many of these problems actual get worse and group knowledge elicitation is even more ìout of focusî.
* Dependent on the application for which we are trying to prune knowledge, the imposed representational typology / knowledge elicitation structure can be a plus or a minus for group knowledge buildup. That is, flexibility (as demonstrated with concept maps) can lead to too much free association / subject fading effect, in contrast to a more well-defined structure (e.g., the IDEF functional decomposition), that often inhibits the ìgenerative-active learning processî that is so important for collective induction in the group. On the other hand, the use of concept maps and/or semantic network structures may have difficulty with team probabilities / fuzziness, conditional attributes, and time progressions. This may come into play for those situations that are heavily reliant on procedural knowledge and inferential use of nodes.
These considerations are also affected by individual differences of the experts being mapped as well. Many engineers seem to like the strict, logical progression of an IDEF decomposition whereas many of the pilots we have mapped prefer the ease of representing their knowledge in the concept map. We have opted towards use of concept mapping for more generalized, front end knowledge elicitation that involve groups, but reserve the IDEF for highly specific, well-defined areas, where the experts tend to have a more homogenous makeup.
* One area of concern within the representational typology is that of addressing semantic descriptors that have dual meanings / multiple connotations. Many of the computer-based elicitation tools we have used have not been ìsmartî enough to interpret meaning, e.g., one participant may use ëinterfaceí to refer to a computer interface while other might interpret this as the ëinterfaceí between the contractor and government entities. This problem is comes into play when individual maps are combined to form summary maps, or when group mapping unfolds as different participants invoke different meanings / interpretations of what is being communicated and then translated into the map. Much of the interpretation is done intuitively by looking at clusters of context to distill the associative context that surrounds individual nodes on the map.
* Another consideration involves the extent to which the knowledge in a group-centered map is scaleable or is ìtransferableî to similar - yet different situations. When one conducts a group session, domain specific knowledge is from variant sources which may make sense for the situation at hand. However, when transferred to a similar situation, much of the variance of the initial map may become highly salient and inappropriate.
* The quantitative / qualitative evaluation of map content is difficult. Our overall focus has been to have multiple sessions to ìsearch for invarianceî across all individual participant maps and then compare to the group-centered map. In this way, comparisons / contrasts can be made that help an evaluation team make recommendations regarding the extent to which elements of the map approach veridicality.
* Other randomly occurring problems that can incur are:
Most of the challenges identified here are problematic for group elicitation and knowledge building. However, we have found that the potential gain in collective induction, and consequently the improved robustness in knowledge buildup, is generally worth the effort. In some cases, problems are intractable and have not yet been solved while in other situations remedies have been successfully implemented. This is a new area, so there are many lessons yet to be learned in the future.
To date, IT products to support the group knowledge elicitation and creation process have been inadequate and the following must be addressed in parallel:
The intent of this section is to present some of the issues that must be addressed in theory development in order to provide better support for group knowledge elicitation and creation. It offers some of our thoughts on the issues that have evolved from our efforts to understand and support this process, rather than any thorough examination of the subject which is beyond the scope of this paper. It is through this reflection and some of our efforts that have convinced us of the need for a multi-disciplinary, multi-theoretic approach to the subject. It also seems that the problem with such an approach is not only the barrier of different languages among people of multiple disciplines, but more importantly, the difficulty and rarity of actually thinking of the question from another discipline's perspective.
In ill-defined emerging situations the assumptions about the environment change from analyzable to unanalyzable. The source of data moves from internal, impersonal to external, personal, and the data acquisition techniques move from regular, routine, reports, many obtained from organizational information systems, to irregular reports from external contacts and feedback from the environment (Daft and Weick, 1984; Dreyfus and Dreyfus, 1986). "The decision maker is faced with a "corporate primordial soup" of customer, industry and technology news; assessments of the news; competitor moves; agent's call reports; alliances; rumors; deals coming and done; problems and solutions; suggestions and scenarios (Lee and Brookes, 1993)." In these situations, organizations actual "create" rather than "discover" their environment through their interpretation process (Daft and Weick, 1984).
In Figure 2, team members engaged in groping to making sense of the situation, react to stimuli from the external environment and interact with each other (verbally & non-verbally) and human artifacts to develop an interpretation of the environment. In fact, there are multiple possible environments, but through this group sensemaking process where they elicit knowledge from various sources and create new knowledge from synthesizing during their analysis process, they operationalize ("create") an environment in which to act. By acting on this interpretation of the environment and finding success in their actions, they have created an environment that validates their particular interpretation. For example (you may want to try this yourself), take a game of 20 questions where one person normally selects in advance some item, while others have up to 20 questions to discern what the item is. Modify this game so that the person does not pre-select an item, but the other people in the group do not know this. As the group asks 20 questions of this person, the person provides an answer that is consistent with the previous question asked. When the group comes up with some answer like "sky", the person responds "correct" because their sensemaking process has produced this answer.
This process involves a complex interaction of cognitive activities by participants engaged in collaboration with others within a context they create through this group interpretation process. Researchers from multiple disciplines, stretching their own cognitive muscles by reflecting on these issues from other disciplines, will help to develop theories to better manage the full cycle of knowledge elicitation/creation and storage/retrieval.
In parallel with theory development, process support changes must be made to provide real value to way groups create and elicit knowledge from one another. While some argue that we must understand the group process of eliciting and creating knowledge before developing tools, from experience, parallel development seems to make more sense. First, the use of the tools can help understand the complex process of group knowledge elicitation and creation, at the same time, the use of tools can actually change the process of group knowledge elicitation and creation. The following sections describe some of the issues related to process support which must be addressed to develop more useful technology:
Need for Better Tool Integration. Each tool available has been built as a separate system. Some problems that have occurred because of this include:
New Visions of Group Interaction. New visions of group interaction must consider social, technological, ecological, and cognitive factors involved in real world collaborations. Situated cognition (Suchman, 1987), communities of mutual learning (Lave, 1991), and in situ understanding of context predominate in guiding our understanding of knowledge building and generative learning in collaborative situations and developing group/computer interfaces. McNeese (1993) has shown the advantages of perceptual learning and collective induction upon knowledge formation, and how these advantages help individual group members transfer knowledge across analogical situations. These findings, in consort with naturalistic case studies involving knowledge building (e.g., group concept mapping), suggest development of group interfaces that may be different from traditional group decision support systems (e.g., GroupSystems, Nunamaker et al. 1991). Traditional supports impose a 'distributed metaphor' by connecting individuals together 'as if' they are engaged in problem solving at remote sites. Interaction is tunneled through computer workstations / individual screens typically enriched with structured tools and voting support. In contrast, a group interface is one in which members are perceptual anchored, face-to-face, and interaction is through mutually accessible group display surfaces. We must look beyond the promise of a strict, distributed metaphor and focus on a naturalistic understanding of the work context as a means to develop important theoretical constructs and the consequent design of group / computer interfaces. Some specific other considerations for group interfaces include:
Group Centered Development of Process Support
Technologies. A user-centered approach means that the userís
domain expertise will have to be incorporated directly into the
design. It also means, in the strictest sense, acquiring knowledge
about the user, the userís work domain, and the userís
design requirements directly from the user. Adapting this approach to
groups means that the design of collaborative technologies for a
given application would necessarily involve the direct participation
of the team for whom the system is being designed (see for comparison
Olson & Olson, 1990). In order to support a user-centered
approach to the design of collaborative technologies, it becomes
necessary to acquire knowledge about the group, the groupís
work domain, and the groupís design requirements directly from
the group. This is in accordance with Grudinís (1988)
observation that CSCW designs need to become more
ëgroup-friendlyí. Hence, a user-center approach adapted
to the design of collaborative technologies is more accurately
referred to as a group-centered approach (McNeese et al. 1992).
More theory-based development of process support technology. While this is not a new criticism of technology, it is still true and worthy of consideration. A more methodical mapping of tool features to our hopefully increasing understanding of the group knowledge elicitation and creation process will allow us to make more substantial gains in process support technology.
Knowledge management involves both the elicitation/creation of knowledge, and the storage/retrieval of meaningful knowledge in future states. In ill-defined, emerging situations, team members grope to make sense of the situation, react to stimuli from the external environment, and interact with each other and human artifacts to develop an interpretation of the environment. A general model of the group knowledge elicitation and creation process for a given situation was presented and lessons derived from experiences in supporting this process were offered. Future gains in knowledge management depend on multi-disciplinary theory development and process support technologies more closely tied to these emerging theories.
Partial support for this work was provided by the National Science Foundation's Division of Undergraduate Education through grant DUE-9452594. Additional support for this work was provided by grants from Lotus Development Corporation, Bell Atlantic Corporation, Lattanze Foundation, and Temple University.
Daft, R., and Lengel, R. 1986. Organization
Information Requirements, Media Richness and Structural Design.
Management Science 32(5):554-571, May.
Daft, R. L., and Weick, K. E. (1984). Toward a
Model of Organizations as Interpretative Systems. Academy of Management Review
Dennis, A. R.; George, J. F.; Jessup, L. M.;
Nunamaker, Jr., J. F.; and Vogel, D. R. (1988). Information
Technology to Support Electronic Meetings. MIS Quarterly
Dreyfus, H., and Dreyfus, S. (1986)
Mind Over Machine, New York: Free Press.
Grudin J. (1988). Why CSCW
Applications Fail: Problems in the Design and Evaluation of
Organizational Interfaces. In Proceedings of the Conference on
Computer Supported Cooperative Work, Portland, OR.
Laughlin, P. R. (1989). A Theory of Collective
Induction: Final Report. Office of Naval Research, Arlington ,
Lee, D. L., and Brookes, C. H. (1993). An
Innovative Corporate Intelligence System at the M.W. Kellogg Company,
Houston, Technical Report, M.W. Kellogg Company, Houston TX.
Lave, J. (1988). Cognition in Practice: Mind, Mathematics, and Culture in
Everyday Life. Cambridge, UK: Cambridge
Newell, A., and Simon, H. A. (1972). Human Problem Solving, Englewood Cliffs, NJ: Prentice&endash;Hall.
McNeese, M. D. (1993). Putting
Knowledge To Use: The Acquisition and Transfer of Knowledge in
Situated Problem Solving Environments," AL/CF-TR-1993-0052. Armstrong
Laboratory, Wright-Patterson Air Force Base, OH.
McNeese, M. D.; Zaff, B. S.; and
Brown, C. E. (1992). Computer-Supported Collaborative Work: A New
Agenda for Human Factors Engineering. In Proceedings of the IEEE
National Aerospace and Electronics Conference (NAECON), 2:681-686.
Dayton, OH: IEEE, Aerospace and Electronic Systems Society.
McNeese, M.D.; Zaff, B.S.; Citera,
M.; Brown, C.E.; and Whitaker, R. (1995). AKADAM: Eliciting user
knowledge to support participatory ergonomics. The International
Journal of Industrial Ergonomics 15(5):345-363.
Mintzberg, H.; Raisinghani, D.; and Theoret, A.
(1976). The Structure of 'Unstructured' Decision Processes.
Administrative Science Quarterly 21, June, 246-275.
Nosek, J.T., and Mandviwalla, M.
(1996). Mobile Group Support Technologies For
Any-Time, Any-Place Team Support. Information Technology and People
9(4). In Publication.
Nunamaker, Jr., J. F.; Dennis, A. R.; Valacich, J.
S.; Vogel, D. R.; and George, JF (1990). Electronic Meeting Systems
to Support Group Work: Theory and Practice at Arizona, Working Paper
CMI&endash;WPS&endash;90&endash;03, Tucson, Az: College of Business
and Public Administration, University of Arizona.
Nunamaker, J.; Dennis, A.; Valacich, J.; Vogel,
D.; and George, J. (1991). Electronic meeting systems to support
group work. Communications of the ACM 34(7):40-61.
Olson, G. M., and Olson, J. S.
(1990). User-centered Design of Collaborative Technology, Technical
Report No. 32," Cognitive Science and Machine Intelligence
Laboratory, University of Michigan, Ann Arbor, MI.
Steiner, I. D. (1972).
Group Process and
Productivity. New York:
Suchman, L. (1987). Plans and Situated Action.
Cambridge, UK: Cambridge University Press.
Whitaker, R.D.; Selvaraj, J.A.; Brown, C.E.; and
McNeese, M.D. (1995). Collaborative Design Technology: Tools and
Techniques for Improving Collaborative Design, AL/CF-TR-1995-0086, Air Force
Materiel Command, Wright-Patterson Air Force Base, Ohio.
Wilson, D.; McNeese, M. D.; and Brown, C. E.
(1987). Team Performance of a Dynamic Team Resource Allocation
Problem: Comparison of Shared Versus Isolated Setting. In Proceedings
of the 31st Annual Meeting of the Human Factors Society, 2:1345-1349.
Santa Monica, CA: Human Factors Society.