Issues for Knowledge Management from

Experiences in Supporting Group Knowledge Elicitation & Creation

in Ill-defined, Emerging Situations

John T. Nosek

Computer & Information Sciences, Temple University

Philadelphia, PA 19122

nosek@thunder.ocis.temple.edu

Michael D. McNeese

Armstrong Laboratory, Fitts Human Engineering Division

AL/CFHI, Bldg 248, 2255 H Street

Wright Patterson AFB, OH 45433-7022

mmcneese@al.wpafb.af.mil

Abstract

Knowledge management involves both the elicitation/creation of knowledge, and the storage/retrieval of meaningful knowledge in future states. In ill-defined, emerging situations, team members grope to make sense of the situation, react to stimuli from the external environment, and interact with each other and human artifacts to develop an interpretation of the environment. A general model of the group knowledge elicitation and creation process is presented and lessons derived from experiences in supporting this process are offered.

Introduction

While organizational environments become more complex and dynamic, there exists limited time to exploit an opportunity or solve a problem. Real time decision makers under stress, for example, combat fighter pilots, come across a fast unfolding situation and must quickly make an acceptable decision. They focus on assessing the situation and taking acceptable actions that present themselves - optimization is not really possible nor sought. Because of increasing time pressures to act quickly, organizational decision makers are beginning to find themselves in a similar position to real time decision makers -- situation assessment is critical and acceptable actions are sought. We are not trying to say that fighter pilots and organizational decision makers work within the same split second time frames, however, the changes for fighter warriors and organizational decision makers can inform our understanding of group sensemaking. For example, with the increasing use of ìreal time information in the cockpitî, pilots will be able to view advanced imagery and communicate/collaborate using highly sophisticated technologies. As these advancements become prevalent, the view of the pilot as individual warfighter becomes obsolete, and barriers between real time and organization decision making breakdown. Within this complex, broad bandwidth decision space, there are many possible actor-to-actor or actor-to-agent couplings that underlie group sensemaking.

Complex, ill-structured, situation domains span such areas as command and control, business planning, new-product design, process reengineering, and information systems development. However, in all cases, the growing importance of group sensemaking, i.e., the elicitation and creation of group knowledge relevant to an emerging situation, is becoming clear -- those that do a better job of it will have a better chance of survival and increased competitive advantage.

Participants engaged in group knowledge elicitation and creation in these ill-structured domains, whether distributed or not, face the existence of multiple and conflicting interpretations about an emerging situation. They are not certain about what questions to ask, and if questions are posed, no clear answer is forthcoming (Daft and Lengel, 1986). They grope through a recursive, discontinuous process of many difficult steps subjected to interference, feedback loops, and dead ends that more closely resembles fermentation than an assembly line (Mintzberg et al. 1976). The participants' experience-based intuitive understanding may depend on insufficient or no-longer-relevant experience(Dreyfus & Dreyfus, 1986). All those involved must effectively and efficiently scan for and filter relevant information to create and maintain a shared mental model. Shared mental models have the problem of knowledge or truth maintenance in that the information that was true for yesterday (or even an hour ago) may have decayed, have subtle changes, or may have demonstrably changed. These changes occurring over the entire decision space can play havoc with meaning, interpretations, and choice of actions, and highlight the need for conflict resolution, multi-source sensemaking, and the social construction of knowledge.

Knowledge management involves both the elicitation/creation of knowledge and the subsequent retrieval of meaningful knowledge in future situations. Therefore, group knowledge that is elicited and created must be stored in a manner that it can be accessed in a meaningful way in future situations requiring group knowledge elicitation and creation. This strong, cyclical relationship between the elicitation/creation of group knowledge and the storage/retrieval of relevant knowledge in future situations demands a better understanding of how to support group knowledge elicitation and creation. This paper discusses some issues that have emerged in our efforts to support this process.

Background

Figure 1 depicts the basic components of a general model of group knowledge elicitation and creation. Each of the following subsections briefly discuss these components:

Group. There are myriad group characteristics which affect the process and subsequent outcomes, and not all will be discussed. For example, individual member characteristics may complicate the process when personality, job status or political agenda introduce conflict (Dennis et al. 1988), and greater numbers of participants multiply these individual effects. Group size affects productivity both positively and negatively (Applegate, 1991). Larger groups increase process complexity with more inputs, viewpoints and expertise to be processed (Nunamaker et al. 1990). At the same time, the greater expertise of larger groups should improve task-related outcomes. Some group attributes can reduce process complexity, such as cohesiveness and positive previous work experience as a group (Nunamaker et al. 1990).

Task. Tasks associated with ill-defined, emerging situations can be highly complex, and task complexity is a key component of group process complexity. One can relate the cognitive effort of information processing tasks to the nature of the task, rather than to the capabilities of the problem solvers, the number of problem solvers, or process characteristics. Complex tasks are non-routine with more uncertainty and which require new knowledge and unique solutions. Wood (1987) rates complex tasks according to the number of acts or information cues, and according to relationships among acts and information cues, as well as changes among them.

Process Support. Process support includes tools, techniques, and methods which can be used to support the group knowledge elicitation and creation process.

Interactions of Group, Task, and Process Support Within a Context. The role of group process loss/gain (Steiner, 1972), as a function of the collective induction (Laughlin, 1989) that ensues in group experiences, impacts the level and quality of the knowledge elicited and created. It is important to consider the extent that these factors are interrelated with task and process support considerations while keeping in mind that the environment determines what is possible and what is constrained when process models are invoked.

Outcomes. Outcomes of the group knowledge elicitation and creation process are categorized as task-related; process-related; and group-related.

Experiences

The authors have attempted to employ various technologies in a variety of situations over a number of years to support group knowledge elicitation and creation. One author has employed leading edge mobile (notebooks, wireless LANS, and WAN connectivity), group support technology (cognitive mapping, document sharing, and desktop conferencing) to provide any-time, any-place support for information systems development (Nosek & Mandviwalla, 1996). The other author has actively employed the AKADAM cognitive engineering methods to find better ways to support future fighter pilots in air warfare (McNeese, Zaff, Citera, Brown, & Whitaker, 1995). The development and use of the AKADAM method has evolved and been iterated many times for different case studies that require individual and/or groups to elicit and create knowledge. In these iterations, depending on the situations encountered, concept mapping, design storyboarding, and IDEF functional decomposition (in various combinations and in decreasing order of frequency and success) were used. While these experiences are diverse, the following sections summarize the lessons learned, and discuss issues to improve knowledge management.

Lessons Learned

As reported in McNeese, Zaff, Citera, Brown, & Whitaker (1995) there are three principles for user-centered / group centered approaches to knowledge elicitation: a) actively employ a means of shared communication, b) facilitate the unconstrained expression of knowledge, c) generate knowledge representations compatible with the capabilities, limitations, and needs of the stakeholders. A basic tenet underlying all of these principles is that individual and group knowledge building is highly participatory and evolutionary often feeding off successive loops of recursive understanding. In some instances, ìtoo many cooks have spoiled the broth; while in other situations ìtwo heads have amounted to be more than oneî. Part of the wisdom associated with knowledge elicitation practices is knowing which of these adages applies to the situation under consideration. There have been many lessons learned especially regarding the constraints of these techniques that can and have been used to continuously improve application effectivity. These lessons successfully show the interdependence of group, task, and process variables with varying constraints on stimulus and environmental conditions as defined by the actual context of knowledge elicitation activities. In this sense, one might say they are derived from "knowledge elicitation in the wildî considerations. Many of the following lessons learned have been selected from the case studies reported in the McNeese et al. (1995) review and are geared toward use of the concept mapping typology as a knowledge management tool:

* We have found that many of our sessions start off with questions asked of experts that are too broad in scope (as defined by the top level node of the map). This ends up creating fractionated knowledge, only attending to surface-level knowledge while never progressing to deep-seated knowledge, and typically results in excessive time expenditures to complete the map. This becomes problematic if the session is time-limited - which has a net effect of just skimming off the surface of the map. Too, if the participants come from varied multi-disciplinary perspectives, then many of these problems actual get worse and group knowledge elicitation is even more ìout of focusî.

* Dependent on the application for which we are trying to prune knowledge, the imposed representational typology / knowledge elicitation structure can be a plus or a minus for group knowledge buildup. That is, flexibility (as demonstrated with concept maps) can lead to too much free association / subject fading effect, in contrast to a more well-defined structure (e.g., the IDEF functional decomposition), that often inhibits the ìgenerative-active learning processî that is so important for collective induction in the group. On the other hand, the use of concept maps and/or semantic network structures may have difficulty with team probabilities / fuzziness, conditional attributes, and time progressions. This may come into play for those situations that are heavily reliant on procedural knowledge and inferential use of nodes.

These considerations are also affected by individual differences of the experts being mapped as well. Many engineers seem to like the strict, logical progression of an IDEF decomposition whereas many of the pilots we have mapped prefer the ease of representing their knowledge in the concept map. We have opted towards use of concept mapping for more generalized, front end knowledge elicitation that involve groups, but reserve the IDEF for highly specific, well-defined areas, where the experts tend to have a more homogenous makeup.

* One area of concern within the representational typology is that of addressing semantic descriptors that have dual meanings / multiple connotations. Many of the computer-based elicitation tools we have used have not been ìsmartî enough to interpret meaning, e.g., one participant may use ëinterfaceí to refer to a computer interface while other might interpret this as the ëinterfaceí between the contractor and government entities. This problem is comes into play when individual maps are combined to form summary maps, or when group mapping unfolds as different participants invoke different meanings / interpretations of what is being communicated and then translated into the map. Much of the interpretation is done intuitively by looking at clusters of context to distill the associative context that surrounds individual nodes on the map.

* Another consideration involves the extent to which the knowledge in a group-centered map is scaleable or is ìtransferableî to similar - yet different situations. When one conducts a group session, domain specific knowledge is from variant sources which may make sense for the situation at hand. However, when transferred to a similar situation, much of the variance of the initial map may become highly salient and inappropriate.

* The quantitative / qualitative evaluation of map content is difficult. Our overall focus has been to have multiple sessions to ìsearch for invarianceî across all individual participant maps and then compare to the group-centered map. In this way, comparisons / contrasts can be made that help an evaluation team make recommendations regarding the extent to which elements of the map approach veridicality.

* Other randomly occurring problems that can incur are:

• not knowing when to stop or start an elicitation session,
• maintaining the visual attention to the map while developing it with participants,
• creating maps with concepts rather than sentences,
• interoperability with follow-on knowledge representations (e.g., translating - bridging knowledge across concept maps, design storyboards, and IDEF typologies) are not all that apparent and disgranular transitions may occur,
• users are not accustomed to working with or ìreading the mapî as they are often ìafraid of getting their feet wetî,
• user-centered knowledge elicitation trades off with the precision requirements of knowledge structure used,
• without audio/video capture of sessions some knowledge may be lost,
• with groups there is a higher probability of shifting topics resulting in disjunctive/immiscible concepts,
• the mapperís inherent knowledge of the expertís subject matter, his/her identity with the group process, and the mapperís inherent capability as an ëexpert mapperí may influence the sessionís effectiveness,
• the mapperís role may be highly diminished if a group directly creates concept maps ìon-lineî using computer-mediated systems,
• and finally the legibility/lucidity of the emergent map can be a factor for group participation which also relates to the mapperís handwriting, the physical parameters/obstructions in the room where mapping occurs, or the computer display legibility/real estate if done on-line.

Most of the challenges identified here are problematic for group elicitation and knowledge building. However, we have found that the potential gain in collective induction, and consequently the improved robustness in knowledge buildup, is generally worth the effort. In some cases, problems are intractable and have not yet been solved while in other situations remedies have been successfully implemented. This is a new area, so there are many lessons yet to be learned in the future.

Issues for Knowledge Management Improvement

To date, IT products to support the group knowledge elicitation and creation process have been inadequate and the following must be addressed in parallel:

• multi-disciplinary theory development focusing on the intersections of cognition, collaboration, and context (Groups engaged in Tasks using Process Support (See Figure 1)).
• Process support improvements in architecture and group interface, with theory-based development of tools.

Multi-disciplinary Theory Development

The intent of this section is to present some of the issues that must be addressed in theory development in order to provide better support for group knowledge elicitation and creation. It offers some of our thoughts on the issues that have evolved from our efforts to understand and support this process, rather than any thorough examination of the subject which is beyond the scope of this paper. It is through this reflection and some of our efforts that have convinced us of the need for a multi-disciplinary, multi-theoretic approach to the subject. It also seems that the problem with such an approach is not only the barrier of different languages among people of multiple disciplines, but more importantly, the difficulty and rarity of actually thinking of the question from another discipline's perspective.

In ill-defined emerging situations the assumptions about the environment change from analyzable to unanalyzable. The source of data moves from internal, impersonal to external, personal, and the data acquisition techniques move from regular, routine, reports, many obtained from organizational information systems, to irregular reports from external contacts and feedback from the environment (Daft and Weick, 1984; Dreyfus and Dreyfus, 1986). "The decision maker is faced with a "corporate primordial soup" of customer, industry and technology news; assessments of the news; competitor moves; agent's call reports; alliances; rumors; deals coming and done; problems and solutions; suggestions and scenarios (Lee and Brookes, 1993)." In these situations, organizations actual "create" rather than "discover" their environment through their interpretation process (Daft and Weick, 1984).

In Figure 2, team members engaged in groping to making sense of the situation, react to stimuli from the external environment and interact with each other (verbally & non-verbally) and human artifacts to develop an interpretation of the environment. In fact, there are multiple possible environments, but through this group sensemaking process where they elicit knowledge from various sources and create new knowledge from synthesizing during their analysis process, they operationalize ("create") an environment in which to act. By acting on this interpretation of the environment and finding success in their actions, they have created an environment that validates their particular interpretation. For example (you may want to try this yourself), take a game of 20 questions where one person normally selects in advance some item, while others have up to 20 questions to discern what the item is. Modify this game so that the person does not pre-select an item, but the other people in the group do not know this. As the group asks 20 questions of this person, the person provides an answer that is consistent with the previous question asked. When the group comes up with some answer like "sky", the person responds "correct" because their sensemaking process has produced this answer.

This process involves a complex interaction of cognitive activities by participants engaged in collaboration with others within a context they create through this group interpretation process. Researchers from multiple disciplines, stretching their own cognitive muscles by reflecting on these issues from other disciplines, will help to develop theories to better manage the full cycle of knowledge elicitation/creation and storage/retrieval.

Process Support Improvements

In parallel with theory development, process support changes must be made to provide real value to way groups create and elicit knowledge from one another. While some argue that we must understand the group process of eliciting and creating knowledge before developing tools, from experience, parallel development seems to make more sense. First, the use of the tools can help understand the complex process of group knowledge elicitation and creation, at the same time, the use of tools can actually change the process of group knowledge elicitation and creation. The following sections describe some of the issues related to process support which must be addressed to develop more useful technology:

Need for Better Tool Integration. Each tool available has been built as a separate system. Some problems that have occurred because of this include:

• Different interfaces and metaphors for interaction. Each tool uses their own interfaces. This makes it difficult to learn and use such systems. Initially, the kinds of problems for which these tools will be used may be ad hoc in nature. The infrequent use of such tools demand that the affordance of the interface be especially high and naturally support the cognitive processes.
• Architecturally each system is different. It is not possible to easily use process support features of one tool with another. The basic storage and manipulation structures are different. Object-oriented structures that can be flexibly manipulated to provide various process support where needed may offer some promise. As noted earlier we need to support group groping, where like a dance, the intermediate steps are as important as the final step. For example, in the exploration of a design rationale using a cognitive mapping tool, it can be useful to vote on some argument, or provide weights based on the group input. Currently, tools for cognitive mapping are perceived differently from those that support group decision support using brainstorming, ratings, rankings, etc. The designers of these tools because of architectural limitations or inability to see the "whole elephant" make it difficult if not impossible to integrate tools.

New Visions of Group Interaction. New visions of group interaction must consider social, technological, ecological, and cognitive factors involved in real world collaborations. Situated cognition (Suchman, 1987), communities of mutual learning (Lave, 1991), and in situ understanding of context predominate in guiding our understanding of knowledge building and generative learning in collaborative situations and developing group/computer interfaces. McNeese (1993) has shown the advantages of perceptual learning and collective induction upon knowledge formation, and how these advantages help individual group members transfer knowledge across analogical situations. These findings, in consort with naturalistic case studies involving knowledge building (e.g., group concept mapping), suggest development of group interfaces that may be different from traditional group decision support systems (e.g., GroupSystems, Nunamaker et al. 1991). Traditional supports impose a 'distributed metaphor' by connecting individuals together 'as if' they are engaged in problem solving at remote sites. Interaction is tunneled through computer workstations / individual screens typically enriched with structured tools and voting support. In contrast, a group interface is one in which members are perceptual anchored, face-to-face, and interaction is through mutually accessible group display surfaces. We must look beyond the promise of a strict, distributed metaphor and focus on a naturalistic understanding of the work context as a means to develop important theoretical constructs and the consequent design of group / computer interfaces. Some specific other considerations for group interfaces include:

• Easier capture of ideas. The speed in capturing ideas in a cognitive mapping session is much less than the speed that participants can propose. Even with two dedicated people to capture discussions, up to a third of the information can be lost (Whitaker et al. 1995). There have been some dramatic improvements in inexpensive speech recognition technology, and this application may be ideal for speech recognition quality that is less than 100%. We know from Cultural Linguistic Anthropology, that people can still communicate well when 20% of words are not understood (Personal conversation with Greg Urban, Univ. of Penn., May 1996). In capturing ideas within a given context, 80% accuracy may prove more than adequate. We need to capture enough to maintain the "context of discourse". Additionally, although less immediately available, natural language software predisposed to certain "contexts of meaning" could speed capture and interpretation.
• Improve viewing of cognitive maps. Cognitive maps can easily become large and the viewing capacity of current technology limits how much of a map can be shown at one time. This causes team participants to become greatly frustrated and severely detracts from usefulness.
• Improve filtering to limit visual confusion and information overload. For example, cognitive maps can become very complicated and visually confusing in very little time. Filtering mechanisms to reduce map density are critical and their availability may be hindered by lack of tool integration.
• Improve techniques to reconcile individual and group cognitive maps.

Group Centered Development of Process Support Technologies. A user-centered approach means that the userís domain expertise will have to be incorporated directly into the design. It also means, in the strictest sense, acquiring knowledge about the user, the userís work domain, and the userís design requirements directly from the user. Adapting this approach to groups means that the design of collaborative technologies for a given application would necessarily involve the direct participation of the team for whom the system is being designed (see for comparison Olson & Olson, 1990). In order to support a user-centered approach to the design of collaborative technologies, it becomes necessary to acquire knowledge about the group, the groupís work domain, and the groupís design requirements directly from the group. This is in accordance with Grudinís (1988) observation that CSCW designs need to become more ëgroup-friendlyí. Hence, a user-center approach adapted to the design of collaborative technologies is more accurately referred to as a group-centered approach (McNeese et al. 1992).

More theory-based development of process support technology. While this is not a new criticism of technology, it is still true and worthy of consideration. A more methodical mapping of tool features to our hopefully increasing understanding of the group knowledge elicitation and creation process will allow us to make more substantial gains in process support technology.

Concluding Remarks

Knowledge management involves both the elicitation/creation of knowledge, and the storage/retrieval of meaningful knowledge in future states. In ill-defined, emerging situations, team members grope to make sense of the situation, react to stimuli from the external environment, and interact with each other and human artifacts to develop an interpretation of the environment. A general model of the group knowledge elicitation and creation process for a given situation was presented and lessons derived from experiences in supporting this process were offered. Future gains in knowledge management depend on multi-disciplinary theory development and process support technologies more closely tied to these emerging theories.

Acknowledgments

Partial support for this work was provided by the National Science Foundation's Division of Undergraduate Education through grant DUE-9452594. Additional support for this work was provided by grants from Lotus Development Corporation, Bell Atlantic Corporation, Lattanze Foundation, and Temple University.

References

Daft, R., and Lengel, R. 1986. Organization Information Requirements, Media Richness and Structural Design. Management Science 32(5):554-571, May.

Daft, R. L., and Weick, K. E. (1984). Toward a Model of Organizations as Interpretative Systems. Academy of Management Review 9(2):284-295.

Dennis, A. R.; George, J. F.; Jessup, L. M.; Nunamaker, Jr., J. F.; and Vogel, D. R. (1988). Information Technology to Support Electronic Meetings. MIS Quarterly December:591&endash;619.

Dreyfus, H., and Dreyfus, S. (1986) Mind Over Machine, New York: Free Press.

Grudin J. (1988). Why CSCW Applications Fail: Problems in the Design and Evaluation of Organizational Interfaces. In Proceedings of the Conference on Computer Supported Cooperative Work, Portland, OR.

Laughlin, P. R. (1989). A Theory of Collective Induction: Final Report. Office of Naval Research, Arlington , VA.

Lee, D. L., and Brookes, C. H. (1993). An Innovative Corporate Intelligence System at the M.W. Kellogg Company, Houston, Technical Report, M.W. Kellogg Company, Houston TX.

Lave, J. (1988). Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life. Cambridge, UK: Cambridge University Press.

Newell, A., and Simon, H. A. (1972). Human Problem Solving, Englewood Cliffs, NJ: Prentice&endash;Hall.

McNeese, M. D. (1993). Putting Knowledge To Use: The Acquisition and Transfer of Knowledge in Situated Problem Solving Environments," AL/CF-TR-1993-0052. Armstrong Laboratory, Wright-Patterson Air Force Base, OH.

McNeese, M. D.; Zaff, B. S.; and Brown, C. E. (1992). Computer-Supported Collaborative Work: A New Agenda for Human Factors Engineering. In Proceedings of the IEEE National Aerospace and Electronics Conference (NAECON), 2:681-686. Dayton, OH: IEEE, Aerospace and Electronic Systems Society.

McNeese, M.D.; Zaff, B.S.; Citera, M.; Brown, C.E.; and Whitaker, R. (1995). AKADAM: Eliciting user knowledge to support participatory ergonomics. The International Journal of Industrial Ergonomics 15(5):345-363.

Mintzberg, H.; Raisinghani, D.; and Theoret, A. (1976). The Structure of 'Unstructured' Decision Processes. Administrative Science Quarterly 21, June, 246-275.

Nosek, J.T., and Mandviwalla, M. (1996). Mobile Group Support Technologies For Any-Time, Any-Place Team Support. Information Technology and People 9(4). In Publication.

Nunamaker, Jr., J. F.; Dennis, A. R.; Valacich, J. S.; Vogel, D. R.; and George, JF (1990). Electronic Meeting Systems to Support Group Work: Theory and Practice at Arizona, Working Paper CMI&endash;WPS&endash;90&endash;03, Tucson, Az: College of Business and Public Administration, University of Arizona.

Nunamaker, J.; Dennis, A.; Valacich, J.; Vogel, D.; and George, J. (1991). Electronic meeting systems to support group work. Communications of the ACM 34(7):40-61.

Olson, G. M., and Olson, J. S. (1990). User-centered Design of Collaborative Technology, Technical Report No. 32," Cognitive Science and Machine Intelligence Laboratory, University of Michigan, Ann Arbor, MI.

Steiner, I. D. (1972). Group Process and Productivity. New York: Academic Press

Suchman, L. (1987). Plans and Situated Action. Cambridge, UK: Cambridge University Press.

Whitaker, R.D.; Selvaraj, J.A.; Brown, C.E.; and McNeese, M.D. (1995). Collaborative Design Technology: Tools and Techniques for Improving Collaborative Design, AL/CF-TR-1995-0086, Air Force Materiel Command, Wright-Patterson Air Force Base, Ohio.

Wilson, D.; McNeese, M. D.; and Brown, C. E. (1987). Team Performance of a Dynamic Team Resource Allocation Problem: Comparison of Shared Versus Isolated Setting. In Proceedings of the 31st Annual Meeting of the Human Factors Society, 2:1345-1349. Santa Monica, CA: Human Factors Society.