Background: A Pattern of Common Mistakes

PROTÉGÉ-II provides a set of tools for developers of knowledge-based systems. PROTÉGÉ-II supports (1) the construction of problem solvers from reusable components (Eriksson et al., 1995), and (2) the generation of domain-specific knowledge-acquisition tools (Eriksson & Musen, 1993). In the PROTÉGÉ-II approach, the developer uses an ontology editor to create ontologies that model domain concepts and relationships. The developer then uses a program that generates automatically domain-specific knowledge-acquisition tools from these ontologies (Eriksson et al., 1994). Domain experts use these knowledge-acquisition tools to create knowledge bases (which consist of instances of classes in the domain ontology). Because these knowledge bases are defined in terms of domain-specific concepts, they must be mapped to domain-independent concepts that can be used by reusable problem-solving methods (Gennari et al., 1994). Figure 1 shows parts of the PROTÉGÉ-II architecture, and illustrates how a critique tool can use information from several representations in PROTÉGÉ-II as the basis for the critique-analysis process. Note that although PROTÉGÉ-II supports automatic generation of knowledge-acquisition tools, these tools can still be subject to critique, because PROTÉGÉ-II allows manual custom adjustments (see Section 4).

 
Figure:   PROTÉGÉ-II and the Critique Tool. A critique tool can use information from many sources in a development environment. In PROTÉGÉ-II, there are three major representations that are relevant for a critique tool. (1) The ontology, which is the output of the ontology editor, provides the basis for ontology critiquing. (2) The tool definition, which is the output of knowledge-acquisition--tool generator, provides the basis for the critiquing the design of a knowledge-acquisition tool. (This definition is sometimes referred to as the Editor Ontology [EO] file.) (3) The knowledge base, which is generated by the knowledge-acquisition tool run-time system, provides the basis for the critiquing of knowledge entered by domain experts. (In the PROTÉGÉ-II approach, the knowledge base consists of instances of the ontology.) In the current prototype implementation of CT, only the tool definition (2) is used as the basis the critique analysis.

We have monitored the use of PROTÉGÉ-II to establish problematic areas and design flaws. We analyzed common complaints, and we approached problems by (1) redesigning tools that caused problems for their users (immediately or at a later time), (2) informing users about use-cases to avoid, and (3) writing critiquing rules for the critiquing tool. The critiquing tool can assist the users by helping them to avoid problems before they occur. A list of the advantages of a critiquing tool for PROTÉGÉ-II follows.

1. Informed users. The critiquing system warns PROTÉGÉ-II users of problematic situations. The users can then decide to follow the recommendations provided, or ignore them when appropriate.

2. Improved quality. A critiquing system can improve the quality of the knowledge-based systems developed with PROTÉGÉ-II. Because the critiquing system performs a systematic search for inconsistencies, it is possible to use the results to improve the quality of the ontologies and the knowledge-acquisition tools generated.

3. Documentation of unsuccessful uses. It is important to document unsuccessful and inappropriate uses of development tools. The development of a critiquing knowledge base supports the documentation and formalization of problematic cases. A critiquing system that searches actively for problems is typically more effective than conventional, paper-based problem documentation.

4. Deferred debugging. In addition to critiquing user mistakes, a critiquing system can help the user to avoid known bugs in development tools. Although the ultimate goal is to correct every known flaw in the tools, correction work must be prioritized. During our incremental development of PROTÉGÉ-II, we often found it advantageous to suspend corrections of certain bugs, and to focus on other parts of the system. As a first step toward correcting a bug, we often add critiquing rules that detect the problematic situation and warn the developer.

Let us examine some of the problems that can be approached by a critiquing system. Examples of common mistakes by the PROTÉGÉ-II users include (1) definitions of is-a relationships in the ontology that should be modeled as is-part relationships, (2) missing or incorrect slot definitions in the ontology, (3) illegal circular references in the ontology, and (4) inappropriate tailor-made layout of forms in knowledge-acquisition tools. A critiquing system can detect some of these mistakes automatically, whereas other mistakes (e.g., confused is-a and is-part relationships) cannot be detected readily.

Not all problems are caused by user mistakes, however. An example of a known type of problem in PROTÉGÉ-II arises from bugs in the layout-designer module that result in incorrect widget layout on forms in knowledge-acquisition tools. The critiquing tool checks for inconsistencies in the window layout, such as windows outside the screen area and widgets outside their window area, and reports these problems to the developer.