Design critiquing in development environments, such as PROTÉGÉ-II, can be performed at several stages in the development cycle. CT, for instance, analyzes the use of PROTÉGÉ-II after the developer has constructed a knowledge-acquisition tool. In the PROTÉGÉ-II approach, developers can use CT to critique knowledge-acquisition tools after generating a new tool and after making substantial changes to a tool. Other types of critiquing tools could potentially analyze the development work at other stages. An alternative to this deferred-critiquing approach is to report problems immediately. For instance, we can accomplish immediate critiquing by incorporating a critique function in the graphical tool that allows developers custom tailor window layouts (Eriksson et al., 1994; Eriksson et al., 1995). The critiquing system can then report immediately layout problems, such as overlapping widgets.
The disadvantage of this immediate approach is that it is often better to report certain critique after the developer has completed a design stage. Some models can be inconsistent internally during design, and the developers sometimes prefer deferred critiquing. Another potential problem is that certain critiquing rules can differ between immediate and deferred critiquing, because such systems use the rules in different contexts. Moreover, architectures that incorporate immediate critiquing in several design stages can be difficult to maintain, because there is no explicit critiquing module. In summary, we can use both immediate and deferred critiquing to support development environments, such as PROTÉGÉ-II. The context of the critique determines the appropriate approach.
Although the CT prototype implementation is a useful tool, the current version has several shortcomings. For instance, there is no support for direct ontology critiquing in CT. Nevertheless, CT can identify certain ontology problems by examining knowledge-acquisition tools generated from the ontologies. A complete implementation of CT should include direct analysis of ontologies, because CT cannot identify many problem types without examining the ontology directly. Also, there is room for improvement of the critiquing of knowledge-acquisition tools, especially concerning the reporting of critique to the developer. For instance, CT reports layout problems textually. By reporting such problems graphically, the developer can understand more easily layout flaws (e.g., overlapping fields in a form layout) than by reading a textual report. Another area where critique support would be useful is the design of mappings between domain-specific and domain-independent concepts in ontologies (Gennari et al., 1994). Unfortunately, critiquing beyond checking for missing mapping definitions is difficult, because the mapping design process requires knowledge of both the domain and the problem-solving method.
A more sophisticated approach to assisting developers than critiquing systems would be an active critiquing system that guides the developer through corrections to the model. Just as TEIRESIAS (Davis, 1979) assists developers and domain experts in correcting and maintaining rules bases, an active critiquing and maintenance system for ontologies and other models could guide the developer through the correction of ontologies and other models. Another example of a knowledge-acquisition tool that performs completeness checking and asks the expert to fill in missing information is Aquinas (Boose & Bradshaw, 1987). In this approach, the critiquing system would, besides reporting problems and providing recommendations, assist the developer in correcting the problem and in dealing with the consequences of the correction.
An important principal problem in the design of critiquing systems is the trade off between syntactic and semantic critiquing. Syntactic critiquing helps developers discover superficial, but sometimes serious problems. Semantic problems, however, are difficult to detect automatically. Semantic checking requires that (1) the model contains redundancy, (2) a reference model is available, or (3) sample data are available. It is rarely realistic for developers to collect sufficient redundant domain knowledge for a critiquing system. Nevertheless, developers can often collect redundant knowledge and reference cases for critical aspects to critique. The design of practical critiquing systems involves balancing syntactic and semantic critiquing.
Fischer et al., 1993 argue that critiquing systems should be embedded in design environment. Although stand-alone critiquing systems, such as CT, have many advantages from a software maintenance perspective, embedded critiquing systems can provide better support for the users than stand-alone systems can. One of the lessons learned from CT is that critiquing modules should be integrated in the architecture of the knowledge-engineering tools at an early design stage. In this approach, several critiquing modules could interact with the design directly, and could provide critique immediately.