Chapter 4 Selection, launch and assessment of test cases

The target for the test case program was to establish as quickly as possible approximately three test cases which would serve as a basis for the development of greater understanding of the issues involved in considering the case for a full program. These pilot projects would test the ability of companies, universities and laboratories to cooperate internationally. This would also yield some insight into the benefits and costs of the process. The end point reached was the selection of five test cases and one study. Final reports on these were submitted to, and assessed at, the sixth meeting of the International Technical Committee (ITC6).

From the outset the ISC gave the ITC a strong challenge. It had to make contact with industry and universities throughout the regions to generate interest in a test case program for a future program which was not certain to follow. This occurred when it was impossible to give even outline advice on the work content and the modus operandi. Potential industrial participants also would be expected to inject significant industrial resources. Happily, the efforts of the ISC, the ITC, regional secretariats and potential partners resulted in the establishment of a comprehensive test case program.

This section reports on the range of activities and issues which characterise the test case program and concludes with an overall impression of the final assessment activity. All the relevant documentation on the test case program from the commencement of work in the six projects is contained in the Report of ITC.

The process of information exchange prior to R&D test case consortium formation is described in Part I, Section 2.4. Interested entities and consortia were provided with a set of explanatory notes giving the requirements to be followed in drawing up a test case.

Proposing consortia were requested to provide administrative data relating to the partners of the consortia and a comprehensive description of the proposed work, resources that would be deployed, the objectives, the benefits expected to be gained, and the organisation and management of the test case. Information was also sought on the metrics, methodology, or measurement system used to evaluate each partner's input to and output from the collaboration to demonstrate equity and balance.

In addition the proposing consortium was requested to submit a consortium cooperation agreement in conformity with the IPR Guidelines for test cases. The explanatory notes contained a timetable allowing 3 months for consortium/proposal formation, 2 months for evaluation and selection, and envisaged that test cases would start at the beginning of 1993 (with the actual start time of the projects being March 1993).

The selection of test cases proceeded in two steps. First a regional evaluation and selection was carried out. In a second step regional short lists for test cases were put forward to the International Technical Committee and the International IPR Committee, that were tasked to make recommendations on the selection of test cases to the International Steering Committee. The selection of test cases was ratified by the International Steering Committee on January 29, 1993.

The criteria for test case selection that were communicated to the proposing consortia are described in detail in the next section.

Of the 11 proposals that were submitted 5 test cases and 1 study were selected. A monitoring and assessment process was elaborated by the International Technical Committee.

The design of the intermediate assessment process was influenced by, first, the work load of the consortia and, second, time constraints during the ITC meetings. International test case coordinators provided reports, made a substantial presentation to the ITC, returned questionnaires, and joined in open discussions at International Technical Committee meetings. International test case assessments took place at ITC5 on June 20, 1993, and at ITC6 on December 1, 1993.

The ITC devoted a three-hour session to the total activity of assessing the performance of each test case by the ITC sub-groups. The chairman of each sub-group submitted the intermediate assessment report in a standard form. The six sub-group reports were reviewed by a committee consisting of the Heads of Delegation and the chairman of the assessment sub-groups. In addition, the ITC, in plenary session, held an open discussion session with the test case presenters. Overview reports based on the assessment of the total range of interchanges between the test case presenters and the ITC were also produced, and the ITC gave feedback to each test case consortia.

At ITC5 the report of Working Group 2, formed to study the problems encountered in consortium formation, was presented. The group conducted a survey of test case consortia, failed test case applicants, regional secretariats and committee members. The results of the surveys were as follows:

Test case assessment followed the procedures described in the paper produced by Working Group 1 (see ITC Final Report Annex II).

4.1 Criteria for test case selection

The Committee adopted several criteria to be followed by those proposing test cases. The agreed criteria were as follows:

4.2 Test cases selected

After a detailed process of assessment, five proposals were selected as test cases for the feasibility program. One proposal (TC2 below) was accepted as a study project but is referred to here for convenience as a test case. Details of the work carried out in each case is contained in the reports of the International Technical Committee.

The titles of the six test cases are:

The six test cases provided a diverse set of activities, including both state-of-the-art studies and generic research. In general, all the consortia had two interests: to assess the value of a full-scale IMS and to build a team for future programs. The test cases are summarised in the following paragraphs. A more comprehensive source of reading on each of the test cases is provided in the ITC's report.

4.2.1 Clean manufacturing in process industries

The purpose of this test case was to carry out a detailed assessment of some of the pre-competitive technologies which will be needed to meet the levels of improvement required over the next 5-20 years if process industries are economically configured to meet increasingly stringent environmental requirements.

Clean Manufacturing in the Process Industries is intended to instigate studies to provide an information source for governments and to establish a framework for future international collaborative research in the area of clean manufacturing in process industries. The project starts by identifying critical aspects of process industries where fundamental improvements in process technology are most likely to be required in order to meet environmental requirements. Chemicals, petrochemicals and pulp and paper will receive particular attention. A forward view of 5 to 20 years is taken. Current state-of-the-art manufacturing/performance/cost approaches will be benchmarked. Future environmental requirements will be estimated. Any potential gaps between practice and the forecast will be determined, and potential technical and system-based solutions outlined. The project is led by ICI Engineering of the United Kingdom and involves Japan, the USA and EFTA as well as the EC. There are twelve consortium members; eight companies, three research institutes and industry associations and one university.

The partners in this project are shown in the table below.

Table of partners: Clean manufacturing in process industries

  Partner                                     Region

  Abitibi-Price                               Canada*

  ICI Engineering                             EC/UK*
  Foster Wheeler Italiana SpA                 EC/Italy
  John Brown Engineers & Constructors Ltd     EC/UK
  Marex Technology bv                         EC/Holland


  Finnish Forest Industries Federation        EFTA/Finland*
  VTT (Technical Research Centre of Finland)  EFTA/Finland
  Finnish Pulp and Paper Research Institute   EFTA/Finland

  Toyo Engineering Corporation                Japan*
  Tokyo Institute of Technology               Japan
  Teijin Limited                              Japan

  Du Pont                                     USA*

                                              * Consortium leader

4.2.2 Concurrent engineering for global manufacturing

This test case comprised a comparative study of global concurrent engineering systems with the objectives of determining best practices, locating major constraints and designing the architecture of such a system.

Global Concurrent Engineering seeks to demonstrate improvements that can be made to global manufacturing capabilities through the implementation of currently known concurrent engineering techniques. The project will establish the extent to which concurrent engineering is practised, and identify any critical constraints in terms of technology, technology management and human resources. The best practices of concurrent engineering would then be synthesised and an architecture of concurrent engineering would be designed. The project is led by Northern Telecom Ltd. of Canada, and involves Europe and the USA. There are sixteen participants; eleven companies, and five universities.

The partners in this project are shown in the table below.

Table of partners: Concurrent engineering for global manufacturing

  Partner                                     Region

  Northern Telecom                            Canada*
  Carleton University                         Canada

  Syntax Factory Automation SpA               EC/Italy
  De Montfort University Leicester            EC/UK
  Transfer Technology Group PLC               EC/UK*

  Noika                                       EFTA/Finland
  VTT (Technical Research Centre of Finland)  EFTA/Finland

  North Carolina State University             USA*
  California Polytechnic                      USA
  Aeroglide Corporation                       USA
  Aritech Corporation                         USA
  Exide Electronics                           USA
  J C Steele & Sons                           USA
  Jacumin Engineering and Machine Co          USA
  Prodelin Corporation                        USA
  Tipper Tie                                  USA

                                              * Consortium leader

4.2.3 Globeman 21

The purpose of this study was to establish criteria for facilitating efficient global manufacturing businesses. Topics include concurrent engineering, inter-enterprise management (including supply chains involving different time zones and enterprises), and enterprise integration.

Globeman 21 or Global Manufacturing in the Twenty-first Century seeks to determine the best practices for global manufacturing organisations. This will be done by examining current practices in a number of volume segments ranging from custom to mass production, finding the generic elements of these best practices and determining the tools and measures available to assist implementation. The project is led by British Aerospace Defence Ltd. and involves all other regions. There are twenty-nine partners; twenty-two companies and seven universities.

The partners in this project are shown in the table below.

Table of partners: Globeman 21

  Partner                                     Region

  CSIRO                                       Australia*

  Spar Aerospace                              Canada
  University of Toronto                       Canada*

  British Aerospace                           EC/UK*
  Alcatel                                     EC/Belgium
  BICC                                        EC/UK
  IPA FhG                                     EC/Germany
  Grai Group                                  EC/France
  IWF                                         EC/Germany
  Pirelli                                     EC/Italy
  Ahlstrom                                    EFTA/Finland*
  Helsinki University of Technology           EFTA/Finland
  Nokia                                       EFTA/Finland
  Partek                                      EFTA/Finland
  Sintef                                      EFTA/Norway
  VTT (Technical Research Centre of Finland)  EFTA/Finland

  Toyo Engineering Corporation                Japan*
  Electrotechnical Laboratory                 Japan
  IBM Japan Ltd                               Japan
  Kyoto University                            Japan
  Mazda Motor Corporation                     Japan
  Ricoh Company Ltd                           Japan
  Takenaka Corporation                        Japan
  Tokyo University                            Japan
  Toyota Motor Corporation                    Japan
  Yokogawa Electric Corporation               Japan

  Carnegie Mellon University                  USA
  Newport News Shipbuilding                   USA*
  University of Virginia                      USA*

                                              * Consortium leader

4.2.4 Holonic manufacturing systems

This project addressed architecture and technology for open, distributed, intelligent, autonomous, cooperation ("holonic") systems for application in Intelligent Manufacturing Systems.

Holonic Manufacturing Systems addresses research, pre-competitive development, systemisation and standardisation of architecture and technology for holonic systems for application in intelligent manufacturing systems. It seeks to establish the technological and organisational basis for the development and implementation of these systems. A "Holon" is defined as an intelligent, autonomous, cooperative agent comprised of an intelligent control system, an information and/or material processing subsystem, and one or more humans who may or may not be present at any given time. A factory may thus be considered a hierarchy of holons, or holarchy. This metaphor has interesting implications for the design of machine controllers and the organisation of manufacturing processes. The project is led by Allen-Bradley of the USA and involves all other regions. There are thirty-two partners; seventeen companies, twelve universities and three research laboratories.

The partners in this project are shown in the table below.

Table of partners: Holonic manufacturing systems

  Partner                                      Region

  BHP                                          Australia*
  CSIRO                                        Australia
  RMIT                                         Australia
  ANCA                                         Australia

  Queens University                            Canada*
  Alberta Research Council                     Canada
  Basic Technologies Corporation               Canada
  University of Calgary                        Canada

  IPA/FhG                                      EC/Germany
  IFW/University of Hannover                   EC/Germany
  Keele University                             EC/UK
  KU Leuven Research and Development           EC/Belgium
  Mandelli                                     EC/Italy
  Nestec York Ltd                              EC/UK
  SOFTING GmbH                                 EC/Germany*
  Tecnologia Grupo INI SA                      EC/Spain
  Tekniker Research Association                EC/Spain

  Aitec Automation Inc                         EFTA/Finland
  VTT (Tech Research Center of Finland)        EFTA/Finland

  Hitachi Ltd                                  Japan*
  FANUC Ltd                                    Japan
  Hitachi Seiko Ltd                            Japan
  Keio University                              Japan
  Kobe University                              Japan
  Toshiba Corporation                          Japan
  Yaskawa Electric Co Ltd                      Japan

  The Allen-Bradley Company                    USA*
  Boston University                            USA
  Carnegie Mellon University                   USA
  United Technologies Research Center          USA
  University of California at Berkeley         USA
  University of Connecticut                    USA
  University of Illinois                       USA
  Consortium for Manufacturing Competitiveness USA

                                             * Consortium leader

4.2.5 Rapid product development

Rapid Product Development was a test case to investigate, develop, demonstrate and deploy a set of technologies and business practices which would be required to significantly improve and shorten product development cycles. For the test case period the project focused on rapid prototyping and measurement technologies.

Rapid Product Development seeks to develop and deploy a set of technologies and business practices which will lead to rapid product prototyping for a wide range of applications. Current practices in terms of technology, measurement, and management practices are assessed. The issues of communication, management, and integration to CAD and other tools will be examined. The result will be a long-term research and development program in these areas. The project is led by United Technologies of the USA. Australia, Canada, Europe and the USA are involved. There are twenty-one partners; seven companies, eleven universities and three research laboratories.

The partners in this project are shown in the table below.

Table of partners: Rapid product development

  Partner                                     Region

  HPM Industries Pty Ltd                      Australia
  QMI                                         Australia
  Swinburne University of Technology          Australia*

  Pratt & Whitney Canada                      Canada*
  Cercast                                     Canada
  Ecole de Technology Superieure              Canada
  Ecole Polytechnique of Montreal             Canada
  McMaster University                         Canada
  National Research Council                   Canada
  University of Western Ontario               Canada

  Daimler-Benz                                EC/Germany*
  Mercedes Benz-AG                            EC/Germany
  RPK, University of Karlsruhe                EC/Germany
  IPA FhG                                     EC/Germany

  United Technologies Corporation             USA*
  Sandia National Laboratories                USA
  Carnegie Mellon University                  USA
  Massachusetts Institute of Technology       USA
  Purdue University                           USA
  Rensselaer Polytechnic Institute            USA
  University of Texas                         USA

                                              * Consortium leader

4.2.6 Gnosis: systematisation of knowledge

The long-term goal of this project was to develop an approach to a post-mass-production paradigm. This test case aims to develop systemisation of knowledge for design and manufacturing, and the concept of the virtual factory.

Gnosis (Knowledge Systematisation: Configuration Systems for Design and Manufacturing) seeks to develop an approach to a post-mass-production paradigm. This paradigm would recognise the imbalances caused by the present global manufacturing environment: the growing scarcity of natural resources, the problem of environmental destruction, and the issue of regional trade imbalances. It also recognises the need for flexibility in production and product configuration leading towards integrated networks of producers and suppliers. The project is led by Mitsubishi Electric Company of Japan and involves all five regions. There are thirty-one partners; twenty-two companies, eight universities and two research institutes are involved. (The University of Calgary has supplied two partners.)

The partners in this project are shown in the table below.

Table of partners: Systematisation of knowledge

  Partner                                     Region

  Alberta Research Council                    Canada*
  DME, University of Calgary                  Canada
  KSI, University of Calgary                  Canada

  ADEPA                                       EC/France*
  Telemecanique                               EC/France
  ARM                                         EC/France
  IBM France                                  EC/France
  ITMI                                        EC/France
  Man Roland                                  EC/Germany
  Brose                                       EC/Germany
  BICC                                        EC/UK
  FAST-LLP                                    EC/France
  IPA FhG                                     EC/Germany
  Cambridge University                        EC/UK

  ABB                                         EFTA/Finland*
  Tehdasmallit Oy                             EFTA/Finland
  Synergy Center Finland                      EFTA/Finland
  FIMET                                       EFTA/Finland
  VTT (Technical Research Centre of Finland)  EFTA/Finland
  Tampere University                          EFTA/Finland
  EPFL                                        EFTA/Switzerland
  ETHZ                                        EFTA/Switzerland

  Mitsubishi Electric Corporation             Japan*
  Nissan Motors Ltd                           Japan
  Fuji Xerox Ltd                              Japan
  Kajima Construction Ltd                     Japan
  Shimizu Construction Ltd                    Japan
  University of Tokyo                         Japan
  Tokyo Institute of Technology               Japan
  Kyushu Institute of Technology              Japan

  Deneb                                       USA*
                                              * Consortium leader

4.3 Assessment of test cases

The test case program provided many important lessons. The experience was sufficiently positive to support a recommendation for a full-scale program.

The test cases themselves were industrially relevant and showed that a research program in the highly competitive area of manufacturing would attract first-class companies and universities and could be operated. The IMS feasibility study was a catalyst to form business-university networks and to resolve potential conflicts. Surprisingly, language and cultural differences did not prove to be a barrier. Rather, these differences added value to the projects and improved their quality in terms of scope, technical content and organisation. On balance, parties contributed to the success of the test cases and should benefit equitably from their results.

Regional funding mechanisms varied greatly, and this delayed the formation of some of the consortia. Intellectual property rights issues in some cases also contributed to delays.

The regional approach was useful in starting projects. However, many projects moved to a more direct form of management. Regional groups did prove useful in resolving some disputes.

The original balance of benefits has been achieved. This is true in the face of changing consortium members in some cases. The consortia benefitted from their members' ties to other networks. This was much of the reason that the test cases were able to come together quickly. Different funding mechanisms have created problems for some of the projects.

The test cases are well designed. The people and other resources are of high quality and up to the task. The attention of senior company and university officials has been engaged. All institutions are strongly committed to their projects. Collaboration is primarily horizontal, engaging technology users, developers, and suppliers.

Overheads are considered high, in the region of 25 - 30% of project costs. This was a learning experience and may be lowered in the future through the use of modern telecommunications practices. The study covered the start-up and early periods of the projects where overhead costs are concentrated.

4.4 Specific lessons learned from individual test cases

Each test case provided answers to many important questions. In what follows is a summary of the findings by the ITC Test Case Monitor and the consortium leader at the assessment conducted in Vienna, December 1993.

TC2 Clean Manufacturing in Process Industries

TC3 Concurrent Engineering for Global Manufacturing TC4 Globeman 21 TC5 Holonic Manufacturing Systems TC6 Rapid Product Development TC7 Gnosis: Systematisation of Knowledge
Next Section, Contents, IMS Page, KSI Page
gaines@cpsc.ucalgary.ca 28-Aug-94