ABSTRACT

The value added by an organization’s IT assets is a critical concern to both research and practice.  Not surprisingly, a large number of IS effectiveness measures can be found in the IS literature. What is not clear in the literature is what measures are appropriate in a particular context.  In this paper we propose a two-dimensional matrix for classifying IS Effectiveness measures.  The first dimension is the type of system studied.  The second dimension is the stakeholder in whose interests the system is being evaluated.  The matrix was tested by using it to classify IS effectiveness measures from 186 empirical papers in three major IS journals for the last nine years. The results indicate that the classifications are meaningful.  Hence, the IS Effectiveness Matrix provides a useful guide for conceptualizing effectiveness measurement in IS research, and for choosing appropriate measures, both for research and practice.

Keywords: IS research frameworks, user satisfaction, effectiveness, IS success

Total annual worldwide expenditure on information technology (IT) probably exceeds one trillion US dollars per year¹ and is growing at about 10% compounded annually. With these huge sums of money being spent on IT, one might expect that managers and researchers would devote considerable efforts to assessing which forms of IT expenditure are most effective. Indeed, there is a thriving industry consisting of trade publications, consultants, in-house IT experts, and academic researchers offering answers to questions about

• new information technologies such as client-server computing, Windows NT vs. Unix, and Enterprise Resource Planning (ERP) systems,
• new ways for organizations to manage IT, such as outsourcing, and
• new ways of designing and building more effective information systems.


However, few clear guidelines exist about how effectiveness should be measured. The purpose of this paper is to provide a clear set of guidelines for IS success measurement.

In their influential article, DeLone and McLean [1992], reviewed 100 papers containing empirical IS success measures that had been published in seven publications during the seven years 1981-1987. They classified the huge range of IS success measures they found into six categories, and towards the end of their paper present their six categories of success measures in the model shown in Figure 1. DeLone and McLean [1992, p. 87] argue that when measuring IS success, researchers should "systematically combine" measures from their six IS success categories.
 
 







Figure 1: DeLone and McLean's Model of IS Success
[DeLone and McLean [1992], Figure 2, p.87]

DeLone and McLean’s paper is an important contribution to the literature on IS success measurement because it was the first study that tried to impose some order on IS researchers’ choices of success measures. However, although it distinguishes between individual impact and organizational impact, the paper does not recognize explicitly that different stakeholders in an organization may validly come to different conclusions about the success of the same information system. By contrast, Seddon’s [1997] re-specification of DeLone and McLean’s model posits that different individuals are likely to evaluate the consequences of IT use in different ways: "IS Success is thus conceptualized as a value judgement made by an individual, from the point of some stakeholder" [Seddon 1997, p.248].
 
 



II. RESEARCH FRAMEWORK

Building on both the preceding studies and the work of Grover et al. [1996], the purpose of this paper is to present an alternative to DeLone and McLean’s model of IS success that we have found useful for framing most questions about IS effectiveness. Our framework is based on the seven questions shown in Table 1 that organizational psychologists, Cameron and Whetten [1983, pp. 270-274], argue must be answered when measuring organizational effectiveness.

Table 1: Seven Questions to Answer when Measuring Organizational Performance
[Cameron and Whetten, 1983]

1. From whose perspective is effectiveness being judged?  2. What is the domain of activity? (depends on tasks emphasized in the organization, competencies of the organization, and demands from external forces) 3. What is the level of analysis? (individual, subunit, organization, population, societal) 4. What is the purpose of evaluation? 5. What is time frame is employed? (short, long) 6. What types of data are to be used? (objective or perceptual) 7. Against which referent is effectiveness to be judged? (effectiveness of this organization compared to: some other organization; some ideal level of performance; stated goals of the organization; past performance of the organization; or certain desirable characteristics)

Looking at Table 2, it is immediately evident that measures of effectiveness appropriate for one cell might be quite inappropriate for another. For example, the IS effectiveness measures appropriate for evaluating the benefits to an individual user of some aspect of a system (row 2, column 1 in Table 2) might be increased speed of task completion and/or increased decision quality. By contrast, the IS effectiveness measures used by Hitt and Brynjolffson [1996] for evaluating the value to a nation of firms’ investments in IT (row 5, column 4) involve macroeconomic estimates of United States consumer surplus. By the nature of their subject matters and stakeholders, the measures in these two types of study need to be very different. Yet both are measures of IS effectiveness.

All but one of the measures shown in Table 2 were selected from the studies examined later in this paper in attempting to test the generality of the matrix^5. Our purpose in selecting these particular measures was to try to convey, in this simple two-dimensional representation, some sense of the range of different effectiveness measures that have been used in the past by different researchers. All the example effectiveness measures in the studies in the body of Table 2 were used by their respective researchers as indicators of whether some stakeholder, be it a person, organization, or nation, was better-off as a result of an investment of time or money in some type of endeavor involving IT.

As one looks at the range of measures in Table 2, it seems obvious that very different measures are necessary for measuring IS Effectiveness in different contexts, and that a "systematic combination" of six different types of measure as suggested by DeLone and McLean [1992], quoted earlier, is not going to work. Based on this observation, we propose that:

• diversity of IS effectiveness measures is to be encouraged, and
• Cameron and Whetten’s seven questions in Table 1 together with the matrix in Table 2 provides a useful framework for selecting appropriate measures for future IS research.

The rest of this paper examines these propositions in more detail.

The different columns in Table 2 describe different types of "system". Moving across the table from left to the right, the focus changes from aspects of information technology, to individual information systems, to types of IT system, and to a firm’s portfolio of IT-based systems. Heavier lines separate the last two columns because, unlike columns 1-4, the systems of interest in these studies are not applications of IT. Column 5 studies are interested in the effectiveness of different methodologies for developing information systems, where the methodology is thought of as "the system". Column 6 studies treat an organization’s IT function as "the system" of interest.

The different rows of Table 2 describe the different stakeholders in whose interests IS effectiveness is measured. Row 1 is used for studies where IS effectiveness is thought to be independent of the needs and wants of different stakeholders. It seems most appropriate for studies where objective measures of effectiveness, such as speed or accuracy, are available. Row 1 is also appropriate for most experiments, where the investigator, not some stakeholder with a personal interest in the system, makes the judgments of effectiveness on some reasonably objective basis. Neither Cameron and Whetten [1983] nor Grover et al. [1996] include independent stakeholders in their frameworks, yet there seems to be a need for such a class of stakeholder in a discipline where objective measures of effectiveness, such as response times and levels of transaction security, are valid measures for some studies. This row was not initially in our matrix, but during pilot testing, we discovered it helped resolve a number of classification difficulties.

Row 2 in Table 2 is for studies that focus on benefits from the point of view of individuals. Benefits individuals receive from use of information technology include increased productivity, better decision-making, faster promotion (if the system helps them perform more effectively than others), and possibly, political advantage. In the research we reviewed, individual benefits were explored for all six types of system in Table 2. Therefore, no cells are empty in row 2.

Row 3 concerns effectiveness measures that relate to groups. Although one could argue that groups are just short-term organizations, the measures in the group decision support (GDSS) literature are so group-specific (e.g., equality of participation) that it seems better to introduce a special class of stakeholder that recognizes the distinctive characteristics of groups. GDSS studies often collect information about both group performance and individual performance/satisfaction. As a result, many GDSS studies use measures of effectiveness from both row 2 and row 3.

Row 4 is for studies where IS Effectiveness is measured from the point of view of the management or owners of an organization. Although the potential difficulties of achieving goal congruence between management at different levels of an organization and the owners is well known, it is assumed in Table 2 that these interests are similar enough to be grouped in one row. IS Effectiveness measures appropriate for row 4 tend to have a strong economic flavor. For example, Weill [1992] says "the focus of this paper is on the firm’s portfolio of systems" [p.311], and he measures firm growth, return on assets, % change in labor, and market share. It is clear that Weill’s measures are based on the point of view of management and owners of the firms, and that because they relate to all IT applications in the 33 firms he studied, they belong in row 4, column 4 of the matrix.

In the last row of Table 2, the interests involved are now those of a country, and the choice of the most appropriate IS Effectiveness measure is expected to change again. As shown in Table 2, e.g., the Hitt and Brynjolfsson [1996] study, the measures of effectiveness most useful for evaluating the impact of different information systems or technologies for a country are very different from those one would use in, say, the top row of Table 2.

The reason for drawing this row-by-row distinction among the different types of stakeholder in Table 2 is when one system is evaluated, by one person, on behalf of different stakeholders, different responses can be obtained. To illustrate, Table 3 shows a small sample of responses from data collected for a recent study of data warehousing success [Seddon & Benjamin 1998]. Column headings show the exact questions asked. Entries in the table are from the tape-recorded transcript. The units of analysis are, first, the data warehousing system evaluated from the organization’s point of view, and second, the same system evaluated from respondent’s point of view. Note that the responses in the right-hand column are more frank, identify different salient issues, shift in focus from "they" to "I", and may come to opposite conclusions! Table 3 demonstrates that those evaluating computer systems must make it very clear (to the respondent, themselves, and their readers) on whose behalf the evaluation is performed.
 
 

Table 3. Transcript Responses from Interviewees about
Data Warehousing Success

The discussion so far focused on measures of effectiveness of the different IT applications in columns 1-4 of Table 2. The measures in columns 5 and 6 are also measures of system effectiveness, but the "system" is now either an aspect of a methodology for building systems, or the IT function in an organization. Recall that column 5 is concerned with the effectiveness of systems for changing information systems. In Column 5 of Table 2, McKeen et al. [1994] measured satisfaction of individual users in their study of the effect of user participation on system effectiveness. Therefore, their effectiveness measure is classified in row 2, column 5. By contrast, Caron et al. [1994] measured cost savings, quality improvement, and customer satisfaction in theirstudy of reengineering at CIGNA insurance. Because their effectiveness measures reflect the (presumed) interests of management, not the individual employee, their measures are classified in row 4, column 5. Column 5 is included in the IS Effectiveness matrix because of the importance of system development methodologies in the application of IT, and the need to compare the effectiveness of different change practices.

In Column 6, the system of interest is the IS/IT function itself. How effective is it? Pitt et al.’s [1995] use of "Service Quality" for evaluating the effectiveness of the central IT functions of three firms is a row 2, column 6 measure. Pitt et al. collected opinions from some hundreds of individual users in each firm, so the stakeholders in their study were classified as individual users. By contrast, Lacity and Hirschheim’s [1993] book on outsourcing, which also involves the assessment of the effectiveness of central IT functions (in 21 organizations), adopts the point of view of senior management. Hence, Lacity and Hirschheim’s measures are classified as more economics-oriented row 4, column 6 effectiveness measures. Although the opinions of individuals within a firm may inform the judgments of senior management in Lacity and Hirschheim’s study, the nature of the evaluations is much more concerned with accounting profitability and return on investment than the opinions of individual users.

The classification scheme in Table 2 looks plausible, but does it work for all studies of IS effectiveness? To test the generality of the matrix, we followed DeLone and McLean [1992], and Grover et al. [1996], and attempted to use our framework (the matrix) to classify the IS effectiveness measures used in prior studies. DeLone and McLean's research methology involved

    1. proposing a list of 6 categories of IS Success measure that seemed, from their point of view, to make sense, then
    2. classifying the measures found in a sample of the literature those six categories.

They note that classification was often not clear cut: “all of these classification decisions are somewhat arbitrary" (DeLone and McLean 1992, pp. 63-4).  Where a study used multiple measures, they classified it into more than one category. Our methodology is similar.  Based on the theoretical work of Cameron and Whetten (1982), we proposed a classification scheme that seemed to make sense (Table 2).  Then we analyzed a sample of papers from the literature attempting to classify the measures used in those papers in terms of our two dimensions.

DeLone and McLean reviewed the literature for the seven years from 1981 to 1987. We decided to review the next nine recent years, from 1988 to 1996. The three journals we decided to review were all major U.S. journals: Management Information Systems Quarterly (MISQ), Information Systems Research (ISR) (from 1990), and the Journal of Management Information Systems (JMIS). These three leading IS journals seemed likely to reveal the best of IS Effectiveness measurement practice used during the last decade. Our objective was to identify and classify all empirical studies where IS Effectiveness was the dependent variable, and in particular, to identify any cases where the variables used did not fit readily into the IS Effectiveness Matrix.

Step 1 in this review process was to identify empirical papers that used IS effectiveness measures as dependent variables. Step 2 was to classify the measures. For both steps, two co-authors of this paper reviewed each article in each journal independently, then met to resolve disagreements. The five cases shown in Table 4 illustrate some of the more difficult decisions we encountered in Step 1

The reason for presenting these five borderline-classification examples is to give the reader some idea of the range of measures included in the analysis. In particular, the last two examples illustrate the broad notion of "system" used in this study. We debated whether the column 5 and 6 measures of effectiveness belonged in the framework at all. On balance, we decided they were worth including because:

• studies in columns 5 and 6 need effectiveness measures of some kind, and
• it is helpful to maintain awareness

(both more precise in their measurement, and easier for the reader to understand) if they had identified the unit of analysis (the stakeholder and system) more clearly.

The result of our classification efforts is available as a 200-row table in the Appendix. A summary of the data is presented in Table 6. Table 6 shows the frequency of occurrence of IS effectiveness measures for each combination of stakeholder and system. The sum of entries in the cells in Table 6 adds to 200, not 186 (the number of papers analyzed), because some papers used measures from the point of view of more than one stakeholder.

Table 6: Frequency of Occurrence of IS Effectiveness Measures for Each Combination
of System and Stakeholder

DeLone and McLean [1992] analyzed 100 empirical papers containing IS effectiveness measures, from 1981-1987, and found a multitude of different measures. After arguing that a reduction in the number of measures was desirable, they classified these measures into six categories. In this paper, we analyzed 186 empirical papers from 1988-1996, and we, too, found a multitude of measures. However, unlike DeLone and McLean, we do not believe that this diversity of measures is a problem.

This paper’s first insight is that in a world of conflicting human interests and vastly different systems, different sharply-focused measures of IS effectiveness are likely to be needed for different purposes. While we adopted a positivist perspective in our research, we do not mean to imply that the impact of a system could be constrained to one group of stakeholders. As the IS research community knows, introducing a system can have unforeseen social and political impacts. Our message is simply that different measures are likely to be needed to assess the impact and effectiveness of a system for different groups of stakeholders. We suggest this is an important message given the growth of empirical IS research studies [Farhoomand & Drury 1999]. Table 2 and the Appendix can assist in identifying:

• appropriate measures that should be combined in a study to assess effectiveness from different stakeholders’ views,
• units of analysis that received little attention from researchers previously.

The second insight of this paper is that :

• Cameron and Whetten’s [1983] seven questions (Table 1) define the construct space for IS effectiveness measurement, and
• two key dimensions of this construct space are the Stakeholder and the type of IT System being evaluated.

These two dimensions define the IS effectiveness matrix shown in Table 2. Table 3 in this paper (from a study of data warehousing success) illustrates how subtle differences in stakeholder perspective can produce significantly different evaluations of systems.

Combining the above two insights, we suggest that Cameron and Whetten’s seven questions and the two-dimensional IS Effectiveness matrix presented in this paper (Tables 1 and 2) provide useful ways of framing most discussions about IS Effectiveness measurement. The matrix approach is simpler than Grover et al.’s [1996] -- simple enough to go in a textbook discussion on IS effectiveness -- yet it captures the essence of IS Effectiveness measurement. It contributes to the IS literature because it helps researchers organize the huge diversity of measures used in IS effectiveness research into a simple two-dimensional framework. Certainly, the IS Effectiveness matrix was useful in clarifying our own thinking when studying and discussing IS effectiveness. Other researchers also report that they found it to be of value.

We also found the matrix useful when talking with practitioners. For example, recently the IT executive from a local government authority approached the first author of this paper concerned that in a recent survey his IT organization had been criticized as being unresponsive to user needs. He was worried, but the survey had been very general, and he really had no idea of what was wrong. His question to us was: "Did we know of a questionnaire he could use to get a clearer understanding of what was wrong?" When asked if he wanted to assess one particular system, all systems, system development methodologies, or service provided to users by his IT department (i.e., the columns of the matrix), it was clear that he had never thought in such terms. Yet we as researchers knew that the questions needed for these different measurement goals are very different! A brief discussion based around the matrix helped us clarify what was needed. The executive wanted individual user views about quality of service offered by his IT department. We pulled out the literature on SERVQUAL [Watson, Pitt & Kavan, 1998] and SERVPERF [Kettinger & Lee, 1997], and he was soon on his way. The IS Effectiveness Matrix helped to clarify his thinking about what sort of measures were required.

For the future, we recommend that anyone requiring an IS Effectiveness measure should endeavor to answer all seven questions from Table 1 before commencing their evaluation. Further, we strongly recommend that when reporting results of IS effectiveness evaluations, authors of reports should always make clear what type of system they were studying, and on whose behalf the evaluation was conducted.

END NOTES

¹ Total revenue for the Datamation (1997) top 100 IT-producing firms in the world was US$502 billion in 1996 (up 13% from $443 billion in 1995).  If in-house expenditure on staff and system development and output from smaller IT firms is included, it seems safe to assume that worldwide IT expenditure is at least double this amount.  Hence the estimate of annual expenditure of one trillion US dollars on IT.
² It is not always useful to combine these two questions.  For example, at the organizational unit of analysis, studies of outsourcing often report that a firm’s IT manager and the chief executive officer (CEO) may have a different views of IT effectiveness.
³We did not use Cameron and Whetten’s “Industry” group, nor Grover et al.’s “IS Personnel”.
⁴The IT function is a system for making IT resources more readily available to other parts of the organization.
^{5The one study not selected from the three journals (ISR, JMIS, and MISQ) is Lacity and Hirschheim’s [1993] book on outsourcing. Judgements concerning the effectiveness of outsourcing arrangements provide an excellent example of senior management evaluation of the effectiveness (row 4) of an IT function (column 6). Lacity and Hirschheim’s work provides a better example of what we mean by a row 4, column 6 study than any of the papers published in the journals studied during 1988-1996. Another good example of a row 4, column 6 study is Lacity and Willcocks [1998], but it is outside the timeframe of this survey.}
 

An earlier version of this paper was published in the Proceedings of the International Conference on Information Systems (ICIS) 1998 held in Helsinki, Finland in December, 1998. We wish to thank the anonymous reviewers and editors of both ICIS and CAIS for their helpful comments and suggestions.

Editor’s Note: This article was received on July 5, 1999. It was with the authors approximately 1 month for 1 revision. The article was published on November 6, 1999.

Alavi, M., B.C. Wheeler, and J.S. Valacich (1995) "Using IT to Reengineer Business Education: An Exploratory Investigation of Collaborative Telelearning", Management Information Systems Quarterly, (19)3, pp. 293-312.

Barki, H. and J. Hartwick (1994) "User participation, conflict, and conflict resolution: the mediating roles of influence", Information Systems Research, (5)4, pp. 422-438.

Bretchneider, S. and D. Wittmer (1993) "Organizational adoption of microcomputer technology: the role of sector", Information Systems Research (4)1, pp. 88-108.

Cameron, K.S. and D.A. Whetten, (1983) "Some conclusions about organizational effectiveness", in K.S. Cameron, and D.A Whetten, (eds). Organizational Effectiveness: A Comparison of Multiple Models, New York: Academic Press, pp. 261-277.

Caron J.R., S.L. Javenpaa, and D.B. Stoddard (1994) "Business reengineering at CIGNA corporation: experience and lessons from the first five years", MIS Quarterly, (18)3, pp. 233-250.

Compeau, D.R. and C.A. Higgins (1995) "Application of social cognitive theory to training for computer skills", Information Systems Research, (6)2, pp. 118-143.

Cronan, T.P. and D.E. Douglas (1990) "End-user Training and Computing Effectiveness in Public Agencies: An Empirical Study", Journal of Management Information Systems, (6)4, pp. 21-39.

Davis, F.D. (1989) "Perceived usefulness, perceived ease of use, and user acceptance of information technology", MIS Quarterly, (13), pp. 319-340.

DeLone, W.H., and E.R. McLean (1992) "Information systems success: the quest for the dependent variable", Information Systems Research, (3), pp. 60-95.

Dos Santos, B.L., K. Peffers and D.C. Mauer (1993) "The impact of information technology investment announcements on the market value of the firm", Information Systems Research, (4)1, pp. 1-23.

Duxbury, L.E., C.A. Higgins, and S. Mills (1992) "After-Hours Telecommuting and Work-Family Conflict: A Comparative Analysis", Information Systems Research, (3)2, pp. 173-196.

Farhoomand, A.F. and D.H. Drury (1999) "A Historiographical Examination of Information Systems", Communications of the Association for Information Systems, (1)19. URL: http://cais.aisnet.org/contents.asp.

Goodhue, D.L. and R.L. Thompson (1995) "Task-Technology Fit and Individual Performance", Management Information Systems Quarterly (19)2, pp. 213-236.

Grover, V., S.R. Jeong, and A.H. Segars (1996) "Information systems effectiveness: The construct space and patterns of application", Information and Management, (31), pp. 177-191.

Hess, C.M. and C.F. Kemerer (1994) "Computerized Loan Origination Systems: An Industry Case Study of the Electronic Markets Hypothesis", MIS Quarterly (18)3, pp. 251-276.

Hitt, L.M. and E. Brynjolfsson (1996) "Productivity, business profitability, and consumer surplus: three different measures of information technology value", MIS Quarterly, (20)2, pp. 121-142.

Kettinger, W.J. and C.C. Lee (1997). "Pragmatic Perspectives on the Measurement of Information Systems Service Quality", MIS Quarterly, (21)2, pp. 223-240.

Kraemer, K.L., J.N. Danziger, D.E. Dunkle, and J.L. King (1993) "The Usefulness of Computer-Based Information to Public Managers", MIS Quarterly, (17)2, pp. 129-148.

Lacity, M.C. and R. Hirschheim (1993) Information Systems Outsourcing: Myths, Metaphors, and Realities, Wiley.

Lacity, M.C. and L.P. Willcocks (1998) "An Empirical Investigation of Information Technology Sourcing Practices: Lessons from Experience", MIS Quarterly, (22)3, pp. 363-408.

Lederer, A.L. and V. Sethi (1996) "Key Prescriptions for Strategic Information Systems Planning", Journal of Management Information Systems, (13)1, pp. 35-62.

Leidner, D.E. and J.J. Elam (1993) "Executive Information Systems: Their Impact on Executive Decision Making", Journal of Management Information Systems, (10)3, pp. 139-155.

Massetti, B. (1996) "An Empirical Examination of the Value of Creativity Support Systems on Idea Generation", Management Information Systems Quarterly, (20)1, pp. 83-98.

McKeen, J.D., T. Guimaraes, and J.C. Wetherbe (1994) "The relationship between user participation and user satisfaction: an investigation of four contingency factors", MIS Quarterly (18), pp. 427-451.

McLeod, P.L. and J.K. Liker (1992) "Electronic Meeting Systems: Evidence from a Low Structure Environment", Information Systems Research (3)3, pp. 195-223.

Mookerjee, V.S., M.V. Mannino, and R. Gilson (1995) "Improving the Performance Stability of Inductive Expert Systems Under Input Noise", Information Systems Research (6)4, pp. 328-356.

Mukhopadhyay, T., S. Kekre, and S. Kalathur (1995) "Business Value of Information Technology: A Study of Electronic Data Interchange", Management Information Systems Quarterly (19)2, pp. 137-156.

Mukhopadhyay, T., S.S. Vicinanza and M.J. Prietula (1992) "Examining the Feasibility of a Case-Based Reasoning model for Software Effort Estimation", Management Information Systems Quarterly (16)2, pp. 155-171.

Nault, B.R. and A.S. Dexter (1994) "Added Value and Pricing with Information Technology", Management Information Systems Quarterly (19)4, pp. 449-464.

Pitt, L.F, R.T. Watson, and C.B. Kavan (1995) "Service quality: a measure of information systems effectiveness", MIS Quarterly (19), pp. 173-188.

Sambamurthy, V. and M.S. Poole (1992) "The Effect of Variations in Capabilities of GDSS Designs on Management of Cognitive Conflict in Groups", Information Systems Research (3)3, pp. 224-251.

Seddon, P.B. (1997) "A Respecification and Extension of the DeLone and McLean model of IS Success", Information Systems Research (8)3, pp. 240-253.

Seddon, P.B. and N. Benjamin (1998) "What do we know about Successful Data Warehousing?" Australasian Conference on Information Systems, Sydney, Australia.

Subramanian, G.H. and G.E. Zarnich (1996) "An Examination of Some Software Development Effort and Productivity Determinants in ICASE Tool Projects", Journal of Management Information Systems, (12)4, pp. 143-160.

Watson, H.J., R.K. Rainer Jr., and C.E. Koh (1991) "Executive Information Systems: A Framework for Development and a Survey of Current Practices", Management Information Systems Quarterly (15)1, pp. 13-30.

Watson, R.T., L.F. Pitt, and C.B. Kavan (1998). "Measuring Information Systems Service Quality: Lessons from Two Longitudinal Case Studies", MIS Quarterly, (22)1, pp. 61-79.

Weill, P. (1992) "The relationship between investment in information technology and firm performance: a study of the valve manufacturing sector", Information Systems Research, (3)4, pp. 307-333.

Ye, L.R. and P.E. Johnson (1995) "The Impact of Explanation Facilities on User Acceptance of Expert Systems Advice", Management Information Systems Quarterly (19)2, pp. 157-172.

Zack, M.A. (1993) "Interactivity and Communication Mode Choice in Ongoing Management Groups", Information Systems Research, (4)3, pp. 207-239.

Classification of measures by stakeholder and type of system, papers sorted alphabetically. Row and column refer to Table 2. The following abbreviations are used:

ISR Information Systems Research

JMIS Journal of Management Information Systems

MISQ Management Information Systems Quarterly