Chapter 4 — Criteria: Definitions, Measures, and Evaluation

Chapter 4 opens with eight learning goals, numbered 4.1 through 4.8, reproduced verbatim below since they map directly onto the chapter's own section headings.

1. 4.1 Define and identify criteria—yardsticks used to assess employee success.
2. 4.2 Distinguish the static, dynamic, and individual dimensions of criteria and their implications.
3. 4.3 Define and measure contextual, task, and counterproductive behaviors.
4. 4.4 Develop criteria that address challenges such as job performance unreliability, unreliability in the observation of performance, and the multidimensionality of performance.
5. 4.5 Consider the importance of situational determinants of performance and develop and evaluate criteria using standards such as relevance, sensitivity, and practicality.
6. 4.6 Develop criteria that will minimize the detrimental impact of criterion deficiency and contamination and choose whether to use composite or multiple criteria.
7. 4.7 Distinguish observed from unobserved criteria and their antecedents.
8. 4.8 Consider nonnormal distributions of performance and their implications in terms of the presence and production of star performers.

Developing adequate, appropriate criteria is at once a stumbling block and a challenge to the HR specialist. Behavioral scientists have bemoaned this criterion problem for decades — the difficulty of conceptualizing and measuring performance constructs that are multidimensional, dynamic, and purpose-dependent (Austin & Villanova, 1992). Nearly every HR policy's effectiveness depends on resolving it: you cannot validate a selection test or fairly promote someone without first deciding what counts as success. The twin goals of criterion research are enhancing the utility of existing procedures and deepening our understanding of the psychological processes behind job performance — ultimately, a comprehensive theory of performance at work (Campbell & Wiernik, 2015; Viswesvaran & Ones, 2000).

Historically the field treated criteria carelessly. Jenkins (1946) observed that early researchers tacitly assumed criteria were either given by God or just to be found lying about — whatever measure was on hand got used, with little scrutiny. Cascio and Aguinis call this regrettable: even today, practitioners often reach for the most expedient criteria when better ones are attainable with more effort. Progress has come from recognizing that criterion measures are samples of a larger performance universe, deserving as much validation effort as predictors get (Campbell & Wiernik, 2015). As Wallace (1965) put it, the answer to "Criteria for what?" must include "for understanding" (p. 417).

Criteria are standards used as yardsticks for measuring employees' degree of success on the job (Bass & Barrett, 1981; Guion, 1965; Landy & Conte, 2016) — adequate when the goal is prediction, establishing a functional relationship between a predictor and a criterion, as in selection, placement, promotion, and succession planning. But sometimes the goal is evaluation without prediction: the chapter's example is an HR department assessing whether a recruitment campaign succeeded at attracting underrepresented groups (e.g., women for STEM roles). Various criteria evaluate the program, but nothing is being predicted.

The Time-Based Distinction Between Predictors and Criteria

Mullins and Ratliff (1979) supply the cleanest test: time. A standard administered before an employment decision (hire or promote) is a predictor; the same kind of standard administered after the decision, to evaluate effectiveness, is a criterion. The identical instrument can be either, depending on when it's used in the employment sequence. This leads to a more general definition: a criterion represents something important or desirable — an operational statement of the goals or desired outcomes of a program (Astin, 1964), an evaluative standard for measuring performance, attitude, or motivation (Blum & Naylor, 1968).

Table 4.1 — Possible Measures of Criteria

The chapter reproduces a taxonomy of criterion measures (modified from Dunnette & Kirchner, 1965; Guion, 1965; and others), organized into six families. Many fall short as adequate criteria alone, but each deserves study toward a comprehensive sample of job performance.

Performance can be defined as what people do or what people produce. Campbell and Wiernik (2015), Aguinis (2019), and Beck, Beatty, and Sackett (2014) define performance behaviorally — actions relevant to organizational goals; Campbell and Wiernik (2015) insist "performance should be specified in behavioral terms as things that people do" (p. 67). Minbashian and Luppino (2014), O'Boyle and Aguinis (2012), and Aguinis, O'Boyle, Gonzalez-Mulé, and Joo (2016) instead define performance as results — what people produce. Viswesvaran and Ones (2000) blend both: performance is "scalable actions, behavior and outcomes that employees engage in or bring about that are linked with and contribute to organizational goals" (p. 216).

The two are related, not competing: effort (behavior) tends to produce better outcomes (results), correlated at nontrivial levels (Beal, Cohen, Burke, & McLendon, 2003; Bommer, Johnson, Rich, Podsakoff, & MacKenzie, 1995). The real question is when to use each. Aguinis and O'Boyle (2014) give the decision rule for results-based measures: appropriate when (a) workers are already skilled in the needed behaviors, (b) behaviors and results are obviously related, and (c) there are many ways to do the job right (p. 316). Results-based definitions also dominate at the organizational level, since firm performance is judged by output, not method (Boudreau & Jesuthasan, 2011; Cascio & Boudreau, 2011a).

The Ultimate Criterion

Thorndike (1949) coined ultimate criterion for the full domain of performance — every behavior and result that ultimately defines job success, with no further standard beyond it. The chapter's worked example: a salesperson's ultimate criterion would include quality of customer interactions, time with customers, product knowledge, total sales volume, new accounts, customer loyalty built, influence on colleagues' morale and sales, and overall effectiveness in planning, expense control, and reporting.

Operational measures of the conceptual criterion vary along several dimensions. Ghiselli (1956) identified three types: static, dynamic (temporal), and individual dimensionality — the chapter's organizing framework for Learning Goal 4.2.

Static Dimensionality

Performance observed at one point in time is already multidimensional: (1) individuals can be high on one facet and low on another, and (2) maximum and typical performance must be distinguished. Rush (1953) found several independent selling skills: learning aptitude (sales-school grades) was unrelated to objective achievement (sales volume, quota), which was independent of general reputation, which was independent of sales technique — four largely uncorrelated facets.

Task Performance vs. Contextual Performance

More broadly, the chapter names task performance and contextual performance (Borman & Motowidlo, 1997) — the latter also called pro-social behavior or organizational citizenship performance (Borman, Brantley, & Hanson, 2014). The two do not necessarily go together (Bergman, Donovan, Drasgow, Overton, & Henning, 2008); an employee can excel at task performance while underperforming contextually (Bergeron, 2007). Task performance is (a) activities transforming raw materials into the organization's goods/services and (b) supporting activities — replenishing supplies, distributing products, planning, coordinating, supervising (Cascio & Aguinis, 2001). Contextual performance is behavior that creates a good environment for task performance, including:

• Persisting with enthusiasm and exerting extra effort on one's own tasks (punctuality, rarely absent).
• Volunteering for activities not formally part of the job (suggesting improvements).
• Helping and cooperating with others (assisting coworkers and customers).
• Following organizational rules and procedures.
• Endorsing, supporting, and defending organizational objectives (loyalty, representing the organization well to outsiders).

The "Dark Side": Workplace Deviance and Counterproductive Behaviors

Researchers have identified a dark side of contextual performance — workplace deviance or counterproductive behavior (Marcus, Taylor, Hastings, Sturm, & Weigelt, 2016; Spector et al., 2006). Though they seem like opposite poles of one continuum, evidence shows contextual performance and deviance are distinct constructs (Judge, LePine, & Rich, 2006; Kelloway, Loughlin, Barling, & Nault, 2002). Workplace deviance is voluntary behavior that violates organizational norms and threatens the well-being of the organization or its members (Robinson & Bennett, 1995); Vardi and Weitz (2004) catalogued over 100 such misbehaviors. Self-reported examples include:

• Exaggerating hours worked; falsifying an expense receipt
• Starting negative rumors; gossiping about coworkers or a supervisor
• Covering up mistakes, or blaming coworkers for them
• Competing with coworkers unproductively
• Staying out of sight to avoid work
• Taking company equipment or merchandise
• Working slowly or carelessly on purpose
• Being intoxicated during working hours
• Seeking revenge on coworkers; presenting colleagues' ideas as one's own

Typical vs. Maximum Performance

Typical performance is an employee's average level; maximum performance is the peak they can achieve (DuBois, Sackett, Zedeck, & Fogli, 1993; Sackett, Zedeck, & Fogli, 1988). Employees hit maximum levels when they know they're being evaluated, accept instructions to maximize the task, and the task is short. A meta-analysis of 42 studies (N = 4,129) found maximum and typical performance correlate only .33 on average (Beus & Whitman, 2012) — a large gap, since general mental ability correlates more strongly with maximum performance (r = .25) than typical performance (r = .16).

Operational measures must be taken at some point in time, and optimum timing varies by situation — results can differ sharply depending on when measurements are taken (Weitz, 1961). For life insurance agents, ability predicts early sales success, but interests and personality matter more later (Ferguson, 1960); the same holds for accountants, where interpersonal skill eventually outweighs technical expertise (Bass & Barrett, 1981). Barrett, Caldwell, and Alexander (1985) identify three ways criteria can be dynamic: changes in average group performance, changes in validity coefficients, and changes in rank ordering — over time.

1. Changes in Average Group Performance

Ghiselli and Haire (1960) tracked investment salespeople for 10 years and found a 650% productivity improvement with no leveling off — but only among survivors of the full 10 years, so experience must be equalized for fair comparison (Ghiselli & Brown, 1955). Other evidence (Barrett et al., 1985) does not consistently show productivity rising over long spans, so this finding does not generalize automatically.

2. Changes in Validity Coefficients

Bass (1962) tracked 99 salespeople for 42 months using ability tests and peer ratings as predictors against semiannual merit ratings. Validities appeared to fluctuate erratically, but statistically there were no significant differences for the ability tests, and only about 20% of peer-rating validity pairs differed significantly (Barrett et al., 1985). Two competing explanations exist for why validities might change: the changing task model (criteria for effective performance shift while ability stays stable) and the changing subjects model (individuals' skill changes while job requirements stay constant) (Henry & Hulin, 1987) — neither fully supported.

3. Changes in Rank Ordering

This third form has drawn the most research attention (Hofmann, Jacobs, & Baratta, 1993; Hulin, Henry, & Noon, 1990): if rank ordering changes over time, future performance becomes a moving target and prediction deteriorates the farther out you go (Keil & Cortina, 2001). Correlations among performance measures over time show a simplex pattern — higher between adjacent time points, lower across longer gaps (Steele-Johnson, Osburn, & Pieper, 2000). Deadrick and Madigan (1990) confirmed this with weekly data from sewing machine operators, concluding relative performance is not stable over time — a conclusion Hulin et al. (1990), Hofmann et al. (1993), and Keil and Cortina (2001) all reached, with implications for how individuals evaluate one another (Reb & Cropanzano, 2007).

Wearable Sensors and Intraindividual Performance Fluctuations

Wearable sensors (devices akin to a Fitbit) now let researchers capture performance fluctuations with far more precision (Chaffin et al., 2017; Tomczak, Lanzo, & Aguinis, 2018) — a smartphone can log ambient sound, proximity to people and devices, GPS location, or check-ins correlated with physiological markers like heart rate (Beal, 2015). This enables monitoring on a monthly, weekly, daily, or hourly basis, opening the research area of intraindividual performance fluctuations (within-person analysis) — big-data collection previously unthinkable in HR research (Harlow & Oswald, 2016), useful for questions like whether daily citizenship-behavior fluctuations trace to the work itself or the social environment (Spence, Ferris, Brown, & Heller, 2012). The major conclusion: within-person variability is not necessarily faulty measurement (Dalal, Bhave, & Fiset, 2014) — performance genuinely moves over time, so prediction should target a prespecified time span rather than assume a fixed score.

Individuals performing the same job may be equally good, yet contribute in quite different ways — requiring different criterion dimensions to evaluate them. Kingsbury (1933) made this point nearly a century ago: some executives succeed as planners but fail as directors, others excel at directing but produce weak plans, and few are equally competent at both. Failing to recognize this in testing and rating, he argued, is a major reason attempts to study and test executives fail — good tests of one kind of executive ability are not good tests of the other (p. 123).

Although Kingsbury described one managerial job, it could plausibly be treated as two — directing and planning — that only differ psychologically. Studying individual criterion dimensionality is thus a useful way to determine whether the same job, performed by different people, is psychologically the same job or a different one for each person.

Competent criterion research remains one of personnel psychology's most pressing needs. Stuit and Wilson (1946) showed that continuing attention to better performance measures yields better predictions — a conclusion Viswesvaran and Ones (2000) confirm still holds.

Ronan and Prien (1966, 1971) frame four basic challenges: reliability of performance, reliability of performance observation, dimensionality of performance, and situational modification of performance. The first three appear below; the fourth is covered in the next section.

Challenge #1: Job Performance (Un)Reliability

Job performance reliability — consistency of performance over time — is a fundamental, usually implicit assumption behind predictive studies. Are the best (or worst) performers at Time 1 also best (or worst) at Time 2? Thorndike (1949) distinguished intrinsic unreliability (personal inconsistency) from extrinsic unreliability (external variability — weather, machine downtime, supply delays), much of which traces to careless observation or poor control.

One remedy is aggregation — averaging behavior over many occasions to cancel incidental factors. Epstein (1979, 1980) ran four studies sampling behavior repeatedly over weeks (self-ratings, others' ratings, objective behaviors, personality inventories, heart rate) and found stability emerges reliably once behavior is averaged over enough occurrences; once reliability is adequate, validity evidence follows. Martocchio, Harrison, and Berkson (2000) similarly found that longer aggregation periods strengthened the validity coefficient between employee lower-back pain and absenteeism.

Two caveats: there is no shortcut to adequately sampling the domain you generalize over, and aggregation is no panacea — systematic effects (sex, race, rater attitudes) can bias entire groups of studies, which is one reason meta-analysis (Chapter 7) exists. Rambo, Chomiak, and Price (1983) showed reliability depends on task complexity and environmental constancy: weekly data from sewing-machine and folding/packaging operators under piece-rate pay showed extraordinary consistency — week-to-week correlations of .94 and .98, year-over-year .69 and .86, and even week 1 vs. week 178 correlations of .59 and .80. Stable, routinized tasks with production-linked incentives produce highly consistent output. A later six-year study of foundry chippers and grinders under individual incentive pay was broadly consistent (Vinchur, Schippmann, Smalley, & Rothe, 1991), though reliability varies by job content.

Challenge #2: Reliability of Job Performance Observation

Different observation methods can yield markedly different conclusions. Bray and Campbell (1968) validated assessment-center sales-potential predictions against field performance six months later, measured two ways: an independent auditor riding along on field visits (unaware of the predictions), and ratings from supervisors and trainers (also unaware). Assessment-center predictions correlated .51 with the auditor's ratings — but showed no significant relationship with supervisors' or trainers' ratings, and the field ratings didn't correlate with the supervisor/trainer ratings either. The lesson: studying performance reliability is only possible once the reliability of judging performance is itself adequate (Ryans & Fredericksen, 1951); there is no silver bullet for improving it (Borman & Hallam, 1991) — Chapter 5 goes deeper.

Challenge #3: Dimensionality of Job Performance

Reviews (Campbell & Wiernik, 2015; Ronan & Prien, 1966, 1971) conclude a unidimensional measure of job performance is unrealistic even for lower-level jobs — single measures like absenteeism turn out far more complex than they appear. Global criteria still tend to work well in most selection situations, but solving a specific problem (e.g., too many quality complaints) calls for a specific criterion, and multiple specific problems call for multiple specific criteria (Guion, 1987).

Individual performance is shaped by its surrounding conditions, yet most research ignores variables beyond the predictors measured. The chapter examines six extraindividual influences, together illustrating in situ performance (Cascio & Aguinis, 2008b): "the specification of the broad range of effects—situational, contextual, strategic, and environmental—that may affect individual, team, or organizational performance" (p. 146).

• Environmental and organizational characteristics — absenteeism and turnover relate to organizational factors (pay, promotion, HR practices), interpersonal factors (cohesiveness, satisfaction with peers/supervisors), job factors (role clarity, autonomy, repetitiveness), personal factors (age, tenure, mood), and shift work (Allen & Vardaman, 2017; Dineen, Noe, Shaw, Duffy, & Wiethoff, 2007; McEvoy & Cascio, 1987; Sun, Aryee, & Law, 2007; Barton, 1994; Staines & Pleck, 1984).
• Environmental safety — injuries and lost time affect performance (Probst, Brubaker, & Barsotti, 2008); a positive safety climate, management commitment, and sound safety communication with goal setting increase safe behavior (Reber & Wallin, 1984) and resource conservation (Siero, Boon, Kok, & Siero, 1989), and can be measured reliably (Zohar, 1980; Hofmann, Burke, & Zohar, 2017).
• Lifespace variables — conditions surrounding an employee on and off the job. Vicino and Bass (1978) used four (first-assignment task challenge, life stability, supervisor–subordinate personality match, supervisor's own success) to explain an additional 22% of variance in management success at Exxon beyond the company's own system (multiple R of .79); personal orientation, career confidence, cosmopolitan-versus-local orientation, and job stress are other candidates (Cooke & Rousseau, 1983; Edwards & Van Harrison, 1993).
• Job and location — Schneider and Mitchel (1980) related six behavioral job functions for 1,282 life-insurance agency managers to five situational variables (agency origin/type, number of agents/supervisors, manager tenure); the most variance any function explained was 8.6%, meaning over 90% of variance in managerial functions came from other sources — company-specific policies and practices matter alongside job demands.
• Extraindividual differences and sales performance — Cravens and Woodruff (1973) used a curvilinear regression on dollar sales volume (corrected R2 = .83), combining extraindividual influences (territory workload, market potential, advertising effort) with individual-difference variables (sales experience, rated effort) to generate a purer estimate of individual performance.
• Leadership — leadership and situational factors are well documented to affect morale and performance (Detert, Treviño, Burris, & Andiappan, 2007; Srivastava, Bartol, & Locke, 2006). Altogether, performance variation stems from individual, group, and organizational characteristics; until total variability can be partitioned into intraindividual and extraindividual components, predictors of individual differences should not be expected to correlate strongly with performance measures heavily shaped by factors outside a person's control.

Guion (1961) outlines a five-step procedure, presented as the practical synthesis of the discussion so far.

1. Analysis of job and/or organizational needs.
2. Development of measures of actual behavior relative to expected behavior identified in the needs analysis — supplementing objective outcome measures like turnover, absenteeism, and production.
3. Identification of criterion dimensions underlying such measures, via factor, cluster, or pattern analysis.
4. Development of reliable measures, each with high construct validity, of the elements identified.
5. Determination of the predictive validity of each predictor for each criterion measure, taken one at a time.

Step 2 distinguishes behavior data from result-of-behavior (outcome) data and recommends behavior data supplement, not replace, outcome data. Step 4 advocates construct-valid measures — construct validity being the judgment that a test or device measures a specified attribute to a significant degree, usable to promote understanding or prediction of behavior (Landy & Conte, 2016; Messick, 1995). These two poles — utility (the highest, most useful validity coefficient) versus understanding (construct validity) — anchor a long-running controversy the chapter revisits in the composite-versus-multiple-criteria debate below.

The chapter offers three yardsticks for judging a criterion's usefulness, corresponding to Learning Goal 4.5.

Relevance

A criterion's principal requirement is judged relevance — a logical relationship to the performance domain in question. Per SIOP's Principles for the Validation and Use of Personnel Selection Procedures (2018): "A relevant criterion is one that reflects the relative standing of employees with respect to an outcome critical to success in the focal work environment" (p. 14). The APA Task Force on Employment Testing of Minority Groups (1969) emphasized that the most relevant criterion for evaluating tests is a direct measure of job proficiency after an appropriate period on the job (roughly six months to a year) — though the most relevant measure is rarely the cheapest, since a well-designed work-sample test or performance system takes real ingenuity and expense (Jackson, Harris, Ashton, McCarthy, & Tremblay, 2000).

Sensitivity or Discriminability

A useful criterion must discriminate between effective and ineffective employees. The chapter's cautionary example: quantity produced under machine pacing, where everyone produces roughly the same amount, has little value as a criterion since it can't distinguish performers; scrap amount or error count might be more sensitive there. Note that criterion variance and relevance are not necessarily linked — a low-variance criterion element (e.g., dollar cost of industrial accidents) can still carry major implications on a different scale, which is why operational measures must be distinguished from the underlying conceptual formulation of what actually matters to the organization (Cascio & Valenzi, 1978).

Practicality

Management must understand the real benefits of carefully developed criteria, but objections arise if record keeping and data collection become impractical and interfere with operations. HR occupies a staff role supporting those directly responsible for profit, growth, and service — not a license to run organizations as research laboratories. The chapter's blunt instruction: keep criterion measurement practical.

Criterion Deficiency

Criterion measures differ in how completely they cover the criterion domain. A university professor's job includes teaching, research, and service; measuring only teaching and service is deficient because it omits research. Likewise, measuring a manager's flexibility through traits alone is deficient, since flexibility is a higher-order construct requiring mastery of opposing behaviors across social/interpersonal and functional/organizational domains (Kaiser, Lindberg, & Craig, 2007). Morrow, Jarrett, and Rupinski (1997) found that training-program utility estimates can differ not because programs differ in effectiveness, but because criterion measures differ in breadth — a measure covering only some trained tasks underestimates a program's true value.

Criterion Contamination

Criteria gathered carelessly, with no checks before use, become contaminated. Maier (1988) demonstrated this with military aptitude tests validated against hands-on performance tests for radio repairers and auto mechanics: untrained sergeants administered the tests with little monitoring and only one administrator per examinee, producing error-filled data. After statistical cleanup, validities rose from .09 and .17 to .49 and .37 — completely changing the interpretation of the tests' validity.

Contamination occurs when the operational criterion includes variance unrelated to the ultimate criterion, split into error (random variation that cannot correlate with anything by definition) and bias (systematic contamination that can correlate with predictors) (Blum & Naylor, 1968). Brogden and Taylor (1950b) define a biasing factor as "any variable, except errors of measurement and sampling error, producing a deviation of obtained criterion scores from a hypothetical 'true' criterion score" (p. 161) — its direction is unspecified, so bias may inflate, deflate, or leave validity unchanged, depending on its correlation with the predictor.

Three Sources of Criterion Bias

Bias due to knowledge of predictor information is especially serious for ratings. If a supervisor sees an assessment center's prediction of a subordinate's potential before rating that subordinate's performance, the rating can be biased upward (confirming a "shooting star" label) or downward (rivalry). Either way, the predictor looks like it's doing a better job than it actually is. Rule of thumb: keep predictor information away from those providing criterion data. Bray and Grant (1966) modeled this correctly, collecting assessment-center data with no bearing on promotion decisions and validating predictions against "promoted vs. not" eight years later.

Bias due to group membership arises when policies, explicit or implicit, govern hiring or promotion for certain groups (e.g., hiring mainly from certain schools, or promoting reservists who were also promoted in their military units) — studies linking such characteristics to career success are necessarily biased. Bias in ratings affects supervisory ratings, the most common criteria in practice (Aguinis, 2019; Lent, Aurbach, & Levin, 1971; Murphy & Cleveland, 1995), which are vulnerable to spotty observation, unequal opportunity to demonstrate proficiency, personal prejudice, or inability to distinguish performance dimensions (Thorndike, 1920) — covered in depth in Chapter 5.

Applied psychologists agree job performance is multidimensional and needs multidimensional criteria. The open question: combine measures into a composite score, or keep them separate — and if combined, by what rule?

The Case for a Composite Criterion

Brogden and Taylor (1950a), Thorndike (1949), Toops (1944), and Nagle (1953) argue the criterion should yield one yardstick of overall "success" or organizational value — even criteria validated separately must eventually combine into a composite when a decision is required. Weighting choices matter: combining external and internal customer service with equal weight implies the organization values both equally, but weighting external service 70% and internal 30% is a deliberate strategic decision that changes predictor-criterion validities. Murphy and Shiarella (1997) found in simulation that 34% of the variance in a selection battery's validity was explained purely by how task and contextual performance were combined — the combination rule is itself a major design decision.

The Case for Multiple Criteria

Advocates of multiple criteria argue demonstrably different variables should not be combined. Cattell's (1957) analogy: "Ten men and two bottles of beer cannot be added to give the same total as two men and ten bottles of beer" (p. 11). Pulakos, Borman, and Hough (1988) found military recruiters' selling, human-relations, and organizing skills were all important but unrelated to each other — the best seller wasn't necessarily the best organizer — making a composite ambiguous and psychologically nonsensical. Guion (1961) names the fallacy directly: assuming "there is a general factor in all criteria accounting for virtually all of the important variance in behavior at work" (p. 145).

Differing Assumptions and Resolving the Dilemma

Schmidt and Kaplan (1971) and Binning and Barrett (1989) note the camps differ on what construct the criterion represents and what validation is for. A composite need not be behavioral — Brogden and Taylor's (1950a) "dollar criterion" is explicitly economic: "The criterion should measure the overall contribution of the individual to the organization" (p. 139). Multiple-criteria advocates (Dunnette, 1963a; Pulakos et al., 1988) hold the criterion should be a behaviorally homogeneous psychological construct, though they concede composites are eventually needed for real decisions.

The resolution ties to purpose: for research aimed at psychological understanding, keep dimensions separate and behavioral. For managerial decisions — job assignment, promotion, capital budgeting, program cost-effectiveness — weight and combine dimensions into a composite representing overall economic worth, regardless of intercorrelations.

Traditional personnel psychology relied on a simple prediction model relating predictors to a composite criterion, neglecting intervening variables. Binning and Barrett (1989) offer a fuller model of the inferences required to rigorously develop criteria (the chapter's Figure 4.2). Managers care most about Inference 9: whether assessment scores predict subsequent job performance. One route to justifying it is Inference 5 — a direct empirical link between predictor and operational criterion, the traditional basis of the term criterion-related validity. But full confidence in Inference 9 also requires Inference 8: the operational criterion measure must relate to the true performance domain it represents. Both links, not just the first, are needed.

Performance domains are made of behavior–outcome units (Binning & Barrett, 1989): outcomes (e.g., dollar sales volume) are what the organization values, and behaviors (e.g., selling skills) are the means to those ends, so behaviors take on different value depending on the outcomes they produce. This maps onto the earlier debate — composite models focus on outcomes, multiple-criteria models focus on behaviors, and together they form the full performance domain, which is why both remain necessary.

Inference 8 (criterion development) is usually justified through job-analysis evidence that all major behavioral dimensions or job outcomes are represented in the operational criterion. What personnel psychologists traditionally call construct validity ties to Inferences 6 and 7: if a test measures a specific construct (Inference 6) shown critical to job performance (Inference 7), inferences from test scores to job performance (Inference 9) follow logically. Inference 7 is justified through deriving job specifications; Inference 10 through developing a job description; Inference 11 through verifying links between job behaviors and outcomes — job analysis (Chapter 9) supplies evidence for all of these.

The criterion problem, in this framework, results from neglecting evidence for Inferences 7, 8, and 10 — fostering a shortsighted view of criterion validation. Two consequences follow: criterion measures end up psychometrically less rigorous than predictor measures, and performance criteria end up less richly embedded in theoretical networks than predictor-side constructs — both limiting theory development and the quality of staffing and career decisions (Binning & Barrett, 1989).

An unspoken assumption runs through most performance-management practice, including the utility calculations in Chapter 14: that criteria, and performance especially, follow a normal, bell-shaped distribution, with most individuals clustered near the center and few extreme performers.

A recent research stream challenges this assumption, showing that for many jobs, performance follows a heavy-tailed distribution instead (Aguinis, Gottfredson, & Joo, 2013b; Aguinis, O'Boyle, Gonzalez-Mulé, & Joo, 2016; Joo, Aguinis, & Bradley, 2017; O'Boyle & Aguinis, 2012). Under a normal distribution, extreme high scores are anomalies to be "corrected" via transformations or outlier removal. Under a heavy-tailed distribution, many star performers (far above the mean) are expected, not anomalous, and a minority of individuals produce a disproportionate share of total output.

The Evidence for Heavy-Tailed Performance

Joo, Aguinis, and Bradley (2017) examined data from over 600,000 workers — publications by 25,000+ researchers across 50+ fields, plus productivity from movie directors, writers, musicians, athletes, bank tellers, call-center employees, grocery checkers, and assemblers — and found at least 75% of distributions are heavy-tailed, not normal. Concretely: a normal distribution predicts roughly 35 researchers with 10+ publications (three SDs above the mean), but 460 actually reached that mark — over 13 times the expected number. Among roughly 3,300 Grammy-nominated artists, a normal distribution predicts 5 with 10+ nominations, but 64 actually achieved it. This pattern doesn't hold for every job (Beck et al., 2014; Vancouver, Li, Weinhardt, Steel, & Purl, 2016), but appears true in most cases studied.

Practical Implications for Producing and Retaining Star Performers

Since stars should not be treated as anomalies to fix, heavy-tailed distributions carry real consequences for research and practice alike (Aguinis & Bradley, 2015; Aguinis & O'Boyle, 2014):

• Minimize situational (ceiling) constraints that block stars from emerging.
• Allow star performers to rotate across teams, widening their network and spreading knowledge to rising stars.
• Invest sufficient resources in stars contributing to core strategic objectives.
• Retain stars by attending to their developmental network (e.g., spouse employment opportunities, long-term contracting with a star's subordinates).
• During budget cuts, protect star performers especially — once they leave, replacements can trigger a downward spiral of production.
• Give stars preferential treatment, but articulate the perks clearly to all workers and apply them fairly to anyone reaching that performance level.
• Invest disproportionate resources in stars to generate greater overall output.
• Avoid non-performance-based incentives, limited pay dispersion, and longevity-based promotion — these discourage stars from emerging by rewarding homogeneous performance.

Cascio and Aguinis close every chapter with an "Evidence-Based Implications for Practice" list. For Chapter 4, it functions as the chapter's own executive summary.

• The effectiveness and future progress of our knowledge of HR interventions depend fundamentally on careful, accurate criterion measurement.
• It is important to conceptualize the job performance domain broadly and to consider job performance as in situ performance — the specification of situational, contextual, strategic, and environmental effects on individual, team, or organizational performance.
• The notion of criterion relevance requires prior theorizing and development of the dimensions that comprise the domain of performance.
• Organizations must first formulate clear ultimate objectives and then develop criterion measures representing economic or behavioral constructs — involving behaviors and/or results. Criterion measures must pass the tests of relevance, sensitivity, and practicality.
• Conclusions reached can depend on (1) the particular criterion measures used, (2) the time of measurement, (3) conditions outside an individual's control, and (4) distortions and biases inherent in the situation or measuring instrument.
• Because there may be many paths to success, a broader, richer schematization of job performance is needed.
• Star performers are responsible for a disproportionate quantity of results.

Chapter 4 ends with nine discussion questions, each paired below with a concise model answer grounded in the chapter's content.

1. For what types of jobs is the use of behavior-based performance measures better than results-based measures, and for which other jobs is the reverse true?

Results-based measures fit best when workers already have the needed skills, behavior-result links are obvious, and there are many valid ways to do the job right (Aguinis & O'Boyle, 2014) — sales roles, where dollar volume speaks for itself. Behavior-based measures fit better when the link is unclear or shaped by factors outside the worker's control — customer-service or safety-sensitive roles, where coaching the right behaviors matters before results appear.

2. Discuss the problems that dynamic criteria pose for employment decisions.

Dynamic criteria mean average group performance, validity coefficients, and rank ordering can all shift over time (Barrett, Caldwell, & Alexander, 1985). If rank ordering shifts, future performance becomes a moving target and prediction deteriorates the farther out you project (Keil & Cortina, 2001), so a test validated today may predict less accurately a year later — long-range predictions are inherently less trustworthy than short-range ones.

3. What are the implications of the typical versus maximum performance distinction for personnel selection?

Since typical and maximum performance correlate only about .33 (Beus & Whitman, 2012), a test capturing maximum performance (a timed work sample) may say little about day-to-day performance. Selection systems built around maximum-performance assessments should be interpreted cautiously as predictors of typical, ongoing performance; organizations focused on sustained output may need criteria that directly capture typical performance instead.

4. Do you agree with the definition of counterproductive behaviors? Are all counterproductive behaviors necessarily bad for organizations—and employees?

The chapter defines workplace deviance as voluntary behavior violating organizational norms and threatening organizational well-being (Robinson & Bennett, 1995). A reasonable answer agrees with the core definition while noting nuance: behaviors like working slowly might, in specific cases, reflect a legitimate response to unsafe pace demands or burnout rather than malice — organizational context matters when judging whether a behavior is truly counterproductive or a symptom the organization should address.

5. What are the implications for theory and practice of the concept of in situ performance?

In situ performance (Cascio & Aguinis, 2008b) insists that situational, contextual, strategic, and environmental effects be built into how we understand performance rather than treated as noise. For theory, models must incorporate extraindividual variables (environment, safety climate, lifespace, job/location, leadership), not assume performance is purely individual. For practice, organizations should not judge or reward individuals solely on outcomes without accounting for the situational constraints that shaped them.

6. Are there ethical implications associated with the use of wearable sensors? What would you do if your employer requires that you use a GPS in your car to collect data that can be accessed by the organization 24 hours a day, 365 days a year?

Wearable sensors and GPS tracking raise genuine privacy concerns, since they can capture data well beyond working hours and the job's scope — the chapter itself flags this as an open ethical question. A thoughtful answer weighs the business rationale (richer data through aggregation) against reasonable privacy expectations, favoring monitoring limited to working hours and job-relevant contexts with transparency about what is collected — echoing Chapter 1's point that employees respond better with input into monitoring design.

7. How can the reliability of job performance observation be improved?

Bray and Campbell (1968) show observation reliability cannot be assumed — different observers can reach entirely uncorrelated conclusions about the same performance. Improvement requires training raters on standardized criteria, ensuring adequate and consistent observation opportunity, reducing rater bias and prejudice, and aggregating observations across more occasions and raters — there is no single silver-bullet fix (Borman & Hallam, 1991).

8. What factors should be considered in assigning differential weights when creating a composite measure of performance?

Weighting should reflect the organization's actual strategic priorities — the chapter's 70/30 external-versus-internal customer service example reflects a deliberate strategic choice, not equal weighting by default. Since Murphy and Shiarella (1997) found 34% of validity variance in a test battery came purely from how components were combined, weights should rest on job-analysis evidence about what matters most, not convenience.

9. Describe the performance domain of a university professor. Then propose a criterion measure to be used in making promotion decisions. How would you rate this criterion regarding relevance, sensitivity, and practicality?

The chapter's own example defines a professor's performance domain as teaching, research, and service; a criterion measuring only teaching and service would be deficient, since it omits research. A better criterion combines teaching evaluations, research output and quality, and documented service (committee work, mentoring, outreach) — relevant if all three domains are genuinely part of the job as analyzed, sensitive if it discriminates strong from weak performers within each domain (a raw publication count may lack sensitivity if quality varies widely), and practical only if the underlying data can be collected without excessive burden on the department.

Every bolded or explicitly defined term in Chapter 4, in one line each, in the order the chapter introduces them.

A single table capturing the chapter's core definitions, its three dimensions of criteria, its three evaluation yardsticks, and its most important debates and findings.