Chapter 16 — Training and Development: Implementation and the Measurement of Outcomes

Chapter 16 opens with eight learning goals, numbered 16.1 through 16.8. They are reproduced verbatim below because Cascio and Aguinis use this exact numbering scheme throughout the book, and instructors sometimes reference goal numbers directly in assignments and quizzes.

1. 16.1 Classify training methods as presentation, hands-on, group building, or technology based.
2. 16.2 Identify key principles of instructional design to encourage active learner participation.
3. 16.3 List the essential elements for measuring training outcomes.
4. 16.4 Explain the advantages and disadvantages of ROI and utility analysis as methods for assessing the financial outcomes of learning and development activities.
5. 16.5 Identify potential contaminants or threats to valid interpretations of findings from field research.
6. 16.6 Distinguish experimental from quasi-experimental designs.
7. 16.7 Describe the limitations of experimental and quasi-experimental designs.
8. 16.8 In assessing the outcomes of training efforts, distinguish statistically significant effects from practically significant effects.

The chapter opens with an uncomfortable finding: despite a massive literature on training techniques, few U.S. companies that conduct training make any systematic effort to assess training needs before choosing a method (Arthur, Bennett, Edens, & Bell, 2003; Saari, Johnson, McLaughlin, & Zimmerle, 1988). In a LinkedIn Learning (2017) survey, only 15% of learning and development (L&D) professionals reported that they lack the data to know which solutions are effective — which sounds reassuring until you realize it implies the other 85% believe they have that data, even though rigorous outcome measurement remains rare. The chapter's diagnosis: firms treat hardware, software, and technique as more important than outcomes, mistakenly ranking "what trainees should learn" as secondary to "which tool delivers it."

New training methods appear constantly. Some are rooted in theoretical models of learning and behavior change (behavior modeling, team-coordination training); others result from trial and error; still others (interactive multimedia, computer-based business games) reflect technological rather than theoretical advances. By 2016, technology-based methods delivered 41% of learning hours at the average organization — nearly 10 points higher than 2008 and more than 15 points higher than 2003 (Association for Talent Development, 2016). The chapter does not attempt to catalog every specific technique (pointing readers to Goldstein & Ford, 2001; Noe, 2017; Wexley & Latham, 2002 for that); instead it organizes the landscape into four categories and then gives a checklist for judging any method's adequacy.

Following Noe (2017), the chapter sorts every training method into one of four categories: presentation, hands-on, group building, and technology based. Knowing which bucket a named method belongs to is a direct, testable skill — quizzes and discussion prompts routinely ask you to classify a method.

Presentation Methods

With presentation methods, an audience receives one-way communication from the trainer, in one of two formats: lectures, or videos (usually paired with lectures to show trainees real-life examples).

Hands-On Methods

Hands-on methods include on-the-job training, self-directed learning, apprenticeships, and simulations.

• On-the-job training — new or inexperienced employees learn in the work setting, during work hours, by observing peers or managers and then imitating their behavior (Tyler, 2008). Examples: onboarding, job rotation, understudy assignments ("shadowing," in which an understudy relieves a senior executive of selected duties so they can learn parts of the executive's job — Dragoni, Park, Soltis, & Forte-Trammell, 2014), and executive coaching (an individualized development process in which a skilled coach helps a leader become more effective in their organizational role — Vandaveer, 2017).
• Self-directed learning — trainees take responsibility for all aspects of learning, including timing and who is involved. Trainers may facilitate, but trainees master predetermined content at their own pace.
• Apprenticeship — a work-study regimen combining on-the-job and classroom training, typically lasting an average of four years.
• Simulations — training that represents real-life situations, with trainee decisions producing outcomes that mirror what would happen on the job.

Simulations themselves take several named forms, each worth distinguishing precisely:

• Case method — representative organizational situations presented in text form, usually to groups who identify problems and offer solutions, learning from each other's feedback.
• Incident method — like the case method, but trainees get only a sketchy outline of an incident; they must question the trainer, attempt a solution, then compare it to the trainer's full-information solution.
• Role playing — including multiple role-playing, where a large group splits into smaller groups that role-play the same problem independently, then reassemble to discuss outcomes with the trainer.
• Experiential exercises — a hybrid technique simulating organizational-psychology experiences, blending elements of the case method, multiple role-playing, and team-coordination training; trainees examine responses first individually, then in their group, then with the larger group and trainer.
• Task model — trainees construct a complex but buildable physical object; a group must then duplicate it using alternative communication arrangements, with only certain trainees allowed to view the object, discussing communication problems as they arise.
• In-basket technique, business games, and assessment centers — covered in Chapter 13.
• Behavior or competency modeling — covered in Chapter 15.

Group-Building Methods

Group-building methods are designed to improve group or team effectiveness, not individual skill in isolation.

• Adventure learning — an experiential method building teamwork and leadership skills through structured activities (wilderness training, outdoor training, improvisational activities, drum circles, even cooking classes — Noe, 2017), aimed at self-awareness, problem solving, conflict management, and risk taking (Greenfield, 2015).
• Team training — designed to improve effectiveness across organizational team types (production, service, project, management teams, committees; see Chapter 15). It targets knowledge (mental models enabling function in new situations), attitudes (beliefs about the team's task and feelings toward teammates), and behavior (actions enabling communication, coordination, adaptation, and task completion) (Salas, Burke, & Cannon-Bowers, 2002).
• Action learning — teams (typically 6–30 employees, sometimes including customers or vendors, often cross-functional) work on real business problems, commit to an action plan, and are held responsible for executing it, usually presenting novel solutions to top executives within two weeks to a month (Malone, 2013; Pedler & Abbott, 2013).
• Organization development (OD) — systematic, long-range programs of organizational improvement through action research: (a) preliminary diagnosis, (b) data gathering from the client group, (c) data feedback to the client group, (d) data exploration by the client group, (e) action planning, (f) action — then the cycle repeats.

Technology-Based Training

Instructor-led, face-to-face classroom training still comprises 49% of available training hours (down from 64% in 2008); counting all instructor-led delivery methods (classroom, online, remote) together, that figure rises to 65% of all learning hours (Association for Talent Development, 2016). Even so, technology-delivered training is expected to keep growing as technology improves, costs fall, demand rises for customization, and organizations recognize the cost savings of tablet-, smartphone-, and social-media-delivered training. Currently seven in 10 organizations incorporate video-based online training, and 67% of people learn on mobile devices (LinkedIn Learning, 2017).

Technology-based training creates a dynamic learning environment: it facilitates collaboration and enables customization (programs adapted to learner characteristics) and learner control — self-pacing exercises, exploring linked material, chatting with peers and experts, choosing when and where to train (Noe, 2017).

The chapter names at least 15 forms of technology-based training (Noe, 2017), reproduced here as a full list because instructors may quiz on recognizing specific formats:

• E-learning, online learning, computer-based training, and Web-based training.
• Webcasts or webinars — live, Web-based delivery to dispersed locations.
• Podcasts — Web-based delivery of audio- and video-based files.
• Mobile learning — through handheld devices such as tablets or smartphones.
• Blended learning — hybrid systems combining classroom and online learning.
• Wikis — websites allowing many users to create, edit, and update content and share knowledge.
• Distance learning — delivered to multiple locations online via webcasts or virtual classrooms, often supported by chat, e-mail, and online discussions.
• Social media — online or mobile technology enabling creation and exchange of user-generated content: wikis, blogs, networks (Facebook, LinkedIn), micro-sharing sites (Twitter), shared media (YouTube).
• Shared workspaces (e.g., Google Docs) hosted on a Web server for sharing information and documents.
• RSS feeds — updated content sent automatically to subscribers instead of by e-mail.
• Blogs — Web pages where authors post entries and readers comment.
• Micro-blogs or micro-sharing (e.g., Twitter) — tools enabling communication in short bursts of text, links, and multimedia.
• Chat rooms and discussion boards — electronic message boards for same- or different-time communication, sometimes moderated by a facilitator.
• Massive open online courses (MOOCs) — enroll large numbers of learners (massive), free and accessible to anyone with Internet access (open and online), using lecture videos and interactive coursework including discussion groups and wikis (online), with fixed start/completion dates, quizzes, and exams (courses).
• Adaptive training — customized content presented based on learner needs.

Is technology-based training more effective than instructor-led training? Two meta-analyses found no significant difference between formats, especially when both teach the same type of knowledge — declarative or procedural (Sitzmann, Kraiger, Stewart, & Wisher, 2006; Zhao, Lei, Lai, & Tan, 2005). The chapter argues the more useful research questions are about mix and sequencing: what is the optimal blend of formats, does the sequencing of technology-based and in-person instruction matter (Bell, Tannenbaum, Ford, Noe, & Kraiger, 2017), does on-demand versus prescheduled timing affect motivation, and how do user experience and gamification affect performance (Thielsch & Niesenhaus, 2017)?

If technology-based training is to be maximally effective, it must be designed to encourage active learning. The chapter lists four principles to build into instructional design (Brown & Ford, 2002):

1. Design the information structure and presentation to reflect meaningful organization ("chunking") of material and ease of use.
2. Balance learner control with guidance that helps learners make better choices about content and process.
3. Provide opportunities for practice and constructive feedback.
4. Encourage learners to be mindful of their own cognitive processing and in control of their learning processes.

A training method can be effective only if used appropriately, and appropriate use means rigid adherence to a two-step sequence: first define what trainees are to learn, and only then choose the method that best fits those requirements. The chapter is blunt about the common failure mode — trainers who choose a method first and force it to fit needs afterward — calling this "retrofit" approach not just wrong but often extremely wasteful of organizational time, people, and money. Its explicit prescription: "It should be banished."

A technique is adequate to the extent that it provides the minimal conditions for effective learning. The chapter offers a nine-item checklist that any proposed technique should satisfy; designers apply it to every candidate method, and if a technique is deficient in an area, either modify it or bolster it with another technique.

• Motivate the trainee to improve his or her performance.
• Clearly illustrate desired skills.
• Provide for the learner's active participation.
• Provide an opportunity to practice.
• Provide feedback on performance while the trainee learns.
• Provide some means to reinforce the trainee while learning (e.g., chatbots — automated, personalized software-to-human conversations that can provide reminders, track goals, assess transfer, and support continued performance; Han, 2017).
• Be structured from simple to complex tasks.
• Be adaptable to specific problems.
• Enable the trainee to transfer what is learned to other situations.

Once a technique passes this checklist and training is conducted, the final step — the subject of the rest of the chapter — is to measure the effects of training and how those effects interact with other organizational subsystems.

"Evaluation" implies a dichotomous outcome — a program either has value or it doesn't — but in practice outcomes are a matter of degree. To assess outcomes, evaluators must document systematically how trainees actually behave back on the job and how relevant that behavior is to the organization's objectives (Brown, 2017a; Machin, 2002; Snyder, Raben, & Farr, 1980), while also considering the evaluation's intended purpose and the sophistication of its intended audience (Aguinis & Kraiger, 2009).

Why Measure Training Outcomes?

Most companies assess training with nothing more rigorous than post-training participant reactions (Association for Talent Development, 2016; Brown, 2005; LinkedIn Learning, 2017; Sugrue & Rivera, 2005; Twitchell, Holton, & Trott, 2001) — which the chapter frames as a missed opportunity, since there are many legitimate reasons to evaluate training (Brown, 2017a; Noe, 2017; Sackett & Mullen, 1993):

• To make decisions about the future use of a training program or technique (continue, modify, eliminate).
• To compare the costs and benefits of training versus nontraining investments (e.g., work redesign, improved staffing).
• To do a comparative analysis of costs and benefits across alternative training programs.
• To make decisions about individual trainees (certify as competent, provide additional training).
• To contribute to scientific understanding of the training process.
• To further political or public relations purposes — increasing the credibility and visibility of the training function by documenting success.

Essential Elements of Measuring Training Outcomes

At bottom, evaluation is counting — new customers, interactions, dollars, hours. The hard part is deciding what to count and building routine methods to count it. The chapter quotes William Bryce Cameron's (1963) famous line: "Not everything that counts can be counted, and not everything that can be counted counts" (p. 13). Campbell, Dunnette, Lawler, and Weick (1970) specify what actually counts in a training context:

• Use of multiple criteria — not for the sake of numbers, but to reflect the multiple contributions of managers to organizational goals.
• Some attempt to study the criteria themselves — their relationships with each other and with other variables, especially internal versus external criteria.
• Enough experimental control to point the causal arrow at the training program — how much is enough depends on possible interaction with the criterion measure and susceptibility to the Hawthorne effect.
• Provision for saying something about the practical and theoretical significance of results.
• A thorough, logical analysis of the process and content of training.
• Some effort to address the "systems" aspects of training impact — how training effects are altered by interaction with other organizational subsystems.

As with any HR program, the first step in judging training's value is to specify multiple criteria, since training is directed at specific components of performance and organizations pursue multiple, sometimes competing, objectives — training may move an organization toward some goals while moving it away from others simultaneously (Bass, 1983). The chapter examines criteria along three dimensions: time, type, and level.

Time

When, relative to training, should criterion data be collected — before, during, immediately after, or long after? Timing can substantially change how results are interpreted (Sprangers & Hoogstraten, 1989). A study of 181 Korean workers (Lim & Morris, 2006) found that the relationship between perceived training utility and perceived on-the-job application (transfer) decreased as the gap between training and measurement widened. Conclusions from immediate pre-post comparisons can differ drastically from conclusions based on the same measures 6–12 months later (Freeberg, 1976; Keil & Cortina, 2001; Steele-Johnson, Osburn, & Pieper, 2000) — yet both measurements matter. A review of 59 studies found measurement time spans of one year or less in 26 studies, one to three years in 27 studies, and more than three years in only six studies (Nicholas & Katz, 1985). What matters most is not the absolute level of behavior (e.g., grievances or accidents per month) but the change in behavior from before training to some time after.

Types of Criteria

Internal criteria are linked directly to performance in the training situation itself — attitude scales and achievement exams designed specifically to measure what the program taught. External criteria assess actual changes in job behavior — for example, following a two-day EEO law training, a written exam at training's conclusion is an internal criterion, while supervisor/peer/subordinate ratings of on-the-job application of EEO principles are external criteria. Both are necessary, and researchers need to understand the relationships between them to draw meaningful conclusions.

Criteria may also be qualitative (attitudinal and perceptual measures from interviews, observation, or written instruments — real-life examples of what quantitative results represent; Eden, 2017) or quantitative (outcomes of job behavior and system performance found in employment, accounting, production, and sales records: turnover, absenteeism, dollar sales volume, accident rates, controllable rejects). Researchers have traditionally favored quantitative measures except in organization development research (Austin & Bartunek, 2003; Nicholas, 1982; Nicholas & Katz, 1985), but the chapter calls this a possible mistake — ignoring qualitative measures misses the richness of how events occurred, and relying on only one type is short-sighted.

Finally, distinguish formative from summative criteria. Formative criteria evaluate training during design and development, usually through pilot testing and qualitative input (opinions, beliefs, feedback from subject matter experts and sometimes customers) — their purpose is to make the program better before it launches. Summative criteria determine whether trainees acquired the outcomes specified in training objectives (knowledge, skills, attitudes, or new behaviors; Noe, 2017) — their purpose is to judge the finished program.

Levels of Criteria

"Levels" can refer to the organizational levels from which criterion data are collected (trainers, trainees, subordinates, peers, supervisors, and policy makers/sponsors, plus group-level sources like work units, teams, and squads providing aggregate morale, turnover, grievance, cost, error, or profit data), or to the relative rigor of the measurement approach.

Kirkpatrick (1977, 1983, 1994) identified four levels of rigor for evaluating training: reaction, learning, behavior, and results. The chapter is careful to note these levels provide only a vocabulary and a rough taxonomy — higher levels do NOT necessarily provide more information than lower levels, and the levels need not be causally linked or positively intercorrelated (Alliger & Janak, 1989).

The chapter lists four specific concerns with Kirkpatrick's framework (Alliger, Tannenbaum, Bennett, Traver, & Shortland, 1997; Holton, 1996; Kraiger, 2002; Spitzer, 2005), and these four concerns are exactly the kind of content a discussion prompt or quiz will ask you to reproduce:

1. The framework is largely atheoretical; to the extent it is theory-based at all, it rests on an outdated behavioral perspective that ignores modern, cognitively based theories of learning.
2. It is overly simplistic, treating constructs like trainee reactions and learning as unidimensional when they are actually multidimensional (Alliger et al., 1997; Brown, 2005; Kraiger, Ford, & Salas, 1993; Morgan & Casper, 2001; Warr & Bunce, 1995) — for example, reactions include both affect toward the training AND its perceived utility.
3. The framework assumes relationships between levels that research does not support (Bretz & Thompsett, 1992) or that do not make intuitive sense. Kirkpatrick argued trainees cannot learn without positive reactions to training, yet a meta-analysis by Alliger et al. (1997) found an average correlation of only .07 between reactions of any type and immediate learning.
4. The framework ignores the actual purposes for evaluation — decision making, feedback, and marketing (Kraiger, 2002).

Kraiger's Integrative Model of Training Evaluation

Figure 16.1 presents Kraiger's (2002) alternative model, built to overcome Kirkpatrick's deficiencies. It distinguishes evaluation TARGETS (training content and design, changes in learners, and organizational payoffs) from data-collection METHODS (e.g., cost-benefit analyses, ratings, surveys). Targets and methods connect through the FOCUS of measurement — for changes in learners, focus might be cognitive, affective, or behavioral change. Finally, targets, focus, and methods all link back to evaluation PURPOSE: feedback (to trainers or learners), decision making, and marketing.

For the organizational-payoffs target specifically, Kraiger's model identifies three possible focuses: transfer of training (transfer climate, opportunity to perform, on-the-job behavior change), results (performance effectiveness or tangible outcomes to a work group or organization), and financial performance resulting from training (return on investment or utility analysis) (Sung & Choi, 2014) — which is exactly the topic the chapter turns to next.

Whatever measures are used, the goal is meaningful inference and ruling out alternative explanations — which requires administering measures according to a logical plan (an experimental design), such as before-and-after measurement against a comparable control group. In assessing on-the-job behavioral change specifically, allow a reasonable period — at least three months — after training before measuring, especially for programs meant to change decision-making skill, attitudes, or leadership style. A large-scale meta-analysis reported an average interval of 133 days (about 4.5 months) before collecting behavioral outcome measures (Arthur et al., 2003). Detecting real change requires carefully developed techniques: scripted job-related scenarios with empirically derived scoring weights (Ostroff, 1991), behaviorally anchored rating scales, self-reports supplemented by subordinate/peer/supervisor reports, critical incidents, or comparisons of trained versus untrained behaviors (Frese, Beimel, & Schoenborn, 2003).

Return on Investment (ROI)

ROI relates program profits to invested capital as a ratio: the numerator expresses profit related to a project, and the denominator is the initial investment (Cascio, Boudreau, & Fink, in press). Worked example from the chapter: an organization invests $80,000 to design and deliver a wellness program, which produces $240,000 in total annual savings from reduced sick days and improved health. ROI = [($240,000 − $80,000) / $80,000] × 100% = 200%, or a net benefit of 2:1 per dollar spent.

ROI's major advantage is simplicity and wide acceptance — it blends all major profitability ingredients into one comparable number. Its two disadvantages: (1) despite its straightforward logic, there is real subjectivity in estimating the inflow of returns, how inflows/outflows occur across future time periods, and how much future returns should be discounted for risk and price inflation (Boudreau & Ramstad, 2006); (2) typical ROI calculations examine one HR investment at a time and fail to consider how investments work together as a portfolio (Boudreau & Ramstad, 2007) — training may add value beyond its cost, but that value might be higher still if combined with proper individual-incentive investments tied to the training outcomes.

Utility Analysis

Financial outcomes may alternatively be assessed via utility analysis (see Chapter 14). The basic formula for assessing training's dollar-value outcomes (Schmidt, Hunter, & Pearlman, 1982) builds directly on the general selection-utility formula (Equation 14.7):

The term dt is the effect size. Starting from the null assumption of no performance difference between trained (experimental) and untrained (control) workers, the effect size tells you whether a difference exists and how large it is. Equation 16.2 gives effect size: dt = (mean performance of trained group − mean performance of untrained group) / (SD of the untrained group's performance × the reliability of the performance measure, ryy — e.g., interrater agreement expressed as a correlation). Equation 16.3 converts dt into a percentage change in performance: % change in X = dt × 100 × (SD of pretest performance / mean of pretest performance), where the second factor is the coefficient of variation, applicable when performance is measured on a ratio scale (Sackett, 1991).

When several studies exist, or when dt must be estimated for a proposed human resource development (HRD) program, the best estimate comes from cumulating results across available studies via meta-analysis (Arthur et al., 2003; Burke & Day, 1986; Guzzo, Jette, & Katzell, 1985; Morrow, Jarrett, & Rupinski, 1997). As such a "menu" of effect sizes accumulates, HR professionals can compute expected utilities for proposed HRD programs before committing resources.

Worked Illustration of Utility Analysis

To estimate the net payoff from a supervisory-skills training program: T = 2 years; N = 100; dt = .31 (Mathieu & Leonard, 1987); SDy = $30,000; C = $4,000 per person. Applying Equation 16.1: ΔU = 2 × 100 × .31 × $30,000 − (100)($4,000) = $1,460,000 over two years.

The chapter immediately flags this figure as illusory on its own, because it ignores several real-world adjustments: (1) $1,460,000 received in two years is worth only $1,103,970 today at a 15% discount rate (per Mathieu & Leonard, 1987); (2) it ignores variable costs and taxes (Boudreau, 1988); (3) it looks at only a single cohort, whereas effective training gets applied to multiple cohorts, whose payoffs must also account for attrition of trained employees and decay in the training effect's strength over time (Cascio, 1989; Cascio et al., in press). Even after all these corrections, the chapter notes the monetary payoff from training can still be substantial and well worth demonstrating.

Why Not Hold All Training Strictly Accountable in Economic Terms?

Economic indexes derived from operating-unit performance are often biased by factors outside a manager's control (turnover, market fluctuations), and those biasing influences are not always obvious enough to correct for. This does not mean financial-impact measures should be abandoned — every effort should still be made to use them — but evaluators must be aware of their limitations, and must weigh the utility of the information-gathering effort itself: if the cost of determining whether a program was beneficial outweighs any possible benefit, the effort may not be worth making. The chapter's synthesis: thorough evaluation considers measures of training content and design, changes in learners, AND organizational payoffs together, because together they serve every purpose of evaluation — feedback, decision making, and marketing.

Influencing Managerial Decisions With Evaluation Data

The real payoff from evaluation data comes when it drives strategically important organizational decisions (Boudreau & Ramstad, 2007; Cascio et al., in press). Mattson (2003) showed that training evaluations expressed in terms of results do influence whether operating managers modify, eliminate, continue, or expand programs — and that organizational cultural values, the complexity and credibility of the information, and its degree of abstractness versus concreteness all affect managers' perceptions of how useful and usable the evaluative data are.

In the Morrow et al. (1997) study, researchers boosted managerial acceptance by presenting the utility model and procedures — framed candidly as fallible but reasonable estimates — to the CEO and senior strategic-planning and HR managers BEFORE conducting the research. This mattered because nearly any field application of utility analysis relies on an effect size calculated from an imperfect quasi-experimental design, so securing buy-in in advance heads off later credibility challenges. Mattson (2003) similarly emphasized framing results in terms managers of the specific business unit already cared about — for a sales-and-service unit, he emphasized sales volume, employee retention, and customer-service improvement, not generic statistics. The framing of the message directly affects its acceptability.

An experimental design is a plan for conceptualizing relations among a study's variables; it implies how to control the research situation and how to analyze the data (Kerlinger & Lee, 2000; Mitchell & Jolley, 2013). Experimental designs work with either internal or external criteria, and researchers use them to make CAUSAL inferences — ruling out alternative explanations so they can say training, specifically, caused an observed change.

Most experimental designs and most training studies cannot point the causal arrow unequivocally at training (Eden, 2017). Three necessary conditions must hold (Shadish, Cook, & Campbell, 2002): (1) the outcome (y) did not occur until after the treatment (x); (2) x and y are actually shown to be related; and (3) — the hardest — other explanations of the x-y relationship can be eliminated as plausible rival hypotheses.

Shadish et al. (2002) identify four broad categories of validity that potential contaminants threaten:

• Statistical-conclusion validity — the validity of inferences about the correlation (covariation) between treatment (training) and outcome.
• Internal validity — the validity of inferences about whether changes in one variable caused changes in another.
• Construct validity — the validity of inferences from the people, settings, and cause-effect operations sampled in a study to the constructs those samples represent.
• External validity — the validity of inferences about the extent results generalize across populations, settings, and times.

The 12 Threats to Valid Inference

The chapter names 12 specific threats worth knowing individually, since Table 16.2 rates each experimental design against them:

1. History — specific events occurring between the "before" and "after" measurements, in addition to training.
2. Maturation — ongoing processes within the individual (growing older, gaining job experience) that are simply a function of time passing.
3. Testing — the effect of a pretest on posttest performance.
4. Instrumentation — the degree to which an instrument measures different attributes at two different points in time (e.g., different raters rating behavior before versus after).
5. Statistical regression (regression to the mean) — changes in criterion scores from selecting extreme groups on a pretest.
6. Differential selection — using different procedures to select people for experimental versus control groups.
7. Attrition — differential loss of respondents from various groups.
8. Interaction of differential selection and maturation — pre-existing group differences compound further through maturational change during training.
9. Interaction of pretest with the experimental variable — something during training interacts with the pretest so it affects the trained group more than the untrained group.
10. Interaction of differential selection with training — when multiple groups are trained, differential selection means they weren't equivalent on the criterion to begin with, so they may react differently to training.
11. Reactive effects of the research situation — the research design itself changes trainees' expectations and reactions, so results can't generalize to future applications of the training outside the research setting.
12. Multiple-treatment interference — residual effects of earlier training experiences affect trainees differently.

Table 16.1 in the chapter lays out four classical experimental designs (A through D), each combining pretest and control-group conditions differently, illustrating the range of causal inferences researchers can draw.

Design A — After-Only, One Control Group

Design A has not been used widely in training research, largely because the pretest concept is deeply ingrained in researchers' thinking, even though it is not essential to a true experimental design (Campbell & Stanley, 1963). Its safeguard against initial group inequality is randomization — within stated confidence limits, randomization can suffice without a pretest (Campbell & Stanley, 1963, p. 25). Design A controls for testing as a main effect and as an interaction, but does not measure them, which limits generalization since you cannot examine whether training interacts with pretest ability level. In organizational settings, variables like job experience, age, or prior performance can serve as covariates, or be used to "block" (match) subjects into pairs before randomly assigning one member of each pair to each group — both strategies increase statistical precision. Design A's chief advantage is avoiding pretest bias and the "give-away" repetition of similar material (as in attitude-change studies); its cost is that it does not prevent maturation, regression, or history effects from occurring after the study begins (Shadish et al., 2002). When bringing participants to an evaluation is costly, after-only measurement of trained and untrained groups is actually the best option (Kraiger, McLinden, & Casper, 2004).

Design B — Before–After, No Control Group

Design B's defining feature is that it compares a group with itself — in theory the best possible comparison, since subject-characteristic variables are all held constant. In practice it is fraught with rival hypotheses: history (events between pre- and posttest), maturation (fatigue, hunger, boredom accumulating over the interval), the reactive effect of the pretest itself changing what is being measured (e.g., an attitude questionnaire changing the attitude; a manager who knows they are observed changing their behavior — use nonreactive measures whenever possible, per Rosnow & Rosenthal, 2008; Webb, Campbell, Schwartz, & Sechrest, 2000), instrumentation (different raters pre/post), and statistical regression (extreme-scoring groups regressing toward the mean regardless of any real training effect). A control group corrects for regression because both groups experience the same regression and other influences — so a posttest difference between groups should be attributable to training. Despite all these problems, Design B is still better than no evaluation at all, and it can adequately assess individual achievement against a targeted performance level (Kraiger et al., 2004; Sackett & Mullen, 1993).

Design C — Before–After, One Control Group

Design C is adequate for most purposes, provided experimental and control sessions run simultaneously. It controls history, maturation, and testing, since events that would produce a pretest–posttest difference in the experimental group should equally affect the control group. Instrumentation is controlled by randomly assigning observers to sessions (with many observers) or by using each observer across both conditions while keeping them blind to which subjects receive which treatment. Random assignment adequately controls regression or selection effects, and Design C's data let researchers assess whether attrition plausibly explains any pretest–posttest gain. When highly unusual test procedures involve deception, surprise, or stress, a no-pretest design (like Design A) is preferable or even essential (Campbell & Stanley, 1963; Rosnow & Rosenthal, 2008). Successful replication of pretest–posttest changes across times and settings increases generalizability. To compare experimental and control results in Design C, use analysis of covariance with pretest scores as the covariate, or analyze change scores for each group (Cascio & Kurtines, 1977; Cronbach & Furby, 1970; Edwards, 2002).

Design D — the Solomon Four-Group Design

The Solomon (1949) four-group design is the most elegant of the classical designs. It parallels Design C but adds two more control groups lacking a pretest: C2 receives training plus a posttest; C3 receives only a posttest, separating the main effect of testing from the interaction of testing with training. If training produces real change, it should replicate across four directional comparisons: (1) the experimental group's posttest exceeds its own pretest; (2) the experimental group's posttest exceeds C1's posttest; (3) C2's posttest exceeds C3's posttest; (4) C2's posttest exceeds C1's pretest. Confirming all four substantially strengthens the design's inferential power, and comparing C3's posttest to the pretests of the experimental group and C1 lets researchers evaluate the combined effect of history and maturation.

Because not all four groups receive a pretest, gain-score ANOVA is ruled out; instead, analyze posttest scores with a 2×2 ANOVA (pretested vs. not-pretested, by training vs. no-training), estimating training main effects from column means, pretesting main effects from row means, and interactions from cell means.

Despite its elegance, the Solomon design has real problems (Bond, 1973; Kerlinger & Lee, 2000): it assumes time and training experience affect posttest scores independently, though some interaction is inevitable, and it demands large samples — 120 participants just to get 30 per group. A field application with only 37 and 58 subjects, using nonrandom assignment, shows how hard this design is to apply realistically (Sprangers & Hoogstraten, 1989). Even Solomon's own suggestion — training the untrained controls afterward to "replicate" the study — runs into trouble: the first training program may have already changed the organization, so a later cohort enters already influenced. Cascio (1976a) demonstrated this: the factor structure of a survey on managerial attitudes toward African Americans shifted across three samples from the same company over time, likely because two years of rising EEO awareness and over 2,200 managers completing the program had already changed attitudes. Even so, when applied with random assignment and proper controls, the Solomon design controls more sources of invalidity than any other design discussed here.

Before turning to quasi-experimental designs, the chapter pauses to place classical experimental design in proper perspective, naming four limitations even the best design cannot escape.

1. Experiments usually settle on a single criterion dimension, limiting how much information they can provide — there is no logical reason to restrict analysis to one dimension, but this is usually what happens. Ideally an experiment is part of a continuous feedback process, not an isolated demonstration (Shadish et al., 2002; Snyder et al., 1980).
2. Meta-analytic reviews show effect sizes from single-group pretest–posttest designs (Design B) are systematically HIGHER than those from control- or comparison-group designs (Carlson & Schmidt, 1999; Lipsey & Wilson, 1993) — design type itself moderates conclusions about training effectiveness, though corrections by dependent-variable type and design type can account for most of this bias.
3. Adequate statistical power matters — power (the probability of correctly rejecting a false null hypothesis; Murphy & Myors, 2003) depends on effect size, measure reliability, the pre-post correlation, sample size, and design type (Arvey, Cole, Hazucha, & Hartanto, 1985). Software for computing power and confidence intervals should make power analysis routine.
4. Experiments often fail to focus on an organization's real goals — the important question may not be whether treatment A beats treatment B, but what performance level to expect from nearly all trainees at acceptable cost, and whether improved performance fits the organization's broader strategic direction.

True experiments require manipulating at least one independent variable, randomly assigning participants to groups, and randomly assigning treatments to groups (Kerlinger & Lee, 2000). In field settings, managers often refuse random assignment — they don't see subordinates as interchangeable pawns and distrust randomness, and some view training evaluation itself as disruptive and expensive. Eden (2017) offers eight strategies for overcoming these deterrents to field experimentation, including avoiding jargon, explaining randomization in lay terms, transforming proprietary data, and using emerging technologies like experience sampling (Beal, 2015).

Despite calls for more rigor (Littrell, Salas, Hess, Paley, & Riedel, 2006; Shadish & Cook, 2009; Wang, 2002), quasi-experimental designs can still provide useful data when a true experiment is impossible (Grant & Wall, 2009). What makes a design "quasi" is the absence of randomly created, pre-experimental equivalence between groups, which degrades internal validity (Eden, 2017). The central purpose of any experiment is eliminating alternative explanations for results — if a quasi-experimental design can eliminate some rival hypotheses, it is worth running, even without full control. Table 16.3 in the chapter rates four quasi-experimental designs (E through H) against the same threats used for Designs A–D.

Design E — The Time-Series Design

The time-series design is especially relevant for training evaluation. It uses a single group, with criterion data collected at multiple points in time, both before and after training. Plotting a curve of criterion scores against time periods should show a discontinuity — a change in slope or intercept — that corresponds specifically to training and occurs nowhere else in the series. It's frequently used for training aimed at improving readily observable outcomes like accident rates, productivity, and absenteeism, and its large number of pre/post observations lets researchers analyze the stability of outcomes over time. To rule out alternative explanations, add comparison groups or a reversal period (where the intervention is withdrawn) (Noe, 2017).

Design F — The Nonequivalent Control-Group Design

Design F looks like Design C, but critically, individuals are NOT randomly assigned from a common population to experimental and control groups — it uses naturally occurring groups (e.g., work group A versus work group B), which is common in applied settings. It's especially valuable when true random-assignment Designs A or C are impossible, since even a nonequivalent control group makes results far less ambiguous than Design B alone. The nonequivalent control group becomes a more effective control as the similarity between the two groups' pretest scores increases. Its major invalidity sources are the selection-maturation interaction (e.g., if the experimental group happens to be younger and less experienced than an older, more experienced control group) and the testing-training interaction, plus regression effects when groups are "matched" (not a substitute for randomization) but their pretest means still differ substantially. Statistical control applied after the fact is NOT a substitute for random assignment (Carlson & Wu, 2012) — but despite these hazards, the chapter encourages increased use of Design F in applied settings, provided researchers stay alert to its specific contaminants.

Design G — The Nonequivalent Dependent Variable Design

Also called the "internal referencing" strategy (Haccoun & Hamtieux, 1994), Design G uses a single treatment group and compares two sets of dependent variables: one training SHOULD affect (experimental variables) and one it should NOT affect (control variables). It applies whenever evaluation is based on a performance test. Its major advantage is controlling two important internal-validity threats — testing and the Hawthorne effect (simply reflecting on one's behavior due to participating in training could itself produce change) — without the risk that an unmeasured variable differentiating a nonequivalent control GROUP (as in Design F) might interact with training (e.g., self-efficacy might already be higher in a nonequivalent control group if volunteers perceive they don't need the training; Frese et al., 2003).

Design G does not control history, maturation, or regression effects, and its most serious disadvantage is that the researcher can control how easy or hard it is to generate significant differences between experimental and control variables simply by choosing how similar or different those variables are. The fix: choose control variables conceptually similar to, but distinct from, the trained content. In a charismatic-leadership training study (Frese et al., 2003), trained (experimental) items included variation of speed, variation of loudness, and use of "we"; untrained (control) items — independently coded for similarity to inspirational speech and chosen to be LEAST similar — included combining serious/factual information with wit, using practical examples, and good organization (a, b, c structure). Results: participants improved far more on trained variables than untrained ones (effect sizes of about 1.0 versus .3), showing the training worked on targeted behaviors specifically rather than producing generic improvement — though long-term, objective effects on organizational performance or subordinate commitment remained unknown.

Design H — The Recurrent Institutional Cycle Design

Design H suits cyclical training where a large population is trained in successive cohorts rather than all at once. Worked example: a large sales organization ran a State Manager Program every two months for small groups (12–15) of middle managers, covering all aspects of retail sales, timed so all roughly 110 state managers were trained over 18 months. Design H combines two or more before–after studies at different times: Group I gets a pretest at time 1, training, then a posttest at time 2; at that same time 2, Group II gets a pretest, training, and a posttest at time 3 — so at time 2, an experimental and control group have effectively been created. Measuring Group I again at time 3, and giving Group II a pretest at time 1, controls for history and shows how training interacts with other organizational events over time.

Cross-sectional comparisons become possible: Group I's time-2 posttest versus Group II's time-2 pretest; Group I's gains (time-2 posttest) versus Group II's gains (time-3 posttest); and Group II's time-3 posttest versus Group I's time-3 posttest (training gains versus gains, or no gains, during a no-training period). Design H controls history and test-retest effects but not selection differences — one fix is splitting a large group into two equated samples, one measured before and after training (Group IIa) and one measured only after (Group IIb), since comparing equated groups' posttest scores is more precise than comparing unequated ones. Its remaining weakness is inadequate control of maturation — minor when training teaches specialized skills, but a real rival hypothesis when the goal is changing attitudes.

As in selection research, the distinction between statistical and practical significance matters for training outcomes: a statistically significant change score may mean little in a practical sense. Practically, researchers must show training affects organizational goals in tangible terms — lower production costs, increased sales, fewer grievances — and practical significance is typically reflected through effect sizes or variance-accounted-for measures (Grissom & Kim, 2014; Schmidt & Hunter, 2014).

A related distinction concerns theoretical significance. Training researchers often stop at showing a program "works," mainly to sell the idea to top management or legitimize an existing investment — but that's only half the story. The real test is whether a NEW training program is superior to previous or existing methods for the same objective, which requires systematic research evaluating independent variables likely to affect outcomes (different training methods, depths of training, or media types). If researchers adopt this two-pronged approach and map independent-variable effects across different trainee populations and criteria, the assessment gains theoretical significance. Using meta-analysis, Arthur et al. (2003) found medium-to-large sample-weighted average effect sizes for organizational training: .60 for reaction criteria, .63 for learning measures, and .62 for behavior/results measures. Statistical significance, while not trivial, in no sense guarantees practical or theoretical significance — the chapter's core caution for this section.

Logical Analysis

Experimental control is only one strategy for responding to threats to internal or statistical-conclusion validity (Eden, 2017; McLinden, 1995; Sackett & Mullen, 1993). A logical analysis of a training program's process and content can further explain why particular results were obtained. Both qualitative and quantitative criteria matter here; qualitative questions worth asking include:

• Were the goals of the training clear both to the organization and to the trainees?
• Were the methods and content of the training relevant to the goals?
• Were the proposed methods used and the proposed content actually taught?
• Did it appear that learning was taking place?
• Does the training program conflict with any other program in the organization?
• What kinds of criteria should be expected to show change as a result of the training?

For every one of these questions, supplement expert subjective opinion with objective data. A quantitative method (Bownas, Bosshardt, & Donnelly, 1985) documents the linkage between training content and job content, generating a list of tasks over-emphasized in training, tasks not being trained, and tasks instructors intend to train but that graduates report being unable to perform. Its steps: identify curriculum elements; identify tasks performed on the job; obtain ratings of training emphasis, how well each task was learned, and its corresponding job importance; correlate the training-emphasis and job-requirement ratings into an overall fit index; and use training-effectiveness ratings to flag over- or under-emphasized tasks.

Cascio and Aguinis close every chapter with an "Evidence-Based Implications for Practice" list — a distilled set of practitioner takeaways. For Chapter 16, this list functions as the chapter's own executive summary.

• Numerous training methods and techniques are available, but each can be effective only if used appropriately. To do that, first define what trainees are to learn, and only then choose the particular method that best fits these requirements.
• In evaluating training outcomes, be clear about your purpose. Three general purposes are to provide feedback to trainers and learners, to provide data on which to base decisions about programs, and to provide data to market them.
• Use quantitative as well as qualitative measures of training outcomes. Each provides useful information.
• Regardless of the measures used, the overall goal is to make meaningful inferences and rule out alternative explanations for results. To do that, administer measures according to some logical plan or procedure (experimental or quasi-experimental design). Be clear about what threats to valid inference your design controls for and fails to control for.
• No less important is a logical analysis of the process and content of training programs, for it can enhance understanding of why the obtained results occurred.

Chapter 16 ends with ten discussion questions. Below, each is paired with a concise model answer grounded directly in the chapter's content — useful both for self-testing and as a starting point if any of these questions resurface in a graded discussion or quiz.

1. Discuss two methods of training that illustrate each of the following categories: presentation, hands-on, group building, and technology based.

Presentation methods include lectures and videos, both one-way communication from trainer to audience. Hands-on methods include on-the-job training (learning by observing and imitating peers or managers in the actual work setting) and simulations such as the case method (analyzing written organizational scenarios in groups). Group-building methods include adventure learning (structured activities building teamwork and self-awareness) and action learning (teams solving real business problems and presenting solutions to executives). Technology-based methods include e-learning/Web-based training and mobile learning delivered through smartphones or tablets — both leverage learner control and self-pacing that the other three categories generally lack.

2. Discuss the advantages and disadvantages of interactive multimedia training.

As a form of technology-based training, interactive multimedia offers learner control (self-pacing, exploring linked material, choosing when and where to train), scalability across dispersed locations, and — per the chapter's meta-analytic evidence — learning outcomes statistically comparable to instructor-led training for the same knowledge type (Sitzmann et al., 2006; Zhao et al., 2005). Its disadvantages track directly to the chapter's four instructional-design principles: without deliberate chunking, guided learner control, built-in practice and feedback, and support for metacognitive awareness, the technology alone will not produce learning (Brown & Ford, 2002) — expensive, polished multimedia can still fail if it isn't designed for active participation.

3. Why are both quantitative and qualitative criteria important to consider in the assessment of training outcomes?

Quantitative criteria (turnover, absenteeism, sales dollars, accident rates) provide the outcome measures organizations use to justify decisions in objective, comparable terms, but on their own they omit the richness of HOW events occurred. Qualitative criteria (attitudinal and perceptual measures from interviews and observation) supply that missing context and process detail. The chapter explicitly warns that exclusive reliance on either type alone is short-sighted — LinkedIn Learning's (2017) finding that L&D professionals' top measures of success were nearly all qualitative (with only vague reference to actual outcomes) illustrates the risk of leaning too far in one direction.

4. What criteria would you use to select one training method over another?

First and always, define what trainees are to learn before selecting any method — choosing a method first and retrofitting it to needs is the exact error the chapter says should be "banished." Then apply the chapter's nine-item adequacy checklist: does the method motivate the trainee, clearly illustrate desired skills, allow active participation, provide practice opportunities, provide performance feedback, provide reinforcement, move from simple to complex, adapt to specific problems, and enable transfer to other situations? A method deficient on any point should be modified or paired with a complementary technique rather than discarded outright.

5. When does it make the most sense to use ROI to assess training outcomes?

ROI works best when measurable outcomes already exist (reductions in errors, sick days, accidents), the training links clearly to an organization-wide strategic goal such as cost reduction or improved customer service, management has genuine interest in the result, and the training reaches many employees (Noe, 2017). ROI is less appropriate when outcomes are diffuse or hard to isolate from other simultaneous HR investments, since typical ROI calculations examine one investment at a time rather than as part of a portfolio (Boudreau & Ramstad, 2007).

6. Describe some of the key differences between experimental and quasi-experimental designs.

True experimental designs require manipulating an independent variable, randomly assigning participants to groups, and randomly assigning treatments to groups (Kerlinger & Lee, 2000) — random assignment is what creates pre-experimental equivalence between groups. Quasi-experimental designs lack that randomly created equivalence, most often because field settings and resistant managers make random assignment impractical (Eden, 2017). As a result, quasi-experimental designs (Designs E–H) cannot rule out as many rival hypotheses as classical designs (A–D) — but they still eliminate SOME threats to valid inference, which the chapter argues makes them worth running when a true experiment simply isn't possible.

7. Your boss asks you to design a study to evaluate the effects of a training class in stress reduction. How will you proceed?

Start by defining the specific outcomes stress-reduction training should change (physiological markers, self-reported stress, absenteeism, error rates), then select criteria across time (immediate post-training AND a follow-up at least three months out, since attitude and behavior change need time to manifest), type (both internal criteria like a post-course knowledge check and external criteria like supervisor ratings of on-the-job composure), and level (individual trainee data plus aggregate unit-level absenteeism or grievance data). Given that stress-reduction outcomes involve attitudes, prefer a design with a control group (Design C or, if random assignment is politically impossible, Design F) to rule out history and maturation, and supplement the statistical result with a logical analysis of whether the training's content and delivery actually matched its stated goals.

8. Your firm decides to train its entire population of employees and managers (500) to provide "legendary customer service." Suggest a design for evaluating the impact of such a massive training effort.

Because 500 people cannot realistically be trained simultaneously, the recurrent institutional cycle design (Design H) fits best: train cohorts in sequence, using each not-yet-trained cohort as a temporary control group for the cohort ahead of it, and later measuring an already-trained cohort again to help control for history. Splitting a later cohort into two equated subgroups (one measured before and after, one measured only after) would sharpen the comparison further. Pair this with quantitative external criteria the business already tracks (customer satisfaction scores, complaint counts, repeat-business rates) and qualitative logical analysis (were goals clear, was content relevant, did it appear learning occurred) to interpret why results turned out as they did.

9. Identify two examples that illustrate the difference between statistical and practical significance.

First, the chapter's own reaction-learning correlation: Alliger et al.'s (1997) meta-analysis found a statistically detectable but practically trivial correlation of .07 between trainee reactions and immediate learning — a real, nonzero relationship with essentially no practical value for predicting whether learning occurred. Second, a training program could show a statistically significant improvement in a performance measure that, once converted to a dollar-utility figure via Equation 16.1, turns out too small to justify its cost once discounting, taxes, and attrition are factored in (as in the chapter's supervisory-training example, where the raw $1,460,000 utility shrank to $1,103,970 in present-value terms) — statistically real, but only meaningfully significant once weighed against these practical costs.

10. What additional information might a logical analysis of training outcomes provide that an experimental or quasi-experimental design does not?

An experimental or quasi-experimental design can tell you WHETHER training produced a statistically detectable change on a chosen criterion, but it typically examines only a single criterion dimension and says nothing about the mechanism behind the result. Logical analysis fills that gap by asking whether the training's goals were clear, whether its methods and content actually matched those goals, whether the proposed content was actually taught, whether learning appeared to occur, and whether the program conflicted with other organizational initiatives — supplementing subjective expert judgment with objective linkage data (e.g., Bownas, Bosshardt, & Donnelly's 1985 method for correlating training emphasis with job requirements). This is what lets evaluators explain WHY a design produced the results it did, not merely THAT it did.

Every bolded or explicitly defined term in Chapter 16, in one line each, in the order the chapter introduces them.

A single table capturing the chapter's method taxonomy, its evaluation frameworks, its two financial formulas, and its experimental/quasi-experimental design menu — everything you need to answer a cold-call question about Chapter 16 without re-reading it.