A Consultant's Casebook
about me: Andres Inn, Ph.D.
|
|
|
The Best Predictor of Future Performance is Past PerformanceFor a number of reasons, I regard Assessment Centers as overrated. It is an arrogant conceit to claim that performance assessed during 1-2 days in a contrived and artificial setting will predict performance better than a careful review of past performance. Recall that one of the candidates in the OSS assessment centers flunked their stress interview, but had actually passed an interrogation by the Gestapo! The eminent psychologists working for the OSS were proud that they had screened out a potential failure. I remain amazed that they screened out the one candidate who experienced success in the field! Having one exemplary candidate fail would make me question the validity of the assessment technology. Instead, the failure to accurately predict (even past) behavior was considered as a positive outcome. Many kinds of data are serially correlated. If the temperature today is -10, it is extremely unlikely that tomorrow will register +30. In the past few years, I've amazed some friends by predicting winter weather during the summer. My predictions have been easy because the summers were unambiguous predictors. A few years ago we experienced an incredibly long, dry summer; so I predicted a short warm winter. Then we suffered a short, cold, and wet summer after which I predicted a long, cold, and snowy winter. I couldn't go wrong! But the point is that human performance is likewise serially correlated. To know how successfully someone will complete a future assignment, look at his past assignments. That's why successful managers and CEOs command so much money when they move from one company to the next. Their future performance is expected to be at least as good as their past performance, and it's a relatively safe bet. The assessment centers purport to capitalize on this fact and attempt to present "face validity." That is, the situational exercises appear to have some relationship to managerial tasks, and assessors assume that candidates who perform well on assessment center exercises will perform equally well on the job and in the future. This argument rests on two assumptions: (1) that assessment center exercises are truly comparable to job requirements, and (2) that assessors accurately measure performance and not some irrelevant variables. It's the time span! 1-2 days of assessment center dubiously transferable tasks vs years of measurable experience. measurable .... see police performance appraisal. Then there's the problem of examples. The best predictor of successful job performance is previous job performance. Anecdotal evidence from the early German, British, American assessment procedures suggest that those who passed through the assessment procedures also performed successfully. Not only that, but those who passed, were able to excel in a broad range of other activities. Recall "Wild Bill" Donavan, a war hero turned spy master. Or Julia Child, a famous graduate of the British Camp X, who mastered cooking, book publishing, and television presentations. But recall also, that these people were already successful before the assessment procedures began. First, they were all of exceptional intelligence. ... so what is it that assessment centers measure? Is it merely intelligence. Beyond our scope to .... but it's an interesting thought. Face ValiditySo, how well do the situational exercises mirror the requirements of actual jobs? Decide for yourself how well the Brook Exercise, the Wall Exercise, the Construction Exercise, and the Stress Interview mirrored the actual requirements of spies, saboteurs, and field agents in Nazi Germany. As regards the standard fare of assessment centers for managerial candidates, I remain unconvinced. First, there is the problem of duration. An assessment center exercise which takes an hour or two cannot compare to the tasks of managers which require a focus on results for a period of months, quarters, or even years. Short-term focus on an assignment with a concrete end has nothing to do with operating in a business environment where the shifting sands of competition, personnel, profits, exchange rates, politics demand a long-term focus on performance goals. Second, there is the problem of rater focus. Assessment Center raters measure simple observables: whether or not the candidate prioritizes, whether or not the candidate promotes her ideas forcefully, whether the candidate organizes and The In-Basket ExerciseThe Leaderless Group DiscussionSuperior/Subordinate InteractionsAssessment Center proponents frequently attempt to substantiate their claims of validity by asking participants whether or not the experience was useful, predictive, satisfying, enjoyable, or whatever. As for liking the experience ... T-Groups, Sensitivity Groups, ... all rely on a standard recipe. Put people into a strange environment where the rules are unclear. Create stress. Have the people emote in response to the stress. Remove the stress and accept their emotions and emotive behaviors. Voila! Instant liking for the group. THis is true whether in Jamestown, FBI Waco Texas, Branch Davidians, or in my case as a hostage by two escaped maniacs at the Cave in Dallas, Texas. Describe the experience! I do my best work during the very early daylight hours, or the late evenings, when my computers don't need to compete for my attention. It was about 11:00 in the evening, when one of my graduate students, Stan Kirk, drove by my office at Southern Methodist University and noticed that my light was on. He had been teaching me pool, and decided that it was too late for work, but just the right time for a beer and a game of pool. We had about $10.00 between us and headed for the corner bar. It was too crowded, and we moved on to the "Inner Circle Bar" at a strip mall right off the North-South Interstate. We were in the middle of our second or third game, when I looked up at Stan and saw him grinning, his hands held high; a cigarette in one, and a beer in the other. Confused, I looked around behind me and saw two men with ski masks; one brandishing a shotgun, the other an automatic rifle. "This is a stick-up," one yelled! "Everyone in the corner!" I'm no fool. I know that a wall of bodies will stop a rifle bullet, and a shotgun blast will penetrate less. I was the first in the corner. "If I can't see your hands, I'll start shooting." "Oh-oh!" I'm cursed with being only 5'8", and I wasn't wearing cowboy boots with heels. I raised up on my tiptoes, waving my arms in the air and pretty much in a panic. As the robbers were rifling the cash register at the bar, I heard sirens. A moment later there was a pounding at the door. "Open up, this is the poh-liis!" As it turned out, the dry-cleaner next door was working late, and heard the commotion in the bar. He telephoned the police, and the police arrived aggressively, and in force. The 25-30 of us were instant hostages. As the night dragged into morning we were threatened, slapped around, mildly tortured (we were not allowed to visit the washroom to return our borrowed beer), and alternately treated with some civility. Between bouts of standing in the corner, or against the wall, we were treated to quarters and allowed to play pool, pee in the trash can, and some of our group chose to drink free beer. Stan and I conferred as to where we would dive if the police broke in and shooting began. By morning I found myself talking to the two lunatics who had broken out of a Texas reform school. I even liked them, knowing full well about the "Stockholm" syndrome and Bruno Bettelheim's descriptions of Jewish concentration camp victims who identified with their captors. The "Stockholm syndrome" refers to the bond that sometimes develops between captor and captive. The case of a woman hostage held at a bank in Stockholm provides the label. She became so emotionally attached to one of her captors that she broke her engagement to another man, and remained faithful to her former captor during his prison term. And, to this day, I remember the experience at the Inner Circle Bar, not with terror, but as a humorous and interesting event in my life. The point is, that social scientists have described a perfect recipe for creating situations which participants find interesting, likable, useful, nurturing, self-actualizing, sociable, whatever. Mix together in an unfamiliar environment::
These factors will always work to produce positive affect. It works in Therapy Groups, Training Groups, Sensitivity Groups, Substance Abuse Support Groups, and Assessment Center Exercises. Place participants into an unfamiliar environment with unclear rules. Create stress by making people perform in an unfamiliar ways (describe deep feelings, emotions, painful experiences, etc.). Relieve the stress by accepting each participant's unique reaction to the experience. Voila! You have created instant liking for the group experience. Circumstances much like these cemented the relationships among the inhabitants of Jamestown, and the Branch Davidians near Waco, Texas.
Likewise, who dares argue with me that a rater who observes my behavior in the contrived setting will be able to evaluate my performance better than the people who have worked with me over the years? Information leakage: The moment I know that there is an "in-basket" exercise in which candidates must respond in writing to the messages placed in their "in-basket," I can anticipate how these behaviors will be judged: I should prioritize messages, I should understand the chronological order of events, I should delegate responsibility where appropriate and make decisions when needed. The moment I know there is a "leaderless group discussion" exercise, I know I need to stand out as a leader. I must not dominate the discussion, but should be seen as guiding the discussion ... etc. In the case of John Teahan's assessment procedures with the Dallas Police, we saw the problems associated with information leakage: scores of candidates improved over the 5 days of the experience.
The argument for use: The assessment task was made more difficult because the assessors could not get adequate job descriptions, the agents may be sent to do tasks different from the ones originally intended, and the shifting theaters of war may lead to a complete change in the situation to which the recruit was trained for and sent. Same problem with managers. Can't get adequate descriptions for how managers "lead," "motivate," "organize," etc. Worse, different managers may use completely different technologies for accomplishing the same results. Assessment Centers are supposedly able to assess the flexible approach assumed to be necessary for management. First, I doubt the claims of validity: despite appearances, I don't believe that Assessment Centers measure what they purport to measure. Second, there is the problem of value for money. Third, I believe that the results are serially contaminated (another problem with validity). And ... ,l The problem of Validity
Expense
Contamination The Detroit Oral Boards circa 1975 The Refinements
|
All material is copyrighted! Not to be used without the written permission of Andres Inn. ©2000.
Send mail to Andres.Inn@mail.ee with questions or comments about this web site.
|
![[Under Construction]](../images/undercon.gif){kind=small}
