Case Study for World Leader in Software Development

Background

The purpose of the case study was to determine the ability of the Harrison InnerView software to predict success for the sales position for a world leading software development company. Eighty-eight profiles of current employees were provided. Eighteen of the profiles were eliminated from the sample due to having a consistency score of less than zero. This indicates that these individuals either attempted to fool the questionnaire or tried to answer the questions too quickly. Thus the actual sample was 70.

All the employees in the sample were rated according to their job performance by supervisors.

Each employee in the sample was rated or each criteria according to the following scale:

90-100	Excellent to Best
80-89	Very Good
70-79	Average to Good
60-69	Below average
40-59	Poor
Below 40	Failed or will soon fail in the position

The company elected to place employees in one of three categories: “high” performers who were given a 90 rating, “medium” performers who were given an 80 rating, and “low” performers who were given a 70 rating.

The sample was split into two groups: Group A consisted of 37 subjects and Group B consisted of 33 subjects. The sample sizes were different due to a slightly poorer than expected response rate for Group B.

InnerView received the performance ratings for Group A. These were used to determine the traits that related to success in this position. The performance ratings were withheld for Group B until the traits were selected and developed into a “job template”. Subjects from Group B were then evaluated using the job template thus obtaining the suitability predictions of InnerView for Group B. Group B predictions were then given back to the company. The company then returned to InnerView the performance ratings for Group B. Thus Group B was the “blind” group used to determine the predictive accuracy of the system. Four of the profiles were incorrectly completed and were thus returned to the Europe office. However, they were not returned until the template was completed and the template configuration and group B predictions were released to the company. Of the four, two were intended for Group A, but since we did not obtain the data until after the results were released, we included all four as part of the B Group (since all 4 were actually blind samples).

The aim of the research was to determine the exact traits that make a Salesperson successful at this company and to determine the relative accuracy of the InnerView system to predict success based upon the “job template” that incorporates those traits. The template formulates those traits into “traits to have” as well as “traits to avoid”. The template is then used to measure future applicants and serve as a developmental guide for current employees. The template is shown by a graph that indicates how a person scores against each of the required traits and then offers a final “bottom line” score between zero and one-hundred that represents the individual’s level of “total suitability” for a particular position. A score of one hundred represents a person who is completely suitable for that position. Assuming the person is eligible for a position (has the education, experience and technical skills), a suitability score of 75 or greater represents a person who has a good probability of performing effectively in that position. A score of 74 or less represents a person who is considered to be unlikely to perform well in that position.

The sample was taken from telephone sales representatives (SSR) as well as from field sales (FSR). The sample included employees from the USA as well as Europe. It was intended to assess differences between Europe and the USA as well as differences between field sales and telephone sales. The sample size was too small to make any conclusions in that regard. However, there were no significant differences from the data obtained.

Accuracy level

The results showed a very high predictive accuracy and a strong correlation between the InnerView suitability score and the actual job performance. This indicates that the template includes a comprehensive set of traits related to suitability for this position.

The results were analyzed in two ways: the predictive accuracy and the variation between the suitability scores and the performance ratings:

1) Predictive accuracy - A prediction is considered accurate if one or more of the following conditions are met:

a) The suitability score is 75 or greater and the performance score is 75 or greater

b) The suitability score is less than 75 and the performance score is less than 75.

c) The suitability score is within 6 points of the performance score.

The logic behind this definition - If the suitability score were 75 or above and the person were eligible for the position, it would indicate a prediction that the person would probably succeed in the position. If the suitability score were less than 75 and the person was moderately eligible (not highly eligible) for the position then this would indicate a prediction of below average performance. Also, if the suitability score were within 6 points of the performance score it would indicate a very close prediction and thus should be considered accurate.

From Group A, 34 of the 37 employees (92%) showed a correlation between the InnerView suitability score and performance to be accurate, according to the definition above.

The blind Group B showed that 30 of the 33 employees (91%) had an accurate predictive rating based upon the comparison of the suitability score and the performance rating.

A further analysis of the detailed reports of the 6 inaccurate predictions (three from each group) showed that at least three of the inaccuracies would have been rectified. The system offers detailed reports about the individual that can identify potential problems that are too complex to be identified in the job template that was the basis of the predictions in this study. For example, xxxx who had a performance rating of 70 and a 99 suitability rating had a 10 score on “Wants Capable Leader”. A high score on this scale indicates that matching this person with the right supervisor is critical to success. In this case, xxxxx also had a 2 on Tolerance of Structure and a 10 on Wants Recognition. If xxxxx were matched with supervisor who was highly structured or failed to give recognition, the performance would likely suffer. This is similar to yyyy who had a performance rating of 70 and a suitability rating of 78. Although this is fairly close, yyyy also had a 9 score on Wants Capable Leader, an 8 on Wants Recognition and a 4 on Tolerance of structure. This would also indicate the same potential difficulties as xxxx, but to a lesser extent. zzzzz had a 70 performance rating and a 99 suitability rating had numerous scores that managed to slip by the temple because each score in itself was just slightly outside the negative areas, but collectively would indicate a significant difficulty. For example, zzzzz’s results showed dogmatism, bluntness, self-centeredness (dominating), and defensiveness, but each narrowly escaped the negative points on the template. However, collectively these indicate an aggressiveness that is likely to interfere with performance. A trained user would easily identify this difficulty. In another example, qqqq scored an 80 on performance but only a 68 on suitability. However, qqqq’s scores indicate a risk for theft. This could be a difficulty that is not yet detected.

2) Variation between the suitability ratings and the performance ratings

The degree of variation between the suitability score and the performance ratings also provide an effective means of understanding the accuracy level of the methodology. The variation was examined in two ways: the average variation between the predictive suitability rating and the performance rating for each group and the percentage of predictions that fall within different ranges of closeness.

The average variation between the suitability score and the performance rating for Group A was 6.0 points. The average variation between the suitability score and the performance rating for the blind Group B was 7.0 points. The average variation between the suitability score and the performance rating for the two groups combined was 6.4 points. This indicates a very close relationship between the suitability score and the performance ratings for both the sample group and the blind group.

Next we examined the percentage of employees whose two scores (performance rating and predicted suitability score) were within 6 points, 8 points, 10 points and 15 points. The table below shows these percentages for each group.

Variation between performance rating and predictive suitability score:

If the two scores were within six points, the prediction would be considered to be extremely accurate. If within 8 points, the prediction would be considered to be quite accurate. If within 10 points, the prediction should still be considered to be reasonably accurate, especially due to the fact that the performance ratings provided were generalized into three categories rather than individually rated. If the two scores are greater than 10 points apart, but within 15 points, it could not be considered to be accurate, but at the same time could not be considered to be very inaccurate. If the two scores are greater than 15 points apart, they should be considered to be inaccurate. Therefore, in Group A, only 4% could be considered to be inaccurate and in Group B only 9%.

Conclusion

The results show the template developed is able to predict job success with a high level of accuracy. The sample size was sufficient to assume reliable results for future applicants. However, it is recommended that after six months to one year of use, a further analysis of results be conducted. This will be particularly useful to determine further traits that may interfere with job success. In addition, it will enable us to determine if there are any small distinctions between FSR and SSR and between Europe and the United States.