What follows will provide you with a basic understanding of the technical and statistical construction of the Culture Index™ Survey. It also serves as a primer for those who need to have a basic understanding of the requirements for determining acceptable standards when evaluating any psychometric instrument.
While the EEOC does not make a legal distinction between tests and personality inventories, a practical distinction exists between the two. Tests usually produce pass or fail scores of a cognitive nature, meaning measurement of learned skills such as mathematics and vocabulary. These types of scores can be easily compared. For example, if two people take a vocabulary test and one scores 75% and the other 85%, then the latter person clearly received the higher score.
Personality and behavior inventories, on the other hand, rarely produce pass or fail scores; instead, they report non-cognitive traits, aptitudes, interest, and other qualities which are not “book learned”. As a result, personality inventories are more difficult to use, particularly when comparing individuals against each other or against job demands. They require specialized training and monitoring. For the sake of simplicity, however, the term “score” will be used in this report in reference to test or inventory results.
Today, there are many common misperceptions and confusion regarding “testing” – some of it due to contradictions in federal law, some due to hearsay information, and some due to people’s opinions. For instance, some people believe that testing is illegal. This is not true. The Supreme Court ruled that testing “is not only legal, but valuable” when done appropriately (Griggs v. Duke Power Co., 1971). Appropriate usage means that a test is relevant to job demands and scores are linked to job performance. For example, why administer a typing test to someone who is applying for a sales position?
Under law, verbal interviews and other common selection practices come under EEOC scrutiny just as do personality assessments. In fact, the vast majority of discrimination suits lodged over the past 20 years have been the result of an interviewer’s statement or question rather than a test score.
Some employers fear that the results of testing will result in stereotyping and pigeonholing employees. This can result where there has been inadequate or inappropriate training and where test results have been extended beyond their proper contexts and applications. It is the responsibility of the trainer and consultant to have the qualifications, and provide client support to prevent these occurrences.
Another major issue is adverse impact – whereby an instrument possesses an inherent bias against a minority (sex, race, religion, national origin). One of the most common areas of adverse impact is test result differences between men and women. If women consistently score differently than men in a specific test, then it has adverse impact and is discriminatory. This alone does not rule out the use of the test, but studies must be done comparing test scores against valid job performance measurements, resulting in statistically significant measures that particular scores do, in fact, accompany higher performance.
A tougher issue is adverse impact between various races/ethnicities. This is due to the difficulty of researchers gathering enough data across the job hierarchy. The Culture Index Survey shows no evidence of bias against different races/ethnicities.
Appropriately used, personality inventories produce a wealth of information to companies. Their results can be used for self-awareness, individual management and coaching, identifying and understanding necessary traits and behaviors for specific jobs, developing job descriptions, outlining organizational strengths and weaknesses, identifying training needs, and for selection and placement. A tool for management to consider is The Culture Index Survey which underwent five years of research, and was introduced to the market in the autumn of 2004. Consequentially, it exhibits current norms of measured traits and behaviors in the contemporary North American work population. The norms are exhibited below.
There were four separate administrations of the survey from 1999 to 2004. The sample sizes of the administrations was 180, 312, 428, and 493 cases, respectively. Each administration reflected job hierarchy samples (e.g. executive, sales, supervisor, and production) and were very near 50% between sexes.
There is no inherent bias or discrimination (adverse impact) between women and men in the survey. Research to date has not found evidence of inherent discrimination in the instrument along racial lines. Literacy, obviously, was necessary for understanding and completion of the survey.
It takes approximately ten minutes to administer and complete the survey.
The survey is a self-report inventory which measures seven personality traits and seven behaviors which most researchers and users consider important to work-related activities.
The seven constructs are Autonomy, Social-Ability, Patience, Conformity, Energy Units, Logic and Ingenuity. Self-report means that an individual completes the survey based upon their own perceptions and beliefs of their personality and required job behaviors.
The Culture Index Survey has also been used as an other-report inventory where by individuals check words which they believe describe another. This has been found to be very valuable in team-building exercises.
The survey format consists of 174 words in each of the two sections. Section One asks the respondents to check those words which describe themselves. Section Two asks the respondents to check those words which describe how they must behave to be successful in their current positions. It is a free-choice technique, meaning an individual can choose to check a word or not, and does not have to pick one word from a series or group (forced-choice).
Words checked are processed and converted to standardized (or Z-) scores, which are in turn converted into centile scores. Some of the words are experimental – they are not included in the calculations but may be used in future research.
Acceptable personality inventories must exhibit statistical evidence of their validity and reliability. VALIDITY means that the instrument measures what it purports to measure. For instance, the Culture Index Survey measures the trait or construct called autonomy (or assertiveness) and there are 21 words in each section which reflect the construct. In order to prove all 21words do, in fact, measure autonomy a construct validity study was conducted using a statistical technique called factor analysis.
An item analysis was performed in which each item correlation with total score was examined; only those items with statistically significant correlation with total score were retained. Users of personality inventories should be extremely wary of a test publisher who cannot or will not exhibit construct validity analyses results. Any claims that the information is proprietary or the only evidence of validity is a comparison of the test’s results against job performance measurements (job or criterion-related validity) should be considered highly suspect.
There are various reliability tests (e.g. test-retest, alternate form, split-half). The Culture Index Survey used the split-half reliability technique (specifically, Cronbach’s Alpha)
Again, reputable test publishers will document their reliability correlation coefficients, and the figures should be at least 0.70 or better.
All inventories can be susceptible to faking (avoidance) – checking words that the respondent believes the employer would like to see checked. It would be absurd to state that any inventory (or resume, for that matter) could not be faked. However, research has shown that few people attempt to fake, particularly when they are told that fakes responses can be detected. It has also been shown that deceptive responses often manifest themselves in work performance.
The trick is how to minimize faking or detect it when it does occur:
Make certain that a rapport and trust is established between the survey administrator and the respondent. The respondent should be told what the inventory’s purpose is and what it is not. However, the administrator must be careful not to explain the inventory’s measurements since this could bias the responses.
The administrator should never coach the respondent. Providing hints or statements as to the type of personality the employer is looking for will lead to biased responses.
An inventory should not be administered when there is a volatile climate in the organization, i.e., impending lay-offs or where there is antagonism between the administrator and the respondent.
In rare instances invalid responses may occur if the respondent lacks insight into their own characteristics, is self-deceptive, is extremely fearful of criticism, or has an inordinate desire for attention or sympathy.
The Culture Index Survey was not designed to be a diagnostic of personality disorders or to be used in a clinical setting. If an employer suspects that an employee or candidate may be unduly anxious or disturbed, we recommend that the employer use the services of a licensed psychologist, social worker, or psychiatrist.
A faked response alone by a job applicant should never be considered a reason for not pursuing the applicant’s qualifications. An inventory or test is only one criterion or component in the selection process.
Lastly, one should be skeptical when interpreting any personality self report inventory of adolescents or people of very low intelligence. These individuals frequently have distorted self concepts. We do not recommend that the Culture Index Survey be used in these circumstances.
The Culture Index Survey should never be used alone to predict job success for a new hire or promotion or placement of a tenured employee.
In selection, promotion, and reorganization decisions other critical variables have to be used and properly considered – interviews, past experiences, job knowledge, skills, intellect, education, job stability, and performance appraisals, etc. The survey should always be integrated and reported within a complete job-related context. It is not designed to ensure job success, but to improve the predictability of job success.
Most importantly, a thorough job analysis should be conducted. A proper analysis can define and weigh the variables important to present and future successful performance. The C-Job Position Analysis Questionnaire™ supplies information pertaining to the behavioral job demands and our consultancy support is available to assist clients in defining and measuring the other components. Culture Index, LLC conducts job validity studies for their clients, often with no additional charge. We are also available and experienced in helping the client develop job performance measurements in order to conduct such studies. This information usually leads to the development of performance appraisal programs for ongoing use by the client.
The interpretation and evaluation of Culture Index Surveys (and other inventories) must be limited to those who have received formal training in its measurements, applications, and limitations. Even after training, expert consultancy support should always be available. Also, passing along a workshop or interpretive manual to another person for a “quick read” is no substitute for the in-depth instruction, discussions and exercises of the formal training.
Employees or applicants should never be coerced into taking any questionnaire; it should always be voluntary. Coercion can lead to faked responses.
The Culture Index Survey should always be administered in the individual’s primary language.
Access to any test or inventory results should always be limited to those who have formal training in the instrument. Confidentiality of these records is mandatory.
Employees can be provided feedback of their survey response. It is mandatory that it be provided by someone who has been formally trained and available to answer any questions. Feedback to applicants is not required, but may be useful in certain circumstances.
Re-administration is advised to determine new job behaviors, when the position or direct reporting has changed.
The use of inventories and tests must always be within the contexts for which they were designed. This may seem an obvious statement, but there has long been an enigmatic aura around testing - particularly around personality and vocational tests.
In purchasing tests, companies are subject to caveat emptor. There are no federal or state laws preventing the sale of poorly constructed or outright bogus tests. Visibility or longevity in the marketplace is no assurance of a test’s validity – we long ago lost our amazement at certain tests’ tenacity in the marketplace.
It is the buyer’s responsibility to determine test quality and its suitability in their company. Test publishers who are secretive about their research should be scrutinized closely. Also, “oldie, but goodie” does not apply to tests unless they have been periodically checked, updated, and supported by documentation.
Reputable firms publish their research or allow it to be reviewed. They point out their products’ limitations as well as applications and advantages, and, finally, they will make certain that the products are properly explained to company personnel and positioned within the company.
If you have any questions regarding the Culture Index Survey, the C-Job Position Analysis Questionnaire or testing in general, do not hesitate to contact us.
.png)
