There are common psychometric issues in self-assessment questionnaires: Personal bias, answer style, and inconsistency. Steps have been taken to insulate the SEI scores from these obscuring influences: - Personal bias: the SEI has been tested to consider the effects of these biases by utilizing a “positive impression” scale. To a very large extent, the SEI functions effectively without correction. However, the Positive Impression factor is reported on the data sheet to provide useful insight to a SEI Assessor interpreting SEI results.
- Answer style: Another common psychometric issue is that different people assign a Likert scale (e.g., 1-5) with different meanings. Some rarely use extremes, others “always leave room for improvement.” To compensate for these differences, the SEI includes an Answer Style index.
- Inconsistency: some test takers are inconsistent in their answers which can reveal a lack of understanding or a lack of focus. These can reduce the value of the results. The SEI includes a test of consistency that also evaluates completion time.
These three indices and "Mood Detection" are presented in the SEI Data Sheet that accompany the full SEI Reports (see sample to the right). The corrective factors apply to all versions of SEI, and are explained in more detail below. ANSWER STYLE INDEX (AS)There is a well known issue with Likert Scales, some people use a more “modest” approach (e.g., rarely using a 5/5) while others rarely use the middle (e.g., 3/5).[1] To compensate, this index adjusts the scores based on the frequency of answers rated 1 or 5 (ie, more extreme answers). Answer Style is calculated through a complex algorithm that examines the test-taker’s use of the Likert Scale and the frequency of using 1s, 2s, 3s, etc. compared to the normative sample. The graph to the right shows the distribution of Answer Style versus uncorrected EQ scores, revealing a biasing effect of Answer Style — which is corrected by this index.
Discussion About Answer StyleThere are many influences on how people take a self-report tool — how they are feeling that day, etc. One is “Answer Style.” Some people approach a five-point Likert scale with more positive self-statements, where others are more modest – some use primarily positive ratings, some use primarily negative ratings, some rarely use extremes, etc. In addition, there appears to be a cultural bias to Answer Style.[2] We’ve accounted for this known psychometric issue and adjusted the scoring to remove this obscuring influence. The AS checks for the style of answering — someone who gives a lot of 5s and/or 1s, will get a high AS. The calculation for AS is complex; the EQ scores are adjusted slightly differently based on the mean answer style for each language/country, but the range for the international sample is from - 4% to + 6%. To check the corrections country by country, click the following link: http://6sec.org/seivs and look at the worksheet named “Answer Style Correction.” From an interpretation point of view, practitioners can, generally, accept this correction as-is. The purpose is to improve the accuracy of the SEI, but this data also provide valuable perspective on the client and the context in which s/he may be using EQ. RELIABILITY OF THE SEI SCORESThe reliability of the SEI questionnaire is further tested through 3 indicators: - Positive Impression Index
- Consistency Index (including two tests for random answering and density of answers)
- Completion Time
These are described on the following pages. These three tests are used to create an overall score called the “Overall Reliability Index.” This score shows on the Data Sheet as Red (possibly invalid), Yellow (out of norm), or Green (in normal range). This test provides important insight about the reliability of the results. The overall reliability index will be: RED if there is at least one indicator (above) that scores in the Red zone GREEN if two or more of the indicators are Green and none are Red YELLOW if two or more indicators are Yellow and none are Red If the Overall Reliability Index is Red, it means that the results may not be valid because one or more of the validity indicators is far outside the normal range. In this case, we recommend that the EQ Assessor speak with the test taker and request s/he repeat the assessment. In this rare case, even if the project is set to deliver the assessment directly to the test taker, the project system will block the report delivery and the SEI Assessor will be notified by email: Dear SEI Assessor, You've received an error on Report for Client: [Test_Taker_Name_Surname] <[Test_Taker_Email]> Project: [Project_Name] The SEI resulted "unreliable" This means that the profile could be invalid (see below). The Report was calculated and is available for you to download from quest.6seconds.it -- but not sent to test taker. Note that your account was not charged for this assessment -- so you have the freedom to re-administer the questionnaire. * Explanation * The SEI includes a Reliability Index including several tests such as consistency and time for completion. This helps ensure the test-taker is paying attention, understanding the questionnaire, and maintaining focus. If there are too many items that are out of the normal range, the system identifies this as an "unreliable" profile and this email is generated. (See the Technical Manual for more detail on this feature.) We suggest you encourage your client to re-take the questionnaire. (Alternately, you may choose to download the report and email it to the test-taker.) POSITIVE IMPRESSION (PI)Positive impression is derived from a scale of six items that are included in the questionnaire. The PI scale is designed to measure the inclination to have an exaggerated view (representing either a negative or positive bias). The score on these six items is compared to the normative score, and the amount of difference between the PI score and the mean score determines one of the following labels in the data sheet: very low, low, average, high, or very high positive impression.
Discussion About Positive Impression
The Positive Impression (“PI”) scale asks questions such as, “I never have bad days.” Answers to these questions create a sort of "rose colored glasses meter." Someone with a “high” PI is wearing “rose colored glasses.” For someone with a “very high” PI the glasses are quite intense. The assessor can use this information to understand how this person falls in a continuum from a very negative self-impression (under-evaluation) to a very positive self-impression (over-evaluation). Someone who scores “very low,” for example, could have unrealistically negative view of her/himself and maybe s/he using gloomy glasses! Very high PI can be an indicator that the test-taker is attempting to manipulate the test results (though it could also signify very high confidence, or a blissful disregard to challenges). Very low PI can be an indicator of low self-esteem. This consideration is the reason why the positive impression feedback is included in the Overall Reliability Index. In case of very high PI, a red light will appear in the data sheet. In case of Very Low, Low or High, a yellow light appears in the data sheet. In case of Average, green light. CONSISTENCY ANSWERS (CA)The consistency index evaluates the frequency of answer choices that the test-taker uses in the 5-point Likert Scale (e.g, “I Agree,” “I Disagree”). This evaluation is based on the elaboration of two indicators: 1. DENSITY INDICATOR - how often does the person use option 1, 2, 3, 4, or 5? Their frequency is compared to the international standard. If one or more options is significantly over-used, the system identifies a potential problem. 2. RANDOM INDICATOR - this test compares way the test taker answers every item with the international standard. If they follow a random pattern in answering, the system detects a potential problem. To increase accuracy, this indicator is also linked to completion time to make the final random feedback more accurate. These indicators are summarized in a single CA index, reported in the Data sheet through 3 lights: Red Light - Low Consistency (problems in one or both indicators) Yellow Light - Moderate Consistency (potential problem in the random index) Green Light - High Consistency (both indicators in a normal range) Discussion About Consistency AnswersConsistency can be a sign of focused attention while taking the SEI. In the case of a “Red Light,” the SEI Assessor should assess the individual situation. We suggest asking the test-taker, “What was it like for you to take the SEI?” The answers might reveal: - The situation was chaotic or distracting. Recommendation: Set aside 10 minutes of quiet time to retake the SEI.
- The test-taker did not understand the questionnaire. Recommendation: Check if the questionnaire should be administered in another language, or of the test-taker should have someone assisting her/him while re-taking.
- The test-taker has a tendency to be inconsistent. Recommendation: The SEI may be accurate as-is.
COMPLETION TIME (CT)As in the other SEI indices, the individual test-taker’s behavior is compared to a large international sample; typically individuals take around eight minutes to complete the SEI. If completion time is unusually fast or slow, it’s a signal of a potential issue. The completion time index is calculated based on main questionnaire, starting after the personal data page, and ending before the final, optional, mood question. If the completion time is: Extremely short: Red Light Short: Yellow Light Average: Green Light Long: Yellow Light MOOD DETECTIONSEI 4.0 (launched August 2015) includes a final optional question. This question helps the EQ Assessor have a better sense of the test-taker’s current situation. In addition, Six Seconds uses this data for research about the relation between EQ competences and mood. The mood model starts by asking the test taker to choose one of eight basic emotions.[3] Next, the test-taker is asked to clarify the intensity level of their mood via one of three options (for each of the eight): Basic Emotion | Low Intensity | Medium Intensity | High Intensity | Anger | Annoyed | Mad | Enraged | Anticipation | Interested | Eager | Vigilant | Joy | Serene | Happy | Delighted | Trust | Accepting | Safe | Admiring | Fear | Uneasy | Worried | Terrified | Surprise | Distracted | Puzzled | Amazed | Sadness | Pensive | Unhappy | Grieving | Disgust | Bored | Disturbed | Loathing |
This final question is optional.
[1] Greenleaf, Eric A. 'Measuring Extreme Response Style'. Public Opinion Quarterly 56.3 (1992): 328. [2] Harzing, A.-W. 'Response Styles In Cross-National Survey Research: A 26-Country Study'. International Journal of Cross Cultural Management 6.2 (2006): 243-266. [3] Plutchik, R. (2001). The nature of emotions. American Scientist, 89, 344–350. |