Quality Calibration

Trainers Tips

Regular calibration of quality assessment forms is an effective way to avoid potential subjectivity within the team of people conducting evaluations. It may happen that each individual team member will interpret the questions on the forms slightly differently. When this happens the agents may feel confused or see that their final scores are inconsistent and unclear. Our Trainers have prepared are several methods for conducting calibrations with Eleveo Quality Management, as well as suggested best practices.

Note that calibrations are not one-time processes, but should be performed at regular intervals, preferably frequently, to achieve optimal results.

Questionnaire for Calibration Best Practice:

Calibrations are often an internal exercise which ensure that the review process is objective. For this reason, the results of the process should not be counted for or against the agent. One way to accomplish this is to select the Questionnaire that will be used for reviews, change its name so that it is more obvious that it is for calibration purposes (e.g., "Calibration Customer Service Review Form"), and then select "Save Copy."

When performing calibrations, using either of the selection methods described below, select this new Questionnaire dedicated to this purpose, instead of the Questionnaire normally used. Since the system stores and trends scores at the Questionnaire level, all calibrations performed will be directly linked to the scorecard created specifically for the calibration.

Note: When creating reports and trends, be sure to select the appropriate questionnaires and avoid calibration data accordingly.

Calibration Methods

System Selected Call Calibrations:

With this method, the Review Scheduler can be used to select a random call for calibrations for a selection of evaluators to review. The advantage of this method is that the evaluation process can be performed at regular intervals with selection criteria built into the selection process (e.g., minimum call duration, call direction, etc.). A disadvantage of this method is that the selected calls can not be replaced, so the calls may turn out to be overly simplistic or may not even meet the requirements of the evaluation. Another point to keep in mind is that if you set up the Review Scheduler to repeat regularly. you will either have to select the same agents you want to use each time, or select all agents and delete the ones you do not want to use in a certain time period.

Instructions on how to use the Review Scheduler can be found on the page named Review Scheduler.

Some additional notes to consider:

If you have created a Questionnaire specifically for Calibrations, make sure you select this questionnaire when prompted to do so.
Select the "For Calibrations" box when prompted in the Review Scheduler. This option will ensure that all reviewers receive the same agents/content to review.
Once the Review Scheduler has been run, have all reviewers perform the evaluations independently of each other.
Run the Compare Evaluators report to review the result (see Calibration Reporting and Interpretation below).
- Note: The last day of the review period is the day the data appears in the Compare Reviewers Report.

Ad Hoc Call Calibrations:

This method uses the Conversation Screen to select a specific call to assign to multiple reviewers for calibrations. The advantage of this method is that complex, nuanced calls can be selected for calibration. However, one disadvantage is the manual nature of the process.

For instructions on how to perform an ad-hoc review using the Conversations Explorer screen, refer to the dedicated page named Selecting a Suitable Review for Calibration.

Some additional notes to consider:

If you have created a Questionnaire specifically for Calibrations, make sure you select this questionnaire when prompted to do so.
have all reviewers perform the evaluations independently of each other.
Run the Compare Evaluators report to verify the result (see Calibration Reporting and Interpretation below)
- Note: The date of the review is the day the data appears in the Compare Reviewers Report.
Repeat these steps at regular intervals (e.g., monthly, quarterly, etc.)

Calibration Reporting and Interpretation:

Regardless of whether you use the Review Scheduler or the ad-hoc method of selecting calls for calibration, run a Compare Reviewers report after the reviewers have completed their assessments. This report is a useful tool that allows you to quickly identify where reviewers may have differing opinions on questionnaire items and the conversation being reviewed.

Instructions on how to use the Compare Reviewers Report can be found on the page named Compare Reviewers Report.

Note: When selecting the date, select either a specific date for the ad-hoc calibration or the last date of the period for the calibration selected by the system.

Interpretation of Results:

When differences are discovered, it is important to find out what may be driving those differences:

Is the question too subjective?
Are the answer choices appropriate for the specific question?
Is there confusion about what the goal of the question might be?
Could the standards for calculating the results of the question (especially if the question might include granular scores) be clarified?

For each question where there are differences, pinpoint either the specific part of the call or the agent behavior that is targeted by the question. Discuss what the goal of the question is and what end goal (through coaching and training) it is trying to achieve. The variance could be from the reviewer's perspective or adjustments may need to be made to the questionnaire and/or answer options.

TIP: It is recommended that you use a specific questionnaire for calibrations. If you use the same Questionnaire to perform the actual reviews, consider whether those calibration results should count for or against the agent.

For the reviews you do not want to include in the reporting it is necessary for the Reviewer to complete the following steps:

Select the completed evaluation on the Reviews screen
Select "More Actions" from the top ribbon menu
Then uncheck the "Include in Reports" checkbox from the drop-down menu.

Linked Documentation Standardization of Calibrations and Reporting Procedures