​Note: Only Site Owners/Admins, select client staff, and KU staff can see the Secondary Scoring section. Go to Other Permission Roles​ and review all available access levels, including “Data Admin – Secondary Scoring” to understand more about your options for those needing to do Secondary Scoring.

Secondary Scoring Overview

The Secondary Scoring feature is designed to serve as a user-friendly tool to assist with quality assurance and accuracy of data entry for Accomplishments. This feature includes entering secondary scores, calculating rates of inter-observer reliability, viewing coding discrepancies, providing comments and feedback on documented events, reflecting on next steps for improving levels of inter-observer agreement, generating recommendations for improvement, and sharing overall feedback and ideas for improvement with those entering the data.

Customized scoresheets can be created for providing feedback on data (for example, on a monthly or quarterly basis). You set the parameters for which data you would like to review, and once the Secondary Scoring Sheets are created, you can save and resume your work at any time. Data entered in Secondary Scoring is synchronized with ​​Accomplishments, so you have the option to display a green or red flag next to that entry in the Accomplishments listing. See the Secondary Scoring Flags section below for details.

We will use the terms Observer 1, Observer 2, and Observer 3 to refer to the following:

  • Observer 1: The primary person entering data for an Accomplishment.
  • Observer 2: The Secondary Scorer, aka the person scoring Observer 1’s entry.
  • Observer 3: The Tertiary Scorer, aka the person who is scoring a scorer. Although not used as often, it provides an additional review process for anyone new to the role of Secondary Scorer by allowing a more experienced scorer (typically KU staff) to double-check Observer 2’s work and make any need recommendations or adjustments.

CCB Owners/Administrators assign visibility and access to Secondary Scoring for users based on their specific needs and responsibilities. Secondary Scoring can be accessed from the left navigation bar in the CCB.​ From this Secondary Scoring Platform, secondary and tertiary scorers can easily access the data they need to review.

  • No Flag: Indicates the record has not yet been scored by Observer 2 or Observer 3.
  • Green Flag: Indicates agreement on scoring by all Observers who have coded the entry.
  • Yellow Flag: Indicates lack of agreement on scoring between Observer 2 and Observer 3.
  • Red Flag: Indicates lack of agreement on scoring between Observer 1 and Observer 2.

Visual example of Secondary Scoring Flags.

Secondary Scoring is an option available to all projects in the CCB. By default, this section is not displayed until it is determined that it will be used by a project. Administrators then provide access to the Secondary Scoring section for anyone serving as Observer 2 or Observer 3. Documenters (Members/Users) do not need access to this tool.

Administrators also need to determine if Secondary Scoring Flags should be visible for all CCB users under View Accomplishments. Enabling this feature will add colored flags (icons) next to each Accomplishment entry that show the outcome of agreement between Observers 1, 2, and 3.

Before you can use Secondary Scoring, you must enable it for the specific CCB form you will score.

  1. Go to Administration.
  2. Select Accomplishment Forms.
  3. Click on the Form Name, and click Edit.
  4. To enable Secondary Scoring slide the Secondary Scoring Enabled toggle to the right.
  5. If you want the Secondary Scoring Flags to appear on the Accomplishments listing, slide the Accomplishment Flag Enabled toggle to the right. It is possible to do Secondary Scoring without showing the flags.

New Form admin settings

Secondary Scoring Form

  1. Click New.
  2. Enter the Title for this specific scoresheet.
  3. Select the Date Range, whether it's monthly, quarterly, biannually, or annually, for example. Enter the related start and stop dates.
  4. Type in your name as the Observer.
  5. Select your Observer Role (either 2 or 3).
  6. Select the Accomplishments Form Name from the list even if there is only one option. When you do that two more fields will appear.
  7. Check the Scoring Field(s) you want to use for this purpose. Normally, it's the Code Field, sometimes referred to as "Activity Type."
  8. Check all the Fields to View​. These are the responses to questions that will help you decide if the Code is correct. The Description Field is one of the best for this purpose, but select any fields that will help you determine the proper Code.

​Note: When you first click on Secondary Scoring, the only action link that will be available is New. After you select a previously created scoresheet, the other options of Edit, Delete, Scoring and Download will be active.

  • To complete a scoresheet, follow the protocols outlined for your project regarding the scoring of entries. If a flag is present, this indicates the entry has previously been scored by another observer. You can change the score selected or leave as-is.​
    1. ​Click once on the scoresheet you created, then click Scoring.
    2. ​You will see the following table with the columns you selected for scoring and viewing.
    3. Review the viewing fields selected previously, then choose the correct option under the scoring column(s), which all have a 2 or 3 after the question, indicating your Observer role.​​
      • Note: There is a separate scroll bar for the scoresheet.
    4. You may add a comment in the Inner Observer Comment field for any entry. Comments can provide encouragement or instructions for improving the coding decisions made by Observer 1.
    5. Continue until you have all the entries scored.
    6. If you have an overall comment, add it in the ​Overall Comments field at the bottom of the screen.
    7. Click Save when you are finished.

Secondary Scoring Scoresheet

(Click the image above to open it full-sized in a new window.)

Once you have completed a scoresheet, you can generate a report from it to reflect on the results yourself, as well as provide it to users documenting entries within CCB.​ To download a scoring report:

  1. ​Select the scoresheet you want to download by clicking on it once ("QA 1st Quarter 2019" is highlighted in screenshot below.).
  2. Click Download.

Secondary Scoring download options

  1. The Secondary Scoring Report will consist of a Cover Page, Score Sheet Data, and at least one Agreement Summary. Each of these will be available on a separate tab at the bottom of your downloaded Excel document.

Cover Page

  • The Cover Page serves as a quick reference to the overall outcome of the Secondary Scoring. It features the scoresheet's Title, Observer Name, Date Range, Number of Entries, and % Agreement between Observers 1 and 2. You can customize this page in Excel to include additional details if appropriate to the project.Scoresheet Cover Page from the Excel document.

 

Score Sheet Data

  • A spreadsheet of the data located in the second tab of the Excel document.
    Score Sheet Data from the Excel document.
    (Click the image above to open it full-sized in a new tab.)

Agreement Summary

  • The Agreement Summaries 1 and 2 (tabs 3 and 4 from the document) display the inter-reliability report of each Observer's scores.  This includes:

    • Number of entries
    • Number of agreement points
    • Observed Agreement
    • Chance Agreement
    • Kappa

     A table will outline the agreement summary and highlight the agreement between observers. Agreement Summary 1:

    Agreement Summary 1 from the Excel document.
    Agreement Summary 2:

    Agreement Summary 2 from the Excel document.

If scoring has been completed for an entry, and if flags are enabled for the CCB form, a colored flag (green, yellow, or red) will appear in the View Data section under Accomplishments to show if there is a match between the Codes provided by Observers 1, 2, or 3. To access the section shown below in your CCB:

  1. Go to the CCB Dashboard.
  2. Click ​View Data.

Secondary Scoring flags shown under View Data in Accomplishments.