1 Introduction

The BJCP is very grateful for the time and energy that our volunteer graders contribute to the program. While we cannot compensate graders financially, grading exams is a key part of the Grand Master Service Requirements (GMSR)  needed to become a Grand Master judge exam graders also earn non-judging experience points for their efforts. In return for enabling this promotion path, BJCP exam graders are asked to complete the grading assignments in a reasonable amount of time. Exam sets occasionally get stalled, and while this is sometimes unavoidable, we want to minimize these incidents. The timeline laid out below establishes a process that keeps the exam program running smoothly and ensures that there are no excessive delays in the path for the examinees to become BJCP judges.

An additional and related aspect of the timeline is a progression path from apprentice grader to lead exam grader. The exams are typically scored by a team of two graders, with one functioning as the lead and the other in a supporting role as second grader. The lead grader ultimately has the responsibility for completing the Report to Participants (RTPs), but the second grader may also assume that role for some exams in the set. Each grader receives GMSR credit as a lead grader for the RTPs they complete, regardless of whom is nominally serving in the capacity of lead grader. It generally takes a few exam sets to become familiar with the exam grading process, but after this calibration period, many graders evolve to the lead grader role. An outline of this progression path is provided at the end of this document.

As stated in the Exam Scoring Guidelines, our target is to complete the grading and review of exam sets in eight weeks from when the exams reach the graders. This requires that graders complete the scoring in no more than four weeks, which should be manageable given the less stringent demands on grading primarily Beer Judging Exams under the 2012 exam structure, as opposed to evaluating the combined essay and tasting format of the Legacy BJCP Exam.

The description of the process is composed of a number of tasks, some have to happen before the exam date and others happen after the exam date.

2 Grading Process Work Breakdown

It is important to realize that the descriptions and the associated Gantt chart only shows a single exam set flowing through the BJCP but at any point in time there are many exam sets being processed by the BJCP grading team. Multiple exam sets are given every month, more sets than there are Associate Directors or Exam Directors. At a higher level, not shown here, that means the processing of individual exam sets can encounter resource conflicts at the AD and ED level. These conflicts are hard to predict since they are driven by when individual sets reach the AD and/or ED for review.

The task numbers in this section refer to items in the detailed Gantt chart that shows all activities.

Given that we are all volunteers, we realize that the “ideal” timeline doesn’t always work given the reality of family, work and other commitments. If this situation occurs, it should be apparent early in the grading process, and it is much better to request that the exams be re-assigned rather than delaying the grading cycle by weeks or even months. We can accommodate some flexibility in the grading schedule, but that requires open communication between the graders and the AD/ED if issues arise during the grading. If this communication does not happen, or if the delays are excessive, the exam sets can and will be reassigned to other graders so that they can be completed in a reasonable amount of time.

2.1 Exam Director’s duties prior to the exam

Prior to the scheduled exam date, the ED may be in discussion with the local exam administrator to work out details such as the proctors and the exam beers, meads or ciders.

Task 1,1; the ED creates the PDF files for the exam site. This includes inserting the exam code, location, and date into the appropriate files.

Task 1.2; the ED emails the files to the local exam administrator a week to 10 days prior to the scheduled exam date.

Task 1.3; the ED sends emails to the graders and AD scheduled to handle this exam set. This notice should reach the graders three to three and one half weeks prior to when they will receive the actual exams.

Task 1.4; the ED sends email to the AHA support staff providing the exam code and the exam date. The shared folder name is created using the exam code.

2.2 Local exam administrator’s duties prior to the exam

The local exam administrator has a number of activities that must be completed prior to the exam day.

Task 2.1; the local administrator works with the ED on exam beers or meads and on approval of proctors – proctors not on the pre-approved lists on bjcp.org must be approved by the ED in advance.

Task 2.2; really a milestone, the receipt of the exam files from the ED.

Task 2.3; the local administrator make the necessary copies of the exam files. The cover letter from the ED details how many copies are required per examinee.

Task 2.4; for Beer Judging or Mead Exams, the local administrator prepares the beers/meads that will be evaluated. The local administrator also completes the beer/mead description document that will be submitted to the ED along with the exams. These descriptions are needed by the graders to understand what was presented to the examinees.

Task 2.5; for all exams the local administrator collects the information needed to fill in the Examinee Data Capture (examinee-data-capture.xls spreadsheet). The local administrator also collects the information needed to fill in the Exam Staff Capture (exam-staff-capture.xls spreadsheet) for Beer Judging or Mead Exams.

Task 2.6; for all exams involving tastings the local administrator prepares the electronic description of the beverages. For Beer Judging, it is <exam code>-BEERS.doc, for mead exams it is <exam code>-MEADS.doc.

2.3 The actual exam is administered

Task 3.0; on exam day, the local administrator follows the instructions from the ED and gives the exam.

2.4 The local exam administrator’s duties after the exam

After the exam is over the local administrator has a number of immediate duties.

Task 4.1; the local administrator makes a copy of all the exam paperwork. This copy has two purposes; first, it is a backup for the originals that are to be sent to the AHA Support staff and two, eventually a copy is provided to each examinee.

Task 4.2; the local administrator emails the Examinee Data Capture (file name examinee-data-capture.xls) , any Exam Staff Capture (file name exam-staff-capture.xls) and any description of exam beers (file name examcode-Beers.doc), meads (file name exam code-MEADS.doc) to the ED.

Task 4.3; the local administrator pays the BJCP via PayPal. The BJCP only accepts payment for the exams via PayPal – only the administrator pays the BJCP, not the individual examinees.

Task 4.4; the local administrator sends the exams to the AHA Support staff via USPS Flat Rate mail. Alternatively, if the ED agrees, a high quality scan of the exams may be acceptable; scans are then directly sent to the ED. If the scans are set directly to the ED then the ED MUST copy the scans to the appropriate shared storage folder.

Task 4.5; the ED acknowledges receipt of the exams in an email to the local administrator. The local administrator will then provide a copy of each examinees exam back to them – only their exam page, not the proctor sheets.

2.5 The BJCP Treasurer’s duties after the exam

Task 5.1; the BJCP Treasurer receives notice from PayPal that the exam set has been paid for and notifies the ED and the Assistant Exam Director. Exam results will not be released if payment is not received.

2.6 The Exam Director’s duties after the exam

After the ED receives the scanned exams the ED has a number of tasks.

Task 6.1; really a milestone, the receipt of the exams by the AHA.

Task 6.2; after the AHA notifies the ED that the exam have been received, the ED emails the local administrator to acknowledge receipt of the exams.

Task 6.4; the ED updates the grader forms for the set. This includes:

• For the RTP:
  • customizing the RTP with the graders’, AD, and ED names
  • the location
  • the exam code
  • the exam date
• For Beer Judging and Mead Judging Exams the EGF including:
  • the graders’ names
  • the location
  • the date
  • the tasting scores
• For Beer Written Proficiency the EGF:
  • The true/false question numbers
  • the examinee answers to the true/false question

Task 6.5; the ED validates that all examinees are eligible to take the exam. The ED notifies ineligible examinees that they will not be receiving exam results or a refund.

Task 6.6; the ED send the exams and forms to the graders and the AD. The forms are emailed to the graders and AD. The exams are transferred to the graders by passing a link to the read-only shared storage space containing the scans.

Task 6.7; the ED updates the Exam Staff Capture form to include the graders, AD and ED information for the set.

Task 6.8; the ED sends the Examinee Data Capture and Exam Staff Capture to the AED. Have the ED do this as soon as possible closes the loop with the examinees quickly and also minimizes issues with sending the final exam results later by getting the examinees entered into the BJCP database so that their moves after the exam date can be recorded.

2.7 The Assistant Exam Director’s duties after the exam

The Assistant Exam Director (AED) handles the initial entry of the exam data into the BJCP database.

Task 7.1; a milestone, the AED receives the Examinee Data Capture form and Exam Staff Capture form from the ED.

Task 7.2; the AED imports the data from the Examinee Data Capture form and the Exam Staff Capture form into the database, BJCP IDs are created as necessary. Various fields in the database are setup for when the exam set is ultimately closed including the names of the ED, AD, and graders.

Task 7.3; the AED validates the information extracted from the Capture forms. This is done for both new and existing members. This information is needs to be correct for when the exams are closed.

2.8 The initial scoring and consensus activities

This set of activities focuses on getting the consensus scores for the exam set.

Task 8.1; a milestone, the graders receive the exams from the ED. Based on recent data this happens about 15 days after the exam date.

Task 8.2; the grades independently assign scores and complete their EGF. This task is allocated three weeks.

Task 8.3; the graders reconcile their scores and add them to the EGF. This task is allocated one week. During this task, the lead grader starts writing the RTPs. If there is no communication with the AD and ED, the AD emails the graders for a progress report.

Task 8.3.1; this task is optional, the graders have communicated with the AD and ED to request an extra week to complete their scoring. This should be an exceptional situation. If after the week there is not communication with the AD and ED, the AD emails the graders for a progress report.

Task 8.3.2; this task is optional, the graders have communicated with the AD and ED to request a second extra week to complete their scoring. This should be a really exceptional situation. If after the week there is not communication with the AD and ED, the AD emails the graders for a progress report.

Task 8.4; both graders send their completed EGF and consensus scores to both the AD and the ED.

If grading has not completed task 8.4 within five weeks (three is desired but five can be tolerated with valid reasons) of task 8.1, one or both graders has/have been unresponsive or unable to fulfill the grading commitment. This is a situation we want to avoid.

If the lead grader is the bottleneck, then the lead grading role is transferred to a back-up grader, an incomplete assignment by the lead grader is noted, and no GMSR credits are granted. Processing goes back to task 6.6 for the ED to send the exams to the back-up grader.

If the second grader is the bottleneck, the exams are assigned to a back-up grader, an incomplete assignment by the second grader is noted, and not GMSR credits are granted. Processing goes back to task 6.6 for the ED to send the exams to the back-up grader.

2.9 The completion of the RTPs

This set of activities focuses on getting the RTPs completed and delivered to the AD and ED.

Task 9.1; the lead grader completes the RTPs. This task is allocated two weeks. The lead grader is responsible for completing the RTPS but the second grader may be writing some of them. If there is no communication with the AD and ED by the end of the week, the AD emails the graders for a progress report.

Task 9.1.1; this task is optional, the lead grader has communicated with the AD and ED to request an additional week to complete the RTPs. This should be an exceptional situation. If there is no communication with the AD and ED by the end of the week, the AD emails the graders for a progress report.

Task 9.1.2; this task is optional, the lead grader has communicated with the AD and ED to request a second extra week to complete the RTPs. This should be a really exceptional situation. If there is no communication with the AD and ED by the end of the week, the AD emails the graders for a progress report.

Task 9.2; the lead grader emails the completed RTPs to the AD and ED. The completed RTPs should not have a score filled in but everything else should be filled in.

If grading has not completed task 9.2 within three weeks (one is desired but three can be acceptable with valid reasons), the lead grader has been unresponsive or has been unable to fulfill the complete grading commitment. This is a situation we want to avoid. Since the lead did complete scoring but not the RTPs, he or she is still given GMSR credits as second grader but not the additional credits normally allocated to the lead grader. The AD and ED will share responsibility for completing the RTPs, and incomplete assignment by the lead grader is noted.

2.10 The Associate Director’s initial scoring of the exams

The AD does an initial scoring of the exams as a baseline for reviewing the graders’ results.

Task 10.1; a milestone, the AD receives the exams from the ED.

Task 10.2; the AD determines preliminary scores for the exams. This task is allocated two weeks.

2.11 The Associate Director’s review of the grading

This set of activities focuses on the AD review of the grading results.

Task 11.1; a milestone, the AD has received the EGF and consensus scores.

Task 11.2; the AD notifies the Exam Scheduler (ES), the ED that handles scheduling of exams and graders, that preliminary scoring is completed. The ES needs this information to coordinate awarding credits to sites trying to increase their seat limits.

Task 11.3; the AD reviews the graders’ scoring. The AD evaluates the scoring vs. the AD’s preliminary scores produced in task 10.2.

Task 11.4; a milestone, the AD receives the finished RTPs.

Task 11.5; the AD notifies the ES that the RTPs have been completed for this set. The ES needs this information to coordinate awarding credits to sites trying to increase their seat limits.

Task 11.6; the AD reviews/revises graders’ scoring and RTPs. This task is allocated three weeks. The AD reviews and revises the scores and RTPs as necessary. Changes to the RTPs are made with track changes active.

Task 11.7; the AD emails the reviewed/revised scores and RTPs to the ED. The AD sends a cover letter with final scores along with the revised RTPs to the ED.

2.12 The Exam Director’s review of the grading

This set of activities focus on the ED’s activities to complete the grading and to close the exam set.

Task 12.1; the ED reviews/revises the scores and RTPs. This task is allocated two weeks. The ED modifies scores and RTPs as appropriate. Changes to the RTPs are made with “track changes” active.

Task 12.2; the ED finalizes the exam scores and RTPs.

Task 12.3; the ED sends the final results and RTPs to the graders, AD and AED. The revised RTPs, and final scores are sent out. In keeping with the BJCP Privacy Policy, the graders and AD are not provided the names of the examinees.

2.13 The Assistant Exam Director’s duties to close out the exam

This set of activities focuses on getting the results into the BJCP database and notifying the examinees of their results.

Task 13.1; the AED enters the results into the database. The final scores are entered into the database along with entering/updating the proctors, administrator, graders, AD, and ED

Task 13.2; the AED sends final results, pins, RTPs, credentials to examinees.

2.14 The graders and Associate Director’s review of the final results

Task 14.1; the graders review the final scores and RTPs (updated during review with “track changes”). The revised documents and scores help them improve. They may question the AD and/or ED about the changes.

2.15 The examinees’ review of their final results

Task 15.1; the examinees receive their scores and RTP. At this point the exam set has finished grading.

2.16 Exam scheduling activities

The activities in this section are related to coordinating the credits for other designated exam sites that are trying to increase their exam seat limits by helping recruit exam graders.

Task 16.1; the ES determines if the graders associated with a designated exam site have completed the grading assignments.

Task 16.2; the ES notifies the local exam administrator and the ED that the site seat limit is increased. This only happens if all the assigned graders complete their work prior to the designated exam date. This task couples into the Gantt chart for a different exam set. Current data indicates that it typically is 10 to 11 weeks after an individual grader receives the exam when credit can be given to increase the seat limit for a designated site.

2.17 AHA support activities

Task 17.1; the AHA receives email from the ED detailing the exam code and the exam date. The exam code is used to identify the shared folder the exam files will be placed in.

Task 17.2; the AHA receives the hard copies of the exam from the local administrator.

Task 17.3; the AHA sends email to the ED containing the exam code and indicating that the exams have been received.

Task 17.4; the AHA scans the exams and places the images in the shared folder indicated in task 17.1.

Task 17.5; the AHA emails the ED a notice that the exams have been scanned.

Task 17.6; the AHA mails the original hard copies to the BJCP

3 Incomplete Grading Assignments

Any incomplete results from either grader will be noted by the ED, and the responsible grader will be notified. Graders are given two “freebie” incompletions, but are put on probation after the second is incurred. If a second grader is responsible for a third incompletion, he or she is removed from the list of active graders. If a lead grader is responsible for a third incompletion, he or she is relegated to the second grader role for future exam sets. This aspect of the graders history is not part of one’s BJCP record and is held in confidence by the Exam Directors. The Exam Directors also reserve the right to modify this policy in specific cases if there are extenuating circumstances beyond the control of the graders.

4 Progression Plan to Lead Grader

The BJCP typically enlists approximately 30 judges as new volunteer graders in a calendar year. Some of these graders only score one set to get a better feel for the examination progress, and this provides useful training for leading CEP or Exam Study courses. However given the extra time and effort it takes to mentor new exam graders, a more preferable outcome is that the judge continues in the grader role, even if he or she is only able to volunteer for a few sets per year. Grading exams at some level is an excellent way to continue one’s beer judging education and give something back to the exam community.

As noted above, the exam grading is typically split into lead and second grader roles. While there are rigorous requirements for being a BJCP exam grader, the distinction between lead and second grader is fuzzier due to our diversity of backgrounds and judging experience. Generally, the lead grader possesses most or all of the following attributes:

1. Active exam grader (has graded at least once in the past two years),
2. Has graded at least four exam sets in the past two years,
3. Is up-to-date on any changes made to the BJCP exam or scoring process,
4. Has completed exam grading assignments on a timely basis,
5. Has demonstrated the ability to complete RTPs with clear, concise and constructive feedback, and
6. Has demonstrated the capability to mentor apprentice and junior graders.

Most of these attributes are teachable, but it does take a few exam sets to move up the learning curve. A typically progression path is as follows:

1. Apprentice grader. These graders have typically scored only one or two sets, or do not score exams on a regular basis. They should review the RTPs, but do not take an active role in their completion. They still provide a valued service by providing supporting scores to the lead grader and AD.
2. After one or two exam sets, these graders take an increasing level of responsibility for completing the RTPs. For example, with most exam sets having 12 exams, they may take on completing two RTPs in their third grading assignment and then four RTPs in their fourth grading assignment.
3. Lead grader. These graders have the experience and ability to take responsibility for at least half of the RTPs in an exam set. Many of these individuals are involved in the mentoring and coaching of new exam graders and also contribute in other ways to the homebrewing community.
4. Associate Exam Director. These positions are appointed by the BJCP Board of Directors upon the recommendation of the Exam Directors. These individuals have typically been involved with the grading program for several years and have scored over 100 exams. They play a critical role in reviewing the consensus scores assigned by the graders and the written feedback on the RTPs. This is an excellent role for increasing ones visibility in the BJCP, and we have historically rotated one or two lead graders into this role each year. While not formally required, most Exam Directors were previously an Associate Director.

Exam Director. These positions are appointed by the BJCP Board of Directors. The qualifications and responsibilities for this position, as well as the Associate Exam Director, are provided on the [highligh]need to provide a link to details[/highlight] BJCP web site. The AD and ED responsibilities are described here.

5 Detailed Chart

The detailed Gantt chart depicting the processing of a single exam set is shown below. For creating this Gantt chart, the exam date was set as January 1, 2014 which results in the examinees receiving their scores and RTPs on May 12, 2014. The May 12^th date is if the extra two weeks are used in Task 8.4 and the two extra weeks are used in task 9.2. Otherwise the result would be available on April 14, 2014.

Timeline Gannt chart – PDF

Objective

This BJCP Scoresheet Guide was initially developed to serve as a rubric for grading the BJCP Beer Judging Exam. However, after assembling a comprehensive list of what constitutes a great score sheet, we realized that this document can also be used as a training tool for current and aspiring judges.

While this document was prepared for the BJCP Beer Judging Exam the same methodology applies to the other BJCP Judging Exams.

Background

The information in this document is intended for use by current judges, by new judges preparing for the BJCP Beer Judging Exam, and by BJCP Beer Judging Exam graders. It was drafted by the BJCP Exam Directorate in response to suggestions[1] for a more transparent and consistent process for grading BJCP Beer Judging Exams. The quantitative metrics were created after evaluation of scoresheets completed by National, Master and Grand Master judges. Additional information was incorporated after review by the BJCP Associate Exam Directors (EDs) and by active BJCP exam graders. The BJCP Scoresheet Guide was created using a data-driven approach that should minimize any subjectivity (or perceived subjectivity) with the exam grading process. It should be noted that as with grading a written exam on any subject, there is generally a learning curve needed to make an accurate and expedient evaluation of the quality of a beer scoresheet. This document should make that learning curve less steep for graders and make the grading process more transparent to our BJCP membership. The BJCP Scoresheet Guide is intended to supplement the Exam Scoring Guide used by BJCP exam graders but should not be used as a recipe for finding faults when evaluating a scoresheet. In other words, the intent is not to incorporate the algorithm into a spreadsheet for graders to enumerate every positive and negative aspect of a score sheet, but to serve as a self-consistent check that the assigned score correlates with the maximum achievable judging level based on the score.

[1] Special thanks to Andrew Luberto for drafting a document outlining his concerns with the grading process and providing an impetus to move forward with this guide

Supporting Documents for Exam Graders

The following document should be reviewed by graders, particularly new graders or graders who have not graded exams within the previous twelve months. These documents can be found on the BJCP website and downloaded by clicking on the provided hyperlinks.

1. Exam Scoring Guide. This document outlines the mechanics of the exam grading process and provides guidance for the expectations for scoresheets that correspond to the different judging levels.
2. Exam Grading Form (EGF), which is located in the Exam Grader Resources section of the BJCP Exam Center. This grading tool is a fairly complex Excel spreadsheet that is meant to streamline the grading process, but it can be intimidating for some graders. The ED will have entered the component and total scores for each beer, and these determine the scoring accuracy. The primary role of the lead and second graders is to enter their scores for Perception, Descriptive Ability, Feedback and Completeness/Communication into the appropriate cells in their respective tabs in the EGF. This will produce a summary report that the graders use to compare and reconcile scores, and it also generates feedback tables that the lead grader will paste into the Report to Participant. Exam graders should ensure that they are using the current version of the EGF, since some of the previous ones have errors that resulted in incorrect scores or tables.
3. Report to Participant (RTP), which is a Word document also located in the BJCP Exam Center. Most graders are using the checkbox version of this form, and the ED will provide each grader team with a template that incorporates the exam code and other information specific to the exam set they are grading. The graders should review the check box fields before they start scoring the exam so that they know what they should be noting for incorporation into the RTP after they have reached consensus scores. In particular, they will want to note:
   1. Characteristics that were not identified (either by omission, unfamiliarity or insensitivity)
   2. Characteristics to which the examinee may be sensitive, e.g. diacetyl
   3. Whether the examinee described the intensity of key beer characteristics
   4. Whether the examinee gave feedback on the stylistic accuracy
   5. Whether the examinee suggested corrective actions for any perceived technical issues
   6. Whether the examinee described characteristics in incorrect sections of the scoresheet, e.g. astringency as an Aroma component
   7. Whether the feedback assumed anything about the process or ingredients
   8. Whether there were an excessive number of blank lines in the scoresheet
   9. Whether the handwriting was difficult to read

BJCP Scoresheet Components

The five components of beer scoresheets evaluated on the BJCP Beer Judging Exam are Scoring Accuracy, Perception, Descriptive Ability, Completeness and Communications, and Feedback. These competencies are discussed below, and guidelines are given for achieving proficiency in each. While these can be quantified to some extent, the scoresheet should be evaluated as a whole and not as the sum of its parts. It should also be noted that the criteria for a Master level scoresheet is 90 that denotes perfection, and that was taken into account in designing the grading rubric.

Scoring Accuracy

Scoring Accuracy is evaluated on the individual exam beers based on the differences between the total scores of the examinee and the consensus score from the proctors, according to the chart below. A variance of less than seven points is required to earn at least 60 of the maximum and well below the passing threshold of 60%. This is low enough to send a message to the examinee without being discouraging, and this reasoning is also applied to the other components of the scoresheet unless there are no comments recorded.

Scoring Accuracy Rubric

Variance from Consensus	Points/Beer
0	20
1.5	19
2.5	18
3.5	17
4.5	16
5.5	15
6	14
6.5	13
7	12
8	11
9	10
10	9

Perception

Perception is arguably the most difficult competency to evaluate because it requires condensing data from several sources into a concise representation of the characteristics of each exam beer. These data sources include not only the proctors’ scoresheets, but the scoresheets completed by the exam participants, the exam beer descriptions supplied by the exam administrator, and the BJCP style guidelines.

Perception Ability

Each exam beer results in a unique set of information that the graders need to distill into a finite number of essential components in the Perception rubric. As with Scoring Accuracy, it is important for BJCP Beer Judge Exam participants and graders to recognize that this competency is not evaluated solely on the comments supplied by the exam proctors. BJCP Beer Judge Exam graders should also be careful to not create a lengthy list of every characteristic independently noted by each proctor with the unrealistic expectation that all of them should be mentioned by the examinee. Finally, judges and graders need to be cognizant that the same characteristic can be perceived differently by different tasters. For example, suppose one proctor perceives a grainy character and another proctor comments that the beer has husky notes. An examinee who mentions either descriptor or a closely related one such as grassiness should be credited with accurate perception for this characteristic. These flavors are neighbors on the Beer Flavor Wheel (versions of this sensory training tool are available at https://www.beerdredge.com/beer-wheel and http://morebeer.com/), so describing the same characteristic differently should not be regarded as a perception error.

Many perception errors on scoresheets from newer judges will simply be an error of omission. For example, an examinee may not comment on the level of hop bitterness in a style where that is a desired attribute, but it should not be assumed that this person cannot taste bitterness. But when grading the scoresheet, it should count as both a perception error and should also results in a deduction for providing an incomplete description. This situation often occurs when the examinee is not well-versed in the style being judged. A process that works for many graders for evaluating perception skills is as follows:

1. Review the BJCP Style Guidelines for the beers served during the exam.
2. Review the information provided by the exam administrator on the background of the exam beers and make note of any property that might impact the judging. For example, was it a classic example of the style, was it doctored, or was it perhaps an aged sample?
3. Begin with the scoresheet of the more experienced proctor and write down the key descriptors that were noted for the malt, hop and other components of the aroma. Try to capture three or primary and secondary characteristics that are at least low to moderate in intensity as well as anything that is out of place or missing in the style being judged.
4. Write down key descriptors that were noted for the appearance, particularly those which may not be appropriate in the style, such as haze or being too dark or light in color.
5. Write down the key descriptors that were noted for the malt, hop character, balance and other components of the flavor. Again, try to capture three or four primary and secondary characteristics that are at least low to moderate in intensity as well as anything that is out of place or missing in the style being judged.
6. Write down any key descriptors that were noted for the mouthfeel.
7. Read through the scoresheets of the other proctor(s) and circle any descriptors which are already captured on the list you created from the scoresheet from the lead proctor. Add any new descriptors that capture the perception of the other proctors(s).
8. Narrow down the list of descriptors from the proctors to a subset of six to eight consensus characteristics that capture the essence of the beer. These are the primary and secondary characteristics that are expected to be noted on exemplary participant scoresheets. These descriptors can be incorporated into an NxM matrix for each exam beer, where N is the number of key descriptors and M is the number of examinees.
9. Print out a hard copy of the matrix after you have defined the list of key characteristics for each exam beer. An example of this matrix approach.
10. As you review the participant scoresheets, mark the descriptors on the matrix that were correctly identified by the examinee with solid circles ● and mark any omitted descriptors with an X. If any characteristic is particularly important (e.g. hop character in an American IPA) or inappropriate (e.g. diacetyl in a German lager), it should be given double weight with two X or two ● symbols in the appropriate cell of the matrix. These symbols will give you a visual indication of the perception skills of each examinee for each exam beer.
11. If the examinee noted a characteristic (such as astringency) that was not among the descriptors used by the proctors, the graders should make a comment in one of the blank grids at the bottom of the perception matrix for that examinee. This could be due to a perception error (a false positive), but it could also be incorporated into an updated rubric for that beer if the graders observe that a significant fraction of the other examinees noted that same characteristic. In this case, it would be appropriate to add astringency to the perception matrix and give credit to the examinees who noted it on their scoresheets, but without making a deduction on the scoresheets of the judges who were silent on that characteristic.

In general, judges should be striving to achieve the following perception competencies which are evident in most scoresheets from Master and Grand Master judges:

1. The primary components of the aroma described on the examinee’s scoresheet are consistent with the consensus of the proctors and/or over one-half of the examinees.
2. There are no significant aroma perception errors, i.e. there are no aroma components perceived at low or higher levels that are not included in the consensus of the proctors and/or over one-half of the examinees.
3. The descriptions of the color, clarity and head are consistent with the consensus of the proctors and/or over one-half of the examinees. It is also desirable to describe other characteristics of the head, but this is not essential information for the brewer.
4. The primary components of the flavor described on the scoresheet are consistent with the consensus of the proctors and/or over one-half of the examinees.
5. There are no significant flavor perception errors, i.e. there are no flavor components perceived at low or higher levels that are not included in the consensus of the proctors and/or over one-half of the examinees.
6. The primary components of the mouthfeel described on the scoresheet are consistent with the consensus of the proctors and/or over one-half of the examinees.

Perception Rubric

1. Master (18-20): No more than two elements of the aroma, appearance, flavor and mouthfeel differ from the consensus of the proctors and/or over one-half of the examinees.
2. National (16-17): No more than three elements of the aroma, appearance, flavor and mouthfeel differ from the consensus of the proctors and/or over one-half of the examinees.
3. Certified (14-15): No more than four elements of the aroma, appearance, flavor and mouthfeel differ from the consensus of the proctors and/or over one-half of the examinees.
4. Recognized (12-13): No more than five elements of the aroma, appearance, flavor and mouthfeel differ from the consensus of the proctors and/or over one-half of the examinees.
5. Apprentice (10-11): Five or more elements of the aroma, appearance, flavor and mouthfeel differ from the consensus of the proctors and/or over one-half of the examinees. The judge did however make an effort to judge the beer.
6. Minimum (8-9): There are minimal comments on the scoresheet, but not enough information is provided to access perception skills.

Descriptive Ability

Descriptive Ability is a measure of the competency to use adjectives and phrases to identify the aroma, flavor and other characteristics of the beer. Beer judges should be able to describe “how much” as well as “what kind of” aroma and flavor components are present. Scoresheets from Master and Grand Master judges typically have a proliferation of descriptive information beyond just “malty” or “hoppy,” and this is important because the map between the perceived characteristics and the BJCP guidelines for any style in part determines the stylistic accuracy and technical merits of the beer. As an example of adding layers or descriptive information, consider an American Pale Ale which an Apprentice judge might describe as having a “hoppy aroma.” A Recognized judge might add a descriptive adjective, stating that it has an “American hop aroma,” while a Certified judge may also note the intensity and write that it has a “moderate American hop aroma.” National judges typically add an additional layer of descriptive information, such as “moderate citrusy American hop aroma,” while a Master judges could take that a step further and write “moderate citrusy American hop aroma, with notes of tangerine and grapefruit.” The key in grading exams is to reward examinees for using descriptive terminology but to not set the bar too high for less experienced judges who are still learning beer vocabulary and how to translate the beer characteristics they perceive into written descriptions on the scoresheet.

Descriptive Ability Competencies

1. Master level: At least five descriptive adjectives[1] or phrases[2] are used to describe the aroma of the beer.
2. Master: At least three descriptive adjectives or phrases are used to describe the intensity of the aromatic components of the beer.
3. Master level: At least three descriptive adjectives or phrases are used to describe the appearance of the beer.
4. Master level: At least five descriptive adjectives or phrases are used to describe the flavor of the beer.
5. Master level: At least three descriptive adjectives or phrases are used to describe the intensity of the flavor components of the beer.
6. Master level: At least three descriptive adjectives or phrases are used to describe the mouthfeel of the beer.

[1] In this context, descriptive means adjectives more specific than generic words such as “malty,” “hoppy,” and “estery.” Note that “No” or “None” are valid adjectives for characteristics which may be appropriate in the beer style being judged but are not present in that particular sample.

[2] In this context, the comment, “A medium spicy noble hop flavor emerges mid-palate,” has two descriptive adjectives for the hop flavor, one for the intensity and one descriptive phrase.

Descriptive Ability Rubric

1. There are a total of twenty one (21) opportunities to use descriptive adjectives or phrases, and the targets for Master/National/Certified/Recognized judge (based on 90/80/70/60 percent of that total) are: Total number of adjectives and phrases:

• Master (18-20 points): 19 or more descriptive adjectives and phrases
• National (16-17 points) = 17-18
• Certified (14-15 points) =15-16
• Recognized (12-13 points) =12-14
• Apprentice (10-11 points) =9-11
• Minimum (8-9): Fewer than 9

Feedback

The BJCP Exam Scoring Guide states that “the brewer should receive useful and constructive feedback explaining how to adjust the recipe or brewing procedure in order to produce a beer that is closer to style. The comments should be constructive and consistent with the characteristics perceived by the examinee as well as with the score assigned to the beer.” This is a great synopsis on the purpose and desired elements of feedback on a beer scoresheet. Note that feedback is not restricted to the Overall Impression section, so careful reading of the entire scoresheet is required to accurately assess this competency. For example, stating in the Flavor section that the balance is appropriate for the style provides feedback to the examinee. The feedback should be specific in terms of how the process or recipe should be corrected rather than just pointing out that something needs to be addressed. Feedback also includes calling out when characteristics are permissible for the style, such as “low levels of yeast character” in an International Pale Lager.

Feedback Competencies

1. (2 points). The examinee provides component and total scores for the beer. This is one form of feedback since the brewer will be able to match the assigned score with the appropriate range in the chart on the scoresheet.
2. (1 point) The feedback is constructive, polite and professional, generally including at least one supportive or positive comment about the beer.
3. (1 point) The feedback is consistent with the score assigned to the beer.
4. (2 points) The judge attempts to provide an accurate diagnosis of any stylistic flaws which impacted the perceived beer quality and/or score.
5. (2 points) The judge attempts to provide an accurate diagnosis of any technical flaws which impacted the perceived beer quality and/or score.
6. (2 points) The feedback given to the brewer is accurate with respect to the characteristics perceived by the judge. This is independent of whether the perception was accurate.
7. (1 point) The feedback does not make any assumptions about the process or ingredients.
8. (1 point) The feedback given in the Overall Impression section is consistent with comments in other sections of the scoresheet.
9. (2 points) Observations of characteristics of the aroma, appearance, flavor and mouthfeel are noted in the appropriate section of the scoresheet, with feedback either given in that section or as part of the Overall Impression. For example, astringency should be discussed in the Mouthfeel section, not Flavor.

Feedback Rubric

1. There are a total of 14 points which may be earned by providing complete and accurate feedback. The targets for Master/National/Certified/Recognized judges are:

• Master (18-20 points): 13 or more feedback elements
• National (16-17 points): 11-12
• Certified (14-15 points): 9-10
• Recognized (12-13): 7-8
• Apprentice (10-11 points): Fewer than 7, but an effort was made to provide feedback
• Minimum (8-9): There are minimal comments on the scoresheet, but not enough information is provided to access feedback skills

Completeness and Communication

Completeness and Communication are measures of the ability of a judge to produce a high level of content in the scoresheet while clearly and effectively communicating information about the beer to the brewer. This is, in theory, the easiest category in which to achieve a high score on the Beer Judging Exam, but it does require some attention to detail. A complete master-level scoresheet not only requires addressing the components of the aroma, appearance, flavor and mouthfeel that are appropriate in a given beer style, but includes book-keeping details such as identifying the beer style, the beer or entry number, the participant number or name of the judge, filling in the scores for the beer, and marking the appropriate checkboxes at the bottom and left side of the scoresheet.

Completeness and Communication Competencies

1. (2 points) All applicable components[1] of the aroma listed on the scoresheet are addressed.
2. (2 points) All applicable components of the appearance listed on the scoresheet are addressed.
3. (2 points) All applicable components of the flavor listed on the scoresheet are addressed.
4. (2 points) All applicable components of the mouthfeel listed on the scoresheet are addressed.
5. (2 points) The Overall Impression section includes a comment on overall drinking pleasure associated with entry (1 point) and if the total score is less than 45, offers at least one suggestion for improvement (1 point).
6. (4 points) Efficient use of vertical space: For perfect score, fewer than two blank lines remain on the completed scoresheet (these are typically only in the Appearance or Mouthfeel sections on a Master level scoresheet). Deduct 0.5 point for every blank line beyond 2, up to a maximum of 4 points deducted. For example, 5 blank lines would be a (5-2)/2 = 1.5 point deduction.
7. (2 points) Numerical values are assigned for all component scores (1 point) and also for the total score (1 point).
8. (1 point) The stylistic accuracy, technical merit, and intangibles boxes are checked. There is no partial credit here.
9. (1 point) Descriptor definitions are checked when applicable (characteristics are either perceived at moderate or higher levels or are flaws in the style being judged). Partial credit may be awarded.
10. (1 point) Comments are well organized and legible.
11. (1 point) There is efficient use of horizontal space: The scoresheet typically has six to seven words per line with a font size and spacing that balances content and legibility. The objective here is to discourage judges from writing in an extremely large font to fill up the space on the scoresheet without conveying much information.

[1] In this context, the phrase “applicable components” means characteristics which are expected in the beer style being judged according to the BJCP Style Guidelines and/or to perceived characteristics which would be regarded as flaws in that particular style.

Completeness and Communication Rubric

1. There are a total of twenty points available on a complete scoresheet. The targets for Master/National/Certified/Recognized judge (approximately 90/80/70/60 percent are:    

• Master (18-20 points): 18 or more completeness competencies
• National (16-17 points) =16-17
• Certified (14-15 points) =14-15
• Recognized (12-13 points)=12-13
• Apprentice (10-11 points): Fewer than 12, but an effort was made to judge the beer
• Minimum (8-9): There are comments on the scoresheet, but an unacceptably low level of effort was made to judge the beer

Summary

This BJCP Scoresheet Guide is intended to provide guidance for the creation of high quality beer scoresheets exemplified by those produced by most Master and Grand Master judges. As with other BJCP documents, it is not intended to be a stand-alone document, but should supplement other material on the BJCP website related to beer styles, beer judging and exam grading. If you have any comments or suggestions, please communicate them to the BJCP Exam Directors at [email protected].