NATAS Transparency Report: Comprehensive Insight Towards Equality and Fairness
NATAS
The National Academy of Television Arts and Sciences (NATAS) released its 2020-2021 transparency report on March 3, 2022. The goal of this report is to provide greater insight into and context around the actions taken by the NATAS and the National Awards Committee. This insight is provided in the hopes of adjudicating fair competitions. The report consolidated data from two years-worth of Emmy Awards competitions. This data provides a comprehensive look at the calculations, discussions, and decisions that led to the award ceremonies. NATAS’s 2020-2021 transparency report also includes a demographic survey for the first time. This survey was conducted in 2021 to better understand the judge’s panel’s representation amongst the diverse community of creators. More details about this survey will be discussed near the end.
The judging process to determine winners of NATAS’s competitions follows a strict criterion of factors. These factors include the scoring scale, nominee count, minimum viability for nomination, tiebreaking, natural cutoffs, judge impropriety, and anonymity. NATAS scored submissions on a scale of 1-10 but has since changed it to 1-7 since 2021, with 10 and 7 being the best scores respectively. Each category has a default set of one winner and five nominees. However, this set can be reduced or increased in proportion to the number of entries when compared to the competition average.
NATAS has a standard to determine a submission’s viability to become a nominee. Currently, entries must have an average score of 4.0 on the 1-7 scale to be qualified. Exceptions can occur for submissions with average scores between 3.0 and 4.0 at the discretion of the National Awards Committee. In the case of a tie, there is a comparison of the highest scores of each entry. If a tie remains, multiple winners may be announced. There are even rare instances of natural cutoffs occurring when a significant distinction appears in organized scores.
Lastly, there are the factors that are affected by the judges. Anonymity is a rule amongst the judges and auditors. All votes are conducted entirely on the down-low and then provided to the competition auditor. No National Awards Committee member is provided info on the submissions, judges, or any determining individuals. Each of the judges’ scores and behavioral patterns is monitored through an algorithmic system designed to spot possible outliers that may indicate improper behavior. If ballots are identified to be improper, they are investigated by the competition auditor and possibly brought to the attention of the National Awards Committee.
Now that the basis of NATAS scoring and tracking has been explained, the discussion shall move towards the three competitions under NATAS: the Daytime Emmy Awards, News & Documentary Emmy Awards, and the Sports Emmy Awards. The analysis will be over the 2020 and 2021 ceremonies for each competition. The first competition is the Daytime Emmy Awards. For the 47th Annual Daytime Emmy Awards in 2020, the ceremony recognized a single winner for ninety-five categories, and two winners in three categories. The ceremony’s special Individual Achievement in Animation category resulted in six winners. There was a total of one hundred seven awards for ninety-nine categories.
The 48th Annual Daytime Emmy Awards in 2021 recognized a single winner in ninety-seven categories, and two winners in two categories. Like what happened in 2020, the Individual Achievement in Animation category also resulted in six winners for 2021. Altogether, there were a total of one hundred seven awards for one hundred categories.
Some judging irregularities did occur during both years. For 2020, ten of the one thousand-ten judges were irregular. Each of these ten judges was disqualified. Three potential judging irregularities were found in the Blue Ribbon Round ballot. As a result, the National Awards Committee nullified the pre-nomination round and issued a rejudging. The remaining judges were identified due to their erratic scoring patterns. The 48th Annual Daytime Emmy Awards also saw its fair share of judging irregularities. Learning from their errors in past competitions, NATAS created an additional, automated methodology system to assist in identifying judge impropriety. Unlike in 2020, not all the judges identified as outliers were disqualified in 2021. Of the nine identified, only five were removed.
Other matters of note for the Daytime Emmy Award competition were an announcement error and an alleged “vote-swapping scheme.” During the 47th Annual Daytime Emmy Awards, an error occurred in the announcement of nominees. Upon delivery of the nominee press release to the auditors, the accountant noticed three categories were missing an entry each. The missing nominees had been identified to NATAS by the accountant at the time of tabulation, but there was a technological error in the transfer of the complete nomination record from the submissions database. The correction of this error resulted in three additional nominees, with one of the additions becoming the winner of its respective category.
NATAS received a report alleging two programs, competing in separate categories, had entered a “vote-swapping” alliance before the 48th Annual Daytime Emmy Awards ceremony. NATAS investigated the allegations by conducting a test. Fifteen cast members of Program A were assigned as final-round judges for the performance categories Program B competed. Respectively, five members of Program B were assigned as judges for the performance categories of Program A. Both instances resulted in no returned ballots, nor were they marked improperly. NATAS was unable to find evidence that would suggest a coordinated “voter scheme,” and as a result, NATAS closed the matter.
Continuing the News & Documentary Emmy Awards competition, the 41st Annual News & Documentary Emmy Awards ceremony recognized a single winner in fifty-two categories and two winners in a single category. Twelve categories resulted in a nonstandard number of nominations because of an unbroken tie for the fifth nomination. The 2020 ceremony had a total of fifty-four awards and fifty-three categories. Among the two hundred seventy-seven nominations, ten were in two categories devoted to regional reporting. In 2021, the 42nd Annual News & Documentary Emmy Awards recognized a single winner in all sixty categories. Eighteen of the sixty categories resulted in a nonstandard number of nominations, with one being the result of an unbroken tie for the fifth nomination spot. In one category, an insufficient number of submissions registered under the newly implemented “Minimum Viability for Nomination” standard of five submissions. 2021 had sixty awards and sixty categories, with twelve of the three hundred nineteen nominations placed in two categories devoted to regional reporting.
There was a judging irregularity in the 42nd Annual News & Documentary Emmy Awards. Following NATAS’s implementation of an algorithmic monitoring system for outlier ballots in 2021, one of the competition’s nine hundred ninety-five judges was marked for review. The National Awards Committee determined, based on additional context from auditors, that the judge’s score pattern intended to skew the results. That ballot was disqualified.
The last competition reviewed under NATAS’s transparency report was the Sports Emmy Awards for 2020-2021. The 41st Annual Sports Emmy Awards recognized a single winner in each of its forty-two categories, with seven categories resulting in a nonstandard number of nominations. There were forty-two awards and forty-two categories for this ceremony. The 42nd Annual Sports Emmy Awards recognized a single winner in all forty-six categories, and like the 41st, seven categories had a nonstandard number of nominations. Similar to the 42nd Annual News & Documentary Emmy Awards, five judges were identified for review thanks to the algorithmic system used in 2021. These five judges also tried to skew the results and were disqualified.
The transparency report closes with the demographic survey conducted in 2021. This survey was conducted to create and track benchmarks for transparency in submissions, judging, dispute resolution, and adjudication. The survey highlights the diversity of judging pools and reveals areas where representation needs to reflect society and the related industries. Over seventeen hundred judges participated in the survey, with fourteen percent requesting anonymity. The results are far from ideal in the goal of equality and representation, but being public with these results emphasizes NATAS’s pledge to proactively pursue dialogue with entities that represent the various minority groups.