Return to Writings index

What Else? The Endless List of Flaws in the IJS
by Sonia Bianchetti
February 2008

The results of the men's event at the 2008 US Nationals generated a lot of discussion on the internet and the media. The topic was: Who really won the men's title? Was the men's title decided by a faulty interpretation of the calculation method?

More importantly, is the software of ICECALC—the software studied and imposed by the International Skating Union to calculate the results in all championships and international competitions—the best one from a mathematical point of view? According to George Rossano, Jack Curtis, Ed Russell and a few other mathematicians that I contacted, it would not seem so.

Rossano is of the opinion that "intermediate rounding steps used in the ISU calculation method, is incorrect and indicates a basic lack of understanding of significant digits and how they affect the precision of the calculation for final results" and that "ISU rule 353 should be revised to require an exact calculation to at least three decimal places, with rounding of the total score only as the last step of the method" (www.iceskatingintnl.com).

Jack Curtis wrote: "Good math practice does not round off during calculations, doing so only at the end number after all intermediate steps are accomplished using all the places. That principle is violated in the ISU software and in the Canadian proposal. It should be evident that the more often you round off during intermediate steps of calculation, the less accurate the final result will be. In IJS at present, this can sometimes change the outcome placements, which occurred at US Nationals in the men's event."

According to Russell: "It appears that IceCalc is the standard and the rules don't matter. The rules are clear: if you are NOT instructed to round, you SHOULD NOT ROUND. It's not an ambiguity of instructions. I'm not instructed to put water in my gas tank, being intelligent, I don't. IceCalc put water in their gas tank—they should have to FIX the program."

Of course, whatever the outcome of this analysis may be, it will not affect the results of the U.S. Nationals because the title to Evan Lysacek was correctly awarded based on the existing tie-break rules, good or bad as they may be. Still, the issue is very intriguing and perhaps some clarification should be provided by the ISU experts themselves!

Mathematics—to my knowledge—is recognised as being an "exact science." There cannot be different opinions on which is the system that provides the fairest results for the competitors. If even math is questionable in the IJS, what is left? This controversy only added one more question mark about the validity of the IJS.

The new judging system introduced in 2004—as is well known—was presented as "the solution to all figure skating judging problems." Its main purpose was to make judging more fair and objective. Whether it reached its goal is still to be proven. What is sure is that it made the judging incomprehensible and inconsistent while killing the artistry, the creativity and the popularity of the sport. And this is not a personal opinion, it is supported by facts!

After the 2002 Olympic scandal in which the French judge, Marie Reine Le Gougne, said she fixed the results in the pairs competition in a deal with the Russians, the International Skating Union leaders decided to remedy the situation by giving the judges anonymity. It cannot be just a coincidence that TV ratings have gone down hill ever since. The sport's new judging system is perhaps fairer to the skaters—at least as far as the technical score is concerned—but remains indecipherable to the viewing public. Whether there's a cause-and-effect here is only a guess, although it wouldn't be a bad idea for the ISU leadership to at least consider that possibility before the last figure skating fan turns off the TV and leaves the room.

All over the world, the arenas where the most prestigious events are held are dramatically empty, and—even worse—today TV money is disappearing. In 1999, ABC signed a five-year contract with the International Skating Union for $22 million a year. Today, they can't find a U.S. network willing to come close to the current $5 million a year contract with ESPN which expires after the 2008 Worlds. The International Skating Union's search for a new contract has so far been unsuccessful.

Ottavio Cinquanta—the ISU President—must really be desperate if he told the Chicago Tribune last month he would give away the rights if necessary to assure a U.S. telecast of the 2009 World Championships in Los Angeles! What will happen from next year? How will they face all the expenses, subsidize the ISU Championships both in figure and speed skating and balance their budget? Without income, the Union very likely will have to dip into its reserve money. How long will it last?

In the United States the skating federation is faced with the same problem. Once one of the most popular sports for American audiences, skating ratings have been sinking from double figures into the 6.0 range. After making $12 million a year with its old TV deal, the USFSA is now receiving absolutely no rights fee with its new TV deal, and State Farm— their main sponsor during the past 12 years—decided not to renew its contract.

Many are the reasons which caused this decline in the fans' interest. Surely one is that the sport has become boring, totally uninteresting, with all the programs looking alike. Johnny Weir—in a recent interview—said: "I like to make an impression, I like to make art. With the new judging system, it's taking away the individuality of the sport. At the moment, it's looking a little bit like tennis, just hitting back and forth and nothing changes. That's a shame to me." Exactly the same feelings were expressed by Stephane Lambiel and Evan Lysacek after the Grand Prix Final in Torino.

Technique in our sport is essential. Without technique there is no way a skater can express himself, can make "art", but if one has nothing to say or is not allowed to express his "art", then even the best technique is worthless. As Philip Hersh wrote in an article published in The Los Angeles Times—February 3 issue, "Figure skating sadly has become a mind-numbing mathematical exercise, where artistry is only paint by numbers."

I have addressed this matter many times and I do not intend to do it once more here. What I will do is to deal with the problems related to judging itself.

If after four years of intense training through judges' seminars, judging is still considered "unsatisfactory", perhaps the blame should not be thrown just on the judges. I cannot believe that several hundred judges, all over the world, have—all of a sudden—become incompetent, ignorant OR STUPID! I do sympathize with the judges who are blamed for their unsatisfactory judging because it is not their fault. It is the fault of the system.

Besides the mathematics of the calculating system, the ISU should consider whether secret judging, the random draw of the judges or the replacement of "relative judging" with "absolute judging" were the best ideas. It remains to be proven whether the judging system has produced more objective and reproducible results in a competition and if it really prevents collusion or other forms of dishonest judging, which was its initial goal.

Secret judging

One of the greatest harms done by the ISU to their sport was adopting an anonymous judging system as a response to the 2002 Salt Lake City Olympic judging scandal, and eliminating the universally recognised trade mark of the sport: the 5.8's, 5.9's and 6.0's.

In a recent interview quoted in the January 24 issue of USA TODAY, Brian Boitano said: "The 6.0 needs to come back. People have to be able to root, they have to be able to understand the scoring system. Round up, add it up, multiply by something, do whatever it takes—but it all needs to get down to a number people know, either a 6.0 or a 10."

The sport also must once again allow to identify the judges with their scores. Audrey Weisiger, a top U.S. coach, added, "Transparency is essential, but the audience also needs to feel that it is participating, too. You used to be able to boo the judge from Bulgaria. Now the audience feels cheated. You can't have a good-guy, bad-guy feeling anymore. I kind of miss it myself."

Every single judge could be cheating now and the public will never know. This is unfair or an "insult" to the skaters and to the public, as Dick Pound—the then Vice President of the IOC—declared, but it is also unfair to the honourable judges who are lumped together with those who cheat. As a matter of fact, secret judging is perceived as a way to hide intrigues or deals among the judges and is detrimental to the credibility of the sport. Judges must be held accountable not only by the ISU Council or Technical Committees but also to the athletes for their results.

And what about ethics? "Transparency" is considered fundamental to the ethical functioning of any institution. All decisions must be open and subject to public scrutiny. The anonymity of the new judging system is an evident violation of this principle.

Besides, the practice of penalizing a judge whose marks diverge too widely from those of other judges is just outrageous! There is a well documented tendency for people to try to bring their views, opinions, decisions in line with what they feel is expected of them. Not only does the ISU not minimize the tendency to self-censorship, they actually maximize it through the practice of penalizing judges for failing to go along with the crowd. This practice is widely recognized by professional ethicists as one of the main sources of unethical behavior.

The random draw of the judges

Studies have proved that there is a wide spread of marks and consequent placements among the judges, even among the top five competitors. In each event, depending on which judges have been selected, the result could easily vary from first to fifth. For instance, it's quite likely that in close competitions there may be two or more different potential winners depending on which judges are dropped at random. Is it right that the winner should be determined by a coin toss? That is not sport, it's gambling and is most unfair to the skaters. It reduces the whole results system to farce.

The random draw is another flaw of the system, especially when it does not guarantee a fairer result but is only used to make secret judging even more secret! The only way to compensate for that is by using the marks of all the judges on the panels, deleting the highest and the lowest.

Absolute judging

At the heart of the new judging system there was a fundamental change in the method of evaluating skating performances. The former ordinal method of scoring was based on the recognition that humans can make relative judgements with greater precision than absolute judgements.

Under the 6.0 system the judges evaluated the performances of the skaters by comparing one with another, using the so-called "relative marking." This skater is better, the mark must go up. How much? The only way to be consistent through the whole event is to be thinking through the whole event whether the marks given now make sense compared to the marks given before—and that is a comparison.

Under the IJS the judges are now asked to evaluate performances on an absolute point scale without comparison to any other performance. It is questionable if this point base approach to scoring can ever work reliably!

If you are expecting judges to focus on evaluating individual elements of a program, you cannot expect them to evaluate the composition and presentation of that program. These are entirely separate ways of thinking. It is another flaw of the system.

A great deal of research has been conducted into the marking skills of human beings in general. It has been found that when marking on an absolute standard, over a period of time, marking will drift quite considerably. Absolute marking, in general—but even more so in a sport like figure skating—is considered inappropriate. It is practically impossible to quantify objectively the quality of any element of a skater's performance. Marking by comparison tends to be more stable.

This is even more true with the Program Components scores where the nebulous nature of their terminology allows for a wide range of interpretation from the judges. What the judges do not have is a procedure to insure that a consistent standard is applied throughout an event. What we used to say and to hear in every judges' school and at every competition—"Don't forget the first warm-up group!"—has now been replaced by "Judge in the moment!" If a person is asked to judge in the moment, and no guidance is included to insure consistency of judgment, obviously the judging will suffer.

Each of the program components scores has a range of possible values from zero to ten. To decide which mark to assign to each of these elements one should know exactly what is to be considered as "perfect" deserving ten! Does anybody know for instance how "Performance/Execution" or "Skating Skills" should look to deserve 10? Has anyone ever seen a precise and detailed definition of these or other elements?

Or, as George Rossano points out in his excellent article "Scoring Drift," in ice dancing what does "Pattern and Ice Coverage" mean? There are no explicit tables of examples, nor pictures or diagrams for what constitutes good or bad ice coverage. There are no tables of values to say what to do with your marks. The judges have very little definite guidance for what marks to give, and are forbidden from comparing the marks they gave at the beginning of an event or to a previous skater. How possibly can a judge assess a correct mark? And on which basis can a certain mark be considered right or wrong?

The purpose of a figure skating competition is to determine which skater gave the best performance on a given day. How can one accurately rank skaters without seeing what he gave a previous skater? The way our minds are, we can't tell if something is 2 inches long or 3 inches long, but we can compare two things and say which one is longer. So "absolute" judging makes no sense, especially in program components, both because it's impossible for the human mind to know whether something is worth 4 or a 5, and because a competition is supposed to compare things—and we should take advantage of that.

Without the help of explicit guidelines and a self consistent process it is no wonder the Program Components marks increase significantly during an event and reflect more the reputation of the competitor than the actual skating.

We can expect there will be additional problems in judging if the next Ordinary Congress approves proposals submitted by the ISU Sport Directorate concerning a new way of determining the starting order, both in free skating and the short program, in all Championships.

For the short program it is proposed that the competitors be divided in two groups according to their ranking. The competitors in the group "skating earlier" will then be divided into two parts: "competitors with and without ranking, so that competitors with ranking will draw for the later starting numbers." The reason for this reads: "To avoid the situation where competitors without ranking skate before competitors with ranking."

Why such discrimination?

And for free skating: "Each group shall skate in the reverse order of the previous segment with the best placed competitor/pair skating last."

This sounds like an invitation to a wedding party for the judges! They cannot help but be influenced by the order in which the competitors skated. How badly will this affect the final results?

The "personal best performance" scores that are so highly publicized are laughable when you look back at tapes of far better performances by those skaters. Is this due to rule changes? Marking drift? Different judges and specialists? The inconsistency in the judging makes the new judging system a farce as well as incomprehensible.

The bottom line is that the judges have no ability to consistently score the skaters because the system puts insurmountable obstacles in their way and makes it impossible for them to do a good job. The judges may be able to accurately tell that a certain skater should be ranked in a certain way, but their ability to reflect that knowledge in their marks is completely taken away from them. That is probably why at the Olympics and the Worlds last year the ordinals were all over the place.