Return to Writings index

Another Monster
Comments on ISU Communication 1327 — The OAC Commission
by Sonia Bianchetti
August 2005

The ISU has recently sent out Communication Nº 1327 - ISU Judging System, Evaluation of judging and technical content decisions.

In this communication, the ISU explains the rules in detail and gives the guidelines on the procedure to be followed in the "assessment" of the judges' activity, which differs from the procedure stated in the Regulations.

Reading the text, I must confess that I was impressed by the extraordinary ability of the ISU experts to make things as complicated and expensive as possible, with very uncertain results.

I will go through the text of the communication and make some comments based on my knowledge and experience in the sport, and particularly on the matter of the judges' "assessment".

A) Identification of the Officials' Assessment Commission (OAC) Pool
The OAC will be appointed by the ISU Council and shall consist of a pool of ISU Referees and Judges for Single and Pair Skating, Ice Dancing and Synchronized Skating "having been trained in the ISU judging system and having attended the Frankfurt Seminar in July 2005". They shall evaluate evident anomalies based on predetermined mathematical criteria confirmed by the Council and shall prepare a report for each competition including all identified cases of serious errors and/or bias by the judges. After each segment (e.g., Short program, Free skating, Compulsory Dance), the assigned OAC members on site will receive the necessary printouts that include the Grade of Execution (GOE) of every element and the points of the Program Components from all judges in a "random sequence without any reference to specific Judges' names".

Two questions come to mind:

• What kind of experience or competence can we expect from an ISU Judge who has perhaps judged a couple of events in the past season and has attended one seminar? Can the judges feel comfortable that they will be "judged" correctly?
• How can the OAC members understand if an "anomaly" is an error or national bias without knowing the name or the country of the concerned judge?

B) Appointment Procedure for ISU Championships and Senior Grand Prix of Figure Skating Events/Final
OAC Pool members must have the following skills relating to their OAC work:

1) a. ability to analyze competition data; ability to work quickly and in an organized fashion;
b. good written English;
c. be familiar with report writing;
d. demonstrate an ability to remain objective in all officiating evaluation matters;

My questions are: who is responsible for checking that the candidates really possess all these skills?
Will they be asked orally or in writing to confirm that they meet these requirements, and will this be considered satisfactory and enough by the ISU Council? But what I like best is that the candidate "must demonstrate an ability to remain objective in all officiating evaluation matters".
I would really appreciate if somebody in the ISU could explain to me how anyone can demonstrate this skill in advance. Will they ask the candidate if he can "remain objective in all officiating evaluation matters" and expect that, if he cannot, he will reply: "No sir, I am a crook"?

• around 20% of the Figure Skating Members do not have ice dance judges at all;
• more or less 50% of the Members having judges in both disciplines do not have any judge who can act both in figure skating and ice dancing (Great Britain, China, Estonia, Israel, France, Finland, Hungary, Italy, Sweden, and Ukraine, just to name some);
• the Judges who can judge both disciplines represent not more than 20% of the judges of each Member, and often they are not the best experts in either one.

The requirement that 2/3 of the OAC members must be qualified to judge both disciplines limits significantly the possibilities of choice and there are no guarantees that the referees or judges selected are really the most competent, knowledgeable, experienced and honest on the list, as it should be, even if this is more expensive.

C) Assignment of OAC members for ISU Junior Grand Prix of Figure Skating Series and World Challenge Cup for Juniors in Synchronized Skating

Yet for ISU Championships, the ISU states "it is recommended that as may be physically possible in each event, that the OAC panel is seated as close as possible to the same site line as the judging panel". By making this stipulation, they are admitting that the only place to sit and have any fair chance of making a correct evaluation of a performance is in the rink, at the same sight line as the judges—and strangely enough, not in their own homes. This inconsistency leaves me speechless!

This ridiculous idea is no doubt born out of a lack of knowledge plus a desire to save money. Therefore, perhaps the next money-saving idea to emanate from the ISU will be to have all the Grand Prix competitions and ISU Championships judged from a live TV link-up in the judges' homes!

D) Judges Evaluation at Other International Competitions

The Council has decided that there will be no individual assessment of Judges' marks at International Senior/Junior competitions other than the Grand Prix events. The Referee will file a formal report reviewing the judges' activity in regard to ethics, behaviour, attendance at Round Table Discussion, and other generalised reporting areas—but not judging! The reason behind this unbelievable decision is that in such competitions there will be no guarantee for the availability and consistency of technology as available at ISU events (e.g., video replay, DVD analysis, on-site OAC personnel); there will be no checks and balances evident to assess these judges' work.

I wonder if anybody in the ISU has considered the impact that such a decision will have on the whole judging system, aside from being an insult to the hundreds of skaters who will take part in these competitions. With this decision, the ISU has officially stated that they could not care less about how these events are judged! Based on my experience I can affirm that this is just another disgraceful decision of the ISU

These so-called "second class international competitions" have always been and are the lifesblood of the sport, not only for the skaters but also for the judges. Without them the sport would die. They are the grounds where the skaters and the judges grow and develop their experience. Nobody is born a champion or an experienced judge, but if we cut the roots out there will be no champions or judges in the future.

Unfortunately, not all ISU Members have a well-thought-out education program for their national judges; therefore, in some cases the technical committees had to trust the Members when they first nominated judges for international competitions as to their level of knowledge. In the former comprehensive evaluation system, the weaker judges were soon identified. But now, the whole educational system of the judges, which has been so successful so far, will collapse if there is no monitoring of the activity of the judges in these senior and junior international competitions. How will the ISU Technical Committees know which judges are good and competent enough to become ISU Championship judges and meet the requirements to pass the Judges' examination?

In the past, all International competition judges were assessed at every event, and had to act successfully at a stated number of international competitions over a certain period of time to be able to participate at the ISU Judges' examination, but now this well-thought-out educational and monitoring/assessing of the Judges will disappear.

At present, the ISU is still profiting from the hard work carried out over the last 50 years by the Technical Committees to instruct and train the judges to guarantee the best possible quality of judging to the competitors at the highest level. They can avail themselves now of a lot of good judges who were educated in the past. But in the future, without all this background work, where will the best judges come from?"

As with the New Judging System, it seems that nobody in the ISU leadership cares about the future of the sport. "After me, the deluge" seems to be Ottavio Cinquanta's motto.

The ISU states that since there will be 6 Grand Prix events and four ISU Championships reviewed by the OAC, the "activity of most judges in the season would 'enter the ISU radar' and come up for review in at least one of these events". I may be wrong, but I sincerely doubt that most of the judges who will officiate in the 29 international competitions (other than the Grand Prix events or the ISU Championships) organized in the next season will also judge at least one of the major events. Normally the judges selected by the Members for these prestigious competitions, with so much money involved, are the best experienced judges—not beginners! This is nothing new; it has always been like this. And, in any case, one event is not enough to decide on the competency of the judge.

E) Mathematical Criteria to Identify Potential Anomalies

The system is very simple—and very simple minded— at the same time. For each Judge the computer calculates the deviation in elements, which is an absolute value of the difference between entered GOE and the trimmed mean of the GOEs. The total deviation points in elements is the sum of the deviation of the individual elements.

A general "corridor of an acceptable number of deviation points" is given to each individual Judge. The "corridor" is based on the number of elements performed, e.g., in the Short Program there are 8 required elements to perform. Each judge may vary in his/her decision with one (1) step as an average per element, which are 8 steps in total for the Short Program (8.0 is the deviation points maximum). More than 8 is considered an anomaly and might get an "assessment".

For the five Program Components, the Judges' "corridor" will be 15% of the maximum total points of 50 points, which means 7,5 deviation points. If the Program Component scores of the OAC members is located outside the "corridor", the "corridor" can be enlarged accordingly. Of course, this proposed enlargement of the famous "corridor" still allows marks that are wrong, but in the original "corridor" before enlargement, to stand unchallenged. Unfortunately, over the past three years, the ISU has had the ill-informed philosophy that the majority, or "corridor", will always be correct. This system sets us back 50 years, when "to be in line" was the goal of all the judges and it was a tragedy for the skaters who were judged on reputation or on the results of the year before.

we witnessed the same situation at the major ISU events last year. The judges do not judge any more! They must return to the system when a judge that was "out of line"—now known as an "anomaly"—could possibly be the only correct judge, and the others who were "in line" or in the "corridor" could possibly be wrong—or the instances of poor judging we all witnessed last year will continue unabated.

To prove how wrong this philosophy is, I can quote a case that speaks for itself. In the past season, a well known American judge who is probably one of the most educated judges in the evaluation of the Program Components received an Assessment of being out of the "corridor" in his specialty! Knowing the person, by instinct, I would guess that he might well have been the only correct judge in the panel!

At the 2005 European Championships in Torino, from the top skaters to the very last one in the classification list, both in men and ladies, the GOE for a given element was more or less the same—0—all the way through. For the GOE points assigned by the judges to the triple Axel jumps in the men's event in Torino, for instance, except in case of a fall or a particularly severe error for which a deduction had to be applied, they were the same from Plushenko down. But this was not correct! The difference in the quality of the jumps was there as bright as the sun! And how is it possible that there were so few elements deserving pluses? Plushenko skated an outstanding program; nevertheless, the only +2 and +3 were awarded to his step sequence. What about all the other elements that were excellent as well? Are the judges really judging? What are they looking for to give a +1 or a +2?

Well, the explanation is simple: they are wisely and safely giving the same marks to everybody to make sure to be within the famous "corridor". On the other hand, this problem was raised and discussed during the last season. Many judges and officials were of the opinion that this assessment system is wrong and must be changed.

But the ISU is of a different opinion and Cinquanta has proudly announced in a press release that the ISU Council is pleased with the second season using the ISU Judging System. In particular, the ISU Council highlighted that the System has introduced a new level of accountability through the Officials Assessment process and: "Contrary to previous seasons, the evaluation process and Assessments were monitored as the season progressed rather than at the end of the season. Judges were informed of specific deviations which caused an Assessment to occur, this proved to be a key factor contributing to error prevention and education throughout the season. The number of Assessments declined steadily throughout the season. The vast majority of 'assessed' Officials were subject only to an Assessment 1 (1^st Warning), which showed the deterrent effect of the system".

To me, the "deterrent effect of the system" seems just the opposite of what Cinquanta claims. The Assessments declined steadily throughout the season after the judges were informed during the season of specific deviations which caused an Assessment to occur, not because the judges improved in their judging but because they understood what they have to watch for: stay within the "corridor"! And this is a tragedy for skating.

G) On "Assessments" the ISU Communication reads:

"If an 'Assessment' is deemed necessary by the OAC and the Technical Committee, the 'Assessment' will be communicated by the ISU Secretariat to the Officials and the Members concerned without delay", and: "In the case that an accumulation of Assessments (at present when reaching 'Assessment 4'), results in the demotion or suspension for the Officials concerned, they shall be notified by the ISU Secretariat of the possible outcome. The Official will have the right to ask within 5 days upon receipt of the notification for a meeting to be held as soon as possible, to give his/her explanation to the relevant "Assessment"—in front of at least 3 members of the respective Technical Committee.
Any travel, board and lodging or other expenses incurred by the Official concerned relating to the explanation meeting will be for the Official's account if all the 'Assessments', despite the explanations received, are confirmed by the respective Technical Committees after the meeting. The ISU shall only reimburse such expenses if at least one of the 'Assessments' would be revoked".

I quoted the whole text because, again, to me it is beyond belief.

a) The ISU decides that an "Assessment" is deemed necessary, and informs the concerned official of the reasons why it was given, but no explanation is asked of the concerned judge, who might be right in his deviation from the "corridor".

b) Only when the concerned Official reaches "Assessment 4" (which might occur after two or three years for an accumulation of "Assessments 1"), and is going to be suspended, is the judge generously offered by the ISU the possibility of being asked to furnish his explanation (within the time limit of 5 days!), not in writing, but in person by travelling from somewhere, maybe on the opposite side of the world at his own expense—not to meet a neutral committee, above the parties, but the same three Members of the Technical Committee which had decided to impose the sanction!!! I wonder how many judges will travel from Japan or Australia, the US or Canada, or even simply from Moscow to meet the same panel of people that condemned him, hoping to get relief! And what kind of explanation can a poor judge give, maybe three years later, of the reason why, in the short program of skater Smith he exceeded the maximum points deviation of 0.2?

Perhaps the ISU should ask the Wige - Data experts to invent another of their marvellous mini computers—to replace the brains of the judges while they are officiating. In this way, judging will be perfect and there will be no "deviations" from the corridor any longer. Then all judging problems will be really solved once and forever, and this idea, if taken on board, could be another "miracle" from the ISU President.

This article , as well as some interesting comments, can be seen on www.figure-skating-judging-scandal.com