2018 Open - Complete Normalized Rankings

[spinny]

This is an alternate view of the 2018 Open data where the rankings of each workout (18.1 - 18.6) and overall ranking are re-calculated using a normalized system.



Athletes who failed to submit a score for one or more workouts are sorted to the bottom of the list (below athletes who scaled all the workouts), and the individual workout rankings are adjusted accordingly.

The individual workout rankings are calculated, the athlete with the highest overall points is assigned the highest overall rank, and his individual workout scores are removed. The individual workout rankings are then re-calculated considering only the remaining athletes to find the next highest overall points as the next highest overall rank. Repeat until reaching overall rank 1. The math is explained in more detail in this previous post.

This means your individual workout score can never be penalized by someone that has been determined to be less fit overall than you. Thus the normalized rank(JM) for a workout will only ever be the same or lower than the original rank(CF).



As an example in Central East, and to prove that I'm not just picking on heavy lifters,

Anthony Davis

ranked 1st in 18.2a (1RM Clean), and had decent scores except 18.4

(placing 4-23-1-63-163-11)

. After dropping less fit athletes and re-ranking workouts, he demonstrated better overall fitness than most of the athletes that beat him in 18.4 and his overall ranking goes from 25th to 7th in the region.



If your overall rank improves, it means that one or more of your scores was inflated by athletes who were determined to be overall less fit than you (or failed to submit a score).

If your overall rank worsens, it means you placed just high enough in an individual workout to avoid a huge points drop. Your individual rankings won't increase during normalization, but other athletes' rankings might have adjusted enough to move them up in the rankings ahead of you. Compare someone above your ranking who moved up (negative overall delta) and look at their scores on the right side of the sheet to see how their performance compared.



Another interesting side effect of this normalization is that it better shows athletes who mixed scaled/rx workouts. If you scaled a workout but blew it out of the water, you still rank below the athlete who performed 1 rx rep for that workout, but the points gap isn't quite as big.



THE SHEETS:

Men - Africa Middle East

Men - Asia

Men - Australasia

Men - Canada East

Men - Canada West

Men - Central America

Men - Central East

Men - Europe Central

Men - Europe North Men

Men - Europe South

Men - Mid Atlantic

Men - North Central

Men - North East

Men - South America

Men - South Central

Men - South East

Men - South West

Men - West Coast



Women - Africa Middle East

Women - Asia

Women - Australasia

Women - Canada East

Women - Canada West

Women - Central America

Women - Central East

Women - Europe Central

Women - Europe North

Women - Europe South

Women - Mid Atlantic

Women - North Central

Women - North East

Women - South America

Women - South Central

Women - South East

Women - South West

Women - West Coast

The columns are labeled for the original overall ranking (CF) and the normalized overall ranking (JM), plus individual workout rankings, overall points, scores of the workouts, count of DNF/scaled workouts, and even athlete id and affiliate id. Lots of great data!

Gray = qualifies by either scoring system

Green = qualifies by normalized system (but not by original)

Red = qualifies by original system (but not by normalized)

Blue = scaled at least one workout

Orange = DNF at least one workout

 

[rom1974]

@spinny When looking at the normalized numbers it becomes even more clear CF has a good ranking system, even with the 1RM outrage. Weaknesses are supposed to be eliminated. 75% good in all modes and all that. Its not like the 1rm "chunk of meat beasts" are qualifying, they just expose the bw rabbits weakness. The dudes with min-max of rank 20-40 absolutely should be ahead of the 1-1-287-134-9 dudes.

 

[olboodog]

@rom1974 Here's a good example using three people I know

99th place: 183rd (393 reps); 139th (5:04); 151 (310lbs); 39th (720 reps); 389th (116 reps); 115th (146 reps)

114th place: 327th (376 reps); 194th (5:18); 96th (320 lb); 180th (685 reps); 151st (135 reps); 197th (137 reps)

122nd place: 99th (406 reps); 41st (4:33); 727th (268 lb); 141st (688 reps); 114th (139 reps); 77th (151 reps)

With normalized scores, 99th becomes 105th place, 114th place becomes 167th, and 122nd becomes 86th.

You could calculate everyone's scores as a percentage of the world's best score and assign points that way too. There's nothing right or wrong with any scoring method, just different ways of ranking the data.

 

[spinny]

@olboodog Do you know the three well enough to know how they generally rank against each other in workouts? And if so, how would you perceive their overall fitness realtive to each other? Would it align better with the original system or the normalized system (or neither system)?

Like you said, there's no right or wrong answer, I'm just curious in the discussion.

 

[olboodog]

@spinny I'd say if you threw them into regionals or the games, 122nd would fair the best. As you can see by the 18.4 scores, he isn't weak compared to the other two as 18.2a might suggest. Maybe he should have slowed down to hit a higher clean (he actually made a mistake in one of the later rounds and would have been closer to 4:25 had he not done so, leaving me inclined to believe he'd have wiggle room to finish before the other two and not hurt the max lift as much). I'm more inclined to agree with the normalized scores.

 

[spinny]

@rom1974

The dudes with min-max of rank 20-40 absolutely should be ahead of the 1-1-287-134-9 dudes

Let's look at the guy who got

1-1-287-134-9

In Central East in 18.2a:

312 lb is 287 points

313 lb is 278-280 (because of tiebreakers)

315 lb is 223-277 (the next weight, there is no 314)

316 lb is 218-222

So there is already some question of should I really gain up to 54 points per lb when someone lifts heavier? If I lifted 312 but someone else lifted 316, would you really consider him to be significantly (70 points worth) fitter than me?

347 lb is 40 points

356 lb is 20 points

At higher weights, it's only 2 points per lb. So in reality, we can say that 40th place is fitter in this workout because he lifted 35 lbs more. But it's really hard to assign a points value. Should it be 70 points (2 pts x 35 lbs), or 1890 points (54 pts x 35 lbs), or somewhere in between (actually turns out to be 247 points)? Or maybe in a different system it could just be simplified to be 1 point per pound?



At some point it's like "Whose line is it anyway?" where the points are made up. If you lifted just enough to avoid a huge points drop, you might come out ok, otherwise it's very hard to make up a gigantic points deficit.



So then let's look at 18.2 in Central East:

3:54 is 1 point

4:16 is 20 points

4:28 is 40 points

In this case, if I beat 40th place by 34 seconds, it turns out to be 39 points (or about 1 point per second). People with much slower times would also suffer the same kind of penalty where every second is worth more and more points.



The story that the ranking numbers show considering those two workouts

1-287 vs 40-40

is way different than the actual scores. In reality, the first guy lifted 35 pounds lighter, but moved 34 seconds faster. To me that seems like a pretty even matchup.

 

[siyanda1978]

@spinny What’s the argument for this over a z-score? Seems that way you could better differentiate true outlier performances as well as not penalize as much for workouts where there’s a lot of clustering. Either way, your method IMO is better than the current methodology and hope you keep churning out the content through regionals.

 

[debzey01]

@spinny great job!

 

[blessing2012]

@spinny this is great. two questions:
1. did you do this in 2017 as well? It would be great to use this to check for progress
2. would it make any sense to use percentiles in addition to rankings? For those of us who are not top athletes, the rankings can be very misleading. It's very difficult to use rankings to judge progress bc it seems like more and more people are participating each year..

 

[spinny]

@blessing2012 I pulled all of the 2017 data, and should be able to post it by the end of next week.

It probably wouldn't be too hard to add a column for percentile. You could do that in the sheet if you download it to excel and do a little googling to set up the formula. I'll consider adding it as a feature to my calculation script.

 

[aaron2025]

@spinny So a negative delta is a "good" result according this sheet correct?

EDIT: Reading comprehension improved

 

[spinny]

@aaron2025 Correct. A negative delta means that your rankings were falsely inflated by people who were less fit overall than you. This can be beneficial for you if you had a bad score in one workout, but really good scores in all other workouts.

 

[christine98]

@spinny Any chance you can add age groups for people like me in Masters groupings?

Thanks, really enjoy seeing the algorithm used in the other thread. Great to see how it affected my gym. Overall, most people stayed in the same order, but their rankings increased. The few that missed a workout were heavily penalized and floated to the bottom of the group instead of appearing in the middle of the group.

For those unfamiliar with Google spreadsheets, you can filter the sheet like:

To filter for just your affiliate:

• Data->Filtered Views->Create New Temporary Filtered View
• Click the 3 bar triangle icon in "Affiliate" field
• Click "Clear"
• Enter "CrossFit XYZ" (select it when it appears)
• Done

I really think that this is how it should be done. Or at least bits of this technique (such as moving people who missed a workout to bottom, so they can still see their workout ranking, but their overall ranking would be penalized).

 

[dawn16]

@spinny holy shit...i moved up almost 100 places to 115 in mid-atlantic! sweet!

 

[spinny]

Here's a real example of someone who isn't even close to making regionals, but can show how the system works:

CF: (4574th) 1480-4172-3319-9089-11034-3393

JM: (3270th) 1226-2727-2302-3233-03251-2790

overallDelta: -1304

He did all 6 workouts, so after removing the rankings of people who failed to complete one or more workouts, he improves:

points due to DNF athletes: 248-691-555-1418-1495-2790

to go down to

newJM: 1232-3481-2764-7671-9539-3177

The remaining delta would be

deltaJM: 6-754-462-4438-8044-387

So there is a big difference in 18.3 and a huge difference in 18.4.

This means that there were people who posted a higher score in 18.4 (he completed 22 reps rx), but they were worse overall when considering all the other workouts.

For example, someone could have done 23 reps rx (or even 200 reps rx) 18.4 but scaled all of the other workouts. In the current system, you are penalized on that one workout, even though you are clearly more fit overall than that athlete because you rx'd every workout.

In the normalized system, if your overall points show that you have already demonstrated more fitness than another athlete, you are not penalized by his score. In this example, it means there were 8044 athletes who posted a higher score in 18.4, but were actually less fit overall.

 

[aphemix]

@spinny Is anyone else having trouble opening these?

 

[christine98]

@aphemix Works great for me.

 

[jamieslilwifeypoo]

@aphemix I am too. Can’t get any of them to open

 

[yahwehsninja]

@spinny This is very interesting. I can definitely agree with many of your points about how to reassess the rankings based on people who do not complete the Open. Do you have a link to the code/algorithm you use to normalize the results?

 

[272727]

@spinny Great work very interesting