Categorizing Variants of Goodhart's Law

the to-do shelf, but it seems like Shirky principle is mostly about institutions and the nature of bureaucracies. Now, a good question would be - when and why Shirky principle does not apply to the institutions. My first thought is the difference between bureaucracies that have "owner" and ones who lost it. When a bureaucracy has an owner (person or group of people) who can change the bureaucracy in response to external events, then it's probably unlikely to have Shirky effect (WHO and smallpox example in comments). However, bureaucracies that have lost their owner will most likely have this effect. The best thing I read on this subject (how bureaucracies work) is Samo Burja's "Great Founder Theory" [3]. If anyone can suggest something along these lines on the fundamental principles of bureaucracies, I would appreciate a lot. [1] https://arxiv.org/abs/1803.04585 [2] https://www.sciencedirect.com/science/article/pii/S266638992... [3] https://samoburja.com/gft/

The computer averages that and takes trimmed mean to produce the final points for the element. Those base values and points are logically assigned by the complexity of elements. More complex elements cost more. Great skaters can perform more complex elements. Sounds logical, right? What has happened, though, is that using these points as a target has changed the way how skaters train. Why spend time learning "easy" elements that give little points? All time must be spent now learning "complex" elements that give more points. In a way, before, IJS skaters were chasing overall greatness; now, they chase points. "Skater's greatness" – whatever that means – is hard to define, let alone measure. The ability to perform complex jumps was a consequence of it, not a cause. Using points as a metric changes the whole system of training, and a new generation of skaters can perform insane quads (often because of pre-puberty body sizes, though), but no one would call them great skaters. I'm not even starting with how these points affected artistry. "Program components" (PCS) that should measure artistry are practically a joke now. You basically can multiply jump scores by some constant and get PCS. [1] All skating programs look the same now and are very predictable because of points maximization. Back to AI. The rules for figure skating become more and more complex each year in order to duct-tape the constantly emerging issues. Many judges admit that complexity is so high that the human brain can't even possibly apply them in real time and has to resort to cognitive shortcuts. There are some early flirtations with using AI for judging figure skating [2], but nothing serious yet. As outlined in the "Categorized Variants of Goodhart's Law" [3], one of the major reasons of this effect in sports is not just the fact that elements != greatness, but the fact that there is no shared definition of what is a True Goal of this sport. What exactly judging system try to measure? What would constitute the best skater? Where there is no explicitly stated True Goal, any proxy variable used as metrics (like "ability to perform complex elements") will be far from good and will be distorting the system even further. Eventually, this boils down to the structure and decision-making in the international federation that governs this sport. And these things cannot be solved by AI. In a way, using AI is a yak-shaving for ever-increasing rules of an inherently wrong judging system. [1] http://libjournals.unca.edu/OJS/index.php/mas/article/view/2... [2] https://www.youtube.com/watch?v=pq_mv1eyZyA [3] https://arxiv.org/abs/1803.04585

"prepare for exams" is a testament to how failed the system is. Sport is similar. The initial idea of the judging system and competitions is to "measure" the performance of the athlete. Rewards and rules are introduced to make this metric clear and unambiguous. Fast-forward a couple of decades, and athletes and coaches don't care about overall performance – they care about winning the medal, so they focus only on the subset of a skill that gives points. This problem is especially horrifying in the sports like figure skating after introducing the IJS system in 2006. Correlation between "true goal" (skating skills) and "metrics" (points given for specific elements) is collapsed – people chase points, not the skill. It's not a simple problem. There are few neat papers on how to approach solving Goodheart's problems ([1] [2]). But on a practical level, the number one tool seems to be to decrease the importance of the metric. Some sports, for example, have "one major competition" once in 4 years (Olympic Games). The price of winning it is insane. Who cares about the true goal if it's almost life-and-death importance? You chase the points. But other sports have a more healthy approach – tennis, for example, doesn't care much about the Olympics. They have Slams and different surfaces competitions throughout the year. You lost a game - not the end of the world. There will be many more. The higher prize is to win all four Slams in your career, so the price of the metric at a single competition is not that huge. And this collapsing metric is a reason why people cheat. You can't cheat on a true goal, only on a proxy metric. [1] Categorizing Variants of Goodhart's Law - https://arxiv.org/abs/1803.04585 [2] Building Less Flawed Metrics: Dodging Goodhart and Campbell's Laws - https://www.researchgate.net/publication/334478956_Building_...