What are vanity metrics?
What are metrics?
About 250 years ago Lord Kelvin said: “I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be.”
In today’s age of big data and data analytics, with its deluge of data, it becomes important to assign meaning to the data.
Metrics are quantitative values of commonly used measures for assessing, comparing, and tracking performance or progress. Often a group of metrics will be used to build a dashboard that management or analysts review regularly to maintain performance.
In this digital age, it’s become easier than ever for educators to track the various metrics of their students. However, the metrics commonly tracked by educators can often be non-factors in the actual success of the educational efforts, while the useful ones may go untracked.
Here is a very good example of a vanity metric :
CBSE, in collaboration with Intel, organised the ‘AI For Youth’ Virtual Symposium from October 13-17, 2020.
During the symposium a Guinness World RecordsTM attempt was made for ‘Most users to take an online Artificial Intelligence lesson in 24 hours’ on October 13 &14, 2020
Vanity metrics can be fickle, misleading, and appealing for all the wrong reasons. Vanity metrics are metrics that make you look good to others ( like being in the Guinness book of records) but do not help you understand your own performance in a way that informs future strategies.
Vanity metrics should be contrasted with actionable metrics which is data that helps you make decisions and helps your mission reach its goals.
How many views make a YouTube video a success? How about 1.5 million? That’s how many views a video an organization, DoSomething.org, posted in 2011 got. It featured some well-known YouTube celebrities, who asked young people to donate their used sports equipment to youth in need. It was twice as popular as any other video that Dosomething.org had posted to date. Success! Then came the data report: only eight viewers had signed up to donate equipment, and zero actually donated.
Characteristics/ attributes/ features of vanity metrics:
It can be very hard to sift through data and figure out what actually helps versus what just looks good. If you’re unsure if a metric is a vanity metric, a quick shortcut is to ask: “Can this metric lead to a course of action or inform a decision?” If the answer is “no” or “I don’t know,” then you should probably re-evaluate it.
Smart, actionable metrics help you make a decision. They provide feedback and context for what your goal is and whether or not you’re moving towards it.
Another clue is whether or not you can manage cause and effect within your data. Observing random occurrences isn’t helpful.
Is the data a real reflection of the situation?
Often, data can be manipulated or spruced up with extra effort. Data fallacies abound. Vanity metrics may be hard to identify. The central question to ask yourself when considering a metric is whether or not it will help your business achieve its goals.
In direct opposition to vanity metrics, there exist actionable metrics. Actionable metrics are actual, useful statistics that can provide educators with reliable data that they can then use to objectively improve their eLearning experiences, making actionable metrics far more desirable than vanity metrics.
Examples of vanity metrics in education:
Any metric can be a vanity metric, it all depends on the analysis. However, below are some of the most common examples of metrics prone to being vanity metrics. The education market space is full of vanity metrics, such as:
- Institutional Ranking in various ranking systems
- Area of the campus
- Number of books in the library
- Salary Packages of your graduating students
- GER in higher education
- Percentage of GDP spent by the Government on education
Some common vanity metrics in relation to eLearning include:
- Module completion rate.
- Course completion rate.
- Module views.
- Course views.
- Number of questions answered.
- Number of words typed.
But do these metrics measure what really matters? Do they convey the achievement of your mission? Do they build credibility for the delivery of your value proposition and brand promise?
Parents enthusiastically embrace traditional talking points (e.g., test scores or elite college lists), as they go out into the community as ambassadors for the school–or at least to defend their school decision. These are numbers they can understand and correlate to the common language of other parents.
Bad metrics are most common when there is lack of clarity on what success looks like and doesn’t have strong, accurate feedback loops to access progress.
Here are some examples of what could be real actionable educational metrics?
- How well are the learners being equipped with skills for the future? https://www.weforum.org/focus/skills-for-your-future
- How well equipped are the learners with the skills of learning how to learn? https://youtu.be/O96fE1E-rf8
- Do the graduates have sufficiently developed learning power? https://youtu.be/JxWybvns1jg
- How agile are your learners? Can they learn, unlearn and relearn as said by Alvin Toffler : https://stanfordmag.org/contents/embracing-the-need-to-learn-and-relearn
- What is their rate of learning? Their learning curve : https://en.wikipedia.org/wiki/Learning_curve
- Human capital index ( https://openknowledge.worldbank.org/handle/10986/34432)
- LAYS: learning adjusted years of Schooling : https://www.sciencedirect.com/science/article/abs/pii/S0272775719300263
- AI readiness : https://www.insightsfromanalytics.com/post/artificial-intelligence-readiness-assessment
- Quantum readiness: https://hbr.org/2020/09/are-you-ready-for-the-quantum-computing-revolution
I am sure that you have thought of many other important parameters that could really be used for meaningful and actionable learning metrics. Look forward to your sharing them.