TikTok in peril, A-level and Scottish Highers algo fiascos – Computer Weekly Downtime Upload podcast

In this week’s episode of the Computer Weekly Downtime Upload podcast, Caroline Donnelly, Clare McDonald and Brian McKenna discuss the fortunes of TikTok, the A-level results, and the algorithm controversies besetting those and Scottish Highers.

Caroline kicks off the podcast by recounting the recent fortunes of TikTok, the short-form video publishing and network platform that US president Donald Trump sees as an arm of the Chinese Communist Party.

Caroline is herself an avid TikTok devotee and especially loves watching “good dog” videos. She also objects to the app being described as for Generation Z, and not millennials. And it’s been a lockdown haven from the doom-scrolling corona dystopia of Twitter, she says. And it’s been downloaded two billion times.

But the site is under threat in the US, where the government sees the service, which is owned by Chinese company ByteDance, as a privacy-busting trojan horse, controlled by the Chinese Communist Party, potentially at least enabling blackmail and corporate espionage.

As Ryan Priest details in CW’s Downtime column, Scottish historian Niall Ferguson also takes a dour view of the service, describing it as “not just China’s revenge for the century of humiliation between the Opium Wars and Mao’s revolution. It is the opium – a digital fentanyl, to get our kids stoked for the coming Chinese imperium”. Heavy stuff.

TikTok’s salvation in the US might come in the form of acquisition by either Microsoft or Twitter, otherwise it might go dark on 20 September, its lip-syncing days over, and its users forced to migrate to Instagram.

On this side of the pond, things look less bleak for TikTok, which will open its first European datacentre in Ireland in 2022, as it looks to curry favour with European data privacy watchdogs.

While these are interesting and potentially dark times for TikTok and its Generation Z (and older) devotees, last week was a very big week indeed for young adults in England, Wales and Northern Ireland: the A-level results were announced on Thursday 13 August – results without exams, but with added algorithmic “standardisation”.

(The podcast was recorded on Friday 14 August).

Clare has been closely tracking the gender split for A-levels, especially with respect to computing. This year there were 10,629 male candidates and 1,797 female candidates for computing, which is disappointing.

However, in 2018 there were 1,475 female candidates, so there has been an increase, and, proportionately, 11% of the female candidates got A* grades, compared with 7.5% of the males. Also, computing students seem to have been less adversely affected by the Ofqual algorithm than have A-level students more generally.

The podcast team then go on to talk about the controversy surrounding the statistical model applied to teacher assessments (and then, later in the podcast, the model applied by the Scottish Qualifications Authority to Highers, Advanced Highers and National 5s).

For convenient reference, this text, from the Ofqual interim report on the full sweep of the England, Wales and Northern Ireland exams, describes the Ofqual model:

“Our preferred model – known as the Direct Centre Performance (DCP) model – works by predicting the distribution of grades for each individual school or college. That prediction is based on the historical performance of the school or college in that subject taking into account any changes in the prior attainment of candidates entering this year compared to previous years. This was fine-tuned to take account of known issues such as centres with small cohorts of students, small-entry subjects and tiered subjects. Decisions were also made on the number of years of historical data included in the model.

“The details of these decisions are set out in this report and are formalised in the regulations we put in place for summer 2020. Where schools and colleges had a relatively small cohort for a subject – fewer than 15 students when looking across the current entry and the historical data – the standardisation model put more weight on the CAGs [Centre Assessment Grades]. Since small teaching groups are more common for AS and A-level than for GCSE, and given that the CAGs tended to be optimistic, it means that the outcomes in some AS and A-level subjects are much higher this year. However, there is no statistical model that can reliably predict grades for particularly small groups of students. We have therefore used the most reliable evidence available, which is the CAGs.”

A Guardian editorial summed up a possible critique of this approach, which was essentially aimed, Brian comments, at suppressing grade inflation:

“By building in a criterion of past school performance to this year’s A-level and GCSE results, Ofqual has tied the fortunes of individual students to pre-existing inequalities of outcome. To try in this way to impose a standard spread of results, which pupils are powerless to affect, is unjust.”

Clare comments in discussion that an option of sitting an exam in the autumn would be stressful indeed, in this year’s circumstances.

Brian then takes the team back to early August, when the results of the Scottish Highers, Advanced Highers and National 5s (the equivalent of GCSEs) were announced. This was the “first time as history” to the rest of the UK’s “second time as farce”.

Again, the results declared were the result of a combination of teacher assessments and the application of a statistical model to previous years’ school results data. The following text sums up the model the SQA chose:

“In simple terms, mathematical optimisation (more popularly called ‘optimisation’) is a family of techniques used to identify the best possible solution to meet a stated object according to one or more defined constraints. Fundamentally, optimisation was selected as the preferred technique for adjusting estimates, because it tests all possible solutions concurrently, in order to identify the ‘best available’ value for an objective function – given a set of constraints, in a robust and efficient manner.

“Furthermore, optimisation techniques are tested and proven, both in industry and literature, and therefore provide a credible approach for undertaking the adjustments required to support this year’s awarding. The optimisation approach applied was based on a mixed integer linear program within a network framework to ensure that the relativity of refined bands on a course as estimated by a centre, was always maintained. Where adjustment was required, the primary objective function of the optimisation process was to minimise the number of candidates moved between grades to meet the centre constraints for each grade and A-C rate.”

This is from the SQA Technical Report National Qualifications 2020 Awarding – Methodology Report.

Brian talks the team through the narrative of the Scottish case, where the Scottish government decided to revert to teacher assessments. First minister Nicola Sturgeon apologised and said the mistake had been to focus too much on the overall system, and not enough on individual pupils.

One of the most damning outcomes of the application of the statistical model, says Brian, is that the most deprived one-fifth of the Scottish students saw their pass rate go down by 15%, while the wealthiest one-fifth saw theirs go up by 7%.

Brian expresses his own view that both governments should have been less hung up on grade inflation this year, given the stress all of society has been under due to the Covid-19 pandemic. As it is, by imposing these statistical models – however good they might or might not be in technical terms – they have merely created rods for their own backs.

Whatever the truth of that, this story will continue to run, especially given the imminent publication of the GCSE results this coming week.

Postscript. In a dramatic U-turn, the government decided, late on the afternoon of Monday 17 August, that A-level and GCSE students in England will be given teacher-estimated grades. 

Podcast music courtesy of Joseph McDade