srhe

The Society for Research into Higher Education

Monthly Archives: April 2022


by 1 Comment

Towards a better, granular, understanding of differences in awarding gaps

by Kathryna Kwok and Siân Alsop

As a sector, we have long known that a student’s ethnicity has a relationship to the average class of degree they are awarded. Students from Black, Asian and minority ethnic (or ‘BAME’) backgrounds can expect to leave UK universities with lower classifications than their White counterparts – up to 20% lower (UUK & NUS, 2019). Universities have been paying attention to how institutional factors like curricula, assessment and staffing decisions might contribute to this awarding gap in a welcome shift away from focusing on the role student background might play. But not much is really changing. The OfS (2022) reports that although the gap has somewhat narrowed in the last five years, Black students in particular continue to receive 17.4% lower degree outcomes than White students. At this point, it is clear that sweeping, quick fix, or isolated interventions are not fit for purpose. The mechanisms underpinning awarding disparities are complex and entrenched. It seems more likely that a sharpened focus and multi-pronged approach to unpicking the multiple ways in which we disadvantage certain students would be useful, if we are serious about change. 

In this vein, we took as our starting point the need to get granular – to dig into ‘who’, as binary ‘BAME’/White distinctions are uninformative at best, and ‘how’, as a degree award is the culmination of many parts. We focused on understanding differences in awarded marks at the module level. But what’s the best way to operationalise module mark differences between two groups? One obvious option would be, simply, to calculate the difference in means (ie mean of group A – mean of group B). However, this has the potential to be misleading. What if, for instance, there were a lot of variation in students’ marks?

Our solution, as we describe in our 2019 SRHE Research Award final report, was to use the formula for the t-statistic, which is a measure reflecting the difference between two groups’ means scaled to group variation and size. In the context of calculating module mark gaps, we refer to this as the difference index (DI) to avoid confusion with the t-values calculated as part of other statistical testing. The formula for DI is below – here, n is the number of students in a group, s is the standard deviation and x̄ is the group mean.

A larger absolute value indicates a larger difference between the two group means (with group size and variation held constant). A positive value means that group A outperformed group B, while a negative value means that group B outperformed group A. If multiple comparisons are being performed with one common baseline group, it is recommended that the baseline group is consistently positioned as group A.

The DI offers a straightforward yet nuanced way to operationalise module mark gaps using data that universities already routinely collect. As a measure of module-level differences, it also offers a way to characterise, monitor and investigate the awarding gap at a granular level – something which percentage point differences in final degree outcomes are much less able to do.

This said, the DI does have its disadvantages. It is not as intuitively interpretable as a simple difference in means. This is perhaps the trade-off for the added nuance it offers. As an illustration, for two groups each with 10 students and an equal spread of module marks (SD = 10.00), a difference of five marks equates to a DI of 1.12, and a difference of ten marks equates to a DI of 2.24. Another disadvantage of the DI is that it can only be used to compare two groups at a time, meaning separate calculations have to be performed for each pairing. Analyses involving multiple groups (or multiple combinations of groups) could thus quickly become unwieldy. In our case study, which we describe in the report linked above, we used regression modelling to investigate whether module characteristics (eg level, credit value, exam weight) could predict DI. This required us to compute one regression model for each ethnicity pairing (White v Asian, White v Black, White v Mixed, White v Other). One of our findings was that module characteristics significantly predicted DI only between White and Black students, which we note ‘highlight[ed] the importance of recognising the heterogeneity of student experiences and needs’ (Kwok & Alsop, 2021: 16). We hope to conduct similar analyses with a much larger dataset, ideally from multiple institutions, which would enable us to utilise multilevel regression techniques to more elegantly capture and explore granular differences in the marks awarded to different groups of students.

To our knowledge, there is no measure that is systematically being used to operationalise the ethnic awarding gap at a level more granular than final degree outcome. We argue that this limits universities’ abilities to understand the awarding gap and identify what can be done to address it at an institutional level. We believe that the DI offers a solution to this. It can be used by researchers, as we have done, to investigate what curriculum-related factors may be contributing to the awarding gap. Those who teach can use the DI to explore module mark differences in their programmes, both within and between cohorts. Those with access to institutional-level data can investigate these trends on an even larger scale, for instance to explore if there are particular areas or time points where modules have consistently high or low (or negative) DIs. The impact of the pandemic can also be explored in this way, for example by using student cohort data either side of Covid-19. Further, while this article has discussed the DI in the context of investigating the ethnic awarding gap, it can also be used to compare students grouped in other ways. It is important that institutions utilise the data they already have to understand awarding discrepancies in a clear and sufficiently granular way – using the DI would help accomplish this.

Kat Kwok is an Educational Researcher at the Oxford Centre for Academic Enhancement and Development at Oxford Brookes University. She is interested in using quantitative methods to investigate race and gender disparity in higher education, and the student experience. Kat recently started her PhD at Coventry University where she will be using a mixed methods approach to investigate the relationships between feedback and student characteristics, and the impact of feedback.

Dr Siân Alsop is Research Fellow in the Centre for Global Learning at Coventry University. She is a corpus linguist whose research areas include attainment disparities in higher education, feedback, and the language of lectures. Siân was previously a Lecturer in Academic Writing and has worked on a number of projects relating to academic discourse, including the development of the British Academic Written English (BAWE) corpus and the Engineering Lecture Corpus (ELC).


by 1 Comment

Do we want social science, or social science-fiction?

By Paul Temple

Because a lot of its research work involves schools and school-age children, research training at the Institute of Education tends to emphasise issues around confidentiality and anonymity in presenting research findings. Anonymity usually happens anyway in large-scale studies with hundreds or thousands of respondents – the National Student Survey, for example – because the point is to get a picture of what a category of people think, rather than the views of particular individuals. Those of us working on higher education research, however, are often in the position of asking relatively small numbers of informants about their professional views – that is, about the knowledge for which their employer is paying them, unrelated to their private lives. A university finance director could of course decline to be interviewed, but it’s hard to see why there should be a confidentiality problem about their explanation of the university’s resource allocation methodology.

When one of my PhD students was planning a study of the closure of the Ripon campus of York St John University, she assumed everything would have to be anonymised. But why?, I asked her. Yes, the closure was controversial, but that’s now history. Your study, I said, will have far more value if you contextualise it in the real-life settings of York and Ripon; just be sure that your respondents know that what they say will be on the record. But will they talk to me on that basis, she wondered: your problem, I said (correctly, as it turned out), will be getting them to shut up when you want to end the interview. Her thesis was fine, and the then YSJ VC seemed pleased that the institution was considered to be an interesting research subject.

The book by John Brennan et al, The University in its Place (2018), which provides four case studies of anonymised UK universities, took me back to these discussions. Brennan and his co-authors actually provide enough context to make it easy to work out the institutional identities: how many Scottish east coast cities have two universities, one of which (“Aspirational U”) obtained university status in 1994 and has about 5000 students? – no Googling, please. But it is the conceptual reason for anonymization that the authors put forward, aside from wanting “to respect the confidentiality of those to whom we talked” about the “sensitive” matter of university-regional relationships (p48), that I find fascinating. Anonymity was apparently necessary because “it is simply not possible to capture all the day-to-day complexity and uncertainty associated with higher education in practice and in place … [the study] necessarily involves selection, both intended and unintended … [so we cannot] present a complete and fully articulated representation either of particular institutions or places” (p48).

But making a selection is the basis not merely for all social science but for all research, of any kind. Show me a piece of research that isn’t the result of making a selection from the infinite variety of natural and social phenomena that initially confronted the researcher. Having then made a selection, no social scientific study will “capture all the day-to-day complexity” of an organisation, or of anything else. This is simply a logical impossibility, as there will always be more facts to be collected, more detail to be recorded, more details of details to be examined – as surely every PhD supervisor knows: “You’ve done enough, just write the thing up!”. What a researcher must do is focus on the chosen detail and (with an explanation) ignore the rest: you want the finance director to tell you about the resource allocation model, not about “the day-to-day complexity” of managing cash flow. In the cases of Brennan et al, no reasonable person would expect a study of university-regional relationships to include “a complete … representation” of, say, the catering arrangements in a particular institution, any more than an account of university catering would be expected to consider the university’s relations with regional public bodies.

I think that removing through anonymisation the context in which the research subject is embedded reduces the quality of understanding: it makes it harder for the reader to create their own view about what the research is saying – they have to go along with what the researchers say. Actually, anonymisation may mislead the reader if they guess wrong about the identity of the subject: “Funny, this doesn’t sound like Aberdeen!”.

It seems to me that Americans take a much more robust view about these matters than we do in Britain: maybe it’s a result of the First Amendment. George Keller’s ground-breaking book on university planning, Academic Strategy (1983), carried weight because he was describing successes and failures by named individuals in named universities. In the same tradition, Mark Kretovics (2020) doesn’t pull his punches, describing named leadership successes and failures (the failures are the best bits), down to the juicy details of expenses scandals. These accounts make you almost feel as if you’re in the room, not wondering if you’ve even got the right city.

Dr Paul Temple is Honorary Associate Professor in the Centre for Higher Education Studies, UCL Institute of Education.

References

Brennan, J., Cochrane, A., Lebeau, Y. and Williams, R. (2018). The University in its Place: Social and cultural perspectives on the regional role of universiites. Dordrecht: Springer.

Keller, G. (1983). Academic Strategy: The Management Revolution in American Higher Education. Baltimore, MD: The Johns Hopkins University Press. Kretovics, M. (2020). Understanding Power and Leadership in Higher Education: Tools for Institutional Change. New York: Routledge.

Image of Rob Cuthbert


by Leave a comment

Who should pay for higher education in England, and how much

by Rob Cuthbert

SRHE News is a quarterly publication, available only to SRHE members, which aims to comment on recent events, publications, and activities in a journalistic but scholarly way, allowing more human interest and unsupported speculation than any self-respecting journal, but never forgetting its academic audience and their concern for the professional niceties. These are some extracts from the April 2022 issue.

Government uses high inflation as cover for hitting students, graduates and universities

The Government sneaked a student loans announcement out on Friday afternoon 28 January 2022. Ben Waltmann (Institute for Fiscal Studies) said: “Today’s announcement … constitutes a tax rise by stealth on graduates with middling earnings. … For a graduate earning £30,000, this announcement means that they will pay £113 more towards their student loan in the next tax year than the government had previously said. … What really matters is how long this threshold freeze will stay in place. If it is only for one year, the impact on graduates will be moderate, and the government can only expect to save around £600 million per cohort of university students. If it stays in place for longer, it could transform the student loan system, with a much lower cost for the taxpayer and a much higher burden on graduates than they thought … when they took out their loans.” However, some well-informed commentators thought that the Minister had made the best of a bad job.

Waltmann followed up in his 10 February ‘Observation’: ”Students will see substantial cuts to the value of their maintenance loans, as parental earnings thresholds will stay frozen in cash terms and the uplift in the level of loans will fall far short of inflation. This continues a long-run decline in the value of maintenance entitlements. The threshold below which students are entitled to full maintenance loans has been unchanged in cash terms at £25,000 since 2008; had it risen with average earnings, it would now be around £34,000. Separately, the student loan repayment threshold will also be frozen in cash terms. … Finally, tuition fees will remain frozen in cash terms for another year, which hits universities and mainly benefits the taxpayer. … as our updated student finance calculator shows, the government is saving £2.3 billion on student loans under the cover of high inflation.”

At last, the government response to Augar

On 24 February 2022 the government finally issued a detailed response to the 2019 Augar Report, setting out a series of policy proposals and further issues for consultation: “Put simply, we need a fairer and more sustainable system for students and institutions, and of course the taxpayer. We need a system that will maintain our world-class universities not just for today, but for the decades to come. And we need a fairer deal for students …”. Rachel Wolf wrote for The Times Red Box on 24 February 2022 that she was encouraged by the Augar response (and the separate consultation on Lifelong Learning Entitlement), because it suggested that there was proper Cabinet government with a sensible Secretary of State for Education, rather than No 10 being in charge of everything. Yes, but … there was precious little welcome for most of the proposals.

Nick Hillman blogged for HEPI just ahead of the DfE announcement, trying desperately to save the Willetts fee policy (he was Willetts’ special adviser) from being labelled as a political failure. That policy was designed to be redistributive and progressive, but in practice the shortfall in repayments (RAB) became much too high and not enough people understood that many students were not expected to repay loans in full. The Theresa May government misunderstood it to the extent that they raised the repayment threshold, which cost the Exchequer much more without giving much benefit to students. The Treasury tolerated it until the national accounting systems were properly changed to show the loans for what they were, rather than spreading them as a cost over 20-30 years. Hillman selectively quoted Moneysavingexpert Martin Lewis, but he would have done better to see the 24 February Lewis quote that loans had now become a graduate tax throughout people’s working lives.

Jim Dickinson for Wonkhe on 24 February 2022 noted the absence of any response to Augar’s chapter on maintenance grants: “Overall then, almost all students will end up paying significantly more for having significantly less spent on their education … we might have at least expected a response on the bits of Augar that were concerned with students’ costs or their maintenance. That they are not even acknowledged tells us quite a bit about what the government thinks about students and graduates.”

Gavan Conlon of London Economics issued his analysis of the government proposals. “Under the current funding system in 2021-22 … the Exchequer contributes approximately £10.630bn per cohort to the funding of higher education. … given that the RAB charge (the proportion of the total loan balance written off) stands at approximately 52.5%, maintenance loan write-offs cost the Exchequer £4.105bn per cohort, while tuition fee loan write-offs cost £5.303bn. The recent freeze in the repayment threshold reduced HMT costs by approximately £300 million. The provision of Teaching Grants to higher education institutions (for high-cost subjects) results in additional costs of £1.222bn per cohort. Higher Education Institutions receive £11.144bn per cohort in net income from undergraduate students … £10.112bn in tuition fee income … £1.222bn in Teaching Grants. … institutions contribute £189 million per cohort in fee and maintenance bursaries (predominantly the latter) in exchange for the right to charge tuition fees in excess of the ‘Basic Fee’ (£6,165 per annum for full-time students). For students/graduates, the average debt on graduation (including accumulated interest) was estimated to be £47,500 (for full-time first degree students), with average lifetime repayments of £35,900 for male graduates and £13,900 for female graduates. We estimate that 88.2% of all graduates never repay their full loan, while 33.0% never make any loan repayment.” The first scenario he modelled involved “removing the real interest rate, reducing the earnings repayment thresholds to £25,000 (and the associated maximum interest rate threshold), and extending the repayment period to 40 years”. That led to savings of £539million for the Exchequer, with no change for HEIs. The average debt on graduation declined following the changes by £1,600. Average lifetime repayments for male graduates  decline by £2,000 but increase by £3900 for female graduates. “However, these are averages and there are important distributional effects associated with these proposals”. Scenario 2 added the introduction of Minimum Entry Requirements and reintroduction of Student Number Controls. The savings for the Exchequer were estimated at £1322million, with a loss for HEIs of £840million. The effect on students was unchanged.

Conlon then co-authored a blog with Andrew McGettigan (independent) for Wonkhe on 25 February 2022, which showed that most of the savings had actually been achieved by changing the discount rate used for student loans, making them more valuable. “… the graduates who will benefit the most are the highest earning – predominantly male – graduates. The messaging has been that lower earning graduates need to pay more to make the system sustainable. In fact it’s the discount rate change that does most of that – with the extra contributions from lower earning graduates helping to fund the reduced contributions from the richest. … It’s hard to see this when there is a lot of smoke and mirrors. What makes all this worse is the government knows that its discount rate change means that the extra payments made by lower earning graduates in years 30 to 40 are doing most of the heavy lifting.” Student finance campaigner Martin Lewis of Moneysavingexpert called it “a very damning piece”.

Ben Waltmann of the Institute for Fiscal Studies wrote on 24 February that: “The largest student loan reform since 2012 will reduce the cost of loans for high-earning borrowers but increase it for lower earners. Today the government has announced the largest changes to the student loans system in England since fees were allowed to triple in 2012. Starting with the 2023 university entry cohort, graduates will pay more towards their student loans each year and their loan balances will only be written off 40 years after they start repayments. For the same cohorts, the interest rate on student loans will be reduced to the rate of increase in the Retail Prices Index (RPI), a large cut of up to 3 percentage points. Maximum tuition fees will be frozen in nominal terms until the 2024/25 academic year. These changes will transform the student loans system. While under the current system, only around a quarter can expect to repay their loans in full, around 70% can expect to repay under the new system. This is partly due to substantially higher lifetime repayments by students with low and middling earnings and partly due to less interest being accumulated on loans. The long-run benefit for the taxpayer will be around £2.3 billion per cohort of university entrants, as higher repayments by borrowers with low or middling earnings will be partly offset by lower repayments of high-earning borrowers.”

Richard Adams in The Guardian on 24 February 2022 pointed out the DfE’s own analysis showed the poorest would suffer: “An equality analysis on the proposals by the Department for Education, states that “those likely to see some negative impact with increased lifetime repayments under the reforms” include younger and female graduates as well as graduates “from disadvantaged backgrounds, or reside in the north, Midlands, south-west or Yorkshire and the Humber”.

On 24 February 2022 Wonkhe’s Debbie McVitty suggested the proposals were looking for “a third way between capping opportunity and letting the HE market run amok”. John Morgan’s article for Times Higher Education on 28 February 2022 had expert commentators describing the Augar response package as a ‘missed opportunity’, with Chris Husbands (Sheffield Hallam VC) saying “what the package essentially does is to kick the difficult questions down the road”. David Willetts, in Times Higher Education on 3 March 2022, thought the Augar response was “balanced tweaks”; he was trying to rescue his fees policy, which worked in theory but not in practice. Nick Hillman was still trying in Research Professional News on 6 March 2022.

Universities Minister Michelle Donelan wrote a Conservative Home-spun version of the changes, in which she interestingly she referred to “Our top university cities – Cambridge, Oxford, Bristol, Manchester and London …”. Donelan’s speech to the Conservative Party Conference in March 2022 set out what the Minister wants the narrative to be. Diana Beech (London Higher) blogged for HEPI on 7 March 2022 about the government response to Augar and the recent flurry of OfS consultations: “… what we are facing now is not a series of seemingly independent consultations concerned with the minutiae of regulation, but a multi-pronged and coordinated assault on the values our higher education sector holds dear.” Alan Roff, former Deputy VC at Central Lancashire, reprised his 2021 argument for a graduate contributions scheme with his 22 March HEPI blog. We assume that still, no-one in government is listening.

Mary Curnock Cook, former UCAS chief executive, was upbeat about the possibility of setting a minimum entry requirement (MER) in terms of grade 4 English and Maths at GCSE, in her HEPI blog on 24 February 2022. However SRHE Fellow Peter Scott pointed out, in his 28 February 2022 HEPI blog, that “In Scotland, universities set MERs to widen access. In England, the State imposes MERs to curb it. So, it is very difficult to claim the UK Government’s package of measures in response to the recommendations made by Augar are somehow progressive, let alone favourable to fair access.”

Rob Cuthbert, editor of SRHE News and Blog, is emeritus professor of higher education management, Fellow of the Academy of Social Sciences and Fellow of SRHE. He is an independent academic consultant whose previous roles include deputy vice-chancellor at the University of the West of England, editor of Higher Education Review, Chair of the Society for Research into Higher Education, and government policy adviser and consultant in the UK/Europe, North America, Africa, and China.

Email rob.cuthbert@uwe.ac.uk, Twitter @RobCuthbert.

To join a global community of scholars researching into HE, see the SRHE website.

Image of Rob Cuthbert


by Leave a comment

Tunnel vision: higher education policy and the Office for Students

by Rob Cuthbert

In January 2022 the Office for Students published three sets of consultations, 699 pages of proposals for the regulation of student outcomes, the determination of teaching excellence, and the construction of indicators to measure student experience and outcomes. These were not separate initiatives, but part of a co-ordinated programme which needs to be seen in the context of the long-awaited government response to the 2019 Augar report, finally published in March 2022[1].

The OfS consultation announced that numerical thresholds will underpin requirements for minimum acceptable student outcomes at both undergraduate and postgraduate level. Universities and colleges not meeting these could face investigation, with fines and restrictions on their access to student loan funding available as potential sanctions. For full-time students studying a first degree, the thresholds require: 80% of students to continue into a second year of study; 75% of students to complete their qualification; 60% of students to go into professional employment or further study.

Not just numbers? OfS say: “we recognise that using our indicators to measure the outcomes a provider delivers for its students cannot reflect all aspects of the provider’s context … If a provider delivers outcomes for its students that are below a numerical threshold, we will make decisions about whether those outcomes are justified by looking at its context. This approach would result in a rounded judgement about a provider’s performance.”

But then: “… such an approach may present a challenge for some providers. This is because they must only recruit students where they have understood the commitment they are making to support their students to succeed, irrespective of their backgrounds. … Most universities and colleges relish this challenge and already deliver on it. However, some do not. While some may offer opportunities for students to enter higher education, we also see low continuation and completion rates and disappointing levels of progression to relevant employment or further study.” A warning, then, for “some”, but not “most”, providers.

The OfS approach will be fine-grained: “We would consider whether a provider has complied with condition B3 in relation to each separate indicator or split indicator. This enables us to identify ‘pockets of provision’ where performance in a specific subject, for students with specific characteristics, or in relation to partnership arrangements, falls below a numerical threshold”.

‘Selecting’ universities might think that ‘contextual judgment’ will rescue them, but may still decide to play safe in subjects where the numbers don’t look so good. ‘Recruiting’ universities, especially in ‘levelling up’ areas, might be looking at the numbers across many programmes and considering their strategy. Everyone will be incentivised to play safe and eliminate what are numerically the most marginal candidates, subjects and courses. And everyone thinks this will discriminate against disadvantaged students. For example, the University Alliance response published on 16 March 2022 said: “The University Alliance is gravely concerned that the proposals outlined by government could have unintended consequences for the least privileged students in society.”

Sally Burtonshaw (London Higher) blogged for HEPI on 26 January 2022: “As the dust begins to settle on the 699 pages of Office for Students’ (OfS) consultations and accompanying documents published on Thursday and providers across the sector begin to draft responses (deadline March 17th), it feels like there is a gaping chasm between the sector and its regulator. Language in the accompanying press release with references to ‘crack downs’, ‘tough regulatory action’ and ‘protecting students from being let down’, jars with a sector which has contributed so much throughout the pandemic.”

Diana Beech (London Higher) blogged for HEPI on 7 March 2022 about the government response to Augar and the OfS consultations: “… what we are facing now is not a series of seemingly independent consultations concerned with the minutiae of regulation, but a multi-pronged and coordinated assault on the values our higher education sector holds dear.” Diana Beech was a policy adviser to the last three ministers for universities.

SRHE Fellow Peter Scott summed it up like this: “This … ‘direction of travel’ is … based on the assumption that we should continue to distinguish between FE and HE, vocational and academic tracks, in terms of their social bases and costs. Of course, that is the current reality. Universities, especially Russell Group ones, draw a disproportionate number of their students from socially-privileged backgrounds, while FE is badly under-funded. This is why it makes (economic) sense for the Government to try to divert more students there. But is that sustainable in a country that aspires to being both democratic and dynamic? Most other countries have moved on and now think in terms of tertiary systems embracing HE, FE, on-the-job training, adult and community learning, the virtual stuff … bound together by flexible pathways and equitable funding – and, above all, by fair access. In the UK, Wales is setting the pace, while Scotland has had its ‘Learner Journey 15-24’ initiative. In England, sadly, there is no echo of such positive thinking.”

Status hierarchies must, it seems, be maintained, and not just between HE and FE, but also between universities. Contrary to expectations the Teaching Excellence Framework will rise from the ashes of the Pearce Review via the OfS’s second consultation. Earlier versions of TEF did not reliably reproduce the existing status hierarchies; some Russell Group institutions even suffered the indignity of a bronze rating. Clearly this could not be allowed to continue. So now: “The proposed TEF process is a desk-based, expert review exercise with decisions made by a panel of experts to be established by the OfS. The panel would consider providers’ submissions alongside other evidence. … TEF assessment should result in an overall rating for each provider. The overall rating would be underpinned by two aspect ratings, one for student experience and one for student outcomes but there would be no rating of individual subjects within a provider.” Such undifferentiated provider-level arrangements will surely be enough to ensure no further embarrassment for those with the highest reputations.

There will still be gold, silver and bronze awards, but not for all. The OfS script is worthy of Yes Minister: “… our minimum baseline quality requirements establish a high quality minimum for all providers. Therefore, quality identified that is materially above the relevant baseline quality requirements should be considered as ‘very high quality’ or ‘outstanding quality’ … ‘Outstanding quality’ signifies a feature of the student experience or outcomes that is among the very highest quality found in the sector for the mix of students and courses taught by a provider. … ‘Very high quality’ signifies a feature of the student experience or outcomes that is materially above the relevant minimum baseline quality requirements for the mix of students and courses taught by a provider.” Is the difference clear? If not, don’t worry, because the TEF Panel will decide.

As Sir Humphrey might have put it: it’s like the Olympics – not everyone will get on the podium. And it’s like ice dancing: judges hand out the marks based on how they rate the performance. The table of “features of excellence” spells out the criteria, for example: “The provider uses research in relevant disciplines, innovation, scholarship, professional practice and/or employer engagement to contribute to an outstanding academic experience for its students.” Whereas for high quality: “The provider uses research in relevant disciplines, innovation, scholarship, professional practice and/or employer engagement to contribute to a very high quality academic experience for its students.” Is the difference clear? If not, don’t worry, because the TEF Panel will decide.

Nick Hillman blogged for HEPI on 21 January 2022 about the OfS initiatives, reflecting on the limited success of previous attempts to shift evaluation towards metricisation, and Debbie Mcvitty blogged for Wonkhe on 24 January 2022 with a helpful potted history.There will be no surprises in the outcomes of the consultations. Whether or not the Titanic is sinking, we are consulted only on how to arrange the deckchairs. As HEPI’s Nick Hillman said: “I vividly recall what Les Ebdon, the former Director for Fair Access, said a few years ago when he was asked, “What will the Office for Students do?” His answer was, “It’s very simple. I can tell you exactly what the OfS will do. It will do whatever the government of the day wants it to do.” And so it has proved.”

Let us, then, look not at the entirely predictable outcomes, but at the style the OfS has adopted to reach them. The consultation on regulation of outcomes is telling. It takes 100 pages to assemble a rational-bureaucratic edifice in rational-bureaucratic language, with chapter headings including: “… making judgments about compliance with condition B3 … Addressing statistical uncertainty in the assessment of condition B3 … Taking regulatory action when a breach is identified …”. There could have been headings like: “How do we know how good the performance is?” or “What if something goes wrong?”. But that would have exposed the deeper questions, for which answers have already been decided. Instead we are drowned with bureaucratic detail. Details are always necessary, but we should be reminded of why they are needed. Instead these documents do their best to obscure the fait accompli which is their starting point, with a grinding remorseless pseudo-rationality which encourages you to lose sight of purposes and values.

In 699 pages of consultation the OfS has done its bureaucratic best to profess transparency, openness and rigour, while diverting our energies and attention from what an experienced ministerial adviser called the ‘assault on the values which our HE sector holds dear’. The consultations amount to a detailed enquiry about how exactly these values should be assaulted. We are in a consultation tunnel with only one track. What we can see is probably not the light at the end of the tunnel, it may be the lights from an oncoming train.

Rob Cuthbert, editor of SRHE News and Blog, is emeritus professor of higher education management, Fellow of the Academy of Social Sciences and Fellow of SRHE. He is an independent academic consultant whose previous roles include deputy vice-chancellor at the University of the West of England, editor of Higher Education Review, Chair of the Society for Research into Higher Education, and government policy adviser and consultant in the UK/Europe, North America, Africa, and China.

Email rob.cuthbert@uwe.ac.uk, Twitter @RobCuthbert.

[1] Covered elsewhere in this issue of SRHE News. SRHE members can read this and previous editions of SRHE News via https://srhe.ac.uk/my-account/


by Leave a comment

Statistical illogic: the fallacy of Jacob Bernoulli and others

by Paul Alper

Bernoulli’s Fallacy, Statistical Illogic and the Crisis of Modern Science by Aubrey Clayton.  

“My goal with this book is not to broker a peace treaty; my goal is to win the war.”    (Preface p xv)

“We should no more be teaching p-values in statistics courses than we should be teaching phrenology in medical schools.” (p239)

It is possible or even probable that many a PhD or journal article in the softer sciences has got by through misunderstanding probability and statistics. Clayton’s book aims to expose the shortcomings of a fallacy first attributed to the 17th century mathematician Jacob Bernoulli, but relied on repeatedly for centuries afterwards, despite the 18th century work of statistician Thomas Bayes, and exemplified in the work of RA Fisher, the staple of so many social science primers on probability and statistics.

In the midst of the frightening Cold War, I attended a special lecture at the University of Wisconsin-Madison on 12 February 1960 by Fisher, the most prominent statistician of the 20th century; he was touring the United States and other countries. I had never heard of him and indeed, despite being in grad school, my undergraduate experience was entirely deterministic: apply a voltage then measure a current, apply a force then measure acceleration, etc. Not a hint, not a mention of variability, noise, or random disturbance. The general public’s common currency in 1960 did not then include such terms as random sample, statistical significance, and margin of error. 

However, Fisher was speaking on the hot topic of that day: was smoking a cause of cancer?  Younger readers may wonder how in the world was this a debatable subject when in hindsight, it is so strikingly obvious. Well, it was not obvious in 1960 and the history of inflight smoking indicates how difficult it was to turn the tide, and how many years it took. Fisher’s tour of the United States was sponsored by the tobacco industry, but it would be wrong to conjecture that he was being hypocritical. And not just because he was a smoker himself.  

Fisher believed that mere observations were insufficient for concluding that A causes B; it could be that B causes A or that C is responsible for both A and B. He insisted upon experimental and not mere observational evidence. According to Fisher, it could be that people who have some underlying physical problem led them to smoke rather than smoking caused the underlying problem; or that some other cause such as pollution was to blame. According to Fisher, in order to experimentally link smoking as the cause of cancer, at random some children would be required to smoke and some would be required not to smoke and then as time goes by note the incidence of cancer in each of the two groups.

However, according to Clayton, Fisher himself, just like Jacob Bernoulli, had it backwards when it came to analysing experiments.  If Fisher and Bernoulli can make this mistake, it is easy for others to fall into this trap because ordinary language keeps tripping us up.  Clayton expends much effort into showing examples, such as the famous Prosecutor’s Fallacy. The fallacy was exemplified in the UK by the infamous Meadows case and is discussed at length by Clayton; a prosecution expert witness made unsustainable assertions about the probability of innocence being “one in 73 million”.

The Bayesian way of looking at things is to consider the probability a person is guilty, given the evidence. This is not the same as the probability of the evidence, given the person is guilty, which is the ‘frequentist’ approach adopted by Fisher, with results which can be wildly different numerically. Another example, from the medical world: there is confusion between the probability of having a disease, given a positive test for the disease:

                        Prob (Disease | Test Positive) ; the Bayesian way of looking at things

and

                         Prob (Test Positive | Disease) ; the frequentist approach

The patient is interested in the former but is often quoted the latter, known as the sensitivity of the test, which might be markedly different depending on the base rate of the disease. If the base rate is, say, one in 1,000 and the test sensitivity is, say, 90%, then for every 1000 tests, 100 will be false positives. A Bayesian would therefore conclude correctly that the chances of a false positive test are 100 times greater than the chances of actually having the disease. In other words, the hypothesis that the person has the disease is not supported by the data/evidence. However a frequentist might mistakenly say that if you test positive there is a 90% chance that you have the disease.

The quotation from page xv of Clayton’s preface which begins this essay, shows how much Clayton, a Bayesian, is determined to counter Bernoulli’s fallacy and set things straight. Fisher’s frequentist approach still finds favor among social scientists because his setup, no matter how flawed, was an easy recipe to follow. Assume a straw-man hypothesis such as ‘no effect’, take data to obtain a so-called p-value and, in the mechanical manner suggested by Fisher, if the p-value is low enough, reject the straw man. Therefore, the winner was the opposite of the straw man, namely the effect/hypothesis/contention/claim is real.

Fisher, a founder, and not just a follower of the eugenics movement, was as I once wrote, “a genius, and difficult to get along with.”  Upon reflection, I consequently changed the conjunction to an implication, “a genius, therefore difficult to get along with.”  His then son-in-law back on 12 February 1960 was George Box, also a famous statistician – among other things the author of the famous phrase in statistics, “all models are wrong, some are useful” – who had just been appointed to be the head of the University of Wisconsin’s statistics department. Unlike Fisher, Box was a very agreeable and kindly person and, as evidence of those qualities, I note that he was on the committee that approved my PhD thesis, a writing endeavour of mine which I hope is never unearthed for future public consumption.  

All of that was a long time ago, well before the Soviet Union collapsed, only to see today’s military rise of Russia. Tobacco use and sales throughout the world are much reduced while cannabis acceptance is on the rise. Statisticians have since moved on to consider and solve much weightier computational problems via the rubric of so-called Data Science. I was in my mid-twenties and I doubt that there were many people younger than I was at that Fisher presentation, so I am on track to be the last one alive who heard a lecture by Fisher disputing smoking as a cause of cancer.  He died in Australia in 1962, a month after my 26th birthday but his legacy, reputation and contribution live on and hence, the fallacy of Bernoulli as well.    

Paul Alper is an emeritus professor at the University of St. Thomas, having retired in 1998. For several decades, he regularly contributed Notes from North America to Higher Education Review. He is almost the exact age of Woody Allen and the Dalai Lama and thus, was fortunate to be too young for some wars and too old for other ones. In the 1990s, he was awarded a Nike sneaker endorsement which resulted in his paper, Imposing Views, Imposing Shoes: A Statistician as a Sole Model; it can be found at The American Statistician, August 1995, Vol 49, No. 3, pages 317 to 319.