srhe

The Society for Research into Higher Education

Image of Rob Cuthbert


Leave a comment

Quality and standards in higher education

By Rob Cuthbert

What are the key issues in HE quality and standards, right now? Maintaining quality and standards with the massive transition to remote learning? Dealing with the consequences of the 2020 A-levels shambles? The student experience, now that most learning for most students is remote and off-campus? Student mental health and engagement with their studies and their peers? One or more of these, surely, ought to be our ‘new normal’ concerns.

But not for the government. Minister Michele Donelan assured us that quality and standards were being constantly monitored – by other people – as in her letter of 2 November to vice-chancellors:

“We have been clear throughout this pandemic that higher education providers must at all times maintain the quality of their tuition. If more teaching is moved online, providers must continue to comply with registration conditions relating to quality and standards. This means ensuring that courses provide a high-quality academic experience, students are supported and achieve good outcomes, and standards are protected. We have worked with the Office for Students who are regularly reviewing online tuition. We also expect students to continue to be supported and achieve good outcomes, and I would like to reiterate that standards must be maintained.”

So student health and the student experience are for the institutions to worry about, and get right, with the Office for Students watching. And higher education won’t need a bailout, unlike most other sectors of the market economy, because with standards being maintained there’s no reason for students not to enrol and pay fees exactly as usual. Institutional autonomy is vital, especially when it comes to apportioning the blame.

For government, the new normal was just the same as the old normal. It wasn’t difficult to read the signs. Ever since David Willetts, ministers had been complaining about low quality courses in universities. But with each successive minister the narrative became increasingly threadbare. David, now Lord, Willetts, at least had a superficially coherent argument: greater competition and informed student choice would drive up quality through competition between institutions for students. It was never convincing, but at least it had an answer to why and how quality and standards might be connected with competition in the HE market. Promoting competition by lowering barriers to entry for new HE providers was not a conspicuous success: some of the new providers proved to be a big problem for quality. Information, advice and guidance were key for improving student choice, so it seemed that the National Student Survey would play a significant part, along with university rankings and league tables. As successive ministers took up the charge the eggs were mostly transferred to the Teaching Excellence Framework basket, with TEF being championed by Jo, now Lord, Johnson. TEF began in 2016 and became a statutory requirement in the Higher Education and Research Act 2017, which also required TEF to be subject to an independent review. From the start TEF had been criticised as not actually being about teaching, or excellence, and the review by Dame Shirley Pearce, previously VC at Loughborough, began in 2018. Her review was completed before the end of 2019, but at the time of writing had still not been published.

However the ‘low quality courses’ narrative has just picked up speed. Admittedly it stuttered a little during the tenure of Chris Skidmore, who was twice briefly the universities minister, before and after Jo Johnson’s equally brief second tenure. The ‘Skidmore test’ suggested that any argument about low quality courses should specify at least one of the culprits, if it was not to be a low quality argument. However this was naturally unpopular with the narrative’s protagonists and Skidmore, having briefly been reinstalled as minister after Jo Johnson’s decision to step down, was replaced by Michele Donelan, who has remained resolutely on-message, even as any actual evidence of low quality receded even further from view. She announced in a speech to Universities UK at their September 2020 meeting that the once-praised NSS was now in the firing line: “There is a valid concern from some in the sector that good scores can more easily be achieved through dumbing down and spoon-feeding students, rather than pursuing high standards and embedding the subject knowledge and intellectual skills needed to succeed in the modern workplace. These concerns have been driven by both the survey’s current structure and its usage in developing sector league tables and rankings.”

UUK decided that they had to do something, so they ‘launched a crackdown’ (if you believe Camilla Turner in The Telegraph on 15 November 2020) by proposing, um, “a new charter aimed at ensuring institutions take a “consistent and transparent approach to identifying and improving potentially low value or low quality courses.” It’s doubtful if even UUK believed that would do the trick, and no-one else gave it much credence. But with the National Student Survey and even university league tables now deemed unreliable, and the TEF in deep freeze, the government urgently needed some policy-based evidence. It was time for this endlessly tricky problem to be dumped in the OfS in-tray. Thus it was that the OfS announced on 17 November 2020 that: “The Office for Students is consulting on its approach to regulating quality and standards in higher education. Since 2018, our focus has been on assessing providers seeking registration and we are considering whether and how we should develop our approach now that most providers are registered. This consultation is taking place at an early stage of policy development and we would like to hear your views on our proposals.”

Instant commentators were unimpressed. Were the OfS proposals on quality and standards good for the sector? Johnny Rich thought not, in his well-argued blog for the Engineering Professors’ Council on 23 November 2020, and David Kernohan provided some illustrative but comprehensive number-crunching in his Wonkhe blog on 30 November 2020: “Really, the courses ministers want to get rid of are the ones that make them cross. There’s no metric that is going to be able to find them – if you want to arbitrarily carve up the higher education sector you can’t use “following the science” as a justification.” Liz Morrish nailed it on her Academic Irregularities blog on 1 December 2020.

In the time-honoured way established by HEFCE, the OfS consultation was structured in a way which made it easy to summarise responses numerically, but much less easy to interpret their significance and their arguments. The core of the approach was a matrix of criteria, most of which all universities would expect to meet, but it included some ‘numerical baselines’, especially on something beyond the universities’ control – graduate progression to professional and managerial jobs. It also included a proposed baseline for drop-out rates. The danger of this was that it would point the finger at universities which do the most for disadvantaged groups, but here too government and OfS had a cunning plan. Nick Holland, the OfS Competition and Registration Manager, blogged on 2 December 2020 that the OfS would tackle “pockets of low quality higher education provision”, with the statement that “it is not acceptable for providers to use the proportion of students from disadvantaged backgrounds they have as an excuse for poor outcomes.” At a stroke universities with large proportions of disadvantaged students could either be blamed for high drop-out rates, or, if they reduced drop-out rates, they could be blamed for dropping standards. Lose-lose for the universities concerned, but win-win for the low quality courses narrative. The outrider to the low quality courses narrative was an attack on the 50% participation rate (in which Skidmore was equally culpable), which seemed hard to reconcile with a ‘levelling up’ narrative, but Michele Donelan did her best with her speech to NEON, of all audiences, calling for a new approach to social mobility, which seemed to add up to levelling up by keeping more people in FE. The shape of the baselines became clearer as OfS published Developing an understanding of projected rates of progression from entry to professional employment: methodology and results on 18 December 2020. After proper caveats about the experimental nature of the statistics, here came the indicator (and prospective baseline measure): “To derive the projected entry to professional employment measure presented here, the proportion of students projected to obtain a first degree at their original provider (also referred to as the ‘projected completion rate’) is multiplied by the proportion of Graduate Outcomes respondents in professional employment or any type of further study 15 months after completing their course (also referred to as the ‘professional employment or further study rate’).” This presumably met the government’s expectations by baking in all the non-quality-related advantages of selective universities in one number. Wonkhe’s David Kernohan despaired, on 18 December 2020, as the proposals deviated even further from anything that made sense: “Deep within the heart of the OfS data cube, a new plan is born. Trouble is, it isn’t very good.”

Is it too much to hope that OfS and government might actually look at the academic research on quality and standards in HE? Well, yes, but there is rather a lot of it. Quality in Higher Education is into its 26th year, and of course there is so much more. Even further back, in 1986 the SRHE Annual Conference theme was Standards and criteria in higher education, with an associated book edited by one of the founders of SRHE, Graeme Moodie (York). (This was the ‘Precedings’ – at that time the Society’s practice was to commission an edited volume in advance of the annual conference.) SRHE and the Carnegie Foundation subsequently sponsored a series of Anglo-American seminars on ‘Questions of Quality’. One of the seminar participants was SRHE member Tessa, now Baroness, Blackstone, who would later become the Minister for Further and Higher Education, and one of the visiting speakers for the Princeton seminar was Secretary of State for Education Kenneth Baker. At that time the Council for National Academic Awards was still functioning as the validating agency, assuring quality, for about half of the HE sector, with staff including such SRHE notables as Ron Barnett, John Brennan and Heather Eggins. When it was founded SRHE aimed to bring research and policy together; they have now drifted further apart. Less attention to peer review, but more ministers becoming peers.

Rob Cuthbert is Emeritus Professor of Higher Education Management, University of the West of England and Joint Managing Partner, Practical Academics


Leave a comment

The Impact of TEF

by George Brown

A report on the SRHE seminar The impact of the TEF on our understanding, recording and measurement of teaching excellence: implications for policy and practice

This seminar demonstrated that the neo-liberal policy and metrics of TEF (Teaching Excellence Framework) were not consonant with excellent teaching as usually understood.

Michael Tomlinson’s presentation was packed with analyses of the underlying policies of TEF. Tanya Lubicz-Nawrocka considered  the theme of students’ perceptions of excellent teaching. Her research demonstrated clearly that students’ views of excellent teaching were very different from those of TEF. Stephen Jones provided a vibrant analysis of public discourses. He pointed to the pre-TEF attacks on universities and staff by major conservative politicians and their supporters. These were to convince students and their parents that Government action was needed. TEF was born and with it the advent of US-style neo-liberalism and its consequences. His final slide suggested ways of combating TEF including promoting the broad purposes of HE teaching. Sal Jarvis succinctly summarised the seminar and took up the theme of purposes. Personal development and civic good were important purposes but were omitted from the TEF framework and metrics.

Like all good seminars, this seminar prompted memories, thoughts and questions during and after the seminar. A few of mine are listed below. Others may wish to add to them.

None of the research evidence supports the policies and metrics of TEF (eg Gibbs, 2018). The indictment of TEF by the Royal Statistics Society is still relevant (RSS, 2018). The chairman of the TEF panel is reported to have said “TEF was not supposed to be a “direct measure of teaching” but rather “a measure based on some [my italics] of the outcomes of teaching” On the continuum of neo-liberalism and collegiality, TEF is very close to the pole of neo-liberalism whereas student perspectives are nearer the pole of collegiality which embraces collaboration between staff and between staff and students. Collaboration will advance excellence in teaching: TEF will not. Collegiality has been shown to increase morale and reinforce academic values in staff and students (Bolden et al, 2012). Analyses of the underlying values of a metric are important because values shape policy, strategies and metrics. ‘Big data’ analysts need to consider ways of incorporating qualitative data. With regard to TEF policy and its metrics, the cautionary note attributed to Einstein is apposite: “Not everything that counts can be counted and not everything that is counted counts.”

SRHE member George Brown was Head of an Education Department in a College of Education and Senior Lecturer in Social Psychology of Education in the University of Ulster before becoming Professor of Higher Education at the University of Nottingham.  His 250 articles, reports and texts are mostly in Higher and Medical Education, with other work in primary and secondary education. He was senior author of Effective Teaching in Higher Education and Assessing Student Learning in Higher Education and co-founder of the British Education Research Journal, to which he was an early contributor and reviewer. He was the National Co-ordinator of Academic Staff Development for the Committee of Vice Chancellors and Principals (now Universities UK) and has served on SRHE Council.

References

Bolden, R et al (2012) Academic Leadership: changing conceptions, identities and experiences in UK higher education London: Leadership Foundation

Gibbs, G (2017) ‘Evidence does not support the rationale of the TEF’, Compass: Journal of Learning and Teaching, 10(2)

Royal Statistical Society  (2018) Royal Statistical Society: Response to the teaching excellence and student outcomes framework, subject-level consultation


Leave a comment

Beyond TEF Cynicism: Towards a New Vocabulary of ‘Excellence’?

By Steven Jones

One might expect that asking a room full of diverse stakeholders to discuss ‘teaching excellence’ would result in all kinds of quarrels and disagreement. In fact, the SRHE’s September 2019 event (The impact of the TEF on our understanding, recording and measurement of teaching excellence: implications for policy and practice) was a refreshingly convivial and creative affair.

Everyone present agreed that the TEF’s proxies for excellence were wholly inappropriate. In fact, there was surprisingly little discussion of existing metrics. We all felt that the consumerist language of ‘value for money’ and the instrumental lens of ‘employability’ were inadequate to capture the nuanced and complex ways in which curiosity can be sparked and orthodoxy challenged in the HE classroom.

Tanya Lubicz-Nawrocka spoke about the absence of the student voice in TEF policy, noting that conceptualisations of ‘excellence’ tend to overlook the very moral and critical components of transformative teaching that students value most highly. Michael Tomlinson drew attention to student-as-consumer framings within the ‘measured market’, noting the inevitability of institutional game-playing, status leveraging and brand promotion in such a relentlessly competitive environment. Both speakers suggested that students were misleadingly empowered, lacking the agency that policy discourses attribute to them.

I tried to push this idea further, beginning my talk by asking whether any lecturer had ever actually changed the way they teach because of government policy. There was broad agreement that while excellence frameworks influenced professional cultures and co-opted university managers, they barely touched academic practice. Lecturers know their own students – and know how to teach them – better than any White Paper.

But is the TEF actually about university teaching at all? Or do too many barriers sit between policy and practice for that to be a realistic aim? Policy enactment in HE is interrupted by institutional autonomy, by academic freedom and, increasingly, by lecturers’ professional identity. The TEF’s real purpose, I would argue, is more about manipulating the discourse. It manufactures a crisis, positioning intractable academics as the problem and students as the victim, thus allowing competition to come along and save the day.

Grade inflation is one area in which the contradictions of top-down policy discourse are laid bare. The market demands that lecturers mark students’ work generously (so that ‘value added’ columns in league tables don’t hold back institutional ranking). Then policymakers wade in, attacking institutions for artificial increases and threatening fines for those who persist. The logic is inconsistent and confused, but this matters little – the discourse persuades voters that their own hard-won education successes are being devalued by a sector overprotective of its ‘snowflake’ customer base.

TEF provider statements offer the opportunity for universities to fight back, but evidence suggests they’re bland and indistinct, tending towards formulaic language and offering little additional clarity to the applicant.

But despite such missed opportunities, 73 Collier Street was full of new ideas. Opposition to metrics wasn’t based on change-resistance and ideological stubbornness. Indeed, as respondent Sal Jarvis noted, we urgently need to measure, understand and close differential attainment gaps in many areas, such as ethnicity. But there was consensus that current proxies for ‘excellence’ were incomplete, and creative thoughts about how they could be complemented. What about capturing graduates’ long-term well-being instead of their short-term satisfaction? Or encouraging institutions to develop their own frameworks based on their specific mission and their students’ needs? How about structural incentives for collaboration rather than competition? And a focus on teaching processes, not teaching outcomes?

The argument that the TEF is less about changing pedagogies than manipulating wider discourses shouldn’t bring any comfort to the sector. I tried to show how the dominant logic of teaching excellence primes the sector for more fundamental policy shifts, such as for-profit providers receiving taxpayer subsidy on pedagogical grounds. One delegate spoke to me at the end of the event to offer another example, explaining how employability-minded managers within his institution were squeezing out critical engagement with cultural theory to allow for further skills-based, professional training. The TEF may not change practice directly, but it retains the power to nudge the sector away from its core public roles towards more privatised and instrumental practices.

The challenge for us is to articulate a confident and robust defence of all kinds of university teaching. We need to explain how our pedagogies bring lifelong gains both to our students and to wider society, even if initial encounters can be difficult and unsettling. Policy has taken us a long way down the market’s cul-de-sac, but what’s reassuring is that we’re now moving on from TEF-bashing towards a coherent counter-narrative. This event confirmed that universities have more meaningful things to crow about than their fleeting goldenness against a bunch of false proxies.

SRHE member Steven Jones is a Professor of Higher Education at the Manchester Institute of Education. His research focuses on equity issues around students’ access, experiences and outcomes. Steven is a Principal Fellow of the Higher Education Academy. He teaches on the University of Manchester’s PGCert in Higher Education. The view expressed here are his own.

Image of Rob Cuthbert


1 Comment

Post-Truth and the Higher Education and Research Bill

By Rob Cuthbert

The Higher Education and Research Bill (HERB) has begun its Committee stage in the House of Lords. With 500 amendments tabled for line-by-line scrutiny, six days were set aside through to 25 January 2017, but on the first day, 9 January, only one amendment was considered. It was however a pivotal proposal, about the nature and purpose of universities, with the rarity of being taken to a vote – the first time since 2012 (on a health bill) that there had been a vote at this stage in the Lords. Debate is likely to be both heated and confused, because the Bill embodies two key contradictions – between centralised control and free market forces, and between two very different appeals to legitimacy: emotion and personal belief, or evidence.

In HE the neoliberal tendency often gets the blame, but, as Paul Temple points out in this issue of News, neoliberalism is not easily reconciled with the centralising and controlling inclinations which are a key part of the Bill. Times journalist Matt Ridley departed from his usual science and environment beat to devote a column on 9 January 2017 to the Bill, headlined ‘Universities are being nationalised by stealth’.  As a hereditary peer Viscount Ridley was no doubt heading for the House of Lords for the Bill’s first day. The Bill is indeed ‘a Whitehall power grab’, as he argued.

So far, so easy to understand. Whitehall’s civil servants always want more control. But why would politicians enamoured of the market choose to go along with it? Continue reading

Ian Kinchin


Leave a comment

Student evaluation of teaching: are we reaching for the wrong type of excellence?

By Ian Kinchin

Over twenty years ago Carr (1994: 49) wrote:

 ‘It is a shallow and false view of education and teaching which takes it to be a matter of the technical transmission of pre-packaged knowledge and skills in the context of efficient management’

However, it seems that this false view is still able obscure a more contemporary and research-informed views of teaching. The on-going drive for ‘teaching excellence’ still seems to focus on actions of the teacher that promote Carr’s ‘shallow view’. That is not to say that the student voice is not important, but we need to ensure that students are asked the right questions so that we do not promote student passivity as learners and do not subvert the student voice for purely political ends.

Fitzgerald et al (2002) wrote Continue reading

Vicky Gunn


2 Comments

The latest TEF Assessment Framework, automated analysis of data, and some Scottish anxiety

By  Vicky Gunn

I had the good fortune to be in Rio for the Paralympics this September. My step-daughter was competing in an endurance road race. For her, the most important thing was improving on previous race times, but she’d hoped to get a medal as well (even though this was not predicted by her ‘metrics’). At the end of September, the Westminster Government through HEFCE, published the TEF2 Technical Specification[i] and I found, to my astonishment that the original differentiating phrases (meets expectations, excellent, outstanding) were to be replaced with medals: Bronze, Silver, Gold. All of this got me thinking about the Teaching Excellence Framework, built like British Cycling on the idea that we can differentiate excellence for competitive purposes and this is a good in itself. I find this comparison deeply troubling. I, like many involved in quality and teaching development in Scottish Higher Education, have invested several years of my professional life to fostering cultures of enhancement. Indeed, the distance travelled to improvement in teaching provision has been a mantra within the totality of Scottish higher education’s stakeholders (academic, government, student bodies alike). In the Quality Enhancement Framework (QEF), we have been more interested in seeing all Scottish institutions getting ‘personal bests’ (hence demonstrating continuous improvement from within their own context), rather than doing better than all the others (final outcome measure).

However, now that we have the first set of TEF indicative metrics, I (like a cycling coach) am assailed with a few doubts about the laudable concentration of raising the quality of the whole Scottish sector. This is an aim of the QEF. This resulted in engaged participants of a quality system which steadfastly refused the divisiveness associated with differentiated institutional quality review outcomes. Yet, if we individually enter it, the TEF will now demand this of us.  Should Scotland then change its QEF substantially with its aspirational collectivism to be consigned to being a phantasm of a previous era? How long can such a discourse last in the face of going for gold? Should I, as an institutional Head of Learning and Teaching, now focus on competing with HEIs, so my institution is seen as outstanding in comparison to all the others and place the sector’s aspirational culture in a box marked ‘soppy idealism’? To put it in British Cycling’s inelegant but superlatively economic phrasing: how will my small specialist institution medal when facing larger, wealthier institutions?  Continue reading