SRHE Blog

The Society for Research into Higher Education

Tag Archives: Assessment


by Leave a comment

Enhancing student learning through innovative scholarship

by Jenny Burnham, Matt Mears, Kennedy Offor, Jane Woodin and Tim Herrick

This blog is a report on the Enhancing Student Learning Through Innovative Scholarship (ESLTIS) conference held at the University of Sheffield in July 2026. The conference involved participants from 24 different universities and nearly 50 sessions.

When higher education faces real pressure, coming together to reimagine how we teach and learn is more than just professional development – it is a vital act of community. In July 2026, 97 educators, researchers, and student partners gathered at the University of Sheffield for two energising days at the Enhancing Student Learning Through Innovative Scholarship (ESLTIS) conference.

Across two inspiring keynotes, 39 talks and oral bites, and seven hands-on workshops, the atmosphere was filled with practical optimism, bold experimentation, and genuine connection. At a time when financial constraints weigh heavily across the sector, ESLTIS 2026 proved that an exceptional, high-impact academic gathering does not have to cost the earth. By keeping delegate fees as low as possible without compromising on quality, the conference demonstrated that transformative scholarship can remain accessible, inclusive, and deeply rewarding for everyone involved.

To capture the spirit of those two days, this blog brings together the key ideas, personal reflections, and practical insights that defined the event. We explore the overarching themes shaping the sector, followed by firsthand perspectives from our keynote speaker, insights from a session chair navigating AI in the curriculum, and the invaluable contributions of our student partners.

Key Conference Themes

With the tone of ‘Dealing with Disruption’ set by Professor Sam Nolan, the conference explored how learning and teaching in higher education is responding to and being shaped by the outside influences bringing both adversity and opportunity into the sector. A recurrent theme of innovations drawing on AI technology encompassed support for diverse student needs, its use in assessment, and as a teaching aid. The conference mirrored the acknowledgement of the evolving higher education demographic, with discussions covering inclusivity, decolonisation, engagement, identity, and self-worth, with structured gap-filling support, and breaking down of participation barriers, busting out of disciplinary silos to support students’ belonging and wellbeing. Wide-ranging talks covered: curriculum both hidden and carefully designed, strategies to augment students’ development of their skills and their readiness for their post-graduation world, and experiences of collaboration with and between students, staff, and professional services teams

Underpinning it all was the continuing ethos of the ESLTIS community that recognises and celebrates the education-focus of academia, exploring this with talks and workshops on academic identity – how it grows and develops, the things that shape it, and how it can be described and evidenced. The conference was not one thing for all delegates, it had something for everyone; areas with which we are familiar and aspects provoking thought and reflection, from which we could learn.  Yet again, I have attended a conference and come away with more than I expected. 

Reflections from a keynote speaker 

Dr Jenny Burnham is a Senior University Teacher in Chemistry at the University of Sheffield. Here she writes about her career and how she shared her experience at the conference.

I am interested in everything and my career highlights have been similarly eclectic. Rather than focus on a single speciality, I used myself as the common thread to weave a story that would be useful, relevant, or inspirational for a diverse audience of enthusiastic and accomplished educators. Taking inspiration from the conference name and conference themes (academic identity, community in learning and teaching, SoTL, supporting student success, and learning and teaching possibilities), my keynote made a virtue of the meanders of my academic career so far.  

My academic journey started in chemistry, shaping my relationship with information (evidence is key) and placing the primacy of experimental investigation at the centre of my disciplinary awareness. My early scholarship interests grew from this, exploring the development of inquiry skills through laboratory education. Spending a lot of time teaching students in the lab and enjoying their individuality shaped a desire to see them all succeed. This developed into an education philosophy that students are clever and capable with enormous potential, which permeates my work. An MEd drawing on what can be learned from sharing practice and insight (Rowland, 1999) transformed my approach to education. The benefits I felt in sharing my work with people who were interested motivated me to extend my networks. These have been a source of inspiration, opportunity and support, and allowed me to define my leadership identity; supporting, facilitating, promoting, and enabling collegial interaction and growth within my community.

My keynote aimed to provide a little of something for everyone, and evidence that I succeeded in this is captured in three different post-keynote interactions. I was thanked for owning my nerves: “it helps” she said. I had lunch with a PhD chemist considering an academic career; and I received an enquiry about a Learning Landscapes initiatives I worked on. There was something for me too: a final, unexpected result of my talk was that pulling it together reminded me of the cool stuff I have done.  Celebrating successes is definitely something I intend to do more of.  

Reflections from a session chair 

Dr Kennedy Offor was a member of the local organising committee and reflects on his experiences at ESLTIS in the role of session chair.

Chairing the ‘AI in the Curriculum’ session gave me with a unique perspective on how colleagues from different disciplines are responding to the rapid emergence of generative AI in higher education. Although the eight presentations ranged from institutional policy to engineering, mathematics, business, humanities, and language learning, a common thread emerged throughout the session – how learning, assessment, and curriculum design should evolve to ensure that students continue to develop critical judgement, disciplinary understanding, and professional responsibility. The discussion reflected a noticeable shift from reacting to AI as a disruptive technology towards designing learning environments that make purposeful and educationally meaningful use of it.

Laurie Wilson opened the session by introducing the University of Sheffield’s developing ‘Common Approach to AI in the Curriculum’, emphasising “harmonisation rather than homogenisation” and arguing that AI should support good disciplinary teaching rather than drive it. Joanne Irving-Walton then offered a thought-provoking perspective by examining the “gaps” where judgement develops, suggesting that AI increasingly participates in the uncertainty, rehearsal and reflection through which students and educators learn. 

Panos Kloukinas, Nick McCullen and Will Roberts built on this by arguing that mathematics and computation education should increasingly prioritise application, interpretation and problem formulation over procedural execution, while Dawn Whitton demonstrated how students could critically evaluate AI itself through a staged business assessment that required them not only to use AI, but also to critique its assumptions, biases and wider societal implications.

The engineering perspective was represented through our own presentation, which used longitudinal Graduate Teaching Assistant reflections to show how observations of student uncertainty, teamwork and AI use informed iterative redesign of an MSc engineering design module. Mireilla Bikanga Ada’s graduate study provided an interesting counterpoint by showing that, despite widespread AI adoption, teamwork remained the strongest predictor of graduates’ perceptions of degree relevance. 

Rachel Johnson illustrated how a simple, low-stakes classroom activity could encourage students critically to examine AI-generated information and its biases, while Franziska Collier demonstrated how AI avatars could provide authentic, low-risk opportunities for students to practise high-stakes language assessments.

The discussions reinforced my own view that the key educational question is no longer whether students should use AI, but how curricula can help them develop the professional judgement needed to use it responsibly. Rather than diminishing the role of educators, the session demonstrated that AI makes thoughtful curriculum design, assessment, and disciplinary expertise more important than ever, and highlighted an encouraging direction of travel for the Scholarship of Teaching and Learning community.

Students as Partners 

Dr. Jane Woodin was a member of the local organising committee and offers her thoughts on working with students as volunteer partners 

The conference ethos of inclusive dialogue was also evident in the organisation of student volunteers who were invited to participate in whatever they wished, from visiting and advising on the venue beforehand, supporting registration and general enquiries, and co-chairing sessions together with a staff volunteer. Responses from the eight volunteers ranged from tentative interest in helping out to requests to ‘have a go at everything’, and others who did the minimum in terms of active volunteering because they found the topics of the talk so interesting and important.

It was heartening to hear one volunteer offer to sit at the front of the large auditorium to oversee a presentation session, handling the Q &A masterfully, including interventions from the co-chair. Another volunteer started out shyly offering to help at the registration desk, but seeing fellow students taking the chairing role, threw herself into it enthusiastically. 

These experiences remind us how the processes surrounding education are as important as the education itself – or perhaps are or should be inseparable, as the ‘way that you do it’ melds seamlessly into ‘what you do’; offering an example of SoTL in action at its best.  

Final Reflections

Dr Matt Mears is a Senior University Teacher in Physics at the University of Sheffield, and was chair of the Local Organising Committee for ESLTIS2026

As Conference Chair, reflecting on the energy across those two days fills me with immense pride and a distinct sense of gratitude. While hosting an event of this scale undoubtedly takes work, the dividend it pays back to the host team and the wider community is extraordinary. Watching colleagues reconnect, draw genuine inspiration from such a rich variety of presenters, and spark new collaborations in real time reminded me why these gatherings are so vital for our sector.

If you have ever toyed with the idea of organising a conference, or perhaps bringing ESLTIS to your own institution, I cannot encourage you enough to take the leap. The effort involved is far outweighed by the connection, camaraderie, and fresh momentum it generates. Whether as an attender, a presenter, or a future host, I look forward to seeing where our community goes next.


Dr Jenny Burnham is a chemistry university teaching specialist with 20 years’ experience teaching and learning about education at the University of Sheffield. 

Dr Matt Mears is a Senior University Teacher in Physics in the School of Mathematical and Physical Sciences at the University of Sheffield. His research focuses on inclusive education practices and laboratory-based teaching.

Dr Kennedy Offor is an engineering educator and researcher with experience teaching undergraduate and postgraduate engineering at the University of Sheffield’s School of Electrical and Electronic Engineering and, more recently, apprenticeship and technical engineering education at the AMRC Training Centre.  

Dr. Jane Woodin is a Senior University Teacher in the School of Languages, Arts and Societies at the University of Sheffield. She has taught and researched in intercultural communication, focusing on peer interaction and dialogue approaches.

Dr Tim Herrick is a Senior University Teacher in the School of Education, and a National Teaching Fellow. He was part of the ESLTIS conference committee and helped co-ordinate this blogpost.

Reference Rowland, S (1999) ‘The role of theory in a pedagogical model for lecturers in higher education’ Studies in Higher Education, 24(3), 303–314. https://doi.org/10.1080/03075079912331379915


by Leave a comment

When assessment goes local, who keeps what students bring back?

by Carlene Kyeremeh

A student sent me a photograph of a handwritten waybill. Dated November 2022, it records a unit price for printed sachet film. It also carries the names of a supplier and a customer and the given name of the person who issued it. The waybill appeared in a term paper on fiscal consolidation as evidence of a cost increase that no statistical series I have found records.

It is in my files. Not the department’s. Mine.

I describe the waybill here but do not reproduce it, identify those named, or give the figure it records. The students who wrote the paper agreed to this account. I have no record that the supplier, the customer or the person named agreed to the document’s retention or wider use, so I have minimised the identifying detail, including the price, which is the most commercially sensitive thing on the page.

I have argued elsewhere that the curriculum can help to thicken a thin documentary record. Ask each student to obtain one item of locally sourced evidence: an interview, a quoted price, a document from a firm or public office, and, over successive cohorts, a course can assemble material that no database previously held. This is no longer only my ambition. Programmes are already redesigning assessment in response to generative AI, and one attractive move is to require evidence a model cannot retrieve: a price obtained in a market, an interview conducted locally, a document held by a firm or public office. That redesign brings third-party material into course files at scale, often without a parallel decision about what should happen to it after assessment.

That is the promise. The waybill in my files is the problem that follows it.

What universities already know how to govern

Research ethics and commercial confidentiality are not routinely ignored in student work. Professionally oriented programmes have long had to decide what an organisation may disclose, who may see it and when it must be returned. For example, McMaster University’s course-based research guidance explicitly covers undergraduate, MBA and graduate courses. The University of Toronto provides both a course-based ethics route and a classroom confidentiality agreement for third-party proprietary information. In the UK, the University of Manchester’s ethics guidance asks researchers to establish first whether an activity is research training conducted as coursework, described as collecting and analysing a small amount of data to gain experience in method, or research proper. These arrangements show that collection, consent, confidentiality and use within a defined project can be governed. My argument is therefore not that universities lack such protections. It begins where a normal project would end.

The problem begins when the course ends

If material is returned, destroyed or retained only with the assessed work, its lifecycle has a clear end. But the design I am proposing depends on the evidence outliving the assignment. A document collected by one student in 2026 should be available to another cohort in 2028 or 2029, alongside prices, interviews and documents gathered in the intervening years. The course is not only assessing students. It is building a local source collection.

That changes the questions. What exactly does the university hold after the assessment period? Who is its custodian? How long may it be retained? May another cohort consult it? What fresh permission is needed if it is later used for research or publication? These procedures can help answer those questions, but not automatically. An agreement that properly confines information to one course may rule out the very reuse on which the proposed archive depends.

The evidence is not the assignment

The student’s analysis is coursework. The material used to produce it is not necessarily the same thing. A waybill may contain personal or commercially sensitive information. Interview notes are harder still: they are neither the source’s own record nor the student’s analysis, but one person’s written account of what another person said, made for an assignment and now held by a lecturer.

Where the activity is classified as teaching rather than research, this third-party material can still raise a lifecycle question: do institutional rules cover its custody, retention and reuse after the assessment for which it was collected? The Manchester distinction is instructive here. A course collection built to outlive its cohorts is not a small amount of data gathered to practise a method, but neither does it fit neatly within research as that definition intends. It is a lifecycle problem created by a particular curricular ambition.

The reuse is the point

Ghana’s Data Protection Act, 2012 (Act 843) requires personal data to be collected for a specific, explicitly defined and lawful purpose, limits retention and requires further processing to be compatible with the original purpose. Material obtained for one student’s term paper and later made available to another cohort may involve further processing. Nothing in the waybill I received establishes permission for it to become part of a standing course collection.

The Act contains provisions for historical, statistical and research uses where the relevant conditions are met. Those provisions offer a governed route, not a retrospective blank cheque. They do not turn a lecturer’s folder into an institutional archive. The underlying principle is not unique to Ghana: UK guidance likewise treats purpose, storage and research-related reuse as connected but distinct questions.

Ordinary assignment governance is adequate for ordinary assignments. The problem appears when coursework is asked to do something additional: to enlarge a locally available documentary record over time.

The archival value and the governance risk arise from the same feature. An archive that cannot be consulted next year is not an archive. A collection that can be consulted next year without the source having been told is not a defensible one.

A small institutional answer

The answer need not be a new bureaucracy. It could begin with a short institutional instrument that separates use in a student’s assessment from retention in a course source collection. Before collection begins, it should state the teaching purpose, what may be collected, who may later consult it, how long it will be kept, which unit is responsible and what choices the source has. Research use or publication should require a separate decision rather than being hidden inside the phrase ‘educational purposes’.

The remaining requirement is custody. What one lecturer can require, only an institution can keep. A named custodian, controlled access, a retention and disposal schedule, and a record of what the course has produced would convert a personal folder into an accountable institutional practice. Existing MBA, ethics and confidentiality processes provide models; the task is to extend their discipline to the intended afterlife of the evidence.

Proportionality matters. If the process becomes heavier than the assessment it supports, colleagues may stop asking students to gather local evidence. But a department unable to provide a clear collection notice, a retention rule and somebody accountable for what returns should not require the fieldwork. Otherwise, institutional risk is transferred to students and to the businesses that agree to speak with them.

Readers can test this in ten minutes. Take your own institution’s course-based ethics route or classroom confidentiality agreement and look for two things: whether either says anything about third-party material after the assessment period, and who is named as responsible for it. If neither appears, ask where else the institution governs the evidence after the assignment ends. If there is no answer, the arrangements may govern the assignment without governing its evidentiary afterlife.

Before we ask for more

The curriculum can enlarge the archive. The qualification is that the archival ambition creates an additional governance obligation. The question is not only whether the evidence was collected ethically and kept confidentially. It is whether its intended custody, retention and reuse were made explicit before collection began.

I have a waybill in my files and no good institutional answer to those questions. That is not an argument against asking students to go and find out. It is an argument for deciding what we will do with what comes back, before we ask them to go.

Dr Carlene Kyeremeh is Associate Professor and Vice President, University Advancement, Recruitment and Research at All Nations University, Koforidua, Ghana. Her research addresses curriculum decolonisation, higher education policy in Africa and the Caribbean, and the political economy of educational technology.This post develops a governance question raised in Issues 14 and 15 of The Decolonized Curriculum, her LinkedIn newsletter.


by Leave a comment

Walk on by: the dilemma of the blind eye

by Dennis Sherwood

Forty years on…

I don’t remember much about my experiences at work some forty-odd years ago, but one event I recall vividly is the discussion provoked by a case study at a training event. The case was simple, just a few lines:

Sam was working late one evening, and happened to walk past Pat’s office. The door was closed, but Sam could hear Pat being very abusive to Alex. Some ten minutes later, Sam saw Alex sobbing.

What might Sam do?

What should Sam do?

Quite a few in the group said “nothing”, on the grounds that whatever was going on was none of Sam’s business. Maybe Pat had good grounds to be angry with Alex and if the local culture was, let’s say, harsh, what’s the problem? Nor was there any evidence that Alex’s sobbing was connected with Pat – perhaps something else had happened in the intervening time.

Others thought that the least could Sam do was to ask if Alex was OK, and offer some comfort – a suggestion countered by the “it’s a tough world” brigade.

The central theme of the conversation was then all about culture. Suppose the culture was supportive and caring. Pat’s behaviour would be out of order, even if Pat was angry, and even if Alex had done something Pat had regarded as wrong.

So what might – and indeed should – Sam do?

Should Sam should confront Pat? Or inform Pat’s boss?

What if Sam is Pat’s boss? In that case then, yes, Sam should confront Pat: failure to do so would condone bad behaviour, which in this culture, would be a ‘bad thing’.

But if Sam is not Pat’s boss, things are much more tricky. If Sam is subordinate to Pat, confrontation is hardly possible. And informing Pat’s boss could be interpreted as snitching or trouble-making. Another possibility is that Sam and Pat are peers, giving Sam ‘the right’ to confront Pat – but only if peer-to-peer honesty and mutual pressure is ‘allowed’. Which it might not be, for many, even benign, cultures are in reality networks of mutual ‘non-aggression treaties’, in which ‘peers’ are monarchs in their own realms – so Sam might deliberately choose to turn a blind eye to whatever Pat might be doing, for fear of setting a precedent that would allow Pat, or indeed Ali or Chris, to poke their noses into Sam’s own domain.

And if Sam is in a different part of the organisation – or indeed from another organisation altogether – then maybe Sam’s safest action is back where we started. To do nothing. To walk on by.

Sam is a witness to Pat’s bad behaviour. Does the choice to ‘walk on by’ make Sam complicit too, albeit at arm’s length?

I’ve always thought that this case study, and its implications, are powerful – which is probably why I’ve remembered it over so long a time.

The truth about GCSE, AS and A level grades in England

I mention it here because it is relevant to the main theme of this blog – a theme that, if you read it, makes you a witness too. Not, of course, to ‘Pat’s’ bad behaviour, but to another circumstance which, in my opinion, is a great injustice doing harm to many people – an injustice that ‘Pat’ has got away with for many years now, not only because ‘Pat’s peers’ have turned a blind eye – and a deaf ear too – but also because all others who have known about it have chosen to ‘walk on by’.

The injustice of which I speak is the fact that about one GCSE, AS and A level grade in every four, as awarded in England, is wrong, and has been wrong for years. Not only that: in addition, the rules for appeals do not allow these wrong grades to be discovered and corrected. So the wrong grades last for ever, as does the damage they do.

To make that real, in August 2025, some 6.5 million grades were awarded, of which around 1.6 million were wrong, with no appeal. That’s an average of about one wrong grade ‘awarded’ to every candidate in the land.

Perhaps you already knew all that. But if you didn’t, you do now. As a consequence, like Sam in that case study, you are a witness to wrong-doing.

It’s important, of course, that you trust the evidence. The prime source is Ofqual’s November 2018 report, Marking Consistency Metrics – An update, which presents the results of an extensive research project in which very large numbers of GCSE, AS and A level scripts were in essence marked twice – once by an ‘assistant’ examiner (as happens in ‘ordinary’ marking each year), and again by a subject senior examiner, whose academic judgement is the ultimate authority, and whose mark, and hence grade, is deemed ‘definitive’, the arbiter of ‘right’.

Each script therefore had two marks and two grades, enabling those grades to be compared. If they were the same, then the ‘assistant’ examiner’s grade – the grade that is on the candidate’s certificate – corresponds to the senior examiner’s ‘definitive’ grade, and is therefore ‘right’; if the two grades are different, then the assistant examiner’s grade is necessarily ‘non-definitive’, or, in plain English, wrong.

You might have thought that the number of ‘non-definitive’/wrong grades would be small and randomly distributed across subjects. In fact, the key results are shown on page 21 of Ofqual’s report as Figure 12, reproduced here:

Figure 1: Reproduction of Ofqual’s evidence concerning the reliability of school exam grades

To interpret this chart, I refer to this extract from the report’s Executive Summary:

The probability of receiving the ‘definitive’ qualification grade varies by qualification and subject, from 0.96 (a mathematics qualification) to 0.52 (an English language and literature qualification).

This states that 96% of Maths grades (all varieties, at all levels), as awarded, are ‘definitive’/right, as are 52% of those for Combined English Language and Literature (a subject available only at A level). Accordingly, by implication, 4% of Maths grades, and 48% of English Language and Literature grades, are ‘non-definitive’/wrong. Maths grades, as awarded, can therefore be regarded as 96% reliable; English Language and Literature grades as 52% reliable.

Scrutiny of the chart will show that the heavy black line in the upper blue box for Maths maps onto about 0.96 on the horizontal axis; the equivalent line for English Language and Literature maps onto 0.56. The measures of the reliability of the grades for each of the other subjects are designated similarly. Ofqual’s report does not give any further numbers, but Table 1 shows my estimates from Ofqual’s Figure 12:

 Probability of
 ‘Definitive’ grade‘Non-definitive’ grade
Maths (all varieties)96%4%
Chemistry92%8%
Physics88%12%
Biology85%15%
Psychology78%22%
Economics74%26%
Religious Studies66%34%
Business Studies66%34%
Geography65%35%
Sociology63%37%
English Language61%39%
English Literature58%42%
History56%44%
Combined English Language and Literature (A level only)52%48%

Table 1: My estimates of the reliability of school exam grades, as inferred from measurements of Ofqual’s Figure 12.

Ofqual’s report does not present any corresponding information for each of GCSE, AS or A level separately, nor any analysis by exam board. Also absent is a measure of the all-subject overall average. Given, however, the maximum value of 96%, and the minimum of 52%, the average is likely to be somewhere in the middle, say, in the seventies; in fact, if each subject is weighted by its cohort, the resulting average over the 14 subjects shown is about 74%. Furthermore, if other subjects – such as French, Spanish, Computing, Art… – are taken into consideration, the overall average is most unlikely to be greater than 82% or less than 66%, suggesting that an overall average reliability of 75% for all subjects is a reasonable estimate.

That’s the evidence that, across all subjects and levels, about 75% of grades, as awarded, are ‘definitive’/right and 25% – one in four – are ‘non-definitive’/wrong – evidence that has been in the public domain since 2018. But evidence that has been much disputed by those with vested interests.

Ofqual’s results are readily explained. We all know that different examiners can, legitimately, give the same answer (slightly) different marks. As a result, the script’s total mark might lie on different sides of a grade boundary, depending on who did the marking. Only one grade, however, is ‘definitive’.

Importantly, there are no errors in the marking studied by Ofqual – in fact, Ofqual’s report mentions ‘marking error’ just once, and then in a rather different context. All the grading discrepancies measured in Ofqual’s research are therefore attributable solely to legitimate differences in academic opinion. And since the range of legitimate marks is far narrower in subjects such as Maths and Physics, as compared to English Literature and History, then the probability that an ‘assistant’ examiner’s legitimate mark might result in a ‘non-definitive’ grade will be much higher for, say, History as compared to Physics. Hence the sequence of subjects in Ofqual’s Figure 12.

As regards appeals, in 2016, Ofqual – in full knowledge of the results of this research (see paragraph 28 of this Ofqual Board Paper, dated 18 November 2015) – changed the rules, requiring that a grade can be changed only if a ‘review of marking’ discovers a ‘marking error’. To quote an Ofqual ‘news item’ of 26 May 2016:

Exam boards must tell examiners who review results that they should not change marks unless there is a clear marking error. …It is not fair to allow some students to have a second bite of the cherry by giving them a higher mark on review, when the first mark was perfectly appropriate. This undermines the hard work and professionalism of markers, most of whom are teachers themselves. These changes will mean a level-playing field for all students and help to improve public confidence in the marking system.

This assumes that the legitimate marks given by different examiners are all equally “appropriate”, and identical in every way.

This assumption. however, is false: if one of those marks corresponds to the ‘definitive’ grade, and another to a ‘non-definitive’ grade, they are not identical at all. Furthermore, as already mentioned, there is hardly any mention of marking errors in Ofqual’s November 2018 report. All the grade discrepancies they identified can therefore only be attributable to legitimate differences in academic opinion, and so cannot be discovered and corrected by the rules that have been in place since 2016.

Over to you…

So, back to that case study.

Having read this far, like Sam, you have knowledge of wrong-doing – not Pat tearing a strip off Alex, but Ofqual awarding some 1.5 million wrong grades every year. All with no right of appeal.

What are you going to do?

You’re probably thinking something like, “Nothing”, “It’s not my job”, “It’s not my problem”, “I’m in no position to do anything, even if I wanted to”.

All of which I understand. No, it’s certainly not your job. And it’s not your problem directly, in that it’s not you being awarded the wrong grade. But it might be your problem indirectly – if you are involved with admissions, and if grades play a material role, you may be accepting a student who is not fully qualified (in that the grade on the certificate might be too high), or – perhaps worse – rejecting a student who is (in that the grade on the certificate is too low). Just to make that last point real, about one candidate in every six with a certificate showing AAA for A level Physics, Chemistry and Biology in fact truly merited at least one B. If such a candidate took a place at Med School, for example, not only is that candidate under-qualified, but a place has also been denied to a candidate with a certificate showing AAB but who merited AAA.

And although you, as an individual, are indeed not is a position to do anything about it, you, collectively, surely are.

HE is, by far, the largest and most important user of A levels. And relying on a ‘product’ that is only about 75% reliable. HE, collectively, could put significant pressure on Ofqual to fix this, if only by printing “OFQUAL WARNING: THE GRADES ON THIS CERTIFICATE ARE ONLY RELIABLE, AT BEST, TO ONE GRADE EITHER WAY” on every certificate – not my statement, but one made by Ofqual’s then Chief Regulator, Dame Glenys Stacey, in evidence to the 2 September 2020 hearing of the Education Select Committee, and in essence equivalent to the fact that about one grade in four is wrong. That would ensure that everyone is aware of the fact that any decision, based on a grade as shown on a certificate, is intrinsically unsafe.

But this – or some other solution – can happen only if your institution, along with others, were to act accordingly. And that can happen only if you, and your colleagues, band together to influence your department, your faculty, your institution.

Yes, that is a bother. Yes, you do have other urgent things to do.

If you do nothing, nothing will happen.

But if you take action, you can make a difference.

Don’t just walk on by.

Dennis Sherwood is a management consultant with a particular interest in organisational cultures, creativity and systems thinking. Over the last several years, Dennis has also been an active campaigner for the delivery of reliable GCSE, AS and A level grades. If you enjoyed this, you might also like https://srheblog.com/tag/sherwood/.


by 1 Comment

Becoming a professional services researcher in HE – making the train tracks converge

by Charlotte Verney

This blog builds on my presentation at the BERA ECR Conference 2024: at crossroads of becoming. It represents my personal reflections of working in UK higher education (HE) professional services roles and simultaneously gaining research experience through a Masters and Professional Doctorate in Education (EdD).

Professional service roles within UK HE include recognised professionals from other industries (eg human resources, finance, IT) and HE-specific roles such as academic quality, research support and student administration. Unlike academic staff, professional services staff are not typically required, or expected, to undertake research, yet many do. My own experience spans roles within six universities over 18 years delivering administration and policy that supports learning, teaching and students.

Traversing two tracks

In 2016, at an SRHE Newer Researchers event, I was asked to identify a metaphor to reflect my experience as a practitioner researcher. I chose this image of two train tracks as I have often felt that I have been on two development tracks simultaneously –  one building professional experience and expertise, the other developing research skills and experience. These tracks ran in parallel, but never at the same pace, occasionally meeting on a shared project or assignment, and then continuing on their separate routes. I use this metaphor to share my experiences, and three phases, of becoming a professional services researcher.

Becoming research-informed: accelerating and expanding my professional track

The first phase was filled with opportunities; on my professional track I gained a breadth of experience, a toolkit of management and leadership skills, a portfolio of successful projects and built a strong network through professional associations (eg AHEP). After three years, I started my research track with a masters in international higher education. Studying felt separate to my day job in academic quality and policy, but the assignments gave me opportunities to bring the tracks together, using research and theory to inform my practice – for example, exploring theoretical literature underpinning approaches to assessment whilst my institution was revising its own approach to assessing resits. I felt like a research-informed professional, and this positively impacted my professional work, accelerating and expanding my experience.

Becoming a doctoral researcher: long distance, slow speed

The second phase was more challenging. My doctoral journey was long, taking 9 years with two breaks. Like many part-time doctoral students, I struggled with balance and support, with unexpected personal and professional pressures, and I found it unsettling to simultaneously be an expert in my professional context yet a novice in research. I feared failure, and damaging my professional credibility as I found my voice in a research space.

What kept me going, balancing the two tracks, was building my own research support network and my researcher identity. Some of the ways I did this was through zoom calls with EdD peers for moral support, joining the Society for Research into Higher Education to find my place in the research field, and joining the editorial team of a practitioner journal to build my confidence in academic writing.

Becoming a professional services researcher: making the tracks converge

Having completed my doctorate in 2022, I’m now actively trying to bring my professional and research tracks together. Without a roadmap, I’ve started in my comfort-zone, sharing my doctoral research in ‘safe’ policy and practitioner spaces, where I thought my findings could have the biggest impact. I collaborated with EdD peers to tackle the daunting task of publishing my first article. I’ve drawn on my existing professional networks (ARC, JISC, QAA) to establish new research initiatives related to my current practice in managing assessment. I’ve made connections with fellow professional services researchers along my journey, and have established an online network  to bring us together.

Key takeaways for professional services researchers

Bringing my professional experience and research tracks together has not been without challenges, but I am really positive about my journey so far, and for the potential impact professional services researchers could have on policy and practice in higher education. If you are on your own journey of becoming a professional services researcher, my advice is:

  • Make time for activities that build your research identity
  • Find collaborators and a community
  • Use your professional experience and networks
  • It’s challenging, but rewarding, so keep going!

Charlotte Verney is Head of Assessment at the University of Bristol. Charlotte is an early career researcher in higher education research and a leader in within higher education professional services. Her primary research interests are in the changing nature of administrative work within universities, using research approaches to solve professional problems in higher education management, and using creative and collaborative approaches to research. Charlotte advocates for making the academic research space more inclusive for early career and professional services researchers. She is co-convenor of the SRHE Newer Researchers Network and has established an online network for higher education professional services staff engaged with research.


by Leave a comment

Gaps in sustainability literacy in non-STEM higher education programmes

by Erika Kalocsányiová and Rania Hassan

Promoting sustainability literacy in higher education is crucial for deepening students’ pro-environmental behaviour and mindset (Buckler & Creech, 2014; UNESCO, 1997), while also fostering social transformation by embedding sustainability at the core of the student experience. In 2022, our group received an SRHE Scoping Award to synthesise the literature on the development, teaching, and assessment of sustainability literacy in non-STEM higher education programmes. We conducted a multilingual systematic review of post-2010 publications from the European Higher Education Area (EHEA), with the results summarised in Kalocsányiová et al (2024).

Out of 6,161 articles that we identified as potentially relevant, 92 studies met the inclusion criteria and are reviewed in the report. These studies involved a total of 11,790 participants and assessed 9,992 university programmes and courses. Our results suggest a significant growth in research interest in sustainability in non-STEM fields since 2017, with 75 studies published compared to just 17 in the preceding seven years. Our analysis also showed that Spain, the United Kingdom, Germany, Turkey, and Austria had the highest concentration of publications, with 25 EHEA countries represented in total. The 92 reviewed studies were characterised by high methodological diversity: nearly half employed quantitative methods (47%), followed by qualitative studies (40%) and mixed methods research (13%). Curriculum assessments using quantitative content analysis of degree and course descriptors were among the most common study types, followed by surveys and intervention or pilot studies. Curriculum assessments provided a systematic way to evaluate the presence or absence of sustainability concepts within curricula at both single HE institutions and in comparative frameworks. However, they often captured only surface-level indications of sustainability integration into undergraduate and postgraduate programmes, without providing evidence on actual implementation and/or the effectiveness of different initiatives. Qualitative methods, including descriptive case studies and interviews that focused on barriers, challenges, implementation strategies, and the acceptability of new sustainability literacy initiatives, made up 40% of the current research. Mixed methods studies accounted for 13% of the reviewed articles, often applying multiple assessment tools simultaneously, including quantitative sustainability competency assessment instruments combined with open-ended interviews or learning journals.

In terms of disciplines, Economics, Business, and Administrative Studies held the largest share of reviewed studies (26%), followed by Education (23%). Multiple disciplines accounted for 22% of the reviewed publications, reflecting the interconnected nature of sustainability. Finance and Accounting contributed only 6%, indicating a need for further research. Similarly, Language and Linguistics, Mass Communication and Documentation, and Social Sciences collectively represented only 12% of the reviewed studies. Creative Arts and Design with just 2% was also a niche area. Although caution should be exercised when drawing conclusions from these results, they highlight the need for more research within the underrepresented disciplines. This in turn can help promote awareness among non-STEM students, stimulate ethical discussions on the cultural dimensions of sustainability, and encourage creative solutions through interdisciplinary dialogue.

Regarding factors and themes explored, the studies focused primarily on the acquisition of sustainability knowledge and competencies (27%), curriculum assessment (23%), challenges and barriers to sustainability integration (10%), implementation and evaluation research (10%), changes in students’ mindset (9%), key competences in sustainability literacy (5%), and active student participation in Education for Sustainable Development (5%). In terms of studies discussing acquisition processes, key focus areas included the teaching of Sustainable Development Goals, awareness of macro-sustainability trends, and knowledge of local sustainability issues. Studies on sustainability competencies focussed on systems thinking, critical thinking, problem-solving skills, ethical awareness, interdisciplinary knowledge, global awareness and citizenship, communication skills, and action-oriented mindset. These competencies and knowledge, which are generally considered crucial for addressing the multifaceted challenges of sustainability (Wiek et al., 2011), were often introduced to non-STEM students through stand-alone lectures, workshops, or pilot studies involving new cross-disciplinary curricula.

Our review also highlighted a broad range of pedagogical approaches adopted for sustainability teaching and learning within non-STEM disciplines. These covered case and project-based learning, experiential learning methods, problem-based learning, collaborative learning, reflection groups, pedagogical dialogue, flipped classroom approaches, game-based learning, and service learning. While there is strong research interest in the documentation and implementation of these pedagogical approaches, few studies have so far attempted to assess learning outcomes, particularly regarding discipline-specific sustainability expertise and real-world problem-solving skills.

Many of the reviewed studies relied on single-method approaches, meaning valuable insights into sustainability-focused teaching and learning may have been missed. For instance, studies often failed to capture the complexities surrounding sustainability integration into non-STEM programs, either by presenting positivist results that require further contextualisation or by offering rich context limited to a single course or study group, which cannot be generalised. The assessment tools currently used also seemed to lack consistency, making it difficult to compare outcomes across programmes and institutions to promote best practices. More robust evaluation designs, such as longitudinal studies, controlled intervention studies, and mixed methods approaches (Gopalan et al, 2020; Ponce & Pagán-Maldonado, 2015), are needed to explore and demonstrate the pedagogical effectiveness of various sustainability literacy initiatives in non-STEM disciplines and their impact on student outcomes and societal change.

In summary, our review suggests good progress in integrating sustainability knowledge and competencies into some core non-STEM disciplines, while also highlighting gaps. Based on the results we have formulated some questions that may help steer future research:

  • Are there systemic barriers hindering the integration of sustainability themes, challenges and competencies into specific non-STEM fields?
  • Are certain disciplines receiving disproportionate research attention at the expense of others?
  • How do different pedagogical approaches compare in terms of effectiveness for fostering sustainability literacy in and across HE fields?
  • What new educational practices are emerging, and how can we fairly assess them and evidence their benefits for students and the environment?

We also would like to encourage other researchers to engage with knowledge produced in a variety of languages and educational contexts. The multilingual search and screening strategy implemented in our review enabled us to identify and retrieve evidence from 25 EHEA countries and 24 non-English publications. If reviews of education research remain monolingual (English-only), important findings and insights will go unnoticed hindering knowledge exchange, creativity, and innovation in HE.

Dr. Erika Kalocsányiová is a Senior Research Fellow with the Institute for Lifecourse Development at the University of Greenwich, with research centering on public health and sustainability communication, migration and multilingualism, refugee integration, and the implications of these areas for higher education policies.

Rania Hassan is a PhD student and a research assistant at the University of Greenwich. Her research centres on exploring enterprise development activities within emerging economies. As a multidisciplinary and interdisciplinary researcher, Rania is passionate about advancing academia and promoting knowledge exchange in higher education.


by Leave a comment

Promoting equity and employability using live briefs

by Lucy Cokes

‘Live briefs’ are used in Higher Education programmes, and I suggest that they can help promote equity and employability if they are used in very specific ways. The use of live briefs takes place not only in the creative industries, but also across more practical or core subjects in HE. and has many parallels with a wide range of other teaching tools.

Live Briefs have been part of my teaching to students on the Creative Advertising degrees at Falmouth University for the past 10 years, with the last four years using live paid briefs as part of assessment. Done right I passionately believe that live briefs, with their ability to test students through an authentic task, develop creative problem-solving skills, and in turn, enhancing student satisfaction, are a valuable tool.

How live briefs are usually used

A live brief is defined as “a type of design project that is distinct from a typical studio project in its engagement of real clients or users, in real-time settings” (Sara, 2006, p. 1). Often, lecturers believe they are assigning ‘live briefs’, but frequently these are merely ‘simulations’ or ‘mock briefs’ using either outdated, or fictional client briefs which lack a genuine and immediate client need. Distant cousins of live briefs include the use of case studies in teaching, or the use of authentic tasks. However, I believe that the use of a live brief should be the unrivalled method to enhancing students’ employability skills and prospects at university in comparison to these other approaches. Typically, live briefs are sourced through lecturers’ professional networks and are presented to students most frequently as an extracurricular opportunity. These opportunities have often resulted in students securing paid or unpaid placements at agencies or being offered full-time positions post-graduation. By not fully embedding these live opportunities into assessments, there is an inadvertent disadvantage to those already disadvantaged.

How live briefs could be used

Live briefs can be, with effort, integrated into the students’ assessment brief for their modules. Students are often asked to deliver a pitch to the client as part of their assessment with one of the ideas chosen by the client. The winning students should ideally be paid for their time, with full guidance from tutors acting to provide feedback and project manage the process. Course leaders need to use caution when explicitly stating a particular module will contain live paid briefs, as they are often hard to come by. Instead, it is suggested that modules be designed in such a way they can be ‘plugged in’ when accessible.

There are many challenges in using live briefs, these include:

  • Planning in good time prior to start of a module.
  • The need to fit timings with pre-established assessment deadlines.
  • Additional time required for lectures to source the live briefs and manage the ‘clients’.
  • Potential administration constraints with invoicing ‘clients’, paying students and suppliers.

Live briefs seem particularly well-suited to non-profits, small businesses, or government agencies.  Experience has shown that these types of organisations tend so see the partnership with a university and students to be more cost effective, providing social benefits, whilst also being able to be more flexible around the module deadlines.

Organisations benefit from bringing their projects to the university, as they gain a dedicated fresh set of minds working on their problem. The same clients often come back year after year. Chris Thompson of Safer Futures shared that he “…thought the standard, confidence and professionalism of the student pitches and research was exceptional.”

The hierarchy of live briefs has been produced to assist lecturers in deciding how best to use live briefs in their teaching and push for the gold star of having paid opportunities embedded into assessment.  This is a call for a shift in culture and attitudes toward the use of live briefs, so we are not inadvertently decreasing social mobility in the UK through their use.

Live Brief Hierarchy

The hierarchy has been designed to help lecturers navigate the options whilst considering the ever-increasing demand for improved employability equity.

Figure 1: Hierarchy of the use of Live Briefs in University Teaching.
Ranked based on perceived equity and employability status.

Working on live briefs enhances the students’ employability by improving general employability skills, and providing the ability to include this work in their portfolios and CVs. The approach of using live briefs outside of assessment does not provide equal opportunities to students from diverse backgrounds. Less privileged students often work nearly full-time during evenings and weekends to support themselves financially while studying. Indeed, 55% of UK students now work an average of 13.5 hours a week meaning they have less availability to participate in extracurricular assignments  (BBC, 2023). The Social Mobility Commission has noted that “unpaid internships are damaging for social mobility”  (Milburn, 2017). I see a parallel between the use of extracurricular briefs and unpaid internships, so I advocate that we discourage the use of unpaid extracurricular briefs, as they reduce our chances of ‘levelling up’ in the UK.

The Gold Star of Live Briefs

Justyna, BA Creative Advertising graduate, shared her thoughts on working on a paid live brief. “It gave me more motivation to produce the best possible work. But it was mainly because I was excited about the opportunity to actually make a campaign, still as a student. It was a great way of getting work experience and seeing how the industry works. I believe that the campaign I made is one of the most valuable experiences on my CV”.

Embedding live briefs into briefs assessment, producing work for clients, and compensating students for their contributions present significant challenges. However, I believe incremental improvements to the existing practice of utilising live briefs outside of formal assessment without remuneration should be pursued. The deliberate consideration of these options and the effort to implement such changes will gradually shift the culture and attitudes toward the use of live briefs among both university academic staff and external organisations. This progressive adaptation will enhance the integration of live briefs into the curriculum, ultimately benefiting the student experience, learning and employability whilst simultaneously resulting in clear knowledge exchange advantages for the external organisations.

Lucy Cokes is a senior lecturer at Falmouth University, School of Communications. She has been working in higher education for the past ten years and is a Fellow of Advance HE. She leads the Behaviour Change for Good modules on the Advertising courses and started the inhouse agency ‘BE good’ to manage the live projects which have included a number of government funded campaigns around VAWG and Healthy Relationships. Prior to this she ran a highly successful digital marketing agency with 80 staff in the UK across 3 offices.

References

BBC (2023) Most university students working paid jobs, survey shows. [Online] Available at: https://www.bbc.co.uk/news/education-65964375 [Accessed 23 August 2023]

Milburn, A. (2017) Unpaid internships are damaging to social mobility. [Online] Available at: https://www.gov.uk/government/news/unpaid-internships-are-damaging-to-social-mobility [Accessed 22 August 2023].

Sara, R. (2006) Live Project Good Practice: A Guide for the Implementation of Live Projects, s.l.: Centre for Education in the Built Environment


by 3 Comments

The hidden layers of transparency in UK HE assessment practices

by Chahna Gonsalves and Zhongan Lin

Transparency in assessment practices is a critical component of the UK’s higher education sector, but it is a term that carries many layers of meaning. This blog post explores a study that examined how transparency is framed in assessment policies across 151 UK higher education institutions (HEIs). The findings reveal that while institutions strive for transparency, they often overlook the complexities and multidimensional nature of the concept.

Understanding transparency: more than just clear documentation

Transparency in assessment is often associated with clear documentation of criteria, grading practices, and feedback mechanisms. However, this techno-rational approach, which emphasizes explicit documentation and information dissemination, is just one facet of transparency. Our study highlights the need for a more nuanced understanding that includes socio-cultural practices and socio-material enactments.

Techno-rational approaches: the dominant paradigm

The study found that techno-rational approaches dominate the transparency discourse in HEI policies. These approaches focus on ensuring that assessment criteria, learning outcomes, and grading standards are clearly articulated and accessible. For example, many policies mandate the use of detailed assessment briefs, rubrics, and grade descriptors. While this approach aims to make evaluative processes clear and consistent, it often falls short in addressing the dynamic and interpretive nature of academic standards.

One of the most compelling findings was the over-reliance on explicit standards documents, which presume that written criteria can universally ensure fairness and consistency. This static view overlooks the reality that academic standards are co-constructed within specific social and cultural contexts. Without acknowledging this, policies may fail to convey the nuanced, tacit knowledge necessary for fully understanding and applying assessment criteria.

The limitations of techno-rational transparency

Simply providing clear documentation does not guarantee that all stakeholders will understand or effectively use the information. For instance, non-native English speakers and students with varying levels of academic literacy may struggle with the language used in assessment criteria. Moreover, policies often fail to specify effective methods for disseminating this information, relying heavily on static documents rather than interactive or diverse formats that could enhance understanding.

Socio-cultural practices: engaging stakeholders in meaningful dialogue

Beyond documentation, transparency also involves socio-cultural practices that engage stakeholders in ongoing dialogue and clarification of assessment criteria. Policies that promote discussion between educators and students, co-creation of assessment criteria, and collaborative marking processes can foster a deeper understanding and shared meaning of what is expected. For instance, involving students in the creation of rubrics and providing opportunities for mock marking can enhance their evaluative judgment and assessment literacy.

One interesting insight from the study was the importance of dialogue in building a shared understanding of assessment standards. Policies that encourage discussion about assessment criteria not only help students grasp what is expected but also allow educators to refine and clarify their expectations. This dynamic, interactive process contrasts sharply with the static dissemination of information typical of techno-rational approaches.

Socio-material enactments: the role of tools and artefacts

The study also highlights the importance of socio-material enactments, where transparency is realized through the interaction between social practices and material artifacts. This includes the use of digital platforms, rubrics, exemplars, and other assessment tools that facilitate a tangible understanding of assessment criteria. Effective use of these tools can bridge the gap between educators’ tacit knowledge and students’ understanding, fostering a more comprehensive view of transparency.

For example, the use of digital platforms to share assessment criteria and feedback can significantly enhance transparency. These platforms allow for continuous access and interaction with assessment materials, making it easier for students to understand and engage with the criteria. However, the study found that detailed guidance on such platforms is often scant in policies, pointing to a significant area for improvement.

Who benefits from transparency? A multifaceted audience

Transparency in assessment is not solely for students. It also encompasses other stakeholders, including markers, external examiners, tutors, and even employers. The study found that while most policies address the need for transparency for students and markers, they often neglect other crucial stakeholders. This oversight can lead to inconsistencies in how assessments are interpreted and applied, potentially undermining the fairness and effectiveness of the evaluation process.

A particularly intriguing aspect of the study was the identification of specific roles and responsibilities for promoting transparency. By clearly defining who is responsible for ensuring transparency – whether it be module leaders, programme teams, or tutors – institutions can better align their policies with the needs of various stakeholders. This clarity can help avoid the pitfalls of ambiguous roles and ensure a more consistent application of assessment criteria.

Methodology: building the framework

To develop a comprehensive framework we conducted a detailed content analysis of assessment policy documents from 151 UK HEIs. The data collection process involved systematically retrieving and examining these publicly accessible documents, which included academic manuals, assessment policies, feedback strategies, and codes of practice. We excluded documents that were outdated or inaccessible, resulting in a final corpus of 264 documents. Through both deductive and inductive coding methods, we analysed the texts to identify recurring themes and patterns related to transparency. This process involved categorising the data into three main discourses – techno-rational, socio-cultural, and socio-material – guided by Ajjawi, Bearman, and Boud’s (2021) framework. The iterative coding and categorization helped us build a nuanced understanding of how transparency is conceptualized and communicated in HEI assessment policies.

Towards a holistic framework for transparency

Our study proposes a holistic framework that integrates techno-rational, socio-cultural, and socio-material approaches to transparency. This framework emphasizes the need for clear, accessible documentation, active engagement with stakeholders, and effective use of assessment tools and artifacts. By recognizing the diverse needs of all stakeholders, HEIs can develop more inclusive and effective assessment policies.

One of the key contributions of this study is its challenge to the notion of transparency as a static attribute. Instead, transparency is presented as a dynamic, contextually situated practice that requires continuous negotiation and interaction among stakeholders. This perspective shifts the focus from merely providing information to actively engaging stakeholders in the assessment process.

Figure 1. Framework of assessment transparency in Higher Education

Implications for policy and practice

To improve transparency in assessment, HEIs must move beyond merely publishing information to actively engaging with stakeholders through dialogue and interaction. Policies should be clear about the roles and responsibilities of different stakeholders in ensuring transparency. Furthermore, the use of diverse and interactive dissemination methods can enhance understanding and support students’ academic success.

For policymakers, the study suggests that transparency should be explicitly defined within institutional contexts, with guidelines that emphasize both the dissemination of information and the engagement of stakeholders. Educational practitioners are encouraged to adopt participatory practices in assessment design, involving students in creating and understanding assessment criteria, which is pivotal in promoting transparency.

Conclusion: enhancing transparency for a fairer education system

Transparency in assessment is a complex, multifaceted concept that goes beyond clear documentation. By integrating techno-rational, socio-cultural, and socio-material approaches, HEIs can foster a more inclusive and effective assessment environment. This study underscores the importance of comprehensive policies that not only provide clear information but also engage stakeholders in meaningful ways, ultimately contributing to a fairer and more equitable higher education system.

Reflecting on our roles as stakeholders

As readers, it is crucial to reflect on our roles within the higher education assessment ecosystem. Whether we are students, educators, policymakers, or external examiners, we each play a part in fostering transparency. Understanding the nuances of transparency and actively engaging in dialogue and interaction can help us contribute to more equitable and effective assessment practices. By recognizing and fulfilling our roles, we can collectively enhance the transparency and quality of education in our institutions.

Reference

Gonsalves, C and Lin, Z (2024) ‘Clear in advance to whom? Exploring ‘transparency’ of assessment practices in UK higher education institution assessment policy’ Studies in Higher Education, 1-17. https://doi.org/10.1080/03075079.2024.2381124

Chahna Gonsalves is a Senior Lecturer in Marketing (Education) at King’s College London. She is Senior Fellow of the Higher Education Association and Associate Fellow of the Staff Educational Development Association. Her interest in rubrics and the language of assessment is an extension of her role as Department Education Lead.

Zhonghan Lin is a Doctoral Researcher based at the Center for Language, Discourse and Communication, the School of Education, Communication and Society, King’s College London. Her research interests include urban multilingualism, education in ethnically and linguistically diverse societies, and family language policy.


by 1 Comment

Doctoral progress reviews: managing KPIs or developing researchers?

by Tim Clark

All doctoral students in the UK are expected to navigate periodic, typically annual, progress reviews as part of their studies (QAA, 2020). Depending on the stage, and the individual institutional regulations, these often play a role in determining confirmation of doctoral status and/or continuation of studies. Given that there were just over 100,000 doctoral students registered in the UK in 2021 (HESA, 2022), it could therefore be argued that the progress review is a relatively prominent, and potentially high stakes, example of higher education assessment.  However, despite this potential significance, guidance relating to doctoral progress reviews is fairly sparse, institutional processes and terminology reflect considerable variations in approach, empirical research to inform design is extremely limited (Dowle, 2023) and perhaps most importantly, the purpose of these reviews is often unclear or contested.

At the heart of this lack of clarity appears to be a tension surrounding the frequent positioning of progress reviews as primarily institutional tools for managing key performance indicators relating to continuation and completion, as opposed to primarily pedagogical tools for supporting individual students learning (Smith McGloin, 2021). Interestingly however, there is currently very little research regarding effectiveness or practice in relation to either of these aspects. Yet, there is growing evidence to support an argument that this lack of clarity regarding purpose may frequently represent a key limitation in terms of engagement and value (Smith McGloin, 2021, Sillence, 2023; Dowle, 2023). As Bartlett and Eacersall (2019) highlight, the common question is ‘why do I have to do this?’

As a relatively new doctoral supervisor and examiner, with a research interest in doctoral pedagogy, in the context of these tensions, I sought to use a pedagogical lens to explore a small group of doctoral students’ experiences of navigating their progress review. My intention for this blog is to share some learning from this work, with a more detailed recent paper reporting on the study also available here (Clark, 2023). 

Methods and Approach

This research took place in one post-1992 UK university, where progress assessment consisted of submission of a written report, followed by an oral examination or review (depending on the stage). These progress assessments are undertaken by academic staff with appropriate expertise, who are independent of the supervision team. This was a small-scale study, involving six doctoral students, who were all studying within the humanities or social sciences. Students were interviewed using a semi-structured narrative ‘event-focused’ (Jackman et al, 2022) approach, to generate a rich narrative relating to their experience of navigating through the progress review as a learning event.

In line with the pedagogical focus, the concept of ‘assessment for learning’ was adopted as a theoretical framework (Wiliam, 2011). Narratives were then analysed using an iterative ‘visit and revisit’ (Srivastava and Hopwood, 2009) approach. This involved initially developing short vignettes to consider students’ individual experiences before moving between the research question, data and theoretical framework to consider key themes and ideas.

Findings

The study identified that the students understood their doctoral progress reviews as having significant potential for supporting their learning and development, but that specific aspects of the process were understood to be particularly important. Three key understandings arose from this: firstly, that the oral ‘dialogic’ component of the assessment was seen as most valuable in developing thinking, secondly, that progress reviews offered the potential to reframe and disrupt existing thinking relating to their studies, and finally, that progress reviews have the potential to play an important role in developing a sense of autonomy, permission and motivation.

In terms of design and practice, the value of the dialogic aspect of the assessment was seen as being in its potential to extend thinking through the assessor, as a methodological and disciplinary ‘expert’, introducing invitational, coaching format, questions to provoke reflection and provide opportunities to justify and explore research decisions. When this approach was taken, students recalled moments where they were able to make ‘breakthroughs’ in their thinking or where they later realised that the discussion was significant in shaping their future research decisions. Alongside this, a respectful and supportive approach was viewed as important in enhancing psychological safety and creating a sense of ownership and permission in relation to their work:

“I think having that almost like mentoring, which is like a mini mentoring or mini coaching session, in these examination spots is just really helpful”

“I’m pootling along and it’s going okay and now this bombshell’s just dropped, but it was helpful because, yeah, absolutely it completely shifted it.”

“It’s my study… as long as I can justify academically and back it up. Why I’ve chosen to do what I’ve done then that’s okay.” 

Implications

Clearly this is a small-scale study, with a relatively narrow disciplinary focus, however its value is intended to lie in its potential to provoke consideration of progress reviews as tools for teaching, learning and researcher development, rather than to assert any generalisable understanding for practice.

This consideration may include questions which are relevant for research leaders, supervisors and assessors/examiners, and for doctoral students. Most notably: is there a shared understanding of the purpose of doctoral progress reviews and why we ‘have’ to do it? And how does this purpose inform design, practice and related training within our institutions?

Within this study it was evident that in this context the role of dialogic assessment was significant, and given the additional resource required to protect or introduce such an approach, this may be an aspect which warrants further exploration and investigation to support decision making. In addition, it also framed the perceived value of the careful construction of questions, which invite and encourage reflection and learning, as opposed to seeking solely to ‘test’ this.

Dr Timothy Clark is Director of Research and Enterprise for the School of Education at the University of the West of England, Bristol. His research focuses on aspects of doctoral pedagogy and researcher development.


by 3 Comments

Lessons from learning analytics

by Liz Moores and Rob Summers

Why bother collecting learning analytics data?

Some of the reported benefits of using learning analytics data include enabling personalised learning and narrowing attainment gaps. Indeed, a quick dip into some of the recent TEF feedback summaries to higher education institutions seems to suggest that use of learning analytics is valued by TEF panels. But can we learn more from the data to influence teaching practice? Aside from the potential benefits for a more personalised learning experience, we think that it’s a good way of understanding the learning process more generally. Over the past few years, we’ve been analysing some of the data generated from Aston University.

Last minute cramming is not effective in improving attainment.

Yes, your parents were correct – it’s much better to work consistently! Early engagement with studies really appears to matter. In fact, the average attainment levels of those first-year students whose engagement remained at the lowest relative levels throughout the year was very similar to those whose early engagement was lowest in the first three weeks but became the very highest in the last three weeks. In contrast, those who started off enthusiastically, but then lost interest, were awarded higher average marks than any of the groups that started off slowly, regardless of how much or whether their engagement peaked later. The consistency of the data – in that those who started off with high engagement tended to finish with high engagement – was remarkable. Also noteworthy were the effects of early engagement on attainment. For the chart below, we divided students into activity quintiles based on only their first three weeks of engagement (Q5 being the highest engagement) and on end of year mark quintiles (Q5 being the highest attainment). The width of the lines connecting engagement quintile to mark quintile is indicative of the proportion of students linking the two measures. The results highlight how few students pass from higher activity quintiles to lower mark quintiles and vice versa.

Of course, these results come with the usual caveats that we cannot infer cause and effect (it could be that the lower engagers in the first three weeks were just low achieving students). However, for us, this highlights the importance of a good induction into academic life – possibly enhanced by some structured engagement exercises to help get first years into good habits (ie tell them how they should be engaging, and the different ways that they can, not just that they should be doing so). There were probably a fair few students represented in this figure that were not even sure what they were supposed to be doing with all their ‘spare’ time

Behaviour outweighs demographics when predicting attainment

The recent pandemic generated much discussion about digital poverty, suggesting that who we teach might be important – at the very least in terms of access to technology. Our recent evidence suggests that both how you teach and who you teach mattered. However, it is important to note that behaviour outweighed demographics in predicting attainment, albeit that in this case behaviour was probably also influenced by demographics. The gap between disadvantaged students’ attainment and their peers widened during online teaching and assessment conditions, and disadvantaged students were also less likely to obtain all 120 module credits on their first try. We also observed changes in their patterns of engagement, although less so for synchronously delivered teaching (as compared to recorded lectures). Students with the lowest engagement were the ones driving the widened gap; those who engaged well with synchronously provided teaching (even if online) fared much better.

So, we should stop teaching online and get people into the classroom early?

No – not necessarily. We don’t want to claim that all online teaching is bad – instead we need to understand what forms of online teaching work, what good looks like, and how our various teaching strategies affect different groups. Anecdotally, many students have appreciated the flexibility of online teaching, particularly where it has included facilities such as the ability to ask questions anonymously. And if you want to reuse those pre-recorded videos, there has been some interesting research from other research groups on ‘watch-parties’. With the cost-of-living crisis, many students will appreciate being able to log into a lecture from home rather than forking out a bus fare or missing out on some part time work. What is important is to understand what works – and for whom.

Professor Liz Moores is Deputy Dean in the College of Health and Life Sciences at Aston University and has research interests in the evaluation of higher education, particularly as applied to widening participation issues.

Dr Rob Summers is research manager at the Centre for Transforming Access and Student Outcomes (TASO). Before joining TASO, Rob worked in the student outreach team at Aston University managing a randomised controlled trial of two post-16 outreach programmes as part of the TASO MIOM (Multi-intervention, Outreach and Mentoring) project.


by Leave a comment

The ongoing saga of REF 2028: why doesn’t teaching count for impact?

by Ian McNay

Surprise, surprise…or not.

The initial decisions on REF 2028 (REF 2028/23/01 from Research England et al), based on the report on FRAP – the Future Research Assessment Programme – contain one surprise and one non-surprise among nearly 40 decisions. To take the second first, it recommends, through its cost analysis report, that any future exercise ‘should maintain continuity with rules and processes from previous exercises’ and ‘issue the REF guidance in a timely fashion’ (para 82). It then introduces significant discontinuities in rules and processes, and anticipates giving final guidance only in winter 2024-5, when four years (more than half) of the assessment period will have passed.

The second surprise is, finally, the open recognition of the negative effects on research culture and staff careers of the REF and its predecessors (para 24), identified by respondents to the FRAP consultation about the 2028 exercise. For me, this new humility is a double edged sword: many of the defects identified have been highlighted in my evidence-based articles (McNay, 2016, McNay, 2022), and, indeed, by the report commissioned by HEFCE (McNay, 1997) on the impact on individual and institutional behaviour of the 1992 exercise:

  • Lack of recognition of a diversity of excellences including work on local or regional issues because of the geographical interpretation of national/international excellence (para 37). Such local work involves different criteria of excellence, perhaps recognised in references to partnership and wider impact.
  • The need for outreach beyond the academic community, such as a dual publication strategy – one article in an academic journal matched with one in a professional journal in practical language and close to utility and application of a project’s findings.
  • Deficient arrangements for assessing interdisciplinary work (paras 60 and 61)
  • The need for a different, ‘refreshed’, approach to appointments to assessment panels (para 28)
  • The ‘negative impact on the authenticity and novelty of research, with individuals’ agendas being shaped by perceptions of what is more suitable to the exercise: favouring short-term inputs and impacts at the expense of longer-term projects…staying away from areas perceived to be less likely to perform well’. ‘The REF encourages …focus on ‘exceptional’ impacts and those which are easily measurable, [with] researchers given ‘no safe space to fail’ when it came to impact’.
  • That last negative arises in major part because of the internal management of the exercise, yet the report proposes an even greater corporate approach in future. The evidence-based articles and reports, and innovative processes and artefacts that arise from our research will have a reduced contribution to published assessments on the quality of research, though there is encouragement of a wider diversity of research outputs. More emphasis will be placed on institutional and unit ‘culture’ (para 28), so individuals disappear, uncoupled from consideration of culture-based quality. That culture is controlled by management; I spent several years as a Head of School trying to protect and develop further a collegial enterprise culture, which encouraged research and innovative activities in teaching. The senior management wanted a corporate bureaucracy approach with targets and constant monitoring, which work at Exeter has shown leads to lower output, poorer quality and higher costs (Franco-Santos et al, 2014).

At least 20 per cent of the People, Culture and Environment sub-profile for a unit will be based on an assessment of the Institutional Level (IL) culture, and this element will make up 25 per cent of a unit’s overall quality profile, up from 15 percent from 2021. This proposed culture-based approach will favour Russell Group universities even further – their accumulated capital has led to them outscoring other universities on ‘environment’ in recent exercises, even when the output scores have been the other way round. Elizabeth Gadd, of Loughborough, had a good piece on this issue in Wonkhe on 28 June 2023. The future may see research-based universities recruiting strongly in the international market to provide subsidy to research from higher student fees, leaving the rest of us to offer access and quality teaching to UK students on fees not adjusted for inflation. Some recognition of excellent research in unsupportive environment would be welcome, as would reward for improvement as operated when the polytechnics and colleges joined research assessment exercises.

The culture of units will be judged by the panels – a separate panel will assess IL cultures – and will be based on a ‘structured statement’ from the management, assessing itself, plus a questionnaire submission. I have two concerns here: can academic panels competent to peer-assess research also judge the quality and contribution of management; and, given behaviours in the first round of impact assessment (Oancea, 2016), how far can we rely on the integrity of these statements?

The sub-profile on Contribution to Knowledge and Understanding sub-profile will make up 50 per cent of a unit’s quality profile – down from 60 per cent last time and 65 per cent in 2014. At least 10 per cent will be based on the structured statement, so Outputs – the one thing that researchers may have a significant role in – are down to only 40 per cent, at most, of what is meant by overall research quality (the FRAP International Committee recommended 33 per cent). Individuals will not be submitted. HESA data will be used to quantify staff and the number of research outputs that can be submitted will be an average of 2.5 per FTE. There is no upper limit for an individual, and staff with no outputs can be included, as well as those who support research by others, or technicians who publish. Research England (and this is mainly about England; the other three countries may do better and certainly will do things differently) is firm that the HESA numbers will not be used as the volume multiplier for funding (still a main purpose of the REF), though it is not clear where that will come from – Research England is reviewing their approach to strategic institutional research funding. Perhaps staff figures submitted to HESA will have an indicator of individuals’ engagement with research.

Engagement and Impact broadens the previous element of simply impact. Our masters have discovered that early engagement of external partners in research, and 6 months attachment at 0.2 contract level allows them to be included, and enhances impact. Wow! Who knew? The work that has impact can be of any level to avoid the current quality level designations stopping local projects being acknowledged.

The three sub-profiles have fuzzy boundaries and overlap. Not just in a linear connection – environment, output, impact – but, because, as noted above, for example, engagement comes from the external environment but becomes part of the internal culture. It becomes more of a Venn diagram, that allows the adoption of an ‘holistic’ approach to ‘responsible research assessment’. We wait to see what those both mean in practice.

What is clear in that holistic approach is that research has nothing to do with teaching, and impact on teaching still does not count. That has created an issue for me in the past since my research feeds (not leads) my teaching and vice versa. I use discovery learning and students’ critical incidents as curriculum organisers, and they produce ‘evidence’ similar to that gathered through more formal interview and observation methods. An example. I recently led a workshop for a small private HEI on academic governance. There was a newly appointed CEO. I used a model of institutional and departmental cultures which influence decision making and ‘governance’ at different levels. That model, developed to help my teaching is now regarded by some as a theoretical framework and used as a basis for research. Does it therefore qualify for inclusion in impact? The session asked participants to consider the balance among four cultures of collegial, bureaucratic, corporate, entrepreneurial, relating to the degrees of central control of policy development and of policy delivery (McNay, 1995).  It then dealt with some issues more didactically, moving to the concept of the learning organisation where I distributed a 20 item questionnaire, (not yet published, but available on request for you to use) to allow scoring out of 10 per item, of behaviours relating to capacity to change, innovate and learn, leading to improved quality. Only one person scored more than 100 in total and across the group the modal score was in the low 70s, or just over 35%. That gave the new CEO an agenda with some issues more easily pursued than others and scores indicating levels of concern and priority. So my role moved into consultancy. There will be impact, but is the research base sufficient, was it even research, and does the use of teaching as a research transmission process (Boyer, 1990) disqualify it?

I hope this shows that the report contains a big agenda, with more to come. SRHE members need to consider what it means to them, but also what it means for research into institutions and departments to help define culture and its characteristics. I will not be doing it, but I hope some of you will. We need to continue to provide an evidence base to inform decisions even if it takes up to 20 years for the findings to have an impact.

SRHE itself might say several things in response to the report:

  • welcome the recognition of previous weaknesses, but note that a major one has not been recorded: the impact of RAE/REF on teaching, when excellent research has gained extra money, but excellent teaching has not, leading to an imbalancing of effort within the HE system. The research-teaching nexus also needs incorporating into the holistic view of research. Teaching is a major element in dissemination of research (Boyer, 1990) and so a conduit to impact, and should be recognised as such. That is because the relationship between researcher/teacher and those gaining new knowledge and understanding is more intimate and interactive than a reader of an article experiences. Discovery learning, drawing on learners’ experiences in CPD programmes can be a source of evidence, enhancing the knowledge and understanding of the researcher to incorporate in further work and research publications.
  • welcome the commitment to more diversity of excellences. In particular, welcome the commitment to recognise local and regionally directed research and its significant impact. The arguments about intimacy and interaction apply here, too. Research in partnership is typical of such work and different criteria are needed to evaluate excellence in this context.
  • welcome the intention to review panel membership to reflect the wider view of research now to be adopted.
  • urge an earlier clarification on panel criteria to avoid another 18 months, at least, trying, without clarity or guidance, to do work that will fit with the framework of judgement within which that work will be judged.
  • be wary of losing the voice of the researchers in the reduction of emphasis on research and its outputs in favour of presentations on corporate culture.

References

McNay, I (1997) The Impact of the 1992 RAE on Institutional and Individual Behaviour in English HE: the evidence from a research project Bristol HEFCE