The Society for Research into Higher Education

1 Comment

The ‘Holy Grail’ of pedagogical research: the quest to measure learning gain

by Camille Kandiko Howson, Corony Edwards, Alex Forsythe and Carol Evans

Just over a year ago, and learning gain was ‘trending’. Following a presentation at SRHE Annual Research Conference in December 2017, the Times Higher Education Supplement trumpeted that ‘Cambridge looks to crack measurement of ‘learning gain’; however, research-informed policy making is a long and winding road.

Learning gain is caught between a rock and a hard place — on the one hand there is a high bar for quality standards in social science research; on the other, there is the reality that policy-makers are using the currently available data to inform decision-making. Should the quest be to develop measures that meet the threshold for the Research Excellence Framework (REF), or simply improve on what we have now?

The latest version of the Teaching Excellence and Student Outcomes Framework (TEF) remains wedded to the possibility of better measures of learning gain, and has been fully adopted by the OfS.  And we do undoubtedly need a better measure than those currently used. An interim evaluation of the learning gain pilot projects concludes: ‘data on satisfaction from the NSS, data from DHLE on employment, and LEO on earnings [are] all … awful proxies for learning gain’. The reduction in value of the NSS to 50% in the most recent TEF process make it no better a predictor of how students learn.  Fifty percent of a poor measure is still poor measurement.  The evaluation report argues that:

“The development of measures of learning gain involves theoretical questions of what to measure, and turning these into practical measures that can be empirically developed and tested. This is in a broader political context of asking ‘why’ measure learning gain and, ‘for what purpose’” (p7).

Given the current political climate, this has been answered by the insidious phrase ‘value for money’. This positioning of learning gain will inevitably result in the measurement of primarily employment data and career-readiness attributes. The sector’s response to this narrow view of HE has given renewed vigour to the debate on the purpose of higher education. Although many experts engage with the philosophical debate, fewer are addressing questions of the robustness of pedagogical research, methodological rigour and ethics.

The article Making Sense of Learning Gain in Higher Education, in a special issue of Higher Education Pedagogies (HEP) highlights these tricky questions. Continue reading

Ian Mc Nay

Leave a comment

Universities Ancient and Modern

By Ian McNay

This seems to have been a recurrent theme in my life over the last three months, as well as a constant issue through working in a modern university occupying ancient buildings – anomalous, anachronistic and dissonant. One key factor recently was an invitation to contribute a piece to the Sage International Encyclopaedia of Higher Education on ‘Modern Universities’. I had 1000 words, so assumed it meant within the UK – a focus of other entries, in a publication mainly dealing with Anglo-Saxon countries. A mistaken assumption, but I defy anybody to compose a comprehensive coverage of the topic at global level in 1000 words. It has been accepted. The commission led to surfacing something of which I had been vaguely aware: all universities established or designated in the 20th century were secular. The last one with church links was Aberdeen, founded by papal bull and a charter from the local bishop in 1495, but also given a royal charter. In the 21st century, there have already been designations of 15 state universities which were, originally, church foundations, now labelled the Cathedrals Group, and claiming to be the only group in UK HE ‘based on ethical principles informed by faith-based values’. Heythrop College was also a member, but is now closing down; Trinity St. David’s is the only one outside England, and was formerly part of the University of Wales. Their main emphasis, reflecting history and the churches’ role in their communities, is teacher training, but they also have nearly half of the UK undergraduates studying theology and divinity.

Other newly designated universities also have narrow disciplinary bases, allowed by changes in criteria for designation; eight cover creative and performing arts and agriculture. The four private universities designated are similar – a focus on Law and Business and Management Studies, and mainly sold off to hedge funds based outside the UK, unconstrained by charity law or other checks and balances. As I have reported previously, some private HEIs differ in another way from the normative model: their student body has more men than women, a significant majority of BME students, and a mean entry age in the early thirties, not the late teens, so providing for a market segment under-represented elsewhere in the system, possibly conditioned by fee levels and debt aversion. At Coventry, the private arms in Scarborough, London and elsewhere were set up by Deputy Vice-Chancellor Ian Dunn in a separate structure from the university because the innovative features, enjoyed by students, would have been difficult to get approved.

That raises wider questions: is the current isomorphic state provision fit for all students, or, taking Ansoff’s strategy matrix, is it the same product for a new market for which it is less appropriate? The league tables promote a single model, with a characteristic of exclusionary entry levels preferred over diversity of access, for example. If we/you were establishing a university ab initio [the spell check changed that to ‘ignition’!], what would it be like

– to cope with the rapid growth in the 18+ age cohort in the next decade,

– with a projection by Graeme Atherton (Director of NEON) and colleagues of only 26 per cent of students from London being ‘white’, though higher elsewhere, and

– some of those with parents from elsewhere in the EU, countries with lower fees to which they might return for their higher education experience

– as well as developments in curriculum thinking and technology in learning?

Two ‘recent’ foundations have been very different: the Open University and the University of the Highlands and Islands. Do they offer lessons/models?

As it happens, other summer experiences continued the ancient/modern dichotomy.

A seminar at UCL IoE examined Nazarbayev University in Kazakhstan, which took it first students in 2010. It has an international academic staff (over 75 per cent are from elsewhere around the world) and student body – there are 20 flags on its website, aiming to get it high in league tables within 20 years. The emphasis is on research and graduate programmes – an MBA and Master’s awards in Education up to now. The only unit on which there are details is the engineering faculty, where all the listed research centres are located. It is also claimed as a ‘template’ university with an objective defined as helping other Kazakh universities to develop. There are about 40 of them, with mainly a regional focus and catchment. There is a problem: most of their staff do not speak English, the lingua franca of Nazarbayev [the university, not the president of the country], so there are communication issues in any ‘trickle down’ model. A second issue is that the new university gets over 25 per cent of the national HE budget. It is autonomous and has no reporting responsibility to the Ministry of Education, but may gain from lack of state control. There are echoes here of a recent article in World University News by Philip Altbach and Hans de Wit, for both of whom I have had a lot of respect. Their thesis is that most university academic staff should stop doing research, or not start, encouraged by closing down journals to make publication more difficult, and so, they claim, less likely to be a major pressure in academics’ lives. So, in Australia, just the Group of Eight, in the UK just the Russell Group [whose past pronouncements suggest they would support such a policy]. But, back to the question, which has system level implications: if you were planning a new university, would it build in research as an essential? Would a hierarchical system be good, or can diversity displace hierarchy, with different excellences recognised and rewarded? Does it have to have a significant international/global ethos?

I move on, to a holiday, in Portugal and Spain, which took in ancient universities in Coimbra and Salamanca, and modern, or at least younger ones in the Douro Valley, at Vila Real and a second one in Salamanca. Salamanca’s newer, private, university was founded by the pope when the state closed down the faculties of theology and canon law in the ancient one in 1854, though it did not get a charter until the 1940s. The new one took them over and now offers courses across the range of humanities and social sciences, with about 6,500 students. It has a world expert on dogmatic theology [is there another kind?] who recently won the Ratzinger prize. No science, technology or medicine; you have to go to the Pontifical University in Madrid for those. There is some confusion about identity because the pontifical university occupies old buildings in the city centre beside the cathedrals – the one to be replaced by the second has never been demolished and they are conjoined with entrance to the older [preferred by most visitors] through the newer. The walls of the university have names of doctoral graduates written in bull’s blood from the animal killed, cooked and eaten in celebration of success. Not done now, but a fee allows a name to be painted. ‘Fees’ triggers the memory that they are ten times higher in the private pontifical university than in the state university, now on the outskirts of the city and with approaching 30,000 students.

Coimbra is a major name in HE history in Europe, particularly during the Renaissance. It dominates the town both from its hilltop site and through its role as by far the main employer in the city. It has a Wow! library, where the books cannot be touched without official permission, and a chapel where the organ has 20,000 pipes, regularly played for masses, ceremonies and public events. There is a student uniform [really], of black and white, topped by a calf length black cloak, worn with one end slung over the left shoulder [not the right – that is a basic error]. A cross between Dr Who and Zorro. Personalised by badges showing students’ origins, disciplines and involvement in university life. After graduation, the uniform cannot be worn, except for the cloak on special occasions like selling souvenir pencils to tourists to help pay off debts, and singing Fado, the local folk music.

The modern university in this part of the review is the University of Tras-os-Montes and Alto Douro in Vila Real, Portugal. It was set up as a polytechnic in 1973, and given university status in 1986 because of quality research with relevance [there is still a strong polytechnic sector in Portugal]. It was founded for two reasons – to support regional development in a disadvantaged community; to provide a local institution of higher education, since other provision was difficult to get to – distance and under-developed infrastructure. So, initially, it was strong on agriculture, particularly viniculture, tourism, civil engineering (there are magnificent dams on the river) and such subjects as computing and business management to support them. Now it has 35 Bachelor and 38 Master’s courses across four faculties, and the region seems to be doing well, with EU funded projects very evident.

So, modern, or newer universities with a diversity of driving forces behind their establishment. In the UK, successive English governments have changed/reduced the criteria for designation, which has allowed smaller, specialist institutions to qualify, but also allowed privateers to gain a foothold in the sector. It could be that in the middle future, with demographic decline, the private sector will be enhanced by state universities deemed to be failing being sold off as ‘academies’ as in the school sector. That would accelerate the ‘small state‘ agenda. The other motivations have been the protection of an ideology/theology, also reflected in the rise of faith based universities in the UK, and a catalyst for high tech development and international recognition by an emergent nation and a power and glory driven dictator. I return to my earlier question: if HE continues to expand, possibly rapidly in many countries with increasing numbers of young people, what might be the defining characteristics of a university, to be fit for what purpose?

SRHE Fellow Ian McNay is emeritus professor at the University of Greenwich

Each unhappy family is unhappy in its own way

Sam Gyimah MP, Minister of State for Universities and Science

Dear Minister

“Each unhappy family is unhappy in its own way”

If asked to sum up in a single word the direction of higher education policy from 2010 onwards, I think that many of us who try to follow Government thinking on these matters might say that the word would be “markets”. In successive White Papers and speeches, ministers have insisted that fee-paying students should see themselves as customers buying services from a university provider, which in turn should be competing with other providers in the higher education marketplace to offer the best value for money to student customers. In this way, your predecessors have argued, quality would go up and costs would come down, as happens in most markets for consumer goods. The Government has encouraged this trend by demanding that universities provide more information on which student-customers might base their purchasing decisions – most recently the TEF and the LEO data – and by encouraging new entrants into the marketplace with the aim of sharpening competition further.

Many of us in universities rather doubted that trying to create a straightforward market-type relationship between universities and their students was the best way to organise teaching and learning.  For a start, there is little evidence that students themselves want a relationship on these terms: the great majority of students surveyed in the HEPI 2018 Student Academic Experience Survey, for example, arguably preferred a pre-2004 Act, certainly a pre-2011 White Paper, funding model. I think that one reason for this – paying lower fees is no doubt another – is because they understand that in order to learn effectively they must engage with the academic life of the university in a way that is qualitatively different to, say, my engagement with Sainsbury’s when I go shopping there. Sainsbury’s does not expect its customers to help create the products which appear on its shelves; and if I’m unimpressed with them today, and I can see what Tesco are up to tomorrow. I have made no particular commitment to the Sainsbury’s way of shopping. Forgive me if this seems terribly obvious, but it has not always been clear that ministers fully appreciated this distinction.

Although many of us didn’t much like its implications, we did at least think we knew that Government saw our relationship with our students in these transactional, market-based terms. But then, Minister, along you come saying that, on the contrary, we should be in loco parentis to our students, acting (for instance) as go-betweens with their parents or guardians if we have concerns about their mental health (as reported in The Guardian, 28 June). This is not just overthrowing normal market relationships – Sainsbury’s in truth couldn’t care less about my personal well-being – it is redefining universities’ relationships with their students, and in an unhelpful way. (Having a duty of care towards both students and staff members is a different matter.) If I may say so, this has the distinct feel of political grandstanding, wanting to be seen to be acting decisively in response to – what, exactly? Of course, mental illness is desperately serious for the families and friends of those suffering from its various forms, needing the involvement of skilled professionals. A particular concern may be that suicide could result from overlooking a person’s symptoms. (Though suicide in the UK is actually a good-news story – so to speak – as ONS data show that the number of suicide deaths has been falling steadily over recent decades. Middle-aged, disadvantaged men are most likely to commit suicide – and they don’t constitute a large part of the student demographic.)

But what should be the role of a university in relation to its adult students with mental health problems? Nicola Barden writing for WonkHE on 28 June (do you read it, Minister? – you should) identifies a few of the problems which the proposed opt-in system, allowing universities to contact a student’s parents or other nominated individuals in the event of a mental health crisis, will create. Any social worker will tell you that relationships within families can be difficult in ways that outsiders can’t immediately detect: any member of university staff intruding here must be certain that they will not cause further harm – and how can they know that for sure? Imagine a situation where a parent of a student with mental health difficulties believes that the university will contact them in the event of a crisis – only for the student to have withdrawn that consent subsequently, not wanting their family to be involved. The university will then be in an impossible situation, having made commitments to both parties (as they will see it).

We’re operating, Minister, in a Government-mandated market. Universities should support their students in their academic work, but should not set themselves up to fail as substitute families. That historically never was their role; your Government’s market-focused policies have now put it completely beyond reach.

SRHE member Paul Temple, Centre for Higher Education Studies, UCL Institute of Education, University College London.

Leave a comment

Are academic papers written with students cited as often as academic papers written with colleagues?

By James Hartley

This note compares the citation rates for publications written by the author alone with (i) those written by the author and fellow colleagues and (ii) those written together with undergraduates. Although the citation rates for publications written by the author alone, or with colleagues, are higher than those obtained for papers written with undergraduates, the data suggest that some teacher-student papers can make a substantial contribution to the research literature.                                                                                                       


As an academic author I often wonder whether or not my papers with undergraduates are cited as often as my papers with colleagues.  On the one hand, colleagues are often more experienced and generally more familiar with academic writing and publishing than undergraduates.  On the other, undergraduates in the UK sometimes author papers arising from the research that they carried out in their final year supervised by academic staff.  To answer this query I used the website Google Scholar to examine a sample of how often my single-authored publications were cited with respect to (i) those written with colleagues and (ii) those written with undergraduate students. Continue reading

1 Comment

It’s all about performance

by Marcia Devlin

The Australian federal government has indicated its intention to introduce partial funding based on yet to be defined performance measures.

The Mid-Year Economic and Fiscal Outlook (MYEFO) by the Australian government updates the economic and fiscal outlook from the previous budget and the budgetary position and revises the budget aggregates taking account of all decisions made since the budget was released. The 2017-2018 MYEFO papers state that the Government intends to “proceed with reforms to the higher education [HE] sector to improve transparency, accountability, affordability and responsiveness to the aspirations of students and future workforce needs” (see links below). Among these reforms are performance targets for universities to determine the growth in their Commonwealth Grant Scheme funding for bachelor degrees from 2020, to be capped at the growth rate in the 18-64 year old population, and from 1 January 2019, “a new allocation mechanism based on institutional outcomes and industry needs for sub-bachelor and postgraduate Commonwealth Supported Places”. Continue reading


Peer Observation of Teaching – does it know what it is?

by Maureen Bell

What does it feel like to have someone observing you perform in your teaching role? Suppose they tick off a checklist of teaching skills and make a judgement as to your capability, a judgement that the promotions committee then considers in its deliberations on your performance? How does it feel to go back to your department and join the peer who has written the judgement? Peer Observation of Teaching (POT) is increasingly being suggested and used as a tool for the evaluation, rather than collaborative development, of teaching practice.

Can POT for professional development co-exist with and complement POT for evaluation? Or are these diametrically opposed philosophies and activities such that something we might call Peer Evaluation of Teaching (PET) has begun to undermine the essence of POT?

I used to think the primary purpose of peer observation of teaching (POT) was the enhancement of teaching and learning. I thought it was a promising process for in-depth teaching development. More recently I have been thinking that POT has been hijacked by university quality assurance programs and re-dedicated to the appraisal of teaching by academic promotions committees. The principles and outcomes of POT for appraisal are, after all, quite opposite to those that were placed at the heart of the original POT philosophy and approach – collegial support, reflective practice and experiential learning.

In 1996 I introduced a POT program into my university’s (then) introduction to teaching course for academic staff. Participants were observed by each other, and myself as subject coordinator, and were required to reflect on feedback and plan further action. It wasn’t long before I realised that I could greatly improve participants’ experience by having them work together, experiencing at different times the roles of both observer and observed. I developed the program such that course participants worked in groups to observe each other teach and to share their observations, feedback and reflections. A significant feature of the program was a staged workshop-style introduction to peer observation which involved modelling, discussion and practice. I termed this collegial activity ‘peer observation partnerships’.

The program design was influenced by my earlier experiences of action research in the school system and by the evaluation work of Web and McEnerney (1995) indicating the importance of training sessions, materials, and meetings. Blackwell (1996), too, in Higher Education Quarterly described POT as stimulating reflection on and improvement of teaching. Early results of my program, published in IJAD in 2001, reported POT as promoting the development of skills, knowledge and ideas about teaching, as a vehicle for ongoing change and development, and as a means of building professional relationships and a collegial approach to teaching.

My feeling then was that a collegial POT process would eventually be broadly accepted as a key strategy for teaching development in universities. Surely universities would see POT as a high value, low cost, professional development activity. This motivated me to publish Peer Observation Partnerships in Higher Education through the Higher Education Research and Development Society of Australasia (HERDSA).

Gosling’s model appeared in 2002 in which he posed three categories of POT, in summary: evaluation, development, and fostering collaboration. Until then I had not considered the possibility that POT could be employed as an evaluation tool, mainly because to my mind observers did not need a particular level of teaching expertise. Early career teachers were capable of astute observation, and of discussing the proposed learning outcomes for the class along with the activity observed. I saw evaluation as requiring appropriate expertise to assess teaching quality against a set of reliable and valid criteria. Having been observed by an Inspector of Schools in my career as a secondary school teacher, I had learned from experience the difference between ‘expert observation’ and ‘peer observation’.

Looking back, I discovered that the tension between POT as a development activity rather than an evaluation tool had always existed. POT had been mooted as a form of peer review and as a staff appraisal procedure in Australia since the late eighties and early nineties, when universities were experiencing pressure to introduce procedures for annual staff appraisal. The emphasis at that time was evaluative – a performance management approach seeking efficiency and linking appraisal to external rewards and sanctions. Various researchers and commentators c.1988-1993, including Lonsdale, Abbott, and Cannon, sought an alternative approach which emphasised collegial professional development. At that time action research involving POT was prevalent in the school system using the Action Research Planner of Kemmis and McTaggert. Around this time Jarzabkowski and Bone from The University of Sydney developed a detailed guide for Peer Appraisal of Teaching. They defined the term ‘peer appraisal’ as a method of evaluation, that could both provide feedback on teaching for personal development as well as providing information for institutional or personnel purposes. ‘Observer expertise in the field of teaching and learning’ was a requirement.

In American universities various peer-review-through-observation projects had emerged in the early nineties. A scholarly discussion of peer review of teaching was taking place under the auspices of the American Association for Higher Education Peer Review of Teaching project and the national conference, ‘Making Learning Visible: Peer-review and the Scholarship of Teaching’ (2000), brought together over 200 participants. The work of both Centra and Hutchings in the 90s, and Bernstein and others in the 2000s advocated the use of peer review for teaching evaluation.

In 2002 I was commissioned by what was then the Generic Centre (UK) to report on POT in Australian universities. At that time several universities provided guidelines or checklists for voluntary peer observation, while a number of Australian universities were accepting peer review reports of teaching observations for promotion and appointment. Soon after that I worked on a government funded Peer Review of Teaching project led by the University of Melbourne, again reviewing POT in Australian universities. One of the conclusions of the report was that POT was not a common professional activity. Many universities however listed peer review of teaching as a possible source of evidence for inclusion in staff appraisal and confirmation and promotion applications.

My last serious foray into POT was an intensive departmental program developed with Paul Cooper, then Head of one of our schools in the Engineering Faculty. Along with my earlier work, the outcomes of this program, published in IJAD (2013), confirmed my view that a carefully designed and implemented collegial program could overcome problems such as those reported back in 1998 by Martin in Innovations in Education and Teaching International, 35(2). Meanwhile my own head of department asked me to design a POT program that would provide ‘formal’ peer observation reports to the promotions and tenure committee. I acquiesced, although I was concerned that once POT became formalised for evaluation purposes in this way, the developmental program would be undermined.

Around 2008 my university implemented the formal POT strategy with trained, accredited peer observers and reporting templates. POT is now accepted in the mix of evidence for promotions and is compulsory for tenure applications. In the past year I’ve been involved in a project to review existing peer observation of teaching activities across the institution, which has found little evidence of the use of developmental POT.

The Lonsdale report (see above) proposed a set of principles for peer review of teaching and for the type of evidence that should be used in decisions about promotion and tenure: Fairness such that decisions are objective; openness such that criteria and process are explicit and transparent; and consistency between standards and criteria applied in different parts of the institution and from year to year. It always seemed to me that the question of criteria and standards would prove both difficult and contentious. How does a promotions committee decipher or interpret a POT report? What about validity and reliability? What if the POT reports don’t align with student evaluation data? And what does it mean for the dynamics of promotion when one of your peer’s observations might influence your appraisal?

In 2010 Chamberlain et al reported on a study exploring the relationship between annual peer appraisal of teaching practice and professional development. This quote from a participant in the study stays with me, “… the main weakness as far as I’m concerned is that it doesn’t know what it is. Well, what is its purpose?”

POT for professional development is an activity that is collegial, subjective, and reflective. My view is that POT for professional development can only co-exist with a version of POT for evaluation that is re-named, re-framed and standardised. And let’s call it what it really is – Peer Evaluation of Teaching (PET).

Dr Maureen Bell is Editor of HERDSA NEWS, Higher Education Research and Development Society of Australasia; HERDSA Fellow; Senior Fellow, University of Wollongong Australia.

Leave a comment

Mind the Gap – Gendered and Caste-based Disparities in Access to Conference Opportunities

In an interview with Conference Inference [1] editor Emily Henderson, Nidhi S. Sabharwal discussed inequalities of access to conference opportunities in India.

Figure 1: Participation in Conferences by Gender (in a high-prestige institution)Figure 1: Participation in Conferences by Gender (in a high-prestige institution)

EH: Nidhi, can you explain first of all where conferences come into your wider research on inequalities in Indian higher education?

NS: Equitable access to professional development opportunities such as conferences is an indicator of institutional commitment to achieving diversity and inclusion of diverse social groups on campuses. Continue reading