srhe

The Society for Research into Higher Education


Leave a comment

The Impact of TEF

by George Brown

A report on the SRHE seminar The impact of the TEF on our understanding, recording and measurement of teaching excellence: implications for policy and practice

This seminar demonstrated that the neo-liberal policy and metrics of TEF (Teaching Excellence Framework) were not consonant with excellent teaching as usually understood.

Michael Tomlinson’s presentation was packed with analyses of the underlying policies of TEF. Tanya Lubicz-Nawrocka considered  the theme of students’ perceptions of excellent teaching. Her research demonstrated clearly that students’ views of excellent teaching were very different from those of TEF. Stephen Jones provided a vibrant analysis of public discourses. He pointed to the pre-TEF attacks on universities and staff by major conservative politicians and their supporters. These were to convince students and their parents that Government action was needed. TEF was born and with it the advent of US-style neo-liberalism and its consequences. His final slide suggested ways of combating TEF including promoting the broad purposes of HE teaching. Sal Jarvis succinctly summarised the seminar and took up the theme of purposes. Personal development and civic good were important purposes but were omitted from the TEF framework and metrics.

Like all good seminars, this seminar prompted memories, thoughts and questions during and after the seminar. A few of mine are listed below. Others may wish to add to them.

None of the research evidence supports the policies and metrics of TEF (eg Gibbs, 2018). The indictment of TEF by the Royal Statistics Society is still relevant (RSS, 2018). The chairman of the TEF panel is reported to have said “TEF was not supposed to be a “direct measure of teaching” but rather “a measure based on some [my italics] of the outcomes of teaching” On the continuum of neo-liberalism and collegiality, TEF is very close to the pole of neo-liberalism whereas student perspectives are nearer the pole of collegiality which embraces collaboration between staff and between staff and students. Collaboration will advance excellence in teaching: TEF will not. Collegiality has been shown to increase morale and reinforce academic values in staff and students (Bolden et al, 2012). Analyses of the underlying values of a metric are important because values shape policy, strategies and metrics. ‘Big data’ analysts need to consider ways of incorporating qualitative data. With regard to TEF policy and its metrics, the cautionary note attributed to Einstein is apposite: “Not everything that counts can be counted and not everything that is counted counts.”

SRHE member George Brown was Head of an Education Department in a College of Education and Senior Lecturer in Social Psychology of Education in the University of Ulster before becoming Professor of Higher Education at the University of Nottingham.  His 250 articles, reports and texts are mostly in Higher and Medical Education, with other work in primary and secondary education. He was senior author of Effective Teaching in Higher Education and Assessing Student Learning in Higher Education and co-founder of the British Education Research Journal, to which he was an early contributor and reviewer. He was the National Co-ordinator of Academic Staff Development for the Committee of Vice Chancellors and Principals (now Universities UK) and has served on SRHE Council.

References

Bolden, R et al (2012) Academic Leadership: changing conceptions, identities and experiences in UK higher education London: Leadership Foundation

Gibbs, G (2017) ‘Evidence does not support the rationale of the TEF’, Compass: Journal of Learning and Teaching, 10(2)

Royal Statistical Society  (2018) Royal Statistical Society: Response to the teaching excellence and student outcomes framework, subject-level consultation

Vicky Gunn


Leave a comment

Notes from North of the Tweed: Valuing our values?

By Vicky Gunn

In a recent publication, Mariana Mazzucato1. pushes the reader to engage with a key dilemma related to modern day capitalist economics. ‘Value extraction’ often occurs after a government has valued work upfront through state investment and accountability regimes. The original investment was a result of the collective possibilities afforded by a mature taxation system and an understanding that accountability can drive positive social and economic outcomes (as well as perverse ones). The value that is extracted is then distributed to those already with both financial and social capital rather than redistributed back into the systems which produced the initial work via support from the state in the first place. This means that the social contract between the State and its workers (at all levels) effectively has the State pump prime activity, only to watch the fruits of these labours be inequitably shared.

I find this to be a useful, powerful and troubling argument when considering the current relationship between State funded activity and the governance of UK HE. As a recipient of multiple grants from bodies such as the Higher Education Academy (now AdvanceHE) and the Quality Assurance Agency (now a co-regulatory body in a landscape dominated by the Office for Students), I have observed a similar pattern of activity. What this means is that after a period of state funding (ie taxpayers’ money), these agencies are forced through a change in funding models to assess the value of their pre-existing assets. The change in funding models is normally a result of a political shift in how they are valued by the various governments that established and maintained them. The pre-existing assets are research and policy outputs and activities undertaken in good faith for the purposes of open source communication to ensure the widest possible dissemination and discussion, with an attendant build up in expertise. After valuing these assets, necessary rebranding may obscure the value of this state-funded work behind impenetrable websites in which multiple prior outputs (tangible assets) are pulled into one pdf.  Simultaneously, the agencies offer intangible assets based on relationships and expertise networks back to membership subscribers through gateways – paywalls. This looks like the unregulated conversion of a value network established through the collaboration of state and higher education into a revenue generating system, restricting access to those able to pay.2. If so, it represents a form of value extraction which is limited in how and where it redistributes what was once a part of the common weal.

Scottish HE has attempted to avoid this aspect of changes in the regulatory framework in two ways:

  • Firstly, by maintaining its Quality Enhancement Framework (QEF) in a recognisable form.3. Thus: the state continues to oversee the funding of domiciled Scottish student places; the Scottish Funding Council remains an arms-length funding and policy agency which commissions the relevant quality assurance agency; Universities Scotland continues as a lobbying ‘influencer’ that mediates the worst excesses of external interventions; and the pesky Office for Students is held back at the border, whilst we all trundle away trying to second guess what role metrics will play in the quality assurance of an enhancement-led sector over the next five to ten years. Strategic cooperation and value co-creation remain core principles. And all of this with Brexit uncertainty.
  • Secondly, by refocusing the discussion around higher educational enhancement in the light of a skills agenda predicated not on unfettered economic growth, but on inclusive and sustainable economic growth.4.

Two recent outputs from this context demonstrate the value of this approach: The Creative Disciplines Collaborative Cluster’s Toolkit for Measuring Impact and the Intangibles Collaborative Cluster’s recent publication.5. Both of these projects were valued for the opportunity they provided of collaborative problem solving across Scottish HEIs. Their outputs recognise it is now more important than ever to demonstrate the impact of what we do. Technological advances in rapid, annualised data generation is driving demands to assess the  value of our higher education. The prospect of this demand requiring disciplinary engagement means academics leading their subjects (not just Heads of Quality, DVCs Student Experience, VPs Learning and Teaching) need to be more aware of frameworks of accountability than before. Underneath the production of these outputs has remained a belief in the value of cooperation over the values of competition.

However, none of this means that those of us trying to maintain a narrative of higher education as the widest possible state good can rest on our laurels. If we are to seize this particular moment there are some crucial tensions to problematise and, where appropriate, resolve. We need formal discussion around the following:

  • What is to be valued through State influence in Scottish HE? How does the ‘what is to be valued’ question relate to the values and value of this education socially, culturally and economically?
  • How are these values and value to be valued through the accountability framework for higher education in Scotland?
  • What will the disruptions created by a new regulatory framework in England (based on a particular understanding of value and values) mean for how Scottish institutions continue to engage with the QEF, when they will probably also have to respond to a framework that would like to see itself as UK-wide?
  • How can we protect years of enhancement work from asset stripping and value extraction? How can we continue with an enhancement framework with social, cultural, and economic benefits for Scotland and its wider relationship with the world, at the same time as supporting reinvestment into the enhancement of Scotland’s higher education?
  • There is a push to revalue ‘success’ as simple economic outcomes, away from inter-relational outcomes that capture intangible but nonetheless critical aspects of that education – social coherence, wellbeing, cultural confidence and vitality, collective expertise, innovation, responsible prosperity. That path of value extraction may result in more not less inequality: how can we mitigate it?
  • How can all of this be done without merely retreating to the local? Bruno Latour has noted how locality is a cultural player in the current political inability to engage effectively with the planetary issue of the day: climate crisis.6. He notes the sense of security in the local’s boundaries and a perception across Europe that we somehow abandoned the local in the push to be global. The local is important. Yet, he clarifies, climate regime change means withdrawal into the local in terms of value and values – without interaction across political boundaries at a global level – is tantamount to wilful recklessness. How we can enable higher education to secure the local and the global simultaneously is surely the big question with which we are grappling. How can Scotland’s HE leaders engage to ensure the value and values we embody through our accountability regime do not get mired in local growth agendas unable to measure the impact of that growth within a global ecology?

Sitting within a creative arts small specialist institution, these questions seem both overwhelmingly large (how can a minnow lead such a conversation, surely only a BIG university can do this?) and absolutely essential. In the creative arts our students are, in their own frames of reference, already challenging us on the questions of value, values, environmental sustainability and inequality through their artistry, designerly ethics, and architectural wisdoms. I am, however, yet to hear such a recognisable conversation occurring coherently across the various players (political, policy, institutional) in the wider sector, except in activities related to the localities of cultural policy, the creative economy, and HEI community engagement.7.

Perhaps it is time for sector leaders, social, cultural, and economic policy-makers, and student representatives to work together to identify the parameters of these questions and how we can move forward to resolve them responsibly.

SRHE member Professor Vicky Gunn is Head of Learning and Teaching at Glasgow School of Art.

Notes

  1. Mazzucato, M (2018) The Value of Everything: Making and Taking in the Global Economy,  Penguin, p xv
  2. Allee, V (2008) ‘Value network analysis and value conversion of tangible and intangible assets’, Journal of Intellectual Capital, 9 (1): 5-25.
  3. This 2016 description of the sector’s regulatory framework of enhancement remains broadly the same:  https://wonkhe.com/blogs/analysis-devolved-yet-not-independent-tef-and-teaching-accountability-in-scotland/
  4. See the Scottish Funding Council’s latest strategic framework: http://www.sfc.ac.uk/about-sfc/strategic-framework/strategic-framework.aspx
  5. Enhancement Themes outputs: Creative Disciplines Collaborative Cluster: https://www.enhancementthemes.ac.uk/current-enhancement-theme/defining-and-capturing-evidence/the-creative-disciplines
    Intangibles Collaborative Cluster: https://www.enhancementthemes.ac.uk/current-enhancement-theme/defining-and-capturing-evidence/the-intangibles-beyond-the-metrics
  • Latour, B (2018) Down to Earth: Politics in the New Climatic Regime Polity Press, p 26
  • Gilmore, A and Comunian, R (2016) ‘Beyond the campus: Higher education, cultural policy and the creative economy’, International Journal of Cultural Policy, 22: 1-9


Leave a comment

Academic capitalism – or worse?

[1] By Phil Pilkington

There has been much discussion on academic capitalism, the neo-liberalism of universities, the new entrepreneurial management which is transformative rather than transactional. Much has been done to reinforce this change and self-perception by government (in the UK) from the creation of the HE market, the panoply of measuring instruments (however flawed) and the imperative of third stream income to compensate for the loss of state funding.

However, ‘academic capitalism’ is a misnomer: it does not and cannot exist except in the ‘for profit’ sector. Exogenous forces require universities to generate income, sometimes leading to operational surpluses, making universities appear to act as if they are for profit. This is seen in two ways, neither of which are endogenous: the need for growth to remain operationally viable, and the imposition of regulatory controls of quality, responsiveness to ministerial opinions etc, as a consequence of the irony of a Hayekian model of the public sector[2]. But this is appearance and not reality. The marketisation of HE is an outcome of applying Hayekian principles to the public sector; it is not in itself capitalism. Academics are employees who may have ‘professional ethics’ and hard-earned specialisms which are bought in the labour market. Nevertheless, universities are not generating surplus value, a necessary condition for capitalism, as they have no shareholders with which to extract value from the means of production as capital which can become independent of the labour that produces the profit/surplus. The substantial critique of capitalism was that surplus value could be lost or expropriated from the locale of production and converted into more capital. No matter how much university management and governance may copy, or are required by government to enact, business practices in the neoliberal era (measuring performance, contracting out, international competition, and general entrepreneurialism), the challenge to produce operating surpluses is no different for the National Trust, the Royal Society for the Protection of Birds and other large charities which ‘gift back’ the surplus to the charity from trading activities. Students’ Unions as social enterprises have been doing this for decades with their wet sales supporting their charitable purposes.  Perhaps a discursive view of universities as social enterprises tout court may be helpful for their governance and for deciding how operational surpluses should be applied, perhaps ring-fencing cost centres of undergraduate teaching and research.

Nor is the call for a more skilled and ‘employment ready’ workforce a lowering of universities by promoting an HE sector relevant to and supporting the economy. This purpose was one of the planks of the Robbins Report[3]. This instrumentalist view of education is not new; and this knowledge-servicing role applies to the students as well as the institutional purpose. The polytechnics, and their actual geography, were intended to support the industries of the time, particularly aerospace, steel, chemicals, shipbuilding, mining and the motor industry. What is new is the dissidents’ complaint about the new capitalism of universities, which perhaps obscures some deeper and more disturbing concerns about the situation of universities in the knowledge economy.

The complaints are familiar: a dystopian decline of Western civilisation; the creation of the academic precariat; jeopardising the academic mission; moral decay; avarice; Faustian bargains; dumbing down with student-consumer as sovereign; and so on. But the symptoms of capitalism in HE are shared with other sectors; the precariat as an outcome of the loss of employees’ rights or employers’ opportunities in a flexible labour market; new management systems, and so on. Globalisation is not an essential condition of capitalism but is a late capitalism feature. Global reach has long been a feature of imperial higher education, as in the founding of the LSE, sans academic capitalism. Big science has been international and collaborative for more than a century.

The trappings of capitalism – changes to employment practices, the Taylorism of activity measurement, the creation of a market and the management response, including branding strategies for competitive survival – do not make higher education a capitalist system. The extremes of private enterprise branding go further than universities striving to gain market share and a sense of corporate identity. For example, Sir Philip Green’s channelling money out of BHS ‘in effect monetised the firm’s history as a reliable counterpart for workers and lenders’ (Woodruff, 2018). There are close comparisons in the HE sector in monetising heritage/status: the 100-year financing bonds of Oxbridge suggest a strong leverage through historical branding, compared to say the ability of the University of West London to borrow. Marketisation or branding reinforces the stratified inequalities of the HE system and entrenches the market (Brown, 2018). Nevertheless, this is still not extracting surplus value. Similarly, the contracting out of cleaning services in a local hospital does not make the hospital capitalist. It is the surplus value extracted by the contractor that is capitalist in that, again, the net profit creates capital. Building roads, providing a police force, maintaining state schools etc, are essential for the existence of capitalism (and much else): they are not intrinsically capitalist but act as support services for capital.

Sheldon Rothblatt’s (1997) heart-warming celebration of universities as the second oldest western tradition, that has offered so much in our journey of progress and civilisation, begs the debatable current status of universities as capitalist or indeed even as enduring institutions. (I am reminded of Hobbes’ (1655) paradox of the ship of Theseus which each year came into harbour for a refit:  when is it no longer the same ship? Rothblatt sees similar changes in society and universities but doesn’t argue for a causal relationship.) Universities mutate or emerge over time within the material conditions of power and the economy – from Papal Bull and Royal Charter through Parliamentary legislation, ministerial statutory instruments and finally independent (sic) agency.  Will their function and form now will mirror the current neoliberal conditions of globalisation, public sector regulation and deregulation of the private sector? Not quite. The new cycle of knowledge economies is unlike previous regimes for the supporting universities. It is a different economy for universities and for all of us in some deeper ways.

Universities were, and in some respects still are, the providers of new technologies which advance production and manufacturing processes, new materials and industries as well as long term global opportunities and risk assessments via scientific understanding. Universities have fulfilled that role since the late nineteenth century. What is different is specifically the nature of the new economy which is a new form of market, as capital must search for new markets. This is a new model of business enterprise distinct from manufacturing and traditional service industries – the mining or rather possession (as intellectual property) and exploitation of data as privately-owned property. The case for knowledge as a public good is strong (Marginson, 2013) but the change in what counts as knowledge comes with a stricter control of social conditions. The possession of knowledge begets a new ontology and epistemology. It is the thread that runs through capitalism: the transfer of the public good to the private, from the enclosures onwards, so that old ontological claims appear as delusional, fictions and myths.  Universities and their students are both agents within the knowledge economy and the raw materials.

Universities have been exploited by business in the new technologies with significant growth in profit margins for business, which has enabled the financialisation of business rather than its technological development. The exhortations for universities to provide the materiel for innovation and development obscures another trend.  This is eloquently and passionately explained by Mariana Mazzucato in The Entrepreneurial State (2013). Businesses reduce research costs by contracting out to universities, whilst reaping the major and rapidly increasing profit share which is then used to buy back stockholdings to increase share value. This process continues as competition continues to drive down business research costs , increasing share value (which is capital), and increasing reliance on universities as subcontractors to allow for this business strategy. The knowledge economy is ‘cutting edge’ but universities become a contracting out service industry not just for ‘pushing the technological boundaries’ per se but to be used for capital gain (which is then ‘lost’ to the production process and to the universities’ research centres as surplus value). As Mazzacuto might say, the universities become not so much capitalist agents of the new economies as the exploited.

The knowledge economy flowed from state intervention in the US and the UK but is invisible in plain view. Large corporate R&D centres (Bell, Dupont, Xerox, et al) have largely disappeared and university research in the public domain is used by the new technological businesses. For example, all the Apple innovations of touch screen, GPS, internet, microelectronics, and voice activation were government funded developments.

There is another aspect of the cognitive economy of data and intellectual property ownership which is intimately connected to universities in the new wave of entrepreneurism. This is the monetisation of data within the sector. Not the creation and control of patents resulting from research, but the monetisation of data not previously considered. This new wave of capital is not limited to education. The harvesting of data in the health sector (and the contingent insurance industry) has been a site of contention and dispute in several countries in the last five years[4].

The sale of tranches of the student loan book (losses to the Treasury estimated by the National Audit Office for one tranche as over £600 million) will be a carefully calculated risk for the buyers. In the US student loans are considered as approaching sub-prime liabilities, now with more than a trillion dollars of debt. In the UK part of the value of purchasing student loans stems from the personal data to be harvested for the next 40 years in the interchange between the debtor and the new loan-holders.  A more spectacular example of information converted to a form of capital is the sale of Turnitin.  This is no elevated concept of innovation and development other, simply the ability of capital to create a new ontology of products to be marketed within the cognitive economy. Turnitin was sold in 2008 to Warburg Pincus, then in 2014 to a Singapore-based wealth fund (for $752 million) and finally to a holding company of Condé Nast in 2018 for $1.75 billion. Turnitin has itself acquired other companies in 2018 such as Vericite and Gradescope. Jesse Stommel of the University of Mary Washington, Virginia, noted: “How much of that $1.75 billion is going to the students who have fed their database for years? I have a pretty good guess; zero billion.”

The charge of academic capitalism is misplaced when there should be a growing concern about how late capital will find new ways and practices to exploit the university sector.

Phil Pilkington is Chair of Middlesex University Students’ Union Board of Trustees, a former CEO of Coventry University Students’ Union, an Honorary Teaching Fellow of Coventry University and a contributor to WonkHE.

References

Brown R (2018) ‘Neoliberalism, Marketisation and Higher Education’, Professorial Lecture, University of West London

Hobbes T (1655) De Corpore

Marginson S, (2013) ‘The Impossibility of Capitalist Markets in Higher Education’, Journal of Education Policy 28(3)

Mazzucatto, M (2013) The Entrepreneurial State, Anthem Press

Rothblatt S (1997) The Modern University and its Discontents, Cambridge

Woodruff D (2018) ‘Profits Now, Costs Later’, London Review of Books 40(22)

[1] I am extremely grateful to Ian McNay for his advice and support; the faults here remain mine

[2]  Hayek F (1944) The Road to Serfdom. It is difficult not to take an ad hominem approach to Hayek as a friend of dictators, but also as a paradoxical, confused and failed political theorist; his concept of price as information when human thought is irrational is a foundation of the current dispensation. For neo-liberal policy makers (ie the government/OfS) the uniformity of price in the UK HE sector offends against rational market efficiencies driving down prices. The consequent conspiracy hypothesis of cartel price fixing is another neo-liberal trope: the sabotage of government policies by self-interested public sector management and civil servants. Some university leaders have supported the neo-liberal project on the rationality of price levels by suggesting that the artificial limit set by government at the top end should be lifted and the market could be liberated to compete with Ivy League fee levels (prices). There are alternative models of pricing: a holistic model for price for HE could include prior costs (school fees, private tuition, the housing market reflecting catchment areas, etc).

[3]   My thanks to Ian McNay for a reminder that the Robbins Report included ‘the instruction in skills suitable to play a part in the general division of labour…’ and pointing out the skills support by polytechnics for the heavy industries in the north and north east of England. The assumption throughout the 1940s and 50s, from the 1944 Education Act and Claus Moser’s statistical planning of HE for Education Minister David Eccles, was that 80% of the workforce would be engaged in manufacturing and manual work.

[4] In the UK, following the Health and Social Care Act (2012) there was a requirement that all GPs’ case notes be returned to the central care.data to be exploited commercially, this appears to have been abandoned in 2016 after a campaign in part organised by GPs. In Denmark there was a rescinding of a similar arrangement. Data sharing (sic) between Deep Mind (the AI branch of Google) and a London NHS Trust was considered by the Information Commissioner’s Office to be a breach of law. In Italy a deal was made with IBM in 2016 for access to health data (Source: New Scientist, April 2016).

Paul Temple


Leave a comment

Merchant rankers

by Paul Temple

I blogged a while back on THE’s transformation from a publisher of news and opinions on higher education to a producer and vendor of rankings data. Every issue of the magazine it seems now comes with the latest rankings publication, often thicker than the parent publication. The latest one that I’ve seen gives the “2019 University Impact Rankings”. You’ve got to admire the ingenuity of THE’s Chief Knowledge Officer, Phil Baty, and his team in dreaming up ever-more varied ways of ranking universities – and the cleverness of these latest rankings, examining contributions to the UN Sustainable Development Goals (SDGs), is that a wider range of universities than just the usual suspects can claim their place in the sun. So expect to see Kyung Hee University in South Korea boasting of its top world ranking under SDG 11, “Sustainable cities and communities”.

The UN has developed 17 SDGs taking in a wide sweep of worthwhile objectives, including peace, health, welfare, equalities, sustainability, and more. Probably all universities contribute in different ways to many of these goals, but how should their varying achievements in this field be ranked? Well, the difficulty of adding incommensurables together to produce a single number in order to create a league table has never so far got in the way of people with a ranking product to sell. So you won’t be surprised to hear that it turned out to be a piece of cake to add a university’s contribution to, say, “good health and wellbeing”, to a number reflecting its work on “gender equality”, to its number on “climate action”, to compare that total number to a number from a university on the other side of the world which says it contributes to a different set of SDGs – and to come up with a league table. (The University of Auckland came top, since you ask.)

As I said in my earlier blog about the THE annual university awards, you might think, where’s the harm in universities doing a bit of mild boasting about their contributions to perfectly worthwhile aims? Well, I think there are a couple of problems. One was brought home to me recently at a graduation ceremony, where the speech by the presiding member of the UCL brass was almost entirely about how well UCL and its constituent parts had done in the recent QS rankings. This both misleads families and friends, and probably many graduates, into thinking that rankings are some sort of unarguable, football league-style assessment, with a university’s work being counted in the same way as a team’s goals. But it also misses an opportunity to tell your own institutional story – “we are a terrific university, and this is why” – rather than sub-contracting the job to someone with a commercial axe to grind. What happened to institutional self-confidence?

The other problem is that the more universities appear to buy in to rankings like these, the more THE and other rankers are encouraged to offer consultancies based on their rankings. This is dangerous territory. Rather than claiming, however implausibly, that their consultancy services are entirely separate from their rankings activities, THE goes out of its way to link them. Imagine then a marketing director of a university in difficulties of some sort reading the several full-page ads for THE’s consultancy services in the Impact Rankings publication, with their offers of “expert guidance” and “tailored analysis for advancement” drawing on THE’s “deep expertise” with THE experts becoming “an extension of…universities’ marketing departments”. It wouldn’t be surprising if they thought, “Hmmm, maybe working with these guys might help us move up some of these rankings – at least we’d understand more about how they’re put together and we might then make some changes in what we do….”

So sets of methodologically worthless data become turned into income streams for rankings producers because university leaderships take them seriously, which in turn will drive universities’ policy-making in the direction of moving up one league table or another, which in turn will encourage rankers to produce even more league tables in order to exert more power. How on earth did we allow this to happen?

SRHE member Paul Temple, Centre for Higher Education Studies, UCL Institute of Education, University College London.


Leave a comment

What is an academic judgement?

By Geoff Hinchliffe

Academics make academic judgements virtually every working day. But what exactly is an academic judgement? As a starting point, one might have recourse to appropriate statutory documents: for example, the 2004 Higher Education Act mentions that a student complaint does not count as a ‘qualifying’ complaint if it relates to matters pertaining to an ‘academic judgment’ (Higher Education Act, 2004, p5, Section 12). The Office of the Independent Adjudicator (OIA) helps to provide a gloss on the term:

“Academic judgment is not any judgment made by an academic; it is a judgment that is made about a matter where the opinion of an academic expert is essential. So for example a judgment about marks awarded, degree classification, research methodology, whether feedback is correct or adequate, and the content or outcomes of a course will normally involve academic judgment.” (OIA, 2018, Section 30.2)

But although it is heartening to see that some deference is paid to academic judgment, little light is thrown on what it actually is. This can, of course, be useful: for example, Cambridge University’s (2018) complaint procedure quotes the OIA definition without further elaboration. Providing no-one is prepared to question the nature of academic judgement, who are we to complain? But, at the risk of disturbing sleeping dogs, I propose to enquire more closely as to what constitutes an academic judgement.

Two points are worth making at the outset. The first is that academic judgements should not be construed as the special preserve of those designated as ‘academics’. Students also make academic judgements along the same lines as academics, so it’s not the case that academics make special judgements that students couldn’t possibly understand. The second point is that the object of judgements – what is being judged – may vary considerably, but the kind of judgement being made is still of the same type. The elements of judgement remain the same whether the object of scrutiny is a first year undergraduate essay or a paper in a leading journal.

It seems to me that there are four basic types of academic judgement and they frequently operate in combination – it is this that gives the whole business of judgement its mystique and rarefied quality.

1. Process judgements

When we evaluate any kind of process that has (or is supposed to have) a definable outcome we tend to use a set of criteria, although the latter may be carefully delineated or may operate as background guidelines. Examples are the following of some clinical procedure (where the criteria are strict) or following laboratory protocols (ditto). But they could also include the evaluation of the methodology in a piece of empirical research, in which we consider the suitability of the methodology used and its success or otherwise. Process judgements are also entailed in the evaluation of essays in terms of structure and argumentation leading up (hopefully) to a conclusion. In this case, we evaluate the process of writing and structuring an essay or report; we consider, for example,  whether the author has ‘signposted’ the argument so that its trajectory has some kind of sense and cohesion. Process judgements tend to be ‘rule-governed’. But as I have indicated, in some cases the rules are pretty clear and in other cases there is much room for flexibility. So, regarding essays, there are no fixed rules for demonstrating a process of argumentation; but neither are there no rules at all. In the film Pirates of the Caribbean it is explained that the Pirate’s Code is not to be adhered to strictly at all times because it is not so much a fixed code as ‘guidelines’. Very sound advice too.

For those interested in philosophy, process judgements are roughly akin to what Wittgenstein thinks of as ‘following a rule’ (see Wittgenstein, 1958, paras 201-2). In this case, what Wittgenstein has in mind are the rules for using words so that they have a meaning that is understood. But since meanings are never fixed (unless through prior stipulation) then they are indeed ‘guidelines’. Who would have thought a pirate would be reading Wittgenstein?

2. Epistemic Judgements

In the case of epistemic judgements we are assessing claims to knowledge. That is, we are assessing whether the claimant has sufficient evidence and reasons for making a knowledge claim. Of course, we are also interested in the context – that is whether and to what extent the claimant is aware of relevant context that may affect the claims they are making. Furthermore, we are often reluctant to make positive judgements if the claim simply asserts a proposition – to the effect that such-and such is the case – even if the proposition is true. We need to see the evidence and reasoning that back up the knowledge claim – bare assertions are usually not enough.

There are two further features of epistemic judgements: first they are objective, in the sense of being propositional – they purport to say ‘how the world is’. Second, they are universal in the sense that in making such a judgement I am claiming that everyone will reach the same conclusion as myself. Of course, others may disagree but the idea is that, in principle, these disagreements are adjudicable (Steinberger, 2018, p38).

Notice that if I am marking a student paper then I am assessing the kinds of epistemic judgements the student is making – whether the claims made are true and whether they are well founded. It’s not the case that the student is doing one thing and I am doing something else – the writer of the paper and the assessor are both making the same kind of judgements. That is, the student needs to be in the habit of assessing their own epistemic judgements as to evidence and reasoning in exactly the same way that I, the assessor, am doing. The process is the same: the only difference is that in the one case the outcome is a paper and in the other, the outcome is a mark.

3. Reflective Judgements

This kind of judgement is tricky to explain but I think readers will see its importance. By ‘reflective’ I don’t mean the reflective judgement beloved of writers of practitioner manuals where ‘reflective’ means ‘self-reflective’. Thus interpreted, reflection is usually reflection on a procedure and one’s part in it. In other words, practitioner self-reflective judgements are really a kind of process judgement.

What I have in mind as ‘reflective’ is when we think of a piece of data, a theory, or a concept in functional or relational terms. We look for a broader framework within which phenomena can be better understood. Thus, Kant thought that when we reflect on a natural phenomenon we situate nature in a purposive or teleological framework, in order to provide a kind of interpretive unity (Kant, 2000, p67).  More generally, we can think of reflective judgements as contextual: we look for links and relationships in order to make sense of the object of study, to bring some sense of order and unity to bear. Reflective judgements can be highly creative when links are made between phenomena that weren’t thought of before. Quite a lot of Foucault’s work was of this type – for example, the way in which he related different kinds of formative behaviours into the notion of the disciplinary: in seeing that behaviours in schools, prisons, hospitals and the like were produced and reproduced he was able to fashion a new concept – the ‘disciplinary’ – which gives us real insight into how modernity works.

Peter Steinberger (2018, pp47-50) explains the nature of reflective judgements rather well, in my view – he sees reflective judgements as different from what I have called epistemic judgements (following Kant, he calls the latter ‘determinate’ judgements). What a reflective judgement does is to provide an interpretive context in which different knowledge claims can be related and thus better understood. Reflective judgements operate at the level of meaning. When we ask our students to make the links both within and across modules we are asking them to think reflectively.

4. Normative Judgements

We need to be clear that normative judgements are not the same as ethical judgements. I see the latter as delivering a verdict on the worth or rightness of a person or action. As such, ethical judgements play (or should play) a minor role in academic research and production. It is irritating if a historian gives us ethical verdicts on her subject matter (Henry was a good king, but King John was a bad one) – ethical judgements are almost always unnecessary.

But normative judgements are something else. They involve according due sensitivity to the values and norms associated with the subject matter under consideration. By their very nature, normative judgements are contextual. For example, if a student fails to appreciate the nature of religiosity in fifteenth century England (for example, by seeing it in terms of ignorance and superstition because of a modernist, secular approach which the student brings to bear) then it is the normative judgement that has gone awry. Or, if a research methodology fails to take due account of the needs of confidentiality, again, it is a normative judgement that is deficient. Normative judgements often operate in combination with other judgements (especially with process and reflective judgements) and this is one of the reasons why academic judgement can be complex.

Conclusion

If I am right then there are four basic elements to an academic judgement. Typically in any assessment all four elements are operating together – process, epistemic, reflective and normative. The judgements we use are precisely those that we want our students to develop. We can see straight away that attempts to categorise and tabulate all of these elements may be helpful but are unlikely to be comprehensive. The precise nature of the judgement will vary according to subject matter and no set of assessment criteria that I have seen comes anywhere near to giving full justice to the complexity of the judgements involved. Moreover, most of those outside academia (government ministers, MPs, media people and the like) are just clueless regarding how much they know of this complexity.

Complex, yes: but not so complex that we can’t attempt to say what is involved in giving an academic judgement. But the above sketch cannot be the last word – if I have succeeded in suggesting some initial ‘guidelines’ then that is a start.

SRHE member Geoff Hinchliffe teaches undergraduates in the School of Education at the University of East Anglia. This blog is partly based on a paper he gave at the 2018 SRHE Research Conference.

Bibliography

Cambridge University (2018) Student Complaints https://www.studentcomplaints.admin.cam.ac.uk/general-points-about-procedures/academic-judgment

Higher Education Act (2004) http://www.legislation.gov.uk/ukpga/2004/8/pdfs/ukpga_20040008_en.pdf

Kant, I (1933) Critique of Pure Reason, trans. Kemp-Smith, N, London: Macmillan

Kant, I (2000) Critique of the Power of Judgement, trans. Guyer, P and Matthews, E, Cambridge: Cambridge University Press

OIA (2018) Guidance Note on the OIA Rules http://www.oiahe.org.uk/rules-and-the-complaints-process/guidance-note-on-the-oias-rules.aspx#para30

Steinberger, P.J (2018), Political Judgement, Cambridge: Polity Press

Wittgenstein, L (1958) Philosophical Investigations, Oxford: Blackwell


1 Comment

Having faith in the university

by Søren SE Bengtsen and Ronald Barnett

A heightened gap between the university and society is now evident. On the policy level, discourses of excellence, world-classness and value-for-money press upon universities while, on the societal level, there are calls for impact, skills, employability and marketable knowledge. Additionally, in a post-truth and fake news era, universities struggle to establish their legitimacy, and some students even report that they may actually be doing themselves a disfavour by taking a higher education degree. All this is symptomatic of a wide societal, and even worldly, sudden loss of faith in the university. Continue reading


1 Comment

Metrics in higher education: technologies and subjectivities

by Roland Bloch and Catherine O’Connell

The changing shape of higher education and consequent changes in the nature of academic labour, employment conditions and career trajectories were significant Continue reading


1 Comment

The ‘Holy Grail’ of pedagogical research: the quest to measure learning gain

by Camille Kandiko Howson, Corony Edwards, Alex Forsythe and Carol Evans

Just over a year ago, and learning gain was ‘trending’. Following a presentation at SRHE Annual Research Conference in December 2017, the Times Higher Education Supplement trumpeted that ‘Cambridge looks to crack measurement of ‘learning gain’; however, research-informed policy making is a long and winding road.

Learning gain is caught between a rock and a hard place — on the one hand there is a high bar for quality standards in social science research; on the other, there is the reality that policy-makers are using the currently available data to inform decision-making. Should the quest be to develop measures that meet the threshold for the Research Excellence Framework (REF), or simply improve on what we have now?

The latest version of the Teaching Excellence and Student Outcomes Framework (TEF) remains wedded to the possibility of better measures of learning gain, and has been fully adopted by the OfS.  And we do undoubtedly need a better measure than those currently used. An interim evaluation of the learning gain pilot projects concludes: ‘data on satisfaction from the NSS, data from DHLE on employment, and LEO on earnings [are] all … awful proxies for learning gain’. The reduction in value of the NSS to 50% in the most recent TEF process make it no better a predictor of how students learn.  Fifty percent of a poor measure is still poor measurement.  The evaluation report argues that:

“The development of measures of learning gain involves theoretical questions of what to measure, and turning these into practical measures that can be empirically developed and tested. This is in a broader political context of asking ‘why’ measure learning gain and, ‘for what purpose’” (p7).

Given the current political climate, this has been answered by the insidious phrase ‘value for money’. This positioning of learning gain will inevitably result in the measurement of primarily employment data and career-readiness attributes. The sector’s response to this narrow view of HE has given renewed vigour to the debate on the purpose of higher education. Although many experts engage with the philosophical debate, fewer are addressing questions of the robustness of pedagogical research, methodological rigour and ethics.

The article Making Sense of Learning Gain in Higher Education, in a special issue of Higher Education Pedagogies (HEP) highlights these tricky questions. Continue reading


1 Comment

It’s all about performance

by Marcia Devlin

The Australian federal government has indicated its intention to introduce partial funding based on yet to be defined performance measures.

The Mid-Year Economic and Fiscal Outlook (MYEFO) by the Australian government updates the economic and fiscal outlook from the previous budget and the budgetary position and revises the budget aggregates taking account of all decisions made since the budget was released. The 2017-2018 MYEFO papers state that the Government intends to “proceed with reforms to the higher education [HE] sector to improve transparency, accountability, affordability and responsiveness to the aspirations of students and future workforce needs” (see links below). Among these reforms are performance targets for universities to determine the growth in their Commonwealth Grant Scheme funding for bachelor degrees from 2020, to be capped at the growth rate in the 18-64 year old population, and from 1 January 2019, “a new allocation mechanism based on institutional outcomes and industry needs for sub-bachelor and postgraduate Commonwealth Supported Places”. Continue reading


Leave a comment

How likely are BTEC students to enter higher education?

By Pallavi Amitava Banerjee

 

Business and Technology Education Council (BTEC) qualifications are seen by some as prized qualifications for the labour market which draw on work-based scenarios. Providers claim these career-based qualifications are designed Continue reading