SRHE Blog

The Society for Research into Higher Education

Tag Archives: epistemic judgement


by 1 Comment

The ‘unpoliceability’ of AI journal policies

by Dirk Lindebaum

Core to the mission of SRHE is the commitment to high-quality research and scholarship to advance understanding of higher education. This focus makes the conditions under which research is produced, including editorial and journal policies concerning the publication process, a topic of natural interest to readers of the SRHE blog.

The publication process, and how stakeholders interact in that process, however, is profoundly changing due to the widespread use of generative AI (hereafter AI). Consider the irony that authors can now offload their writing to AI; that reviewers can use AI to perform reviews on AI-produced text; and that editors can use AI to make decisions on papers written by AI, and grounding decisions in AI-produced reviews. It is machines chatting with machines, and machines reacting to machines, just without the informed doubt, the willingness of daring to know, the realization that we as social scientists always condition and are conditioned through our work and social practices that follow from our work.

Notwithstanding this development, the use of AI in the research process is routinely and boldly argued to be ‘inevitable’. What is less often understood is that this narrative likely accelerates an already existing crisis of credibility in some fields (eg in management research). A sobering look at the available empirical evidence (summarized here) underlines that the crisis is real.

First, empirical studies suggest that, in 2025, more than 140,000 non-existing references have infiltrated scientific records. Second, while the quality of scholarship is decreasing (as indicated through a measure of language complexity), the use of AI helps researchers to produce more papers and faster (with direct effects on individual careers). All this unfolds against an already disadvantageous baseline, where disruptive and innovate research occurs less often. Third, while many AI journal polices exhort scholars to be the ‘human-in-the-loop’ to check the accuracy of AI outputs, empirical studies suggest that AI biases (caused by human biases in the training data) transfer and magnify in humans through epistemic feedback loops in Human-AI interactions. Fourth, many authors neglect to disclose AI-use, the consequence being that many AI journal policies fail to ensure transparency or constrain AI adoption. Hence, many AI journal policies concerning the use of AI may attempt to ‘police the unpoliceable’, despite the well-intending motives behind such policies.

After years of editorial frontline experience, I believe two policy moves are necessary to counter these developments which, if unaddressed, likely compromise the integrity of science. On the one hand, qualitatively speaking, I propose a set of evaluative criteria concerning how to evaluate (as editors and reviewers), and prepare submissions (as authors), in ways that mobilise judgment to strengthen the voice of authors in terms of mastery of the literature, method, analysis, contribution and recommendations for practice. Developing editor and reviewer judgment to strengthen author voices, and for authors to force themselves to nurture said judgment to inform their own voices distinctly and uniquely, is a matter of paramount importance when the defining scarcity of our time is not access to information or even knowledge, but rather the capacity to develop and apply sound judgment in the presence of seemingly boundless AI slop. I treat judgment here as epistemic judgment concerning reflexive justifications for truth claims – to oneself and others under conditions of uncertainty – and as expressed through the authors’ voice.

This effort has to be earned. This means persisting with developing judgment over time, not despite the struggle involved, but because of it. This is exactly why this struggle matters for AI advocates. While some argue that one needs to be an expert to use the AI in research, this point is often based on decades of building expertise before the arrival of AI. Yet, current socialization practices are such that junior colleagues and PhD students may not find the conditions in which to develop expertise. This does and will negatively affect expertise levels now and in future. The result is that the conditions of re-building expertise shrink, and that means the possibility to recover lost expertise decreases. It seems to me a caution that needs to be put plainly – this should concern usall. However, these ‘qualitative’ criteria are unworkable without also creating the conditions in which editors and reviewers can manage the current deluge of journal submissions.

Therefore, the second, quantitative, proposal is to introduce a submission cap per authors per annum (as assessed by ORCID IDs), given record numbers of submissions to many journals across science subjects. This would provide necessary ‘breathing space’ for often overstretched (volunteering) editors and reviewers, while promoting equitable access to editorial resources, reduce congestion in the peer review pipeline, and discourage the use of the journal as a preliminary check on manuscript submission readiness. Lastly, a submission cap can also counter the creeping ‘junkification’ of science – the mounting volume of low-quality content that permeate academic publishing, which will only intensify in the age of Gen-AI.

Subject to amendments to suit the needs of a particular journal, the proposal would work like this: each author can submit three new submissions per year to a journal, but with a twist to encourage authors to improve the quality of first submission. For each invitation to revise and resubmit a paper, authors could submit one additional paper to a journal (3 + 1). And if they are lucky to get a second (or third) invitation to revise and resubmit a paper, then that number would be 3 + 2 etc. The expansion is author-specific; if submission A gets a revise and resubmit, every author listed on submission A receives +1 to their personal limit at this journal. This expansion would apply to new submissions, and not to invited revisions.

Some colleagues may have reservations about the proposal due to a perceived limit to their productivity. However, let’s appreciate that even if a prolific scholar manages to publish three articles in each of the best five journals in their fields per year, these 15 articles can hardly be said to be detrimental to the scholar’s career prospects.

A submission cap is consistent with scientific progress because it helps editors and reviewers regain vital space to write quality letters and reviews based on the proposed new set of evaluative criteria. Hence, it is key to consider the complementary nature of both proposals, with the benefit of enabling the conditions under which published works can be trusted (more) in the age of Gen-AI. Epistemic agency would flow back to journals and their editors, for academics would set the agenda (rather than big tech).

Rather than advocating contra AI in the research process, my stance here is pro developing human judgment. If developed and assessed well in the research process, the focus on judgment will render the debate around AI journal policies of secondary relevance. Why try to police the unpoliceable if investing in human judgment is one of the best ways to safeguard the integrity of science?

Dirk Lindebaum is Professor of Management & Organisation at the University of Bath, and advocate and owner of “AI-Free Scholarship” which is a registered (noncommercial) trademark that applies to his writing and teaching.


by 1 Comment

Vibe-coding and co-creation: human-AI collaboration in higher education

by Andrew Williams

The debate around artificial intelligence (AI) in higher education has largely focused on students. Are they using ChatGPT in assessments? How do we maintain academic integrity? How should teachers and curricula adapt?

Much less attention has been given to academics working with generative AI (GenAI) as a creative collaborator or co-pilot, rather than merely attempting to mitigate its impact. The potential for productive and effective AI collaboration deserves greater consideration because it challenges longstanding assumptions about expertise, creativity and educational design.

Ever since the release of ChatGPT in November 2022, I have been fascinated by how intuitive the interaction with GenAI tools can be and the ease with which ideas can be explored through ordinary conversation. GenAI is a sophisticated technology that allows users to test possibilities, refine outputs, and rapidly turn preliminary concepts into usable material.

My background is in biological sciences, with no grounding or expertise in computer science, software engineering or coding. For me, this technology has helped me overcome technical barriers and allowed me to explore new ways to teach and to develop bespoke educational resources. I envisioned a learning experience for my undergraduate medical science students (studying human genetics and evolution) that moved beyond static textbook diagrams and dry explanations to a more interactive and visually stimulating format. Realising that vision required a simulation that let students tweak parameters, observe real‑time population dynamics, and grasp complex evolutionary ideas.

Traditionally, such a project would rely on grants, external software developers, or months of self‑taught coding. For most of us, the gap between an innovative pedagogical idea and a digital reality was confined by technical expertise. Today, through iterative collaboration with GenAI, natural language can function as a design interface, allowing educators to transform ideas into working educational tools.

I was also intrigued by the ever improving, and often publicised, capabilities of GenAI tools as coding agents. With no coding experience myself, I decided to try my hand at ‘vibe-coding’.

Natural language as a design interface

‘Vibe-coding’ is an emerging term within technology communities that describes a process where functional intent is communicated via natural language prompts, allowing AI to generate and refine code. This allows the user to focus on what the tool should do rather than how it must be coded. Essentially, ‘vibe-coding’ involves directing the AI to write and co-develop code aligned with your vision and pedagogical objectives. In other words – no code, no problem!

My recent research describes an iterative prompting strategy and the outcomes of a human-AI collaboration to create an interactive simulation that models certain aspects of evolution (Williams, 2026a). Instead of learning JavaScript or working with software developers, I co-created an application with ChatGPT using only natural language prompting. The result was a browser-based simulation capable of supporting inquiry-based learning and formative assessment in undergraduate education.

Rather than blindly generating content based on loosely defined instructions, the human-AI collaboration involved a structured iterative prompting strategy, careful testing of AI-generated output and re-prompting until the desired aims of the project were met. This enabled the creation of an interactive simulation in which students could investigate biological systems, manipulate variables, export datasets and construct evidence-based explanations (figure 1). This allowed students to move beyond consuming information, to generating and interpreting data themselves in real-time.

Figure 1. Snapshot of ‘Survival Island’ – an interactive simulation.

The Illusion of automated expertise

There is a prevailing narrative that AI democratises expertise, in that it allows anyone to do anything more quickly and with less effort. My experience of ‘vibe-coding’ suggested something different. The  credibility of the final simulation depended on actively embedding disciplinary expertise directly into the co-design process.

While the AI had removed the technical barrier (the coding), it had increased the importance of my academic expertise and oversight. The AI could generate a thousand lines of code in seconds, but it couldn’t tell me if the code represented a plausible scientific simulation. My epistemic judgement was required to verify the AI-generated output. The final student-ready simulation required multiple rounds of iteration, testing and re-prompting, until the collaboration achieved a credible and biologically grounded application suitable for teaching.

Rather than replacing my expertise, ChatGPT provided capabilities I do not possess: writing HTML; JavaScript; debugging code. I contributed disciplinary knowledge, pedagogical intent and continuous evaluation. Every feature reflected educational decisions rather than technical ones. The simulation succeeded because the educational need was clearly defined before AI generated the code.

Higher education has often treated GenAI as an automated assistant or as a problem to be mitigated. Increasingly, it may be more productive to think of AI as a collaborator whose contributions depend upon human judgement. Expertise remains indispensable, as humans ultimately remain responsible for judging whether outputs are educationally, scientifically and ethically sound.

Beyond the prompt: the AI literacy gap

Universities have understandably invested considerable effort in developing student AI literacy. This project led me to a broader appreciation of the need to promote teacher AI literacy across higher education, a critical area often overlooked.

AI literacy for teachers is not just about knowing how to write a good prompt or choosing the best online GenAI tool. Rather, it is a multi-dimensional competency that considers foundational knowledge (prompting, tool use), ethical, emotional and technical proficiencies. For educators to co-create teaching resources in collaboration with AI, for example through vibe-coding, more than technical curiosity is required.

The iterative ‘vibe-coding’ process I followed mirrors the ‘critical evaluation’, ‘epistemic’ and ‘innovation’ dimensions of an integrated AI literacy framework for higher education teachers (Williams, 2026b). This recently published conceptual framework comprises eight core AI literacy dimensions, which you can access in full using the link.

In the current context, the important  AI literacy dimensions include:

  • Critical evaluation: the ability to treat the AI model as a “black box” and rigorously verify outputs against authoritative sources.
  • Epistemic authority: a clear understanding of the probabilistic nature of GenAI and the domain-specific expertise to judge AI output, thereby retaining human agency during interactions with AI.
  • Innovation mindset: the willingness to move from being a consumer of tools to a co-creator of bespoke learning environments, including an understanding of how to collaborate effectively with GenAI.

These are becoming essential academic capabilities rather than optional technical skills. This transition presents considerable challenges, as academic practice has historically been predicated on either individual resource creation or the outsourcing of technical development to third parties. GenAI now enables academics to become directors of increasingly sophisticated creative processes. They define the educational challenge, AI performs the technical implementation, and academics evaluate and verify the final outcome.

As technical production becomes easier with GenAI, educational judgement becomes increasingly important. Our value is no longer found in the technical labour of production, but in our ability to define the educational challenge, critique the iterations, and curate the final outcome. The “vibe” we are coding is our pedagogical intent.

Key takeaways

  • AI is a tool, not a replacement.
  • Human-AI collaboration enables pedagogical creativity.
  • ‘Vibe-coding’ uses AI’s coding capabilities but demands epistemic oversight.
  • Teacher AI literacy is multi‑dimensional.
  • Institutional support is required to build that AI literacy.

GenAI is redefining the landscape of higher education, not through incremental improvements to traditional teaching materials, but by enabling entirely new educational content to be created. However, despite the promises of this new technological revolution, we still need robust evidence about student learning gains, engagement, accessibility and the sustainability of AI-generated educational resources.

It is easy to provide AI tools for HE educators to use. The challenge is to give educators the technical, critical and ethical literacy required to use them effectively.

Andrew Williams is a Professor (Teaching) in the Faculty of Medical Sciences at UCL. He teaches immunology and cell and molecular biology to undergraduate and postgraduate students. His research aims to explore the impact of artificial intelligence (AI) on assessment and learning in higher education and the ways in which we can promote student and staff AI literacy.


by Leave a comment

What is an academic judgement?

By Geoff Hinchliffe

Academics make academic judgements virtually every working day. But what exactly is an academic judgement? As a starting point, one might have recourse to appropriate statutory documents: for example, the 2004 Higher Education Act mentions that a student complaint does not count as a ‘qualifying’ complaint if it relates to matters pertaining to an ‘academic judgment’ (Higher Education Act, 2004, p5, Section 12). The Office of the Independent Adjudicator (OIA) helps to provide a gloss on the term:

“Academic judgment is not any judgment made by an academic; it is a judgment that is made about a matter where the opinion of an academic expert is essential. So for example a judgment about marks awarded, degree classification, research methodology, whether feedback is correct or adequate, and the content or outcomes of a course will normally involve academic judgment.” (OIA, 2018, Section 30.2)

But although it is heartening to see that some deference is paid to academic judgment, little light is thrown on what it actually is. This can, of course, be useful: for example, Cambridge University’s (2018) complaint procedure quotes the OIA definition without further elaboration. Providing no-one is prepared to question the nature of academic judgement, who are we to complain? But, at the risk of disturbing sleeping dogs, I propose to enquire more closely as to what constitutes an academic judgement.

Two points are worth making at the outset. The first is that academic judgements should not be construed as the special preserve of those designated as ‘academics’. Students also make academic judgements along the same lines as academics, so it’s not the case that academics make special judgements that students couldn’t possibly understand. The second point is that the object of judgements – what is being judged – may vary considerably, but the kind of judgement being made is still of the same type. The elements of judgement remain the same whether the object of scrutiny is a first year undergraduate essay or a paper in a leading journal.

It seems to me that there are four basic types of academic judgement and they frequently operate in combination – it is this that gives the whole business of judgement its mystique and rarefied quality.

1. Process judgements

When we evaluate any kind of process that has (or is supposed to have) a definable outcome we tend to use a set of criteria, although the latter may be carefully delineated or may operate as background guidelines. Examples are the following of some clinical procedure (where the criteria are strict) or following laboratory protocols (ditto). But they could also include the evaluation of the methodology in a piece of empirical research, in which we consider the suitability of the methodology used and its success or otherwise. Process judgements are also entailed in the evaluation of essays in terms of structure and argumentation leading up (hopefully) to a conclusion. In this case, we evaluate the process of writing and structuring an essay or report; we consider, for example,  whether the author has ‘signposted’ the argument so that its trajectory has some kind of sense and cohesion. Process judgements tend to be ‘rule-governed’. But as I have indicated, in some cases the rules are pretty clear and in other cases there is much room for flexibility. So, regarding essays, there are no fixed rules for demonstrating a process of argumentation; but neither are there no rules at all. In the film Pirates of the Caribbean it is explained that the Pirate’s Code is not to be adhered to strictly at all times because it is not so much a fixed code as ‘guidelines’. Very sound advice too.

For those interested in philosophy, process judgements are roughly akin to what Wittgenstein thinks of as ‘following a rule’ (see Wittgenstein, 1958, paras 201-2). In this case, what Wittgenstein has in mind are the rules for using words so that they have a meaning that is understood. But since meanings are never fixed (unless through prior stipulation) then they are indeed ‘guidelines’. Who would have thought a pirate would be reading Wittgenstein?

2. Epistemic Judgements

In the case of epistemic judgements we are assessing claims to knowledge. That is, we are assessing whether the claimant has sufficient evidence and reasons for making a knowledge claim. Of course, we are also interested in the context – that is whether and to what extent the claimant is aware of relevant context that may affect the claims they are making. Furthermore, we are often reluctant to make positive judgements if the claim simply asserts a proposition – to the effect that such-and such is the case – even if the proposition is true. We need to see the evidence and reasoning that back up the knowledge claim – bare assertions are usually not enough.

There are two further features of epistemic judgements: first they are objective, in the sense of being propositional – they purport to say ‘how the world is’. Second, they are universal in the sense that in making such a judgement I am claiming that everyone will reach the same conclusion as myself. Of course, others may disagree but the idea is that, in principle, these disagreements are adjudicable (Steinberger, 2018, p38).

Notice that if I am marking a student paper then I am assessing the kinds of epistemic judgements the student is making – whether the claims made are true and whether they are well founded. It’s not the case that the student is doing one thing and I am doing something else – the writer of the paper and the assessor are both making the same kind of judgements. That is, the student needs to be in the habit of assessing their own epistemic judgements as to evidence and reasoning in exactly the same way that I, the assessor, am doing. The process is the same: the only difference is that in the one case the outcome is a paper and in the other, the outcome is a mark.

3. Reflective Judgements

This kind of judgement is tricky to explain but I think readers will see its importance. By ‘reflective’ I don’t mean the reflective judgement beloved of writers of practitioner manuals where ‘reflective’ means ‘self-reflective’. Thus interpreted, reflection is usually reflection on a procedure and one’s part in it. In other words, practitioner self-reflective judgements are really a kind of process judgement.

What I have in mind as ‘reflective’ is when we think of a piece of data, a theory, or a concept in functional or relational terms. We look for a broader framework within which phenomena can be better understood. Thus, Kant thought that when we reflect on a natural phenomenon we situate nature in a purposive or teleological framework, in order to provide a kind of interpretive unity (Kant, 2000, p67).  More generally, we can think of reflective judgements as contextual: we look for links and relationships in order to make sense of the object of study, to bring some sense of order and unity to bear. Reflective judgements can be highly creative when links are made between phenomena that weren’t thought of before. Quite a lot of Foucault’s work was of this type – for example, the way in which he related different kinds of formative behaviours into the notion of the disciplinary: in seeing that behaviours in schools, prisons, hospitals and the like were produced and reproduced he was able to fashion a new concept – the ‘disciplinary’ – which gives us real insight into how modernity works.

Peter Steinberger (2018, pp47-50) explains the nature of reflective judgements rather well, in my view – he sees reflective judgements as different from what I have called epistemic judgements (following Kant, he calls the latter ‘determinate’ judgements). What a reflective judgement does is to provide an interpretive context in which different knowledge claims can be related and thus better understood. Reflective judgements operate at the level of meaning. When we ask our students to make the links both within and across modules we are asking them to think reflectively.

4. Normative Judgements

We need to be clear that normative judgements are not the same as ethical judgements. I see the latter as delivering a verdict on the worth or rightness of a person or action. As such, ethical judgements play (or should play) a minor role in academic research and production. It is irritating if a historian gives us ethical verdicts on her subject matter (Henry was a good king, but King John was a bad one) – ethical judgements are almost always unnecessary.

But normative judgements are something else. They involve according due sensitivity to the values and norms associated with the subject matter under consideration. By their very nature, normative judgements are contextual. For example, if a student fails to appreciate the nature of religiosity in fifteenth century England (for example, by seeing it in terms of ignorance and superstition because of a modernist, secular approach which the student brings to bear) then it is the normative judgement that has gone awry. Or, if a research methodology fails to take due account of the needs of confidentiality, again, it is a normative judgement that is deficient. Normative judgements often operate in combination with other judgements (especially with process and reflective judgements) and this is one of the reasons why academic judgement can be complex.

Conclusion

If I am right then there are four basic elements to an academic judgement. Typically in any assessment all four elements are operating together – process, epistemic, reflective and normative. The judgements we use are precisely those that we want our students to develop. We can see straight away that attempts to categorise and tabulate all of these elements may be helpful but are unlikely to be comprehensive. The precise nature of the judgement will vary according to subject matter and no set of assessment criteria that I have seen comes anywhere near to giving full justice to the complexity of the judgements involved. Moreover, most of those outside academia (government ministers, MPs, media people and the like) are just clueless regarding how much they know of this complexity.

Complex, yes: but not so complex that we can’t attempt to say what is involved in giving an academic judgement. But the above sketch cannot be the last word – if I have succeeded in suggesting some initial ‘guidelines’ then that is a start.

SRHE member Geoff Hinchliffe teaches undergraduates in the School of Education at the University of East Anglia. This blog is partly based on a paper he gave at the 2018 SRHE Research Conference.

Bibliography

Cambridge University (2018) Student Complaints https://www.studentcomplaints.admin.cam.ac.uk/general-points-about-procedures/academic-judgment

Higher Education Act (2004) http://www.legislation.gov.uk/ukpga/2004/8/pdfs/ukpga_20040008_en.pdf

Kant, I (1933) Critique of Pure Reason, trans. Kemp-Smith, N, London: Macmillan

Kant, I (2000) Critique of the Power of Judgement, trans. Guyer, P and Matthews, E, Cambridge: Cambridge University Press

OIA (2018) Guidance Note on the OIA Rules http://www.oiahe.org.uk/rules-and-the-complaints-process/guidance-note-on-the-oias-rules.aspx#para30

Steinberger, P.J (2018), Political Judgement, Cambridge: Polity Press

Wittgenstein, L (1958) Philosophical Investigations, Oxford: Blackwell