Review usability - AHRC

As part of our ongoing transition to the UK Research and Innovation (UKRI) Funding Service, we have updated our approach to review usability. We always take great care to ensure that reviews are appropriate and of value to both applicants and our moderating panels.

In line with UKRI’s guidance on writing good reviews, reviews must be based on evidence, and this should be clearly shown within the comments. Reviewers have sight of and utilise the same published funding opportunity-specific assessment criteria as applicants, which can be found on the funding opportunity page.

Reviewers should:

  • familiarise themselves with any specified assessment factors or scoring matrix
  • provide comments and recommendations that are consistent with, justify, and explain their scores
  • ensure their comments are comprehensive, but concise
  • clearly identify strengths and weaknesses of the application in a constructive manner
  • raise any concerns in the form of questions for the applicant
  • avoid jargon, panellists may not be specialists in their field
  • consider other aspects of the proposed research or innovation, like its methodology, experimental design and data management plan, as well as its potential impact (and the pathways to achieving it) including any ethical and social issues

Usable reviews

We are only able to use a review during the assessment process if it meets the following criteria:

  • it includes enough information to help panellists and UKRI staff make an informed judgement on the application
  • the comments are only based on information that’s included in the application
  • the application has not been reviewed negatively because of any equality, diversity and inclusion requirements (for example, decisions to work part-time or past absences for health reasons)
  • the comments are not speculative, inflammatory or damaging to applicants
  • journal metrics, conference rankings or personal metrics have not been used as a substitute measure for assessing the applicant’s contributions
  • the reviewer does not have a conflict of interest with the application and has not revealed their identity. If reviewers are in any doubt as to whether they have a conflict of interest, they are encouraged to contact us via operations@ahrc.ukri.org before accepting and completing a review

If we feel that there are elements of a review that do not meet our criteria, the review will either be marked as ‘unusable’, or there may be an opportunity for the reviewer to amend their comments.

Unusable reviews

Reviews may be deemed ‘unusable’ if:

The review lacks sufficient detail

Our decision-making processes rely on expert review of the applications submitted.

To moderate an application effectively, our panels need to have a clear idea of the strengths and weaknesses of an application as identified by reviewers.

Comments should be evidence-based, comprehensive and justified overall, with critical analysis of the application, so that the moderating panel can make an informed decision based solely on the expert reviews and applicant response received.

The reviewer has not addressed the areas expected for effective expert review, and there is little or no critical examination of the application

When writing expert reviews for AHRC, reviewers must utilise the core assessment review criteria which are available within the Funding Service, considering:

  • the vision
  • the approach
  • the applicant and team’s capability to deliver the project
  • ethics and responsible research and innovation
  • the resources and cost justification

Reviewers should engage with the application in front of them, and not as it could be if changes were to be made.

Reviewers should identify the strengths and weaknesses of an application for the applicant to respond, and for our panels to moderate effectively.

The reviewer’s comments are framed in an offensive manner, or the tone and language used is confrontational or emotive

Reviewers must adhere to the principles and core values within UKRI’s principles of assessment and decision making.

Decisions must be made impartially, fairly and based on merit, using the best evidence and without discrimination.

We want to foster a positive research culture for everyone, by everyone, where everybody is appreciated, respected and valued.

Within this, we require reviews to be balanced and objective.

Therefore, it is important for reviewers to remain constructive, factual and non-confrontational, to provide the applicant with comments that they can respond to effectively.

For most funding opportunities, the project lead will have sight of the expert reviews received as part of the right to reply process.

Amendments to make reviews usable

Where minor issues occur we will continue to work with reviewers to resolve these, if we feel that small amendments could make the review ‘usable’.

We may ask for amendments if:

The comments and grades throughout a review contradict the overall grade given by the reviewer

In these instances, we will liaise with the reviewer if we question whether an incorrect score has been entered, and amendment of this will be a chance to change the score given, as opposed to a reviewer revising their comments.

Revisiting the grading descriptors when awarding a grade can help reviewers to ensure that their comments are consistent with the overall grade given.

Reviewers should focus on the application and its content as it stands, and not as it might be if the opportunity to resubmit were available.

The commentary on how a project has been conceptualised refers to the protected characteristics of an individual involved in the project

We welcome constructive comments that directly relate to a project, the way in which it has been conceptualised and would be managed.

Comments should focus on whether applicants have appropriate skills, expertise and experience, and whether the project has been designed in an appropriate way to achieve its goals.

Any concerns should be articulated in an objective and evidence-based way.

Our aim is to ensure that the substance of valid academic criticisms remain, while ensuring that applicants are not subject to comment regarding their protected characteristics.

The reviewer explicitly or implicitly identifies themselves within their comments

The applicant and panel are not made aware of the identity of expert reviewers.

Reviewers must not mention their name or any personal information that could identify them within their comments.

Anonymity of expert review is important to ensure that reviewers can express their views freely and openly.

Applicant response to reviews

Applicants will have the opportunity to respond to their reviews at the applicant response stage (if applicable to the funding opportunity), and panellists will read all usable reviews at the moderating panel.

We value academic freedom and want to facilitate productive and constructive assessment.

Therefore, project leads should feel empowered to clarify, rebut and resolve any issues within their applicant response, and we encourage panellists to consider any issues within reviews where necessary at the panel meeting.

If applicants are concerned about reviews that they have received, we encourage them to get in touch with us before they complete their applicant response, so we can investigate.

 

Please note, we reserve the ability to mark a review as usable or unusable at any time.

Read more about our peer review processes and guidance on writing a good review and scoring.

Last updated: 16 February 2024

This is the website for UKRI: our seven research councils, Research England and Innovate UK. Let us know if you have some quick feedback or help us improve your experience by taking three minutes to tell us what you think of the UKRI website.