5. Policy and procedure

Is recognition and reward for open research adopted as institutional policy, and included in all relevant policies and procedures, e.g. those related to recruitment, probation, promotion, performance and development review, and other activities involving the appraisal of researchers?

Why is this important?

• Recognition and reward for open research must be incorporated into policy, with defined expectations and responsibilities, in order to be effectively implemented and to be able to support long-term cultural change. A number of policies and procedures will be affected – those related to recruitment, probation, promotion, performance and development review, and possibly others. If policies are not aligned and co-ordinated, this may result in inconsistent practices and mixed messages, which will undermine the policy objectives.

• It will be essential to secure the buy-in of related policy owners and the support of those responsible for their implementation. Changes to systems and processes and responsibilities of support staff may be required and would need to be discussed and agreed.

Maturity scale

No Action	Emerging	Evolving	Sustained
Recognition and reward for open research is not referenced in policies or procedures related to or involving the use of research assessment.	Recognition and reward for open research is mentioned in some relevant policies and procedures, but on a limited basis and with little evidence of integration or effective use.	Recognition and reward for open research is included in most relevant policies, with evidence of effective integration into practice.	Recognition and reward for open research is included in all relevant policies and procedures. There is evidence of increasing effective use of open research criteria by candidates and assessors in research assessment activities. Policies are reviewed regularly.

Progress actions

Here are suggestions for key actions that can be taken to progress from one level of the maturity framework to the next. These can be considered when you develop an institutional action plan.

No Action to Emerging

• Include some mention of open research and open research criteria in some key policies, e.g. for promotion, although the broader policy framework and supporting procedures, training, etc. may not yet be developed.

Emerging to Evolving

• Develop and publish a research assessment policy or statement aligned to the Principles of the Agreement on Reforming Research Assessment, which includes a commitment to recognise and reward open research, with definition of open research and reference to an open research statement or policy.

• Identify relevant policies and procedures involving the assessment of researchers e.g., those concerning recruitment, probation, promotion, performance and development review, and engage policy owners to discuss and agree required policy, including support and resource requirements.

Evolving to Sustained

• Ensure that all relevant policies and procedures have been updated to integrate recognition and reward for open research consistent with institutional policy and open research policies.

• Ensure that policies and procedures are operating effectively and have been refined as necessary in response to feedback. Various routes should be available to provide feedback for substantive and iterative development of policy and procedure.

• Review policies and procedures on a regular basis to align with developments in open research practice.

Main areas of activity

Research assessment policy implementation

Recognition and reward for open research will be relevant to and require incorporation in a range of institutional policies and procedures, including those concerning recruitment, academic probation, promotion and professorial review, performance and development review, and any other processes involving research appraisal and reward allocation. It will be important to ensure that institutional systems of reward and recognition are aligned and consistent between themselves.

We believe the most effective way to achieve this alignment and ensure consistency across relevant policies is to adopt an institutional research assessment policy to which other policies can be linked. Such a policy would set out the general principles of research assessment, which would include expectations and responsibilities related to open research. Relevant provisions could then be adopted into policies and procedures that involve the assessment of research or researchers.

This is not necessarily the only option for implementing relevant policy provisions. They might also be progressively integrated into existing policies, for example, beginning with the academic promotion policy, then moving on to recruitment and probation, performance and development review, etc. This may be a more practicable option in some institutions.

Formulating expectations related to open research

The emphasis on open research as a dimension of research that should be considered in research assessment is relatively recent. (This is discussed in the Introduction.) Where institutions have adopted policies relating to research assessment, so far these have mostly been focused on publications and the responsible use of publication metrics.¹ There is consequently both a need to update existing research assessment policies and a lack of established models for policies that reflect a broader concept of responsible research assessment, especially any that include explicit recognition of open research.

There is also the challenge of articulating open research expectations and requirements fairly, and in a meaningful and realistic way, within any research assessment policy, given that current awareness and practice on the part of researchers are at a relatively low level. Any policy must take account of various factors, in particular:

• researchers will come from different institutional and cultural backgrounds, which will have influenced the degree to which they are aware of and have had opportunity to use open research practices;

• the relevance of open research practices and benchmarks will vary by discipline and type of research. Some disciplines may have more advanced cultures of data sharing or using pre-registration than others; relevant of open research practices and policy expectations will depend on the type of research: data sharing expectations cannot apply where research has not involved collecting data. Policy expectations must be formulated so that they can be applied in ways that are meaningful for specific disciplines and, within disciplines, for specific types of research;

• the existence and scale of any track record in open research will depend on the career stage of the applicant and their employment history, which may include career breaks, or employment in industry or other areas that have not provided opportunities to use open research practices.

Institutions will approach the development of policy, and the inclusion of open research within it, in different ways, but it will be important to ensure the process of policy development is inclusive and the end result embodies the values of the institution. The SCOPE Framework for Research Evaluation developed by the Institutional Network of Research Management Societies (INORMS) is an excellent practical model that can be used to support the an inclusive process of policy co-development and implementation. It includes a number of case studies that demonstrate how different institutions have used the Framework to develop research assessment policies.

‘There was a need to make allowances for adjustments in Schools to support individual and discipline-specific requirements and approaches. This was particularly true for open research practices, which are currently adopted to different degrees across disciplines. Similarly, career stage was a consideration, as earlier-career researchers tend to be more familiar with open research practices than senior researchers.’ University of Bristol

Integrating open research criteria in relevant policies and procedures

Policies and procedures that would need to take account of open research criteria include those relating to recruitment, probation, promotion, and professorial review, performance and development review, and other forms of reward and recognition, e.g. awards of institutional funding and other prizes.

The process of updating policies and procedures is likely to require a substantive collaborative development and consultation phase and could be driven by a research assessment reform group, an existing group with authority in these areas, or an empowered institutional lead. There may be discussion with policy owners over the precise nature of the changes, how and by whom any support requirements will be met, and the provision of additional resources to support implementation. It will be important to reach agreement on these matters if the policy is to be implemented effectively. It will also be necessary to ensure there is appropriate reference to open research expectations/requirements in any update of related policy and procedure documents, and guidance and support are signposted. For example, an academic promotion framework may need to update its criteria to include specification of open research, with information for both promotion panels and candidates providing links to any relevant policy information, guidance and training.

Academic assessment frameworks and open science approaches

Academic promotion and professorial review policies customarily define assessment frameworks in which a number of assessment criteria are identified, usually under broad categories such as Academic citizenship and leadership, Research, and Teaching and learning.

There have been some efforts to create models of academic assessment frameworks in which open science is a defining dimension of academic activity and assessment. These have been European initiatives, where ‘open science’ may encompass not just open research activities and outputs, but other knowledge-related activities such as innovation, the creation of impact, public engagement, and teaching and supervision. These models of academic assessment frameworks may be of use to institutions that are planning or undertaking a review of academic development and assessment pathways and frameworks.

In such frameworks, recognition can be given not just for practising open research directly, but for contributing to a culture in which open research is enabled and practised, for example by delivering training in open research practices,² by using open research products in teaching, or by developing or curating infrastructure that supports open research, e.g. community data standards. These frameworks are consistent with the ambition of the Coalition for Advancing Research Assessment to expand the range of activities and outputs recognised in research assessment.

In 2017 the EU report ‘Evaluation of research careers fully acknowledging Open Science practices’³ proposed an Open Science Career Assessment Matrix (OS-CAM) in which open science is a guiding principle, and the full spectrum of open science practices, including open access to publications, open data, open peer review, research integrity, citizen science and stakeholder engagement, is taken into account. In the proposed framework all aspects of a researcher’s career, output and activities are included in the assessment, and all outputs and activities are assessed on the basis of their degree of openness.

The OPUS project, which began in 2022, has built on the foundation laid by OS-CAM to produce a comprehensive researcher assessment framework which includes an open science dimension designed to support recognition and reward for open science practices. The framework structures indicators in four categories of activities, for Research, Education, Leadership and Valorisation, thus providing an ‘open’ lens with which to view the full range of an academic’s activities. The framework is being tested and refined in collaboration with pilot institutions. It uses a modular design so that the framework can be customised to meet the requirements of the individual institution.

Case study

Including open research in the University of Bristol’s Academic Promotions Framework

Footnotes

1. In the survey of UK institutional policies and practices undertaken by the OR4 project in 2023, 44 or 73% of 60 respondents stated that their institutions had a statement or policy on responsible research assessment or the responsible use of metrics. The majority of these were focused on the use of publication metrics. In scope and terminology many of these statements follow and reference DORA and the Leiden Manifesto.

2. The Open Research Programme is supporting partners to develop their capacity to deliver open research training: https://www.ukrn.org/ws1-training/.

3. Working Group on Rewards under Open Science (2017), ‘Evaluation of research careers fully acknowledging Open Science practices’. https://data.europa.eu/doi/10.2777/75255.