IRRA (Institutional Repositories and Research Assessment)
RAE Software for Institutional Repositories

Leslie Carr & John MacColl, IRRA Project Managers

RAE 2008 is the latest UK research assessment exercise designed to inform the selective distribution of public funds for research by the UK's higher education funding bodies. Like previous exercises, RAE 2008 will be based upon expert review by discipline-based panels considering written submission from participating institutions. It is managed by the Higher Education Funding Council for England (HEFCE) on behalf of all UK HE Funding Councils.

IRRA for Data Collection

Every researcher from each discipline must provide a shortlist of their best publications, together with information about their prestigious research activities (journal editorships, invited keynotes, committee memberships, etc). These are amalgamated within an institution into a single submission for each discipline (or "Unit of Assessment") and transmitted through a set of web services to the RAE team within HEFCE, who collate them and pass them on to 25 discipline-specific panels. Each panel will then need to refer back to the papers originally submitted by the individual researchers to be able to form (and validate) their judgments. The process by which six years of research output for each institution are selected to field the strongest possible set of submissions across a whole institution will involve a complex set of decisions taken by academic managers within institutions. The most accurate and up-to-date information from each researcher is critically important. The previous RAE process required each researcher to provide printed postprints as evidence of their research quality. This time, the IRRA project is facilitating the use of institutional repositories as a potential mechanism for collecting research quality evidence from each researcher in an institution as well as for providing access to that work for the panels.

In each institution, the researchers simply select their best articles from the Institutional Repository. These are routed through to the institution's Research Management System which prepares the final submission for RAE 2008. The RAE panels may access pre- or postprints from the RAE Repository.

IRRA Workflow Diagram

The aim of the IRRA project is to ease the data collection task by embedding it into the processes of the institutional repository. Most UK universities engaged in significant amounts of research now either have an institutional repository, or are actively planning to introduce one. Our Project has identified two types of institutional repository which are being created in response to institutional policies on research publications:

a) Repositories which act as comprehensive research publication repositories (therefore accommodating a mix of metadata-only and metadata+full-text items)

b) Repositories which act as open access research publication repositories, holding metadata+full-text items only.

IRRA will provide solutions for both types. In the case of type (a), the repository should hold all of the researchers' publications, and should be updated as a matter of course. In such an environment, the researcher need only make a simple form-based selection to have their submission passed on to their academic administrator. In trials at the University of Southampton, where the use of a repository is accepted, this has radically reduced the effort required by (and complaints from) the researchers and lecturers. Project IRRA will build on this trial work, which resulted in Southampton's repository being extended with functionality that permits its use for research quality assessment:

This screen shot shows the use of an extended EPrints institutional repository to support RAE processes at Southampton University. IRRA will provide this functionality for all sites which use Eprints or DSpace

In the case of type (b), IRRA will provide a web application which will interface with the open access institutional repository, but will permit the loading of records both directly and by import from other systems (such as a separate publications repository). This solution will also be the sole solution for DSpace sites, since the DSpace software makes type (a) repositories more difficult to achieve (the software assumes metadata+full-text deposits).

The project therefore aims to produce a plug-in for the GNU EPrints platform and a separate web application for DSpace designed as a tool to create an institutional RAE publications repository. Both of these software products will interoperate with the many Research Management systems which are being developed by UK universities to cater, amongst other things, for the non-research outputs data required by RAE2008.

IRRA for RAE Panels

The use of Institutional Repositories for providing RAE panels with access to submitted publications is something that is intuitively obvious. An Open Access repository provides access to the research output for all comers and all purposes. Members of RAE Assessment panels are no longer provided with official printed versions of every paper and they may not have subscriptions to the journals in question, consequently an IR is the natural place to turn to.

For all that theoretical simplicity, the solution is resisted. The major issue that causes disquiet is trust, that is, for the purposes of assessment, can a panel trust that an authors' postprint (final, post-refereeing version) is really equivalent to the publisher's PDF version. The bibliographic facts of publication are certified by the publishers' online table of contents, it is only the papers' contents that are open to question.

Some are taking the position that RAE panel members need (for all papers) the official publisher's version, with the official publisher's imprint as a guarantee of veracity. In response to this, some repository managers are seeking permission from each publisher to create a private set of publishers' PDFs within the repository, accessible only to RAE administrators and panel members.

The IRRA project will continue to promote the use of repositories for RAE, and seek to influence the discussions, disseminating best practice for repository managers.

Technical Overview

The project will provide the tools to permit sites to create an RAE portal for RA2 research output and RA5a research environment and esteem, RA5b individual staff circumstances and RA5c category C staff circumstances. Sites which take IRRA software will be able to use their institutional repository either as a repository for all full-text publications that will be electronically submitted into the 2008 RAE by their university, or as a data source for such a repository. The application will consist of the following components:

Deliverables

The final versions of IRRA RAE software products for DSpace and GNU Eprints will be available in summer 2006 (earlier development releases will be available from January 2006). Workshops will also be run across the UK to train institutions in use of IRRA software in the summer 2006.


IRRA was commissioned by JISC from the Universities of Southampton and Edinburgh, and the project is located within JISC's Digital Repositories programme. Southampton is the home of the GNU Eprints software, and Edinburgh has done considerable development work with DSpace, including the creation of a plug-in for managing E-Theses and Dissertations. It also runs a DSpace-based institutional repository. The contact at Southampton is Les Carr (lac@ecs.soton.ac.uk), Project Manager, and the Edinburgh contact is John MacColl (john.maccoll@ed.ac.uk).

The IRRA project page is irra.eprints.org. If you would you like to be kept informed of the workshops and the availability of the software, please register your interest on the IRRA wiki at http://wiki.eprints.org/w/IRRA/TheRegister.