E LECTRONIC LIBRARY PERFORMANCE INDICATORS : THE EQUINOX PROJECT

In developing a set of performance indicators for the electronic library, the EQUINOX project has also developed a decision support software product, a quality management framework, and an XML Data Type Definition (DTD) to assist interoperability of management data. Work continues on developing the model that underlies the performance indicators.


Introduction
There is considerable interest across all library and information service (LIS) sectors in methods of measuring the performance of digital, networked services.Since for most libraries and information-based organisations such services are becoming an ever-greater proportion of the whole, and with the development of `hybrid library' concepts which demand close integration of all services regardless of medium and delivery mechanism, this interest is hardly surprising.However, to date most work in this area is still experimental and there are no agreed approaches.The use of web server statistics, although common, is acknowledged to be flawed and provides little basis (at least at present) for developing robust and comparative indicators.The situation is compounded by the fact that electronic services are more than just add-ons, but are changing the very nature of the service itself.Furthermore, across many sectors there is evidence of convergence between libraries, computing services and others, making boundary definitions difficult.
Thus, while a considerable body of work exists on performance indicators for traditional library services 1 , this is not as yet paralleled by work for the electronic library.Studies in the USA have been led by McClure, Bertot and others 2 , and more recently by the Association of Research Libraries 3 , while in Europe the major activity has been undertaken within the EC-funded EQUINOX project, the subject of this paper.

Peter Brophy is Director of the Centre for Research in Library & Information Management at
The Manchester Metropolitan University e-mail: p.brophy@mmu.ac.ukInformation Management (CERLIM) at the Manchester Metropolitan University and involved technical partners from the UK and Ireland, with libraries in Ireland, Germany, Sweden and Spain.Towards the end of the project, which formally finished at the end of November 2000, a further 40+ libraries were brought into testing of EQUINOX products.Its primary foci have been on achieving consensus on a core set of performance indicators (PIs) for electronic library services and on developing a software-based decision support tool for library managers.
The project delivered four major outputs.These were: • The set of PIs itself, including prescriptive instructions on the datasets and methodologies needed to calculate them • The EQUINOX software product • The quality management framework, which underpinned the work • An XML Data Type Definition (DTD) developed to assist interoperability of management data.
These four outputs are described briefly below, and are followed by some suggestions as to the directions in which future work may progress.Documentation from the project is available on the project web site at http://equinox.dcu.ie

Performance Indicators for the Electronic Library
There was a variety of starting points for work on the development of suitable performance indicators, but key work included: • the prior EQLIPSE project, undertaken by a consortium, which included many of the same partners, and which had investigated traditional library approaches, as well as testing a quality management (QM) approach integrated into ISO9000: in the end it was decided that this was too limiting to be adopted by the majority of LIS and that a broader basis was needed; • CERLIM's Management Information for the Electronic Library Programme, and in particular an eLib Supporting Study undertaken as part of that programme 4 ; • Studies undertaken in the United States, referred to above, and in particular the work of Charles McClure and John Bertot and their colleagues 5,6 ; • The work of SCONUL's Advisory Committee on Performance Indicators, and especially a publication produced by a working group set up by the four UK higher education funding councils, The Effective Academic Library 7 ; • International work by bodies including IFLA and ISO's TC46 SC8 sub-committee, especially with the publication of Information and documentation ± library performance indicators (ISO 11620) 8 in 1998.
EQUINOX team members were active in organising consultative workshops at venues across Europe, so as to gather as much feedback as possible on the views of library managers from many different national, cultural and sectoral backgrounds.Some of this activity was co-ordinated with the EC-funded Concerted Action CAMILE, which was live during the first part of EQUINOX and in which CERLIM played an active role.A wide variety of views emerged from these debates, but the overwhelming need was found to be for a set of electronic library indicators that could complement the more traditional ones, such as book issues per user or library space per user.Many libraries in Europe ± and especially those in less-developed regions ± are only just starting to implement electronic services, and radical approaches would have been counterproductive.This is a point considered further in section 7 below.
EQUINOX developed a trial set of performance indicators, complete with robust definitions and guidance on methodologies for collecting and analysing data and, after a number of drafts had been considered, it was agreed to recommend a set of 14 performance indicators, designed to complement ISO11620.These are shown in Figure 1.The full definitions of the indicators and of the datasets needed to calculate them are available on the EQUINOX web site.

The EQUINOX software product
One of the major aims of EQUINOX was to build a decision support tool for library managers, based on the quality management framework developed from that used in the earlier EQLIPSE project (see section 5 below) but without its requirement for ISO9000 compliance.The product incorporates the following features: • The manager can specify a hierarchy of mission, aims and objectives, and can link performance indicators to each.This encourages an approach whereby the purpose of the library and each of its services is first specified and indicators are then linked to these purposes.For example, if the library has an aim to ensure that all users have adequate information skills, and runs training sessions to help achieve this, one of the EQUINOXdefined indicators (Number of attendances at formal electronic library service training lessons per member of the population to be served) might be linked in to the specific aim.So too might library-defined indicators, since there is no requirement to limit usage to the recommended set.
• Thus the indicator set is totally flexible.
Although the software was delivered to demonstrator library sites with the EQUINOX PI set pre-installed, these can be supplemented, amended or discarded at the user 's choice.
• Data can either be input manually or be downloaded from other systems.The former recognised that it is not feasible to capture automatically some of the data needed (examples would be number of workstations or number of library staff).The XML DTD definition gives the basic mechanism for the latter (see section 6 below).
• Links can be built between any statement (e.g. an aim) or indicator and any document within or outside the system.So, for example, a text document detailing user comments can be directly linked to the user satisfaction indicator.Or indicators could be linked to statements within the library's current planning documents.• Indicators can be defined at any number of levels, so that branch libraries can be assessed but aggregate statistics used where appropriate.
• The time series for any indicator can be depicted graphically, together with predefined maxima, minima and threshold values.Thresholds can be used to alert library managers to exception conditions.It would also be possible to use them to monitor performance against competitors' or colleagues' data.
Figure 2 depicts the EQUINOX decision support system, illustrating part of the hierarchy of mission, aims, objectives, performance indicators and datasets.
The EQUINOX software is likely to be developed as a commercial product, with specific versions targeted at different sectors ± so, for example, the public library version would

The quality management framework
Underlying the EQUINOX work there is a firm commitment to a quality management (QM) framework, aligned with work on Total Quality Management and models such as that developed by the European Foundation for Quality Management 9 .The background to this work is documented elsewhere 10 , although it is worth emphasising QM's commitment both to user perspectives on performance assessment and to continuous improvement.
In essence the QM approach adopted in EQUINOX owes much to work on systems models of the library, which stress the interconnectedness of inputs, processes, outputs and outcomes, while placing emphasis on the importance of customer (or `user ' if you prefer) satisfaction.This approach suggests that if user needs and preferences are monitored and reviewed regularly, and used to inform decisions on the service mix to be offered and the way each individual service is offered, the overall quality of service will be enhanced.In higher education, for example, where all aspects of quality are under the microscope, through bodies like the Quality Assurance Agency, taking an explicit QM approach should give an important advantage to library managers.

The XML DTD
During the EQUINOX project it became apparent that a method was needed for downloading data into the decision support system from the many systems that libraries now use, but avoiding the need to write a separate script for every different system.Bearing in mind that each library may give access to hundreds if not thousands of external services, it is clearly highly inefficient if someone has to sit down and write a script for each one.Indeed, that barrier alone would be enough to ensure that management data was never extracted.
After discussion in the team, it was agreed that the way forward was to use an intermediate step whereby data would be output from any contributing system as an XML (eXtensible Markup Language) document using an EQUINOX XML Data Type Definition (DTD).Once this DTD is in place, the formatting of output from a contributing system becomes relatively trivial, since the DTD defines precisely the field tags and formats (the document itself being in simple ASCII text).Equally the EQUINOX system (or any other decision support system for that matter) can input the data without having to identify the system from which it was derived and then find a special script to use with it.
During the project we tested the data input capabilities of the EQUINOX software and satisfied ourselves that the XML DTD functioned correctly.We were also able to demonstrate input from web server logs and spreadsheets using this functionality.
We believe that this approach has considerable significance beyond libraries, since virtually all sectors will have an increasing need to download management data from heterogeneous systems in the future.We are, therefore, seeking ways to The EQUINOX project Peter Brophy Fig. 2: EQUINOX Decision Support System ± overview disseminate the approach widely.Within the EQUINOX team, the XML DTD was developed by Fretwell-Downing Informatics, who have agreed to place it in the public domain.It is available at the EQUINOX web site.

Into the future
EQUINOX has been a highly successful project and received excellent reports from the EC's peer reviewers at its formal Review in Luxembourg in September 2000.However, as with any project, it is clear that there is a lot more work to be done.While exploitation effort is currently being concentrated on the software product, CERLIM is also undertaking work on the underlying model behind the performance indicators needed for libraries in the future.Our view is that, while ISO11620 and the EQUINOX set are helpful for libraries in their current state of development, a rather different approach may well be needed in the future.In particular, we need to move performance measurement away from models of the library which focus on the collection and on access to it, towards the idea of the library as the `expert intermediary' (even if operating in the background).In this model the library maintains detailed intelligence about its users and about the myriad information sources that they might find useful, and the core of its activities lies in dynamically building links between the two.This idea has been explored at length in two recent publications 11,12 .Measuring the performance of such systems is, however, a non-trivial problem to say the least!We are always interested in receiving feedback on the EQUINOX outputs or any other aspect of library performance measurement, and would welcome comments from anyone interested in this area.

7 Serials 1 . Percentage of the population reached by electronic library services 2 . Number of sessions on each electronic library service per member of the target population 3 . Number of remote sessions on electronic library services per member of the population to be served 4 .each electronic library service 5 . Cost per session for each electronic library service 6 . 7 . Percentage of information requests submitted electronically 8 . Library computer workstation use rate 9 .of electronic library services 12 .Fig. 1 :
Fig. 1: EQUINOX recommended electronic library performance indicators EQUINOX was a two-year research and development project funded under the European Commission's 4th Framework Programme.It was led by the Centre for Research in Library &