COMPETENCY N

“Evaluate programs and services using measurable criteria.”

Statement of Competency N

Libraries and other types of information centers provide a foundation for programs and services related to education, recreation, information, research, and academic study. Such organizations help to create a conducive learning environment by providing services such as quiet study areas, internet access, and access to information sources, among others. Programs offered may include library tours (both physical and virtual), information literacy training, research training classes, personal library instruction, and guides and tools for research. In addition to the aforementioned programs and services, one of the most important objectives of a library is to provide beneficial and enriching programs that meet the needs of users. In order to create services and programs that are beneficial to users, library and information professionals must evaluate the programs and services using appropriate criteria; this ensures that the library’s programs and services meet or satisfy the needs of users. Possessing the ability to evaluate programs and services allows information professionals to create programs and services that are high quality and effective, and which adhere to the values of the profession and the information organization. Evaluation is essential for libraries and other information organizations because it allows those involved to reflect on the effectiveness of programs and services, in addition to providing an opportunity for modifications and improvements to them.

Libraries and information organizations have to continually employ evaluation and assessment activities in their strategic planning, as well as in their day to day management (McClure, 2008). Some of the factors or issues that a library must address in its evaluation efforts include: questioning why services and programs are being provided the way they are provided, discussing the quality and effect of current library services and programs, and figuring out how best to serve different groups of users both presently and in the future (McClure, 2008).

Library Evaluation

Library evaluation is the process of determining the impact, outcome, success, cost, or other factors associated with library activities, services, or programs (McClure, 2008). Library evaluation is normally carried out at the end of a program or service; through it can also be conducted continuously to monitor programs or services. Evaluation is particularly important in decision making and planning, as it is a tool that provides data (both quantitative and qualitative) that describes the library’s operations; such data is essential in the assessment and enhancement of library services. Charles M. McClure (2008) notes that evaluation is important to libraries and their stakeholders because it (p. 181):

Determines the degree to which service objectives are accomplished and the degree to which these objectives were appropriate for the library;

Assesses how well services support larger organizational goals, e.g., is the library assisting in the accomplishment of university (or local government) mission and goals?;

Monitors the status of services and operations to produce trend data that assesses the service over time;

Obtains community input for planning and development of current and future services;

Justifies and shows accountability of services to governing boards, government officials, local community members, and others;

Compares the relative costs and benefits of one service versus other or similar services for specific target audience groups;

Provides input for planning and the selection/development of future services for specific target audience groups;

Assists the organization to determine the types of staff training that might be most needed and appropriate;

Informs governing boards as to the nature, needs, and success of library services;

Identifies those aspects of library services that are successful and those that need to be refined or newly created;

Educates the staff as to costs, benefits, impacts, and problems related to the operation of the library;

Provides information and data that can be used by local advocacy groups to promote the importance and impact of the library; and

Forces the organization to think in terms of outcomes…what are the outcomes of these services and are they what you want?

Over the years, many tools and methods have been developed for evaluating the services and programs offered by libraries; these include both qualitative and quantitative techniques. Regardless of their nature, all of these have been applied in libraries to observe value and to measure outcomes (Moran, Stueart, & Morner, 2013). Outcomes and value refer to how users utilize and value information acquired or accessed through the library’s systems and services. In the past, libraries evaluated their services and programs through measures of circulation, quantity of reference questions, and gate count; all of these basic quantitative techniques are no longer sufficient in the evaluation process (Moran, Stueart, & Morner, 2013). In evaluating programs and services, the focus for libraries is now on users and coming up with successful outcomes.

Measures

There are a number of measures that can be employed in an evaluation effort; they include performance measures, output measures, and outcome measures (McClure, 2008). Performance measures are used to determine how well a program or service has performed. For example, the number of repeat visitors to the library website (this indicates the number of times a user has utilized the website) can be said to be a performance measure. Output measures, on the other hand, indicate the impacts the come about due to a program or service (Moran, Stueart, & Morner, 2013). There are different outputs from a program or service which are measurable by accomplishments; for example, the output for a website would be number of page views.

Outcome measurements (also known as outcome assessment, results-based evaluation, or outcome based evaluation) refers to the benefits that indicate the effectiveness of a program or service (Rubin, 2006). Many of these benefits include a change in knowledge, behavior, attitude, condition, or skills. It is important to note that these are achievements that a user may or may not have realized without the services or programs of a library; however, the library can be an important contributor to these benefits (Rubin, 2006). Outcomes can also be said to be the main results sought by library users. In general, outcome measurements help to answer one key question: What impact or effect does a library program or service have on a user? The outcome or consequence of a program or service can be seen in its benefits to users (McClure, 2008). Outcome measurement is a user focused approach to the planning and evaluation of services or programs which are offered to address a specific user need and tailored to bring about change for a user (Rubin, 2006). Although outcome measurement may be a relatively new term, outcomes are a concept that many people are familiar with. Outcomes, in the evaluation of a library programs or services, are the intersection of service goals, community needs, and library service responses (Rubin, 2006).

By monitoring the effect or consequence of their programs and services, libraries are able to assess their effectiveness. Consequently, library and information professionals are able to see the quantity and quality of impact of the library’s programs and services. Unlike other measures, outcome measurement focuses on the results instead of the process. Measures that focus on inputs, outputs, and activities all center on staff as they generally describe what the staff has done. These measures are also an important part of the process of creating and developing programs and services (Rubin, 2006). In the outcome measurement approach, outcomes, interim outcomes, and goals, are all user-centered in nature and are used to report on the impact of programs and services.

Changes to library services or programs are normally made by trustees, managers, and directors; these individuals could decrease undervalued or underused offerings, increase popular or successful programs and services, and try to maximize the library’s collection. Outcome-based evaluation provides a better option in the making of decisions related to programs and services (Dresang, Gross, & Holt, 2006). This is an evaluation process that engages the community by creating programs and services with regard to the goals and needs of the community members; it does not rely on the perceptions of the library staff, trustees, or board (Dresang, Gross, & Holt, 2006). In outcome-based evaluation, goals are created through the assessment of community needs. With this evaluation approach, goals are developed with a focus on what the community will receive, instead of what the information organization will do. For example, rather than developing a goal that aims at providing educational materials for school children, a goal could instead be crafted to provide children with the resources they need to succeed in school.

Evaluation and Planning

Evaluation and planning are closely related processes. Evaluating the performance of an organization, through identifying and creating output and outcome measures and subsequently evaluating their attainment, is an important part of the planning process (Moran, Stueart, & Morner, 2013). The concept of assessing user-focused information services and programs through users’ satisfaction has been integrated in numerous strategic planning processes. This has emerged due to the realization that accountability requires evaluation outcomes, and that both accountability and evaluation efforts are key to success. Unlike before, management strategies today are now stricter in addressing the needs and satisfaction of customers, rather than simply quantifying organizational operations as inputs and tasks (Moran, Stueart, & Morner, 2013). This shift from activities, inputs, and tasks requires library and information professionals to understand the needs and information-seeking patterns of users, and ways to satisfy them. One main cause of this shift is the increase in sophistication of the demands of users and other stakeholders; a situation that has placed more emphasis on flexibility and adaptability in providing services (Moran, Stueart, & Morner, 2013). This pressure on resources brings about a bigger challenge for quality control; however, it also provides an opportunity to carry out reality checks, which are an important aspect of accountability that helps to determine the efficiency and effectiveness of programs and services based on an information organization’s goals. A number of techniques have been developed and used by libraries and other information organizations to measure and determine how well these goals are achieved. These methods involve collecting data from users in order to understand their view, as well as their satisfaction (or dissatisfaction) with the library’s programs and services.

Methods of Evaluation

During the evaluation of library programs and services, data must be collected from users. Qualitative data is collected through methods such as interviewing, observation, focus groups, observation, and case studies (Cassell & Hiremath, 2013). Each of these methods are essential for establishing value, and understanding the behavior of users. The satisfaction and expectations of users are essential in the creation of measures. Measures are not only important in determining the satisfaction of users, but also determining the impact of library programs and services on different users. The findings from these methods can also help improve efforts to collect more raw data from transactional informational sources such as electronic sources, online systems, among others (Moran, Stueart, & Morner, 2013).

Survey is one of the most common methods of collecting qualitative data. This method is a set of questions presented to a defined community in order to acquire qualitative data on their activities, values, or perceptions (Cassell & Hiremath, 2013). Because surveys are aimed at understanding individual preferences, they are most appropriate when personal information is needed. Surveys can be carried out over the telephone, through mailed questionnaires, and through the Internet (on-line surveys). Interviews are utilized to supplement the evaluation efforts of surveys; they are employed to either help in the formulation of survey questions, or to add depth to the survey (Cassell & Hiremath, 2013). Interviews are particularly useful in the collection of data on personal experience and perception; and they can be carried out over the telephone, through the Internet, or in face-to-face interactions. Observation is also an important method in evaluation; it simply involves recording a live occurrence or event (Cassell & Hiremath, 2013). This method is most appropriate when actual behavior is the required area of evaluation. Other methods that have been employed in the evaluation of programs and services include case studies and focus groups.

Once the data has been collected, those involved in the evaluation process must check whether the information meets the established measures and quality standards. If measures and quality standards are not met, librarians and other library staff must take corrective actions in order to improve or modify programs and services.

Evaluation and the Improvement of Programs and Services

The evaluation of a library's programs and services helps librarians understand what users really need or what they don’t want; this is important in making appropriate changes, or creating programs and services that meet the needs of users (McClure, 2008). The evaluation process involves selecting interim and long-range outcomes, which are used to identify important goals; a library can use these goals to help keep a program or service on the right track. The failure to achieve a goal forces library staff to question service delivery, and take corrective actions to meet the goal. Evaluation also provides ideas on how library services are utilized, which enables information professionals to gain a new viewpoint (Rubin, 2006). Evaluation can help motivate library staff by providing evidence that their work is important, and that it makes a difference in the lives of other people. This focus on human impact gives library workers at all levels a common purpose, a factor that is key to the improvement of programs and services.

Evidence

Evidence 1: LIBR-232- El Cerrito Public Library: Services Assessment

This assignment was created for LIBR-232: Issues in Public Libraries. The purpose of this assignment was to create an in-depth report of a selected public library to assess its offered services, and to provide recommendations and suggestions to improve these services. For this course, I focused all of my analysis papers on the El Cerrito Public Library, a small public library branch part of the larger Contra Costa Contra Library system. I chose this library because I lived a few blocks away from it, and I had already visited this library a few other times for work from other classes. Because of this, I had developed rapport with several staff members, including the children’s librarian with whom I had the opportunity to interview for another course’s assignment. Lastly, because I am particularly interested in youth librarianship as a future career, I focused my assessment report on services provided to children and teens by the library.

This report is divided into several parts. The first is a description and overview of the city of El Cerrito. I also provide specific demographic data about the youth populations of this community. This demographic data includes information about the number of students enrolled in schools in the city, in addition to growth targets for Academic Performance Index scores (API), which are based on standardized testing. The second part if the report is a description of the youth services provided by the El Cerrito, which are divided into the broad areas of services to children and services to teens. I discuss the physical and virtual spaces offered to students, which help the library fulfil an important component of its public sphere responsibility. The library also fulfills its educational role in the community by providing literacy services, homework help, textbooks, reference materials, and subject guides to students.

In the next two sections I discuss the particular library service of homework help. The part discusses what homework help is, and how it is offered to students at the El Cerrito Library. For this portion of the paper, I visited the library during a weekday afternoon to observe how youth were served by the library. From my observations and investigation of the library website, I learned that there was no formal homework assistance offered to students using the library during after school hours. I discuss one of the informal avenues of homework help offered by the El Cerrito Library on its website—a Homework Help Subject Guide pathfinder. In the next section, I provide a literature review on the topic of homework help in libraries which provides examples of how public libraries and youth service librarians have created, marketed, expanded, and evaluated in-person and virtual homework help centers. Additionally, this literature review provides rationale about the importance of offering homework help service to youth populations in public libraries.

In the last section of the paper, I provide service recommendations for the El Cerrito Library based on my assessment. Specifically, I urge for the establishment of both web based homework help through the library website, an afterschool homework center staffed by volunteer tutors at the El Cerrito Library, two computer terminals for exclusive usage by children and teens, and more seating and study space for teens. In the Appendix of this paper I provide two charts: services currently offered, and services that should be offered. For each chart, I provide information about the following points: qualifications/eligibility, delivery of service, and method of evaluation. The services currently offered that I discuss in the chart are: physical spaces in the library, virtual spaces for children and teens, homework help, Link+/interlibrary loan, Discover & Go, and ready reference. For the method of evaluation of each of these offered services, I obtained information from library staff about how data is collected from library users. In the second chart, the services that I argue should be offered at the El Cerrito Library are: physical space for teens, physical homework help center, virtual homework help center, and computer terminals. The methods of evaluation are those that I developed to collect data from library users of these services to monitor activity and to help improve delivery of these services.

As I argue in this paper, public libraries are valuable to the communities that they are part of in a multitude of ways. This particular library is faced with many challenges that affect both the quantity and quality of services that it offers to users (and nonusers). Services to youth are especially crucial in this day age, because they help to support the personal, informational, education, and recreational needs of this demographic. The El Cerrito Library currently offers a variety of services to children and teens which fall into categories of public sphere, education, and information in order to fulfill its mission of “Bringing People and Ideas Together.” Due to the aforementioned challenges, the number of services it can offer to this group is quite limited and practically nonexistent in the homework help area. A review of the current literature highlights similar public libraries across the county facing problems similar to the El Cerrito Library, but who have utilized resources in a creative way, have formed community partnerships, and have offered non-traditional web-based homework help to meet the needs of the youth in their communities. Lastly, I argue that evaluation plays a crucial role in accountability to all library stakeholders, in gathering the input of targeted users, and in determining the areas where improvement is needed.

In this assignment, I looked to both the mission of the library system and the benchmarks created by other libraries (discovered through literature review) to assess the services the El Cerrito Library provided to youth. Through evaluation, which used data collection techniques including interviews, observation, and secondary research, I identified the service of homework help to be especially lacking at this library. Using the findings of my assessment report, I create specific suggestions and recommendations to increase and improve services to youth in the El Cerrito community. Conducting evaluation of library services should be an ongoing process in all public libraries. Using the skills that I demonstrate in this assessment assignment will no doubt be a part of my future duties as a professional librarian. As I have shown, sometimes creativity and careful balancing of resources is needed in order to provide better services to our users.

Evidence 2: LIBR-260A- Assignment 3: Children’s Library Program

This assignment was created for LIBR-260A: Programming and Services for Children. The purpose of this assignment was for students to attend a children’s library program (that was not a storytime), write an observation report, and evaluate aspects of the program for effectiveness. For this assignment I attended an El día de los niños/El día de los libros event at the Lakeview Branch of the Oakland Public Library, which I selected because it was not a storytime program and it did not occur at the library I was working at the time (Berkeley Public Library).

The first part of the report provides a bit of background information about the Lakeview Branch, in addition to information about El día de los niños/El día de los libros (Children’s Day/Book Day). Next I provide a report of my experience attending the event at the Lakeview Branch, which included a bilingual (English/Spanish) magician performer and a free book for every child attendee. The second part of the report is an interview that I conducted with the children’s librarian of the Lakeview Branch, who was responsible for orchestrating this program. Questions that I asked during this interview covered the topics of publicity, information about the event’s performer, and budget/funding.

Based on the information that I gathered through my research about this program, my observations attending the event, and my interview with the Lakeview children’s librarian, I was able to conclude that the event was enjoyable and successful for all attendees. For example, though my data collection techniques, I determined that this event was able to draw in at least twice as many families than the number that usually attends regularly held events at the branch. Additionally, I found that the children’s librarian made creative use of her very limited budget by obtaining free books from a local nonprofit organization to give to program attendees (80 books were given away in total). Lastly, the booking of a bilingual performer catered to the unique language needs of the diverse community in which the Lakeview Branch is situated.

In my future career as professional librarian, I will no doubt be in charge of organizing similar programs to engage the community. Using data collection methods such as the ones that I used for this assignment will be important in evaluating the effectiveness of these programs, and also can help in highlighting areas which need improvement. Use of evaluation findings about programs and services serve not just as tools that can be used by librarians to strengthen and improve them, but are also important to advocate for funding from the larger organization, the community, and governing bodies. When we are able to demonstrate how our services and programs have helped to improve the lives of our library users, we show how our institutions are integral parts of our communities.

Evidence 3: LIBR-289- Internship Final Report and IndieFlix Evaluation

This evaluation report was created for LIBR-289: Professional Experience: Internships. During my collection development internship at the Oakland Public Library (OPL), I was tasked with evaluating a digital resource to determine if its purchase would be beneficial for the needs and interests of the Oakland community. For this activity, I was in charge of evaluating the online streaming video service, IndieFlix. The main purpose of this activity was to determine if the content available through this service was also available elsewhere on the Internet for free. Because there is so much content available on IndieFlix it was impossible for me to evaluate every single movie available, so I had to construct a methodology to make the evaluation process more manageable. Focusing my sights on film festival content, which library in Oakland users would especially be interested in, I evaluated these movies to determine if they could be accessed for free elsewhere.

My findings are presented in this evaluation report, which provides a description of the service and its layout, charts about film content, calculations about the percentage of films available for free elsewhere, and my evaluation conclusion about the resource. When I completed my evaluation, I presented my findings and evaluation report to the digital resources committee, which is comprised of staff from various departments of the OPL. The information I presented helped these staff, who hadn’t personally evaluated the resource themselves, to decide if it should be purchased by the OPL.

Evaluation of services that we offer library users, such as IndieFlix, are of paramount importance. Through this evaluation we can determine if they are useful to the needs of library users. For this evaluation I used the criteria of access to films for free outside of the service, and access to award winning films to determine the potential usefulness of this service to Oakland Public Library users. In this particular instance, the library had not yet purchased access to this service, so evaluation was important in determining the pros and cons of doing so. Similar evaluation techniques can also be used to assess services already purchased for use by libraries to determine ongoing usefulness of them as well. In my future profession, evaluating services prior to purchasing will be a necessary task; services must be deemed useful to the needs of library users, otherwise they are a waste of resources. When we provide users with access to services that meet their varied needs, we become integral parts of their everyday lives; with this we will continue to be an integral and relevant part of society.

References

Cassell, K. A., & Hiremath, U. (2013). Reference and information services: An

introduction (3rd ed.). Chicago, IL: Neal-Schuman.

Dresang, E. T., Gross, M., & Holt, L. E. (2006). Dynamic youth services through outcome based

planning and evaluation. Chicago, IL: American Library Association.

McClure, C. R. (2008). Chapter 16: Learning and using evaluation: A practical introduction. In

K. Haycock, & B. E. Sheldon (Eds.), The portable MLIS: Insights from the experts (pp. 179-191). Westport, CT: Libraries Unlimited.

Moran, K., Stueart, R. D., & Morner, C. J. (2013). Library and information center

management (8th ed.). Santa Barbara, CA: Libraries Unlimited.

Rubin, R. J. (2006). Demonstrating results: Using outcome measurement in your

library. Chicago, IL: American Library Association.

Evidence Files

Click to download the following files:

LIBR-232 El Cerrito Library: Services Assessment

LIBR-260A Children's Library Program

LIBR-294 Internship Final Report

LIBR-294 IndieFlix Evaluation Report