Personal tools
You are here: Home Publications GUIDE TO EVALUATION OF INSTRUCTION Data From Students: Systematic Student Evaluations

Data From Students: Systematic Student Evaluations

Table of Contents | Previous Section | Next Section


The Student Evaluation System at UCLA

The Office of Instructional Development's Evaluation of Instruction Program (EIP) helps assess and improve teaching at UCLA by providing instructor evaluation services for instructors and TAs across the campus. At the end of each quarter, instructors have the opportunity to solicit formal evaluations from students enrolled in their classes. EIP distributes, collects, and processes evaluation forms via a network of departmental coordinators. Interested instructors should first contact their department evaluation coordinator, or may reach the centralized Evaluation of Instruction Program at extension 56939 (http://www.oid.ucla.edu/eip).

EIP's Standard evaluation forms, which cover most teaching situations for faculty and TAs, were designed in consultation with faculty committees, national experts on assessment, and recommendations from UCLA and System-wide surveys of faculty, TAs and students. While most departments on campus use the standard campus-wide forms, a few departments and schools have devised their own standard forms that are distributed and administered by EIP. The Faculty Consultation Service can work with departments and individuals to develop questionnaires for special needs.


Policy on Data

The Evaluation of Instruction Program seeks to provide as secure an environment for data as it can. The forms are stored for processing in a physically safeguarded location. Data are compiled centrally only for the numerical scores. In addition to individual instructor scores, larger collective comparison data (e.g. for Divisions, or Schools) are also calculated. These compiled individual and comparison data are returned to Department Chairs along with the original student forms. Each Department may devise its own particular policy on how data are presented, distributed, or made openly available. Most often, the forms are returned to the instructor so that the written comments may be read firsthand. Some departments choose to transcribe the comments for small seminars to protect the identity of the students. The numerical data are provided by the Evaluation of Instruction Program in electronic format, and include both departmental scores and appropriate comparison data. The Evaluation of Instruction Program does not release individual instructor data except to the Department or to the individual instructor. External requests for such data are referred to the Departments. External requests for comparison data (without any individuals identified) are generally granted.

Retaining Data

Most departments develop a reliable system for storing numerical data from teaching evaluations. It is less common for departments to retain written comments. Because such data are often used when compiling dossiers for personnel decisions, faculty should be careful to keep copies of their own evaluations. Even normally reliable systems sometimes have unexplainable lapses, and it can be extremely difficult (if not impossible) to re-establish such data after the fact. In addition, it may be useful to annotate records with information that might provide insight into any anomalous results (e.g. “the class was scheduled into a room that was frequently impacted by construction noise,” or “it was my first attempt to develop group projects and they did not work the way I had hoped.”)

Some departments retain only overall ratings, and again, instructors would be better advised to keep data which encompass all the individual items on the form. Such information can often expand on understanding why certain classes may have been rated higher or lower.

Most evidence of teaching data is used within a period of six years after the time it was collected.Based on actual requests for re-establishing older records, eight years would provide a more certain time frame.Comparison data, such as to other instructors in the department or to the overall University mean, should likewise be kept in order to provide bases for comparison, should later disputes arise.Faculty who are nominated by their departments for teaching awards also find some of the written comments useful in documenting what students find particularly compelling about their teaching.

Disputed Data

In the infrequent situation that the integrity of the data are disputed (e.g. if forms are intercepted by the instructor or the chair before processing, or considerably more forms are returned than the number of students enrolled in the course) what facts are known are forwarded to the Committee on Teaching for any further consideration of action. Other Senate Committees may become involved as appropriate.

top


Student Questionnaire Administration Procedures

Since student ratings are sensitive to a wide variety of situational factors, it is very important that questionnaire administration procedures are consistent. The goal is to get candid feedback focused on instructional issues from as many students in the course as possible. The following general guidelines for collecting student ratings may help instructors achieve this goal:

Since student ratings are sensitive to a wide variety of situational factors, it is very important that questionnaire administration procedures are consistent. The goal is to get candid feedback focused on instructional issues from as many students in the course as possible. The following general guidelines for collecting student ratings may help instructors achieve this goal:

Protect student Anonymity

It is important that students not be identified in any way with their evaluations, otherwise less than candid responses are likely. In particular, students fear that an adverse rating might negatively impact their course grade. It is, of course, difficult to maintain confidentiality of student raters in small classes and individual study courses. There is no simple solution to this problem. One option is to have the departmental coordinator type up the responses in such cases.

Have a third party administer evaluations
In order to protect anonymity, evaluation questionnaires should not be administered by instructors or TAs. Rather, a responsible student or the department evaluation coordinator should be appointed to collect the completed questionnaires and deliver them to the department office.

Time questionnaire administration appropriately
Ratings should be collected during the last two weeks of the quarter. Students should be forewarned that evaluations will be done on a certain date so that they will be in class and will be prepared. Administration of evaluations at the final exam or other test is not recommended.

Emphasize the importance of evaluation
It is advisable to give students some context for the evaluation, especially for first year students. It is useful for them to know that the department and instructor value their comments, and the use to which they will be put. Distributing questionnaires at the beginning of the class period and allowing sufficient time for students to complete them, all contribute to the sense of importance placed upon a student’s opinion, and are hence likely to produce more constructive results.

Suggested Readings:

Arreola, Raoul. 2000. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.

Davis, B.G. (1993). Tools for Teaching. San Francisco, CA: Jossey-Bass

top


Interpreting Quantitative Student Evaluations for a Course
Several weeks after the end of each quarter, instructors will receive their students’ responses to the formal questionnaires, along with a summary sheet of statistical information, including mean, median, and standard deviation of the questionnaire responses. Instructors should keep the following points in mind when interpreting the results:

Procedures
Were reasonable procedures (such as suggested in the preceding section) used to collect the ratings?

Sample size/response rate
How many students in the class completed the questionnaires? Responses from less than two-thirds of the class should be viewed with caution. The minimum number of student raters in a class should be 8 to 10; a sample of 15 or more students increases reliability and generalizability of the results. It should be noted, however, that even in such “small sample” situations the qualitative comments may still be extremely valuable.

Making comparisons
The average ratings can be interpreted according to an absolute scale or relative to the ratings of other courses and instructors. For example, a mean rating of 6.5 on a 9-point scale for overall course evaluation may seem “above average.” Taking them at their word, students rated this course as adequate. It may be, however, that 70 percent of similar courses are rated above 6.5. Thus, relative to other courses, the 6.5 rating was in the lower third. Which interpretation is correct? The answer is that both interpretations are useful. The course was judged positively, but students were not particularly excited by it.

The Campus Context
In making comparisons, it may be helpful to consider the campus context. Means do vary considerably between departments and divisions, or between different kinds of courses. Departments are, therefore, encouraged to keep records regarding their own norms.

Variability of responses
The variability of student responses is important diagnostic information. For example, consider an average course rating of 7 on a 9-point scale with a small standard deviation, say of 1.0 or so. This means that most students rated the course as 6, 7, or 8. On the other hand, the same average rating with a larger standard deviation, say 3.4, indicates a greater range of ratings that may suggest problems with the course. It is also important to look at the total distribution of ratings. For example, there are sometimes bimodal distributions of ratings in which a large group of students really liked the course and another large group of students disliked it. This could indicate two different ability or interest groups in the class, which would be worth exploring further for future iterations of the course.

The importance of other variables
Although student response to a course is important for evaluation and teaching improvement, it should not be used as the only measure of teaching. Student ratings are affected by many variables including course content, amount of work, expected grades, class size, and students’ own needs and attitudes.

Suggested Readings:

Arreola, Raoul. 2000. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.

Cashin, William E. Student Ratings of Teaching: Recommendations for Use. (IDEA Paper No. 22). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1990.

Cashin, William E. Student Ratings of Teaching: The Research Revisited. (IDEA Paper No. 32). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1995.

Davis, B.G. (1993). Tools for Teaching. San Francisco, CA: Jossey-Bass

top


Interpreting Quantitative Student Evaluations for Personnel Decisions
Faculty members are often concerned about how evaluation results will be used by their departments in administrative personnel decision-making. Although details of the process differ among departments, the substantial body of research literature on the use of student course and instructor evaluations suggests that the following practices be observed:

Guidelines for administration and explanations of how ratings will be used should be consistent and well-publicized. Small differences between scores should be kept in perspective. Multiple sets of student ratings should be used in administrative decision-making. Global ratings (i.e., overall ratings of the instructor) are more often used than specific items (such as the instructor’s organization or communication skills) for making personnel decisions. While this may be the most convenient measure, decision-makers might note that global ratings are also those most likely to reflect personal bias on the part of students. Ratings for any single instructor or course should be considered in conjunction with university, college, division, department, and even specific course norms. Multiple sources of information should be used in administrative decision-making. In other words, numerical ratings should be only one piece of the larger picture.

Suggested Readings

Arreola, Raoul. 2000. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.

Cashin, William E. Student Ratings of Teaching: Recommendations for Use. (IDEA Paper No. 22). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1990.

Cashin, William E. Student Ratings of Teaching: The Research Revisited. (IDEA Paper No. 32). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1995.

Davis, B.G. 1993. Tools for Teaching. San Francisco, CA: Jossey-Bass.


top


Further Help for Faculty Concerned about Student Evaluations
Perhaps the most common concern that faculty members express about the evaluation process is that students do not take the evaluations seriously and that students are not aware of the gravity of their input into the tenure and merit review process. Research and experience show that instructors who openly announce to students that they themselves take student input seriously are usually the recipients of the most constructive comments.

The vast body of research on student ratings that has accumulated over the years also suggests that student ratings correlate highly with the ratings conducted by one’s own colleagues and even, in some instances, with self-ratings. The high correlation holds up, however, only when students are asked to judge selected aspects of teaching. Students as consumers are well-equipped to assess the clarity of presentation, to comment on organizational matters, to rate the quality of student/instructor interaction, and even to assess their own learning to some extent. Students are not, however, the best judges of whether what they are learning is current, whether the instructor’s perspective is biased, or even whether the selection of course material was appropriate for the achievement of course goals.

Instructors occasionally deride quarterly evaluations because they believe that students cannot make accurate judgments regarding a course or instructor until the students have been away from the course, or even the university, for several years. While it has proven to be very difficult for researchers to obtain a representative sample in longitudinal follow-up studies, the research shows that, in general, alumni who had been out of school for five to ten years rated instructors much the same as did the students currently enrolled. Current research also provides little substantiation for the widely held belief that only warm, friendly, humorous instructors win the “ratings game.” Most students are quite able to discriminate between glossy presentations with little substance and less flashy lectures with relevant content.

If a faculty member’s number one concern about evaluation of instruction results is how they will be used in personnel decision-making, the number one concern among students is that their feedback will not be acted upon. It is, therefore, crucial that having conducted any type of feedback activity with students, instructors be seen to respond to the results. This may not be as easy as it sounds, since bimodal distributions, for example, can make obvious courses of action elusive.

It is important for faculty—particularly first-time faculty—to remember that some students can be insensitive or may lack the maturity necessary to deliver constructive criticism in a non-threatening manner. At times, their comments are overstated and short on diplomacy. While such comments can be very discouraging, if they come from only a few students, they represent only an extreme point of view. However, if such comments come from a majority of the students, advice from a trusted peer or from an objective consultant might be useful. Even if painful, they may contain insight into teaching issues that can be addressed – but again, only if they present a cogent argument, not just a personal attack.


Suggested Readings

Arreola, Raoul. (2000). Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.

Davis, B.G. 1993. Tools for Teaching. San Francisco, CA: Jossey-Bass

Chen, Y. & Hoshower, L.B. 2003. Student evaluation of teaching effectiveness: an assessment of student perception and motivation. Assessment and Evaluation in Higher Education 28(1): 71-88.

McKeachie, W. and Svinicki, M. 2006. McKeachie’s Teaching Tips. Boston, MA: Houghton Mifflin Company.


top


Consulation
Faculty are well advised to seek consultation when deciphering their teaching evaluations.Quite often an outside perspective offers insight to evaluation data that may not be apparent to the recipient of the evaluation.Speaking with peers or mentors within one’s department or at other universities may help discern new approaches to instructional improvement.

Instructors who wish for advice outside of a departmental context are welcome to contact the Office of Instructional Development’s Faculty Consultation service. The office offers consultation to faculty on all aspects of teaching and learning. The objective is to help them reflect on their classroom practice and suggest strategies for improvement. Faculty Consultation services take the form of individual consultation by appointment, workshops for groups of interested faculty, and seminars in college teaching tailored for individual schools/departments. Professional consultation is given based on an analysis of some initial inquiry, or on the provision of such data as a classroom video taped by Audio Visual Services or a set of evaluation results administered through the Evaluation of Instruction Program (EIP). All consultations are strictly confidential and participation by faculty is entirely voluntary.


Suggested Readings

Brinko, K.T. 1991. "The interactions of teaching improvement." In M. Theall & Franklin (Eds.), Effective practices for improving teaching. New Directions for Teaching and Learning, No.48. San Francisco, CA: Jossey-Bass.

Cohen, P.A. 1980. "Effectiveness of student rating feedback for improving college instruction: a meta-analysis of findings." Research of Higher Education, 13 (4), 321-341.

Seldin, P. et al. 1999. Changing Practices in Evaluation Teaching: a Practical Guide to Improved Faculty Performance and Promotion/Tenure Decisions. Bolton, MA: Anker Publishing Company, Inc.

top

Table of Contents | Previous Section | Next Section

Document Actions
  • Print this
  • ShareThis