You’ve heard the reasons why professors don’t trust RateMyProfessors.com,
the Web site to which students flock. Students who don’t do the work
have equal say with those who do. The best way to get good ratings is
to be relatively easy on grades, good looking or both, and so forth. But what if the much derided Web site’s rankings have a high
correlation with markers that are more widely accepted as measures of
faculty performance? Last year, a scholarly study found a high correlation
between RateMyProfessors.com and a university’s own system of student
evaluations. Now, a new study is finding a high correlation between
RateMyProfessors and a student evaluation system used nationally.
Strike another victory for Web 2.0. Here is more.
…and another defeat for student evaluation of teaching?
“For his part, Sonntag acknowledged that some RateMyProfessors.com reviews are “so mean-spirited† that they aren’t worth anyone’s time. ”
I think that this is the crux of the issue; professors don’t enjoy being insulted anonymously with no chance of correcting the statement. The main thing this study seems to uncover is that the same biases that ruin RateMyProfessors.com credibility ruin university-sponsored student evaluations as well. All RMP adds in are nasty and unmoderated statements. RateMyStudents.com is already taken, unfortunately, because I’d love to have a site where professors could offer the same evaluations of their students. Employers could probably make great use of the site, and the value of academic signaling would be increased, benefiting students who make an effort. Of course, some students would be outraged, because how could anyone dare say something bad about THEM!
I take great joy in the discovery there’s a high correlation between results of student evaluations and RateMyProfessor.com. Wow, Voice of the Customer comes to higher education, now if only someone will listen and respond accordingly.
I write the above as a 53 year old who last year returned to graduate school in the School of Public Policy at GMU; I attended a large public university as an undergraduate and a small private college for an MBA. I also have a daughter finishing her junior year at a small public liberal arts institution and so also have a parent’s perspective
My experience as an undergraduate student in the 1970s, graduate student in the 1980s and currently, alumni and current parent is that all higher education really cares about is accessing my wallet. Customer satisfaction (with customers being defined as students and parents) is an unknown philosophy, with perhaps the exception of dorm space, food service offerings and extracurricular offerings. How teachers actually teach is never addressed, except as a question to the tour guide on your typical undergraduate tour.
I love the feedback on RateMyProfessor; maybe the classifications could use some work and refinement but it’s a great start and there’s great performance information in there.
I can understand academicians don’t like it but it’s time for the academy to move out of the middle ages and compete with performance in the classroom with customer satisfaction one of the metrics.
Don’t student evaluations suffer from the same problem as the web site? Students who don’t do the work can just as easily fill out the evaluation as log onto the site.
I’m finishing up my undergraduate studies, and have always tried to leave well-reasoned feedback on RateMyProfessor.com. However, there is a lot of “I got a bad grade and will therefore complain”.
Good news for Cowen and bad for Tabarrok.
I’d trust ratemyprofessors and especially pickaprof (which includes grading history for some schools) a lot more than the officially sanctioned feedback. Students don’t trust the anonymity of the latter and don’t want to rattle the cages of profs they might see again even if they do trust it. Rate and pick are intended for other students and not meant to be fed back into some bureaucracy where they’ll do nothing.
Sure, they’re gonna skew low: malcontents are more likely to post than champions. But malcontents often have good points.
Student evaluations are nonsense.
At my university, we have group final exams. There are a few (hard) professors who get terrible evaluations. But on the group final (all sections of calculus X take identical finals, except for changed numbers/orders), their students tend to score 5-10% higher than the average.
One of them explained it to me: “my students hate me because they only understand 2/3 of what I say. But I say 2x as much as anyone else, so they come out ahead. They just don’t know or like that.”
I’ve read my own ratemyprofessor rankings and even the bad ones are pretty accurate. I’ve read those of my friends and enemies; also pretty accurate. The problem with ratemyprofessor is that the scale is inverted; easy professors get smiley faces hard ones not. This certainly drives off the lazier students, and depresses course enrolments in a potentially troubling way but no more so than gossip in the dorms. I also think this does not reflect the values of all students and even can act as a signal to those students who want to learn.
The problem is not with student evaluations it is with the way they are used by faculty. I have seen more than one colleague on a tenure committee pick a very witty turn of phrase and use it as the sole basis for a devastating attack on junior faculty member’s teaching ability. In my own experience I would certainly like to throttle the single student who wrote that my philosophy of science class was dismissive of sociology. Not because sociology should not be dismissed but because two of the five people on my tenure review committee are from the sociology department.
As an undergraduate student, I don’t trust a damn thing of what anyone else says. Mostly because what everyone else says is highly variable, and young students tend to be highly emotional while criticizing/complimenting their professors.
I’d say a Rate My Professor rating isn’t really useful until the student has had some time to digest what really happened. I absolutely hated my marketing teacher at the time I took the class, but I learned later on that he really helped us learn about the world
ratemyprofessor.com is a good website because professors aren’t interested in real feedback from students, so don’t listen or blow off criticism with the thought “he’s MY student, what does HE know about teaching this stuff?” Well students may not have first hand knowledge of teaching, but they have first hand knowledge of learning. Most professors make their subjects unnecessarily hard because they don’t want to see themselves as soft professors. This site isn’t for the bad professors, but for students interested in knowing who the bad professors are.
I am in graduate school right now. I’m not sure how much of a difference there is between bad professors and hard professors. You’ll get comments like this, from above:
“One of them explained it to me: ‘my students hate me because they only understand 2/3 of what I say. But I say 2x as much as anyone else, so they come out ahead. They just don’t know or like that.’”
This is pathetic and clearly shows that many professors DO NOT take pedagogic concerns into account, then rationalize their own foolishness. Teachers that do not know how to properly balance their own courses to make sure the amount being taught in the class isn’t so overwhelming that the highest grade isn’t 61% are just as bad as teachers that cannot explain things clearly. Making students unable to understand the material because you’ve decided to present WAY to much material, or my favorite, the same material “from alternative perspectives” (this one is especially idiotic since the ability to appreciate alternative perspectives DEPENDS on understanding at least one perspective all ready) is just as bad as not being able to clearly explain the material to your class.
Any student truly interested about a professor will use any means available to find out about a professor. Without fail, the worst rated professors really do turn out to be the worst professors. And if I have any doubt, I ask my fellow mathematicians, who will tell you straight up if the guy is bad, hard, easy, good to learn from, etc.
Again, websites like this seem to upset faculty and the administration so much because these sites are uncontrollable and the truth of the idiocy of higher education gets exposed. This is something they do not find amusing. Welcome to the information age, fellas.
Indiana Jim:
Are all of those comments from the same course? There is one professor at the university I attend with RMP reviews that average “Horrible course, horrible professor” for a 100 level gen-ed course and “Hard as hell, but you’ll learn a lot” for a 300 level course.
Ken wrote: “websites like this seem to upset faculty and the administration so much because these sites are uncontrollable and the truth of the idiocy of higher education gets exposed. This is something they do not find amusing. Welcome to the information age, fellas.”
Well Ken, I’m not upset in the slightest. In fact, I am happy for RateMyProfessor.com. When students post comments like this guy is “****ed up” they just make it clear that ad homenim is not below them. When the same student argues that I’m an “A” student and got a C- in one of his classes” this indicates that he/she either: 1) does not understand that by getting a C- he/she is no longer an “A” student (if he/she means a straight A student); or 2) he/she believes that because he/she has gotten A’s in the past that he/she deserves them henceforth regardless of performance, field of study, input of effort, etc. Being hectored in such a way is a clear signal to thoughtful readers of the dubious intellectual integrity of the post.
No, I think that the posts can be highly revealing in a way that was in no way the intention of the person posting. This has the ring of the invisible hand.
Indiana Jim,
Are you a good teacher or not? This would help assess whether the comments are predictive or not.
I’m a seasoned student–several undergraduate institutions and graduate school–and in my experience the single best indicator of professor quality is simply whether or not they make their own lecture notes. The ones who do are nearly invariably above average, and the more detailed the notes the better the professor, the more sincerely and explicitly they wish to convey their body of knowledge.
Most students, though, evaluate professors based on about a 50/50 weight of whether or not they: 1) Enjoyed the class and 2) Got good grades. To me, this argues powerfully for the signaling model of education because post-enrollment, these are the two key variables that determine your utility gain from the class (1 determines cost and 2 determines the change in the strength of the ability signal). Neither has much to do with much you learned.
I should say–as someone whose gotten pretty good grades even in advanced, graduate level math classes–that most professors who are notably harder than other profs in their institution/department are just arrogant jerks. They think that there is some real level of difficulty that students ought to experience, but without standardized testing this is just sheer nonsense. You need a large comparison population in order to be able to say that a test really wasn’t as difficult and that some students deserve low grades. All profs grade on a curve, the jerks just enjoy making the curves so that most students suffer.
As well, the profs who don’t make lecture notes are also much more likely to be the hard-grading jerks. This is both because they are not nearly as good at their job as they think and because the lecture notes make information more available for students.
Most profs, though, don’t make lecture notes or prepare for class much and delude themselves that it is somehow their students’ fault when they’re not understood.
D and I.J.,
So, let me be specific, and you can judge for yourself. My overall rating on ratempdc
is 2.7. My easiness number is 1.9, giving me about a 2 to 1 ratio on that. My actual
overall general rating from my over 100 class averages in the official evaluations,
being kept secret for my “benefit,” is over 4.0 definitely, although I have not calculated
it (we are talking about several thousand students, whereas the ratempdc score is based on
somewhat over 40). Many say I am “tough but fair,” and each semester several say I am
“the best” they have ever had, while about 1 in 20 regularly excoriate me as much as they
can. The ones who like me a lot also show up somewhat on rate…, making my ratings a
rather sharply mixed bag. So there; go figure.
Ken,
“I think that your A student getting a C in a class is indicative of a bad teacher rather than a student feeling entitled to an A being pissed he didn’t get an A. Consistently getting A’s is comes from a strong work ethic and mind that doesn’t go away upon entering a new class.”
Forgetaboutit; the inflation of grade that I have collected data on at indicates that getting A’s need not come from either a strong work ethic or a “mind that doesn’t go away.”
Expectations are generally set by people relative to what they observe nearby. My department, economics, in entry level classes assigns grade that do NOT differ significantly from a 2.0. On the other hand, in beginning classes in marketing and management the average grade is closer to 3.0 than 2.0. This difference creates false expectations, obviously, and means that while your comment that A’s come from hard work DOES apply to course in economics it DOES NOT elsewhere.
You may blog what you like (“I think that your A student getting a C in a class is indicative of a bad teacher”), but blogging something does not make it so; and it certainly does not in this case.
I went to college before there were rating systems, but I remember most of my professors pretty well. I would have rated about half of them correctly (that is, I would still regard them as good or bad). The other half would have received highly inaccurate ratings: I would have overrated some professors that entertained me even if they didn’t teach that much and I would have underrated professors that were hardasses and challenged everybody to achieve more than they thought were possible. After the fact, I gained much more from the demanding teachers I disliked at the time I took the course. Now I’m thankful for those people, but at the time I feared and disliked them.
I have been a professor for 30 years now, and have received good to excellent ratings the whole time. But I know I could make them higher by manipulating the system and pandering to my students. Some of my colleagues have cynically done so to their benefit. I cannot see how that is good for education generally.
Ken and others who think that ratemyprof is just fine,
Would you support open publication of official evaluations of professors that are full sample, all the students in the classes they teach? I do, and because I happen to know that what shows up on ratemyprof is in some cases a classic example of small sample bias. It is not always accurate of broader student opinion. It represents a self-selected ssmple, and anybody who knows anything about statistics knows that this can be very misleading.
Regarding whether being a “hard-ass” is a good thing or not, well, I am one, but I shall not get into this further other than to say to those who appear to be arguing for dumbing down material in courses apparently so that students can get good grades, are you really sure that this is what you think should be done? It is certainly my view of what has been done to a lot of textbooks in economics in recent years, especially at the Principles level.
Barkley,
I have no problem with hard material, and I don’t really have a firm opinion on what grade distributions should be in the first-best sense. That’s a very complex question that requires balancing the signaling quality of grades, fairness to students, the role and purpose of the professor, etc. For instance, is it ethical for a single professor to give a grade distribution far below that of similar institutions just because he thinks that those are the grades that are “deserved?” I just plain dunno.
I just wish to emphasize a few things:
1) The level of difficulty of material and the grade distribution are separable. You can teach a really tough course but curve to all hell.
2) The prof who gives very low relative grades and blames it on the students seems to assume an unjustifiably privileged position for himself. He/she is administering a non-standardized test that he himself made up and assuming that he knows what the “right” grade should be. ALL PROFESSORS CURVE; it’s just that the hard-asses like to curve downwards. If you want to say to a student that they deserve a low grade, you ought to be able to back it up by demonstrating that their score was low relative to some representative population test score. Otherwise, you’re basically just saying that they did low relative to the subjective standard that you yourself hold, and that cannot be questioned.
3) Almost all the hard-asses I’ve had (and I’m a pretty good student who has survived even the hard-asses), seemed to me to more or less be lazy (almost none of them made their own lecture notes and distributed them to the class (see my comment above)). How much students learn isn’t just a function of student effort, but teacher effort as well. Hard-asses more often than not assume that almost all the work should be done by the student. I will bet dollars to donuts that if the average hard-ass’s test were administered to a large, representative population that had time to study for it, grades would go up because other, good professors would find innovative ways to teach the material.
4) Hard-asses almost always see themselves as smarter than everyone else. Being profs, they’re certainly smarter than most people, but usually not as smart as they think. They think that students do badly because their stupid; indeed, most students are stupid, but in other areas of life even stupid people can learn a lot IF properly instructed. Stupid people routinely do better on the bar exam and myriad financial exams than they do on the average hard-asses test because even though the former test much more difficult and complicated material (on average), there are standardized, tried and tested ways of learning these things. Often, the hard-asses would just rather not be bothered with putting in the effort to figure out what the better methods are of teaching their material.
indiana jim: Sorry to hear that you are dealing with bad professors.
I appreciate the sentiment. It does come with the territory of being a student, though. I knew that upon returning to school, I just find it more frustrating than ever.
What I am so surprised at is just how bad some professors are. As an undergrad, I seemingly had unusual luck with professors, or was just too young to realize how bad professors were. I am in my 30′s now and my patience with the arrogance of academia runs incredibly thin.
As a grad student, I have much for intimate contact with the professorship and am amazed at just how much professors look down on their students. Of course, this rubs off on the TAs, who do nothing but bash their students as well. They have the attitude that if students do poorly it is clearly the students fault. The fact that the grad students and professors have far more mathematical talent than their average student has no correlation whatsoever on their effectiveness at teaching. Yet they think this must be so.
Teachers generally speaking like to give themselves passes on the poor performance of their students, which I find to be disgusting. I am aware of the laziness of many students and that many bad grades or the result of this. I am not an A student because I prefer to hang out with my friends more than study, and many of my poor grades are my fault. But I am very indignant about the poor grades I receive from poor professors and grown more indignant as I get older. As with students, the following is true of teachers: half of them are below average. And as a lifelong student, I am aware of just how low the bar is for average for both students and teachers.
As with all people, teachers are incredibly immune to any sort of criticism, blaming all others (it clearly cannot be them) for the woes of their students performance. Sorry to rant and rave so much, but I truly find it irksome at the idea that poor performance lies mainly with the student. Any sort of thought given on the subject will lead to the conclusion that poor instruction makes up for half of the poor performance of students. After all, formal learning in a classroom setting requires good instruction and good study skills. Both are equally important in this setting.
Poor professors have their place because they force students to acquire skills needed for independent learning, but this is no excuse and some teachers are so bad that even this fails.
Here’s a rule of thumb for people blaming the students. Once you get more than 30 data points and you still think they are below average, you might want to doublecheck your assumed average.
Ken, check out phdcomics.com for a little therapy.
Okay, last thought. Universities could disaggregate the learning process from the accreditation function (grades). Will they do this? Group finals sounds like a start, but I’d bet they do this more for convenience and cost reduction rather objectivity.
A 300 level class may be too small for group finals, but what if you pooled the whole world and profs everywhere submitted questions? Presumably, if something is worth learning, other people are interested in it. This would also work to standardize the curriculum over time as well as approach better evaluation of the value of a department’s degree.
Barkley: “Why should any university use ratemyprof over official evaluations they make
based on the full sample of students taking each course?”
Because everyone knows what hasn’t been corrected on ratemyprof. That’s one reason. It’s been my opinion that student/employee surveys sponsored by the institution are more about making the underling feel like the institution cares. They don’t, of course, until it gets so bad as to impinge on their survival instinct.
I also always felt like the surveys were done kind of helter skelter. My last couple semesters I made a point to write comments on a sheet of paper so I’d have them ready for survey day.
Also, everyone is talking about RMP as it is. Granted, it’s kind of a joke right now, with the hotness or whatever. What about how good it is likely to get in a few years?
Everyone dealing with grading (on both sides) might benefit from reading a little Deming and quality control methodology. Why do you consider it “caving”? I think it should be more like changing the tolerances to match the process capability.
“Which are more reliable?” To who? The absolute results don’t matter. What matters is where you are relative to your competition. People are smart enough to discount outliers. And when more data populates the database, people won’t even need the comments.
The proper measure of a teacher is how much the students learn. Measuring this is not always easy, but no other measure makes any sense, because what the teacher is hired to do is to cause learning in the students.
I.J.
Something around 2.6 sounds pretty reasonable. I know programs and departments around
here (ones whose majors do not get paid too well), where the GPAs are now well in
excess of 3.5. My wisecrack about a convergence on 4.0 was not a joke.
Perhaps a good modification to the sight would be to require professors to provide an aggregate grade score for the class. Then, when each student signed in, they had to post their grade. With a minimum of say 15 or 20 students,an AVERAGE grade of those posting could be calculated. Now, in order to protect students and professors, the site could post a score showing the DIFFERENCE in values between the overall class grade average verses the average of those on the site. That way, complaints could be taken with a grain of salt, if there was a wide variation in overall scores the prof gave verses those complaining, winnowing out the ‘sore looser’ problem.
Thoughts?
“no other measure makes any sense, because what the teacher is hired to do is to cause learning in the students.”
In my field, professors are hired to bring in research grant money. Teaching is something you do to justify the university’s existence and supplement your income. This problem is a separate but related issue.
Do I have to be a teacher to understand the product? In fact, aren’t non-teachers more objective about the service they provide? How long has it been since you were a student? I think between the two of us, I would have the authority in this argument.
And yes, things are different from place to place. But here’s a generalization for you. If something works pretty well, chances are it has been copied, or needs to be copied, more or less.
Your over-specifics are remarkably empty. Without generalization, what do you have? Details that don’t mean anything to anyone else. I’m overgeneralizing? Okay, fair enough.
“frankly, more change does come from actual evaluations than from ratemyprof in most places, certainly where I am”
Well, is that really saying much considering RMP has probably been around for 5 years, and only popular for about 2, while universities have held court for centuries?
People actually adjust to their reviews? Why? The goodness of their heart?
The last half of your response is a lot of strawman arguments against things I never said. Generalizations one might say. It would be really dumb for people to replace their current system with RMP, if the system is good. I never said RMP is great, just that it provides information and will provide more and more in the future. Its potential is great. It will change academia. That’s a prediction you are free to disagree with.
Department heads who know it is unreliable better reassess every few years.
Can you see the official reviews of other professors in your department? Do you know where you stand in relation?
I never said they are a perfect source, but for some people for some information, they are the only source.
BTW, for me, RMP sucks because only a small fraction of my available profs are in they system.
But, considering how good it is already, contrasted with how bad it is now, virtually ensures dramatic improvement.
And RMP isn’t just a site, it’s a concept. If RMP doesn’t improve, why wouldn’t someone else overtake it. You said you taught economics, right?
If you don’t like RateMyProfessors.com just use SyllabusCentral.com.
My opinion is that reviewing professors is not nice or appropriate. If it is about education student should review classes, like at http://www.5caps.com. After all they need to know that the class is useful, not that “prof is hot†.
The standard university faculty evaluation system has some, albeit limited, value. However, allowing students who may be entitled goof offs to rate faculty members anonymously is an invitation to slander. Only yesterday I read the online evaluations of a faculty member I know and respect. One student complained that he gave no exams, only a lengthy paper assigned two weeks prior to the end of the semester. Another joked that his exams were easy.
I then checked out the online ratings of another faculty member. The three students who bashed this highly respected educator had one thing in common…the inability to write a single sentence without an error.
all the things have two sides
Comments on this entry are closed.