Peter Boettke’s Ph.d grading strategy

by on December 28, 2012 at 1:14 pm in Education | Permalink

…your final draft of your paper will be due on April 29th and I will submit your papers (blind) to external referees as well as myself for assessment, an A grade will be limited to those papers, and only those papers, that are recommended for acceptance or conditional acceptance, a B grade will be assigned to those papers that receive a recommendation of revise and resubmit, and a C grade will be assigned to those papers that are rejected by the external referees and myself.  I will be available throughout the semester to discuss and read your drafts, so don’t be a stranger.

The source is here, and for the pointer I thank Jacob T. Levy.

Scott Cunningham December 28, 2012 at 1:36 pm

This is brilliant. Particularly as I just read this morning from Kahnemans book the chapter on “jumping to conclusions” where he talks about de-correlating errors, and the wisdom of crowds. Very ingenious Peter. I’d be willing to share the cost of hiring some external referees too for my masters level causal inference course.

whatsthat December 28, 2012 at 2:57 pm

Eh, what crowds you talking about? This is only a couple of reviewers.

Claudia December 28, 2012 at 3:28 pm

He should do a referee swap with another PhD program. Part of our labor field at Michigan was writing a referee report of a paper en route to publication. It was a good teaching tool and a guided entry into the world of refereeing. Also grad students make great referees (few have axes to grind yet) and the cross university reviews are bound to shake things up a bit. I kind of wish I could farm out the report I have to write this afternoon as I am still in a post holiday good mood.

As an aside, I think this would be a great system for blog posts…blogs that have editors are much easier to digest and don’t seem to generate silly misunderstandings.

Scott Cunningham December 28, 2012 at 6:54 pm

Yeah, probably that’d work. You’d need to find someone needing students to write reports so that it got back in in a timely manner. I suspect for any decent journal, though, it is still impossible to write a paper capable of getting an R&R within one semesters time. But I like the idea of this.

Claudia December 28, 2012 at 7:38 pm

More important than the turn around time; it would stretch professional ethics some to assign your students submit papers to journals. My professional service as a referee does not include grading for other professors’ classes. Nothing in the syllabus suggests that the blind referees are in the dark about the process and I sure hope no one gets such an idea.

Scott Cunningham December 28, 2012 at 1:37 pm

The only challenge I see is getting these refereed in time to give grades. It takes on average, what, six months from first submission at any decent journal?

Rahul December 28, 2012 at 10:56 pm

OTOH, isn’t a semester a pretty short time to churn out a paper from scratch? If any students did get an A I want to know which the journals were.

genauer December 28, 2012 at 1:52 pm

Since this was over 10 years ago, did it work?
How many papers published?

An interesting way to get authors for a journal, many US academics frown upon

Jeremy H. December 28, 2012 at 7:18 pm

This syllabus is from the time period when Leeson and Coyne were students. I think they have been published well and widely.

Note that this language is gone from a more recent syllabus:

http://www.peter-boettke.com/app/download/6555001204/Econ881_2008_syllabus.pdf

Though note a similar grading policy (without external referees) is in an another course, “Advanced Topics in Austrian Economics” (“Austrian III” at GMU):

http://www.peter-boettke.com/app/download/6555001904/Econ895_Spring2012.pdf

genauer December 29, 2012 at 7:52 pm

thanks a lot for the precise reply!

I enjoyed the links, a lot !
I am still laughing, more than 30 minutes later.

It makes me wonder and very curious about what kind of employment contract /rules Boettke has.

I enjoy the arch austrian anarchist streak, which is most likely lost on you and most other people here, so far.

For tonight I just mention Paul Feyerabend “against method”, and the reflection of him by the present holy father, pope Benedict XVI, bayrischer Spitzbube and former Chief Inquisitor.

whatsthat December 28, 2012 at 1:54 pm

what about desk rejects? and aren’t most papers rejected somewhere anyway before they are published? what is the point of this then? to give everyone a C? in reality, you choose where to submit your paper, how is this working here?

libert December 29, 2012 at 12:09 am

Good point. One influential Nobel Prize-winning paper was initially rejected from publication in several journals

dearieme December 29, 2012 at 4:42 am

You are referring to the pretend Nobel, aren’t you?

Saman December 29, 2012 at 8:01 am

No, he’s talking about the one in physics.

Dear lord, everyone knows it’s only in memorandum, can’t all these wise guys drop the subject? Just like people who correct each other’s spelling in the internet to get attention.

Willitts December 28, 2012 at 1:58 pm

The assignment seems appropriate, but is it reasonable to produce a publishable paper in four or five months with every other class you have to take and teach? Sure, some papers can be churned out fast but most cannot, in my inexperienced opinion. Legal research doesn’t take very long, but I suspect economics is much more difficult.

IMO, one of the cardinal sins of teaching is behaving as if your class is the only thing going on in a student’s life.

What is the average time to degree for a PHD in economics, particularly for a top school?

Greg Ransom December 28, 2012 at 2:36 pm

The majority of the most cited peer reviewed articles in a number of fields cannot be replicated.

The peer reviewed economic literature is stuffed with incompetent and fallacious discussions and references to the economics of F. A. Hayek.

Much of the peer reviewed literature instantly becomes dust attracting space-wasters in the stacks, never again read by anyone.

These are mere examples of a general truth about the peer reviewed literature.

Peer review is highly overrated as far as competent and meaningful contributions to science go.

Dismalist December 28, 2012 at 4:59 pm

“Peer review is highly overrated as far as competent and meaningful contributions to science go.”

Completely agreed! It’s 90% trash, except we don’t know ahead of time which 90%!

Any superior alternatives?

K書中心 December 29, 2012 at 2:36 am

“The peer reviewed economic literature is stuffed with incompetent and fallacious discussions and references to the economics of F. A. Hayek.”

Which journals do you read?

Max December 28, 2012 at 2:39 pm

Seems pretty easy to get a C. That’s passing right?

mavery December 28, 2012 at 2:53 pm

Tends not to be in grad school.

Mark Thorson December 28, 2012 at 3:22 pm

The capitalization should be “Ph.D”. Revise and resubmit.

dead serious December 30, 2012 at 9:09 am

Your peer reviewed comment has garnered an A.

Barkley Rosser December 28, 2012 at 3:47 pm

This just sounds like Pete being lazy and getting others to do his work for him, :-).

Andrew' December 29, 2012 at 2:44 am

He’s a professor.

Zach December 28, 2012 at 4:24 pm

It sounds like a plan written by somebody who has never read a referee report.

What’s the grade for “please add 5 references to the work of obscure academic X”?

DocMerlin December 28, 2012 at 9:31 pm

An A+. The referee wants to be associated with the paper.

Andrew' December 28, 2012 at 4:30 pm

What’s his impact factor?

Jerry December 28, 2012 at 6:17 pm

Note that Boettke will assign an automatic A grade to any paper submitted with a cheeseburger.

liberalarts December 28, 2012 at 9:11 pm

Papers are more or less difficult to place in different disciplines and in different subfields within a discipline. For example, I will go out on a limb and speculate that papers in Austrian economics journals are surely either easier to get an acceptance or and R&R than in monetary econ or public finance journals. If that hypothesis is indeed correct, then the quality bar described in Tyler’s post would be uneven.

similar to liberalarts because L.A. has to "slow down" on the posting December 28, 2012 at 9:15 pm

^ I meant, “either easier or harder” above where I said, “either easier.”

Jack December 28, 2012 at 9:42 pm

As a professor (finance, not Austrian economics), I find this clever but counterproductive:
(1) Most likely, the journal will assign a single reviewer, or desk reject it. There is not a diversity of reviews on the paper.
(2) Reviewers are highly idiosyncratic, see Ivo Welch’s recent survey: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2137119
(3) In most good economics and finance journals, acceptance rates are 3 to 10 percent after revisions and 0 percent upon initial submission, and these are unconditional probabilities. For grad students, *conditional* probabilities are surely much lower. This implies most students would get a C, which might be grounds for dismissal in some PhD programs.

That said, there are some respectable but much lower ranked journals that have much higher acceptance rates. Maybe that is the case for the journals that Prof. Boettke has in mind. Still, I find it hard to believe any respectable journal would give enough accepts and revise-and-resubmits for the students to get decent grades…

genauer December 29, 2012 at 5:44 am

As somebody who left academia after my PhD in physics, just a few numbers (7 year averages, because I have those reports handy) for comparison, personal and from my institute:

yearly output of PhDs 9.6,
yearly publications in peer reviewed journals: 132,
average number of authors per publication about 5

my first paper came out of my diploma thesis (like the US masters, we say: a little better : – )
my PhD thesis lists 25 publication with me as an author, that was on the higher side for my Institute, my h-index 13, my professor 55

I do not recall any paper to have been finally rejected. paper writing was a side business relative to the core task: producing experimental results.

There was one instance of a second request for change, in the journal ” ranked number one in total citations” and nearly top impact factor, dependent which journals you count as relevant : – ) A referee wanted the second time to have 2 remotely related, presumably his, papers referenced, and my prof originally wanted to give in, but as a PhD student I dugg in my heels, and we wrote them a letter, that they can take it as it is, or we will take it somewhere else and bit** about them, generously. Of course in very polite words. They published : – )

Rahul December 29, 2012 at 7:08 am

“my PhD thesis lists 25 publication with me as an author”

I have to say, you are either a genius or I have to be skeptical of the quality of your publications. I doubt the latter since a h-index of 13 is no mean feat. Hence, congratulations.

OTOH, you are by no stretch the average paper author.

genauer December 29, 2012 at 12:35 pm

I am certainly no genius, and since that was before the Great Grade Inflation, I didn’t even get a perfect grade, correctly, ……. in hindsight, LOL.

What I originally wanted, was to provide some contrast to the situation of the commenter above, and to stress more that if you are in a serial (paper) production environment, one paper leaving the house every second working day, profs and post-docs have a pretty good feeling what is accepted in what form at what journal, and the PhD students soak it up pretty automatically. At that time there was no h-index and impact factors, just some general feelings.

paper counts for other PhD thesises I have here are 19, 22, and 9, the later a kind of “get it over with” after the guy worked already for 2 years. So I am on the high side, but definitely within the distribution.

One important thing coveted in my former field semiconductor physics is “to be first”, and not necessarily in the first rate journal. Makes it a little tricky if you dont trust the American editor to be completely unbiased.

The typical number of authors / contributors in material science is higher than in old economics papers. Too many papers with just one or two authors would raise serious questions of social compatibility. But I have seen at blattman, that economics is changing now.

And, some more advice for youngsters: dont get caught with the wrong people, with stuff, you regret 20 years down the road. When I asked my prof why we are not on the author list of a certain paper, despite my samples being the beef of it, he just said “Pons / Fleischmann” and I turned on my heel : – )

Explanation: those were the “cold fusion” guys, and some folks of the neighbor institute had been on a (otherwise respectable) paper with them. But allegedly those 50 year olds broke off lunch in tears, if those names were mentioned

Jack December 29, 2012 at 9:14 pm

@genauer: This is very interesting and you are doubtless a formidable researcher. *But* finance and economics are different. Your numbers would be impossible to achieve in finance or economics (unless one published in lousy journals). Research on publication metrics has shown that the correspondence between physics (and other experimental natural sciences) and finance is 70:1. One finance article in a reputable journal has the same weight as 70 physics articles in physics. People get tenure in finance with two or three articles in good journals. To publish one article per year every year (again, in reputable journals) is to be in the top one percent of the profession. Good journals often request three or more revisions and the process can take two years to complete, sometimes five or six (as a colleague of mine experienced).

So you see why I am flabbergasted at Prof. Boettke’s strategy — unless he has unusual journals in mind.

Rahul December 30, 2012 at 1:47 am

genauer’s statistics are unusual for Physics too. I collaborated with some people from the Physics Dept. of a Top10 US University and the typical number of papers one had for a PhD was around three.

I don’t think it’s fair to use his data to make a Physics versus Finance comparison. His is not a typical case.

I’d love to see your citation for “One finance article in a reputable journal has the same weight as 70 physics articles in physics. ” I’m skeptical of that assertion.

Rahul December 30, 2012 at 1:48 am

genauer’s statistics are unusual for Physics too. I collaborated with some people from the Physics Dept. of a Top10 US University and the typical number of papers one had for a PhD was around three.

I don’t think it’s fair to use his data to make a Physics versus Finance comparison. His is not a typical case.

I’d love to see your citation for the impact of a finance paper being 70x that of a comparable physics paper. I’m skeptical of that assertion.

Jack December 30, 2012 at 8:27 am

@rahul asks for the citation, here it is
http://ideas.repec.org/a/bla/jfinan/v47y1992i1p295-329.html

I do not see why you are skeptical. Physics articles are very short, huge in number, and have loads of coauthors.

Think of research output as a pizza, and the dissemination of research output as how you slice the pizza. Some fields slice the pizza thick (finance or economics) while others slice it thin (physics and other medicl or experimental sciences). This is not to say that one field’s research is “better” in any way. Just that an “article” does not have the same meaning in every field.

genauer December 30, 2012 at 11:07 am

@ John, Rahul, all
I decided, that it will not give away my real name, when I give you the link to my institute.
http://www.wsi.tum.de/Research/Publications/tabid/89/Default.aspx
Please search for the names of the professors, like Abstreiter, and look at the count. Even the theoretician P. Vogl has more than one hundred.

Not everything is a Phys.Rev. Lett. or a Phys.Rev.B, but there are plenty of them. They are most the time the result of months of difficult experimental work, lots of collaboration, and not just on paper. Do you still think I am unusual in this environment?

Of course it is a well run Institute at a well run Technical University of Munich (el presidente Herrmann with more than 1000 publications and an h-index of 75), coming as a close second for German Nobel prizes, but hey, you have to set high goals.

Does the GMU compete for “best of Kansas”?

I said 132 papers / 9.6 PhD grads per year. Take an average of 2 local PhD students on the author list and you arrive at about 2 dozen papers at graduation.

zbicyclist December 28, 2012 at 10:40 pm

Any instance of “C” papers being submitted later to actual journals and accepted? Can you get a retroactive “B”?

Kevin December 29, 2012 at 2:40 pm

This fascinating post deserves a follow up. If the paper in question only counted for 50% of the course grade, what was the final spread? I don’t have any knowledge of the process for writing papers like this. If a class of first-year masters degree students were given the same task, would I be off the mark if I predicted that 1/3 of the class might get a B and 2/3 a C? But that’s only for the paper, so the final grades would be higher. And for this course we’re talking about PhD candidates in their final year, no? Students, presumably, would submit work related to their doctoral theses?

Does anyone want to email Professor Peter J. Boettke and ask?

ChrisD December 29, 2012 at 2:50 pm

This seems like a lot of work for one grade in one class.

Charlie December 29, 2012 at 4:33 pm

The two journals papers are submitted to are The Review of Austrian Economics and The Quarterly Journal of Austrian Economics. That this can be his grading strategy is mostly a commentary on the quality of those journals.

Ansh December 29, 2012 at 6:01 pm

There is no way that any Associate or even a Full Professor can go from paper ideas developed during discussions in a course to a manuscript that would survive a desk rejection in 3 months flat. That is just impossible in Finance. Just fleshing out a good idea takes a lot of feedback, many seminars and brownbag presentations over a many months (more than 3 typically) and then just writing up a professional level document usually takes another 2 to 3 months. And this is the best possible case, assuming that your idea goes somewhere and your tests show something interesting and this doesn’t happen for most papers. I have never seen an idea go to a submit-able document in less than 6 months. Maybe some people can do it but majority don’t.
PS: I’m assuming that the author in question has other academic and real life responsibilities other than this particular course or paper.

genauer December 31, 2012 at 6:36 am

The 2012 version actually sounds like a pretty reasonable stretch goal:

“You are PhD students taking an advanced topics course in a field that
supposedly you have decided to specialize in. If you have a paper that can be
submitted to an SSCI ranked journal and that I deem it to have a shot at
publication, then you will get an A. If you produce a paper that I believe has
potential to develop into a paper that could be submitted, then you will get a B.
Term papers or worse yet random thoughts on paper masquerading as a
coherent paper will receive a C or worse. As a result, pick a topic early and work
hard on it.“

This takes the timing problem of acceptance out of it, and gives Boettke enough wiggle room „I deem it“. In a world, where “Austrians” are under constant assault by foaming before the mouth lefties, e.g. http://delong.typepad.com/sdj/2012/12/at-least-ten-years-too-late-budget-arsonist-veronique-de-rugy-is-sorry.html
you better learn quickly to be tough like nails, if you are in places like Mercatus. I also notice, that Boettke added the necessary “C, or worse”.

The advantage with a paper, is that it is only up to your intelligence, work ethic, and imagination, nothing else.

When I started work as an R&D engineer, from the second year on, I had as one of the goals, defining the variable pay and advancement, 1 full invention disclosure (first step of a patent) per year, all years after that 2, and with 3 inventors it counts only 1/3. But again, something mostly under your own control.

After that you get goals, like getting key prototypes finished on time, shipment qualifications, production targets, profit targets, every time more important than before AND less and less under your control.

At the CEO level, you are responsible for the firm, you may do everything right, good strategy and good execution, but your competition has one more trick up the sleeve, and not only your variable pay, but your job and that of thousands others is gone.

Boettkes way is efficient in separating the wheat from the chaff. It is demanding, but it also fosters a culture of achievement.

@ Tyler Cowen

I think there should be some teaching of German Ordnungspolitik in US Academia Economics, and the Mercatus Center would be a good starting point.

Comments on this entry are closed.

Previous post:

Next post: