Another autumn and a whole stack of promotion and tenure files to look at. The phrase “gold standard,” attached to peer-reviewed articles, always strikes me. In the ladder of the evaluation process, the peer-reviewed article stands proudly and invincibly at the top. The more of these you have, the more unassailable your dossier.
Like all things in life, this invincibility turns out to be a double-edged sword. On the one hand, peer-review ensures a process of quality control. Your work is read by at least two experts who pass judgment on its value. They often suggest revisions for improving the manuscript. Their advice, formal and informal, helps the editor make a decision about publication.
There is no doubt that we, as authors, profit from this procedure, even if we get frustrated by the delays. (Often we are at fault, wanting referees to evaluate our manuscript immediately while we dilly-dally with the manuscripts of others.)
I can say that I have been helped enormously by this practice, having revised extensively everything that I have published. In one evaluation of a book-manuscript, I got eleven, single-spaced pages, of both general and specific comments that encouraged me to rethink the project. This is peer-review at its best. (And dear anonymous colleague, in the unlikely event that you are reading this blog, please know that I am eternally grateful for your generosity and selflessness.)
This type of collegiality is often, of course, counter-balanced by unhelpful commentary, sometimes dipped in sarcasm and a few times in venom. We’ve all had to swallow this type of review that either does not understand the project or simply wishes to destroy it by clipping wings, cutting down to size, or letting blood. (You choose the metaphor.)
But this is not what concerns me here. Rather I worry how the review process -- from the one assessing an article for a journal to the more complicated one appraising a candidate for promotion -- can be transformed into a procrustean table of orthodoxy. It forces scholars, particularly younger ones, to conform literally to what sells. That is, to the reigning assumptions about knowledge in that particular field.
If you need six letters for your promotion (and in some “better” institutions ten or twelve) and if these letters have to be uniformly panegyric in tone, then you wonder what type of scholarship will get promoted – careful and methodical, to be sure, but not one taking risks or striving to change perceptions.
I have written many letters for promotion and I have read hundreds of these during my tenure in departmental, college, and university promotion and tenure committees. And I know the weight of one “nuanced,” let alone a negative letter. You cannot get six people, let alone, twelve, to agree about the power of a candidate if that person is doing unorthodox work. At least one reader will conclude that the work in question is substandard or does not fulfill the criteria of excellence, and this letter can often kill the promotion (since we all want to seem tough in these committees and turn some people down).
At best, the peer review evaluates competence and allows committees to assess the place of the candidate in the discipline. At its worst, it invites sycophancy and ostentation. Read enough of these letters in one sitting where everyone is “great,” “luminous,” and “original” and you reach for your calculator to divide the encomia by ten.
To be fair, this is the faculty version of the grade inflation in the university. Taken together they constitute the great academic bubble. But there is no danger of it ever bursting because we’re all happy with it. Everyone involved is brilliant -- from the parents, to the students, to the professors, to the administrators. So let’s all keep praising our dazzling selves.
But it also encourages scholarship that is cautious, careful, and sensible. Let’s look at the PMLA. Would it be rude or unprofessional to ask if you hold this to be a relevant publication for literary studies, this the flagship journal of the Modern Languages Association? Have you read any articles there that are really path-breaking or cutting-edge? What is wrong with the PMLA and why is no one saying anything about it?
The writing seems so careful, so restrained, so risk averse, so reluctant to disagree or to think of the big picture. But how can you expect authors to take risks when they have to satisfy so many people in the editorial process, from the external referees to the editorial board? And why should younger scholars endanger their career, knowing that they need peer-review publications?
There is no doubt that peer review can evaluate scholarship, catch mistakes, point to new directions in the project, and help the authors come to better grips with their argument.
It can also end up reproducing what is already out there.
This is the trade-off we get for wanting to be original.
[EDITOR'S NOTE: The original post was inadvertently uploaded without paragraph breaks. The author's paragraph breaks have been restored in the current version.]
What you're really saying is that we say we want originality but we don't really mean it. The peer review enforces more modest claims, since more original ones outside the consensus in a given field will be seen as more extreme. We know scholarship often moves in baby steps rather than in bold, extreme challenges, but when it comes to tenure reviews we want someone who has made an impact by moving that consensus more decisively in a particular direction. So the problem is not peer review per se; this problem would persist under any regime of evaluation as long as we hold to those conflicting demands of baby-step competence and field-changing originality. The inflated discourse of external evaluations for promotion and tenure doesn't help, since praise of competence comes off as too "nuanced." As a peer reviewer myself, I try not to penalize originality. Usually, it comes down to issues of competence, with many articles not putting forth cogent arguments in the first place. On the other hand, i try not to penalize "mere competence."
The chapter on peer review in Kathleen Fitzpatrick's Planned Obsolescence is a fascinating read (as is her whole book). Here's the link: http://mediacommons.futureofthebook.org/mcpress/plannedobsolescence/one/
Rohan Maitzen
Department of English, Dalhousie University
"Luminous" is a great word. I'll have to use it. When I write letters of rec these days, I include a disclaimer, saying that they're honest. I think in recommendations and tenure you can tell honest support and such support (in my experience) helps a candidate more than empty praise. But it also can provide an excuse to cost-slashing administrators and over-worked editors.
Anyhow, I agree that peer review tends to stifle real innovation and promote group think. The group think may sometimes imagine it's radical, without being so.
But I love the attitude of a tenured anthropologist friend of mine. He genuinely delights in rejection. He says the average academic journal article has just under two readers. (Like my blogging!) So whenever an article if his is rejected, he sends it to another journal and gets at least one more peer reviewer reading it. Don't try this route to readership unless you have tenure, though.
Christmas always feels like a time of (and "for" too, of course, I now realize) HBO's "Six Feet Under." So I thought about it and watched a bit of it today. One of the best dialogues of the show goes:
(*Pardon me for the bracket inside a bracket. I couldn't help laughing a little, remembering how she said the line. Brenda, isn't she great?)
The dialogue seems to pertain to the profession of the humanities as well. But not in a bad way. More in a .... (a long pause later) I thought of Adorno explaining "the Concept of Contradiction" in Lectures on Negative Dialectics, where he, as an illustration of the concept of contradiction, says:
(Hmm. Is it a "worse" way?)
Just as society's survival depends on the means that may very well destroy it, the humanistic profession too, cannot but incorporate practices inimical to it, so as to continue to exist. Practices that make it a "racket," like peer review?
*This is again an odd point to end a comment. But I'll have to! Please take this as one more "worthless" comment written in the spirit of "There Should Be More Comments."
**Then I looked up "racket" in the OED, because I was suddenly unsure whether my understanding of the word's meaning was correct. (I thought it refers to any business where things are done in such a way that vested interests are created and maintained.) From the OED definitions, it seems what's stressed is not vested interests but "dishonesty." So then the humanistic profession cannot be a racket after all?!
I thank everyone for their comments.
I do not know Kathleen Fitzpatrick's Planned Obsolescence but plan to read it.
Janathan is right in that any system of evaluation will eventually have its problems. Yes, there is a conflict between establishing competence and proving originality. But peer review, in its current deified mode, can stifle originality. Do we ever discuss this during tenure review? Is it part of the conversation in editorial boards? We follow the automatic pilot here more often than not.
On the day I posted my blog I read belatedly the October issue of the PMLA. The guest editor, Jonathan Culler, wrote that the editorial board had rejected all but one of the fifty submissions for that issue. Not being a member of the board, I have no idea why this is so. But is it sobering. On the one hand, it suggests high academic standards when you reject most of the submissions. That should make us all feel good. But, on the other, it brings up the problems of the review process. Can cutting-edge work be published in the PMLA when that work has to satisfy the criteria of so many people? Which leads to the other questions I wanted to raise: Why is no one talking about the PMLA?
I don't know if this makes the literary profession a racket. But it certainly does not openly address some of the problems in the profession. Here are some: The state of its flag-ship journal, the consequences of peer review; the snobbery in the profession despite its liberal rhetoric (try getting a job in a top university if you are not a graduate from the top tier); the extraordinary hurdles high school students have to overcome (taking SAT, SAT II, ACT, AP courses and so on) to enter university and the implications this has on the kind of students the university ultimately gets.
As I've mentioned elsewhere on Arcade, I think Planned Obsolescence is required reading for anyone interested in rethinking peer review. Lee's recent post on online peer review is part of that conversation, and certainly open peer review, preprint-style models, and, as Kathleen recently pointed out on her blog (echoing what she'd already written in her book, of course), a renewed relationship between university presses and the universities whose names they bear are all things we should be thinking about.
But you (Gregory) seem to be getting at a few more striking points as well. One of these is the quantification of quality--as you put it, "it suggests high academic standards when you reject most of the submissions. That should make us all feel good." We've all seen internal pushes to market universities more aggressively, not for any specifically academic purpose, but rather to increase the undergraduate applicant pool and thereby lower the acceptance rate (even if the size and character of the admitted class remains steady). The sciences have moved toward an open review culture much faster than have the humanities, albeit in ambivalent ways, and yet t&p continues to rely on a bean-counting "impact factor" based on old-fashioned peer review. The degree to which these professional procedures are in practice serving as ways to not read and think about one another's work is the degree to which they defeat the ostensible purpose of research.
Thus on one hand we've raised the stakes of publication--you have to have more of them to get promoted--and on the other hand we're proving that those publications are excellent by rejecting most of them. Leaving aside the illogic of supposing that a low acceptance rate or a large number of publications can be equated with quality, such a system by definition screws a lot of good scholars, and not even in the interest of promoting knowledge, but rather in the interest of the appearance thereof. Though I'm always skeptical of The Humanities Are Dead pronouncements, I think these are key factors in the phenomenon that you're diagnosing. The humanities aren't dead, but they're changing, and we'd do well to have a hand in how that plays out.
I want PMLA to use its powers for good by disbanding and reconstituting itself as PLotH.* But that's just me.
(*Not to conflate open access with open peer review by any means.)