Mark Tebeau’s thoughtful post about open peer review addresses some of the terra paene incognita ahead for the Journal of the Digital Humanities in terms of open peer review. I say paene [=mostly] because several prominent projects (Kathleen Fitzpatrick’s Planned Obsolescence, Jack Dougherty and Kristen Nawrotzki’s Writing History in the Digital Age, the New Media issue of Shakespeare Quarterly to name a few) have blazed some trails already. But these vanguard efforts were not expected to sustain indefinitely scholarly communication for a journal. To be sure, much exploring remains to be done.

Mark’s proposal highlighted for me two particular issues that warrant further discussion (as he intended), namely incentivizing participation in open peer review and the obligation of the review platform to facilitate that. Even though i disagree with Mark on a number of points, what follows is not intended as a criticism of the post itself—a deliberately rapid but insightful reaction to a Twitter flurry that followed the announcement about JDH itself—but a short stroll down the important path that Mark has thankfully pointed us down. In particular, i want to explore not only motivational forces, but also the nature and status of the scholarly work that open reviews can produce.

do we need to motivate peer review?

Rightly concerned with a principle obstacle to successful open peer review, Mark addresses the problem of providing motivation for diverse and qualified reviewers to do difficult and time consuming work (at least to do it well) who might not really get credit for doing it. With only a few hours per week for such work, how can such invisible labor be justified against the incessant ticking of a tenure or review clock? Or even other pressing deadlines? Even if we know that the work as useful and stimulating, why should it be a priority when it’s not properly recognized?

Mark offers an elegant plea that it is at least partially incumbent on the journal (or whatever) to incentivize and diversify the peer review process. One of his suggestions: badges. For roughly a zillion reasons that i won’t go into here, i remain highly skeptical of the utility and viability of badges. He might be right: something like badges could conceivably provide a tangible way to assess the productive contributions that an individual has made in evaluating and critiquing the work of colleagues. But Marks’s reason for writing, as I take it, isn’t to demand badges per se, but to highlight the need for a motivating force.

We should keep in mind, however, that contributions to an open peer review (even if not easily measured or validated) are already motivated by the grubby, practical realities of professionalization (thankfully made easier by a generally welcoming DH community). Participating publicly in the conversation, the critique, is a crucial form of community engagement (if not a kind of necessary performance) that is far from invisible. As Mark points out, knowledge creation doesn’t happen in solitary archives, but within broader communities. The open forum—and its space for insightful comments and questions—presents an almost unprecedented opportunity for preening. Akin to the really great question from an audience member that swivels heads after a conference paper (and perhaps is remembered even more than the paper itself), it affords one the chance to strut one’s stuff, so to speak. It helps establish one’s own value to a community. This very real value provides an indispensable foundation for many recommendations (either via formal letters or informal conversations), whether for a new job, tenure, or promotion. Furthermore, such public work can lead to more tangible opportunities as well, like collaborations on high-profile projects, solicitations for contributions to journals or edited volumes, etc—all of the kinds of measurable productivity that even most traditional evaluators want to see on a CV.

All this is to say that at least one major incentive (besides sheer altruism) that helps to cultivate a diverse pool of reviewers and fuel the review process itself already exists. This might be pointing out the obvious, but I would submit that this and other incentives (and their implications) are much more complex than we’ve appreciated so far. In general, we have not typically discussed how peer review participation fits into the larger structures of professionalization because, simply, it hasn’t. Peer review has remained a rather exclusionary tribal ritual where the elite exert a uni-directional downward force on the semi-initiated masses. Yet even as this begins to change in the wake of some early open peer review projects, there has not been much discussion (as Mark points out) about how to situate the products of peer review in a larger professional scholarly context. Of course it all starts with people making comments—so exactly what role should the review platform play in such considerations?

the role of the reviewer

A mechanism like badges—or any similar mechanism—places a significant onus on the review platform to facilitate—if not manage outright—some kind of participation metrics or credentialing process. Partially because of existing motive forces as explained above, I’m not yet convinced this is fully necessary. But it raises what seems to be a large and pressing question: What are the functional obligations of the journal or press that conducts open peer review?

DH projects have well demonstrated that “if you build it, they often don’t come.” Simply enabling and promoting open peer review is no recipe for either getting good comments or for changing the way such knowledge-work and community engagement can be appreciated. There can be little doubt that success in shifting (broader humanities) community values w/r/t open peer review will depend intimately on the aesthetics and interface design and functionality that creates a new kind of scholarly experience (more on why theory helps with this below). But thinking about particular functional or usability requirements that might help incentivize participation at the platform level cannot be our only (or perhaps even first) concern. In order to understand the functional obligations of the publisher and review platform, we need to understand the larger social obligations that such review work might create. As a start, i’d like to affirm the importance of the agency of the reviewer in promoting and establishing the value of review contributions. Up until now, this agency has been approximately zero. But that’s not the case anymore.

To complement Mark’s proposal, i would suggest that considerations of motivations and mechanisms for open peer review (and obligations of the platform) must go well beyond the review platform itself. For example, reviewers themselves might experiment with new approaches to presenting (dare i say marketing?) their review work. Of course it’s not the case that reviewers have simply been lazy about documenting their effort. Review work has remained invisible not just because of individual scholars, but because of the generally cloaked process of peer review, which has remained a highly mediated and private exchange. Open peer review, however, largely alliviates the expectation of future secrecy.

Reviewers who have done work they are proud of (even if unrecognized or unacknowledged) might, for example, cultivate something like a digital portfolio of their review contributions, each in a short (and i mean it!) essay form that describes the broader value of each review. Of course there is the issue of external validation of quality, a concern that something like badges might address in the way they could serve as a community nod to good work. But would such a mechanism be too abstract for its own good? Perhaps editors or authors whose work has been substantially improved by a reviewer could attest to such efficacy with a brief comment in the reviewer’s portfolio. Especially if from a major press or established scholar, could this be viable grist for the promotion committee’s mill? There are some reasons not to do this, not least of which is the extra work it would entail, but bear with me for a moment longer.

I wonder, too, if expectations that we might showcase our own value as a reviewer, and possibly solicit appraisals from relevant parties, wouldn’t help make our critical engagement with each other’s work—even while in progress (it’s like 95% done, right?)—much more permanent and useful. Perhaps we might think in terms of a new social contract of scholarly publishing that presents to readers not just a polished and vetted product (whether a book or blog post), but a glimpse into the forces that have shaped that product. Such machinations won’t always be interesting, but that’s where the magic of jQuery comes in.

Beyond visibility and transparency (to say nothing of the pedagogical potential here) the expectation that we should repackage our review contributions to get more credit for them might encourage reviewers to focus on broadly useful comments, whether concerning historiography, theory, practice, structure, or style. Not all comments should address only the big picture, obviously, but i would argue that most helpful, substantive comments (and ones we hope to get credit for) are more along those lines than of the line-item correction variety (and no, this does not disparage the difficult and under-appreciated work of quality copyediting). But has any scholar not received sadly myopic comments that threw the whole process of peer review into question? Admittedly, any sort of repackaging would be much easier when properly enabled by the review platform. And this emphasizes Mark’s larger aim in writing (in my view): the extent to which the review platform should provide such services (or badges) requires much more discussion.

At any rate, the broader point here is that open peer review affords us new ways of capturing and leveraging our critical engagement with each other’s work. As a result, we need to think more carefully about the lifecycle and status of our review work, even beyond how we might credential it at the site of production. At all levels of rethinking how to motivate and recognize reviewing efforts, we should be limited by creativity, not convention. We cannot simply appeal to existing procedures and venues; we might well need new ones.

where does it fit?

My remarks thus far have obviously privileged the spirit of discussion over practical solutions. As Mark immediately recognized, it’s a conversation well worth having. It will help establish conventions for not only how we can provide more effective critiques, but also how we can incorporate that work into a larger scholarly discourse. As past open review projects have demonstrated, some very insightful comments and discussions take place in the margins—sometimes just as valuable as the original piece.

But then shouldn’t we just do it? Do we really need to talk about doing it?

We might look to Salerno, Italy in second half of the twelfth century for an answer. (perhaps not the most obvious place, i know…) As theory-heavy Greek and Arabic medical (and other philosophical) works were translated into Latin, physicians struggled to assimilate and situate medical knowledge in the larger hierarchy of knowledge and to craft a curriculum for physicians studying at the burgeoning universities. Their main challenge was to address the theory/practice divide. Medicine was seen fundamentally as an art or craft and therefore had little place among the true sciences, like ethics, logic, and metaphysics. Bartholomaeus of Salerno, a physician and teacher who became one of the most influential leaders of the charge to legitimate medicine as a science (and thus worthy of a place in a university curriculum), argued successfully (not for the first time, but perhaps most influentially) that practice cannot exist without theory. This may sound absurd (as in: i do stuff all the time without having theorized about it beforehand!), but of course he was really saying that all practice is informed by theory whether we know how it is or not.

In terms of open peer review, we need to get clear about what we want (both socially and technically) out of new review practices, as well as how they should serve not only a publication and its readership, but also the process of knowledge production. I say “should” because such reflection might involve reshaping practices as much as appealing to them, as was the case at Salerno. And as Salernitan physicians (and their followers) did, we also must consider—and argue for—where review work should stand in the larger ecosystem (if not hierarchy) of scholarly discourse and knowledge. Is open peer review an opportunity to highlight some very important intellectual work that we’ve kept under the table? Is it best left by the wayside? Should our professional websites aggregate and highlight our public review work that can be supported/verified by the community? Would this help or hinder either the quality or quantity of reviews?

Hashing out our underlying needs and wants and the causal forces that will drive them (ie theorizing about best practices) will help ensure that we don’t see our open peer review efforts as deterministically and self-evidently improving peer review by dint of their mere existence. Instead we’ll be able to deliberately pursue and defend targeted strategies to make the entire process and its products more useful, sustainable, and transformative both within and without the digital humanities community.

subscribe

get updates

get tweets

about me

Assistant Professor of History
George Mason University

Director of Digital Scholarship
Center for History and New Media

more »

Categories

• administrata
• AHA
• collaboration
• digital humanities
• early medicine
• history of medicine
• history theory
• maps
• new media
• professionalization
• publishing
• reviews
• STS
• teaching
• technology
• text editing