Lecture 7.1: Reviewing Research

AREC 570 / ECON 530 · Unit 7

Lauren Chenarides

Before class

Readings and course handouts are available on Canvas.

The reviewer’s job

Help the editor assess the paper and help the author understand its strengths and weaknesses.

  • Identify the contribution.
  • Assess the evidence and reasoning.
  • Distinguish essential revisions from optional improvements.
  • Explain the recommendation.

Where the report fits

A typical sequence:

  1. Submission and editorial screening.
  2. Invitation to reviewers.
  3. Reports and editorial decision.
  4. Revision, rejection, or acceptance.

Procedures vary by journal. Follow the instructions for the specific review.

Evaluate the paper in front of you

  • Engage with its question and intended contribution.
  • Do not require the author to write your preferred different paper.
  • Explain problems rather than merely declaring them.
  • Be direct and constructive.

Source: Berk et al. (2015).

AJAE: several dimensions of merit

The assigned criteria address:

  • Contribution and originality.
  • Scholarship and analysis.
  • Appeal to the readership.
  • Appropriate illustrations and length.

A report should explain how the paper meets or falls short of relevant criteria.

Source: Agricultural and Applied Economics Association (2013).

Ethridge: follow the argument

  • Is the objective clear?
  • Does the sequence of ideas lead toward it?
  • Does the organization help the reader?
  • Are important terms precise and consistent?
  • Which general problem matters more than sentence-level edits?

Source: Ethridge (2004).

Discussion: identify the binding weakness

A paper has a clear question, polished prose, and precise estimates, but its comparison group is exposed to the treatment.

  • Which review criterion is affected?
  • Why does the concern matter?
  • What evidence or revision could address it?
  • When would this be fatal to the paper’s main claim?

Hypothetical review exercise.

Structure a referee report

  1. Brief summary of question, method, and main contribution.
  2. Overall assessment.
  3. Major comments, ordered by importance.
  4. Specific comments with locations.
  5. Recommendation communicated as the journal requests.

A summary demonstrates understanding; it should not dominate the report.

Make a major comment actionable

A strong comment contains:

  • Concern: what is wrong or unclear?
  • Consequence: how does it affect the conclusion?
  • Remedy: what analysis or explanation could resolve it?
  • Priority: is it essential?

Avoid an unexplained list of preferred robustness checks.

Exercise: revise a weak comment

The empirical strategy is unconvincing. Add more controls.

Rewrite it to identify:

  • The potential confounder.
  • Why it threatens the interpretation.
  • A feasible diagnostic or revision.
  • What remains uncertain even after that revision.

Discussion: Berk, Harvey, and Hirshleifer

  • How can a reviewer distinguish an essential change from an optional suggestion?
  • What makes a recommendation useful to an editor?
  • Why can unnecessary revision requests slow scientific progress?
  • How should a reviewer handle limits to their own expertise?

Research proposal review: the EPA example

The assigned historical criteria include:

  • Research merit.
  • Responsiveness to the funding call.
  • Project management.
  • Other factors such as innovation and engagement.

A promising idea must also be feasible and appropriate for the call.

Source: U.S. Environmental Protection Agency (2016), the assigned 2016 document.

Discussion: journal versus proposal review

Compare the AJAE and EPA criteria.

  • Which standards overlap?
  • Which apply mainly to work not yet completed?
  • How should uncertainty about feasibility affect a proposal review?
  • How would you revise your proposal after this comparison?

Workshop: a short referee report

Review a partner’s proposal or a selected course paper.

Write:

  1. A two-sentence summary.
  2. Its strongest contribution.
  3. One major concern with a feasible remedy.
  4. One specific clarification.
  5. A reasoned overall assessment.

Before you leave

Check that your most important criticism explains both why it matters and what the author could do about it.

References

Agricultural and Applied Economics Association. 2013. American Journal of Agricultural Economics: Refereeing Criteria for Judging the Merits of Journal Submissions. Reviewer guidelines.
Berk, Jonathan B., Campbell R. Harvey, and David Hirshleifer. 2015. Preparing a Referee Report: Guidelines and Perspectives. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2547191.
Ethridge, Don E. 2004. “Guidelines for Critiquing Papers.” Appendix D in Research Methodology in Applied Economics: Organizing, Planning, and Conducting Economic Research, 2nd ed. Blackwell Publishing.
U.S. Environmental Protection Agency. 2016. Integrating Human Health and Well-Being with Ecosystem Services: Peer Review Process, Evaluation Criteria and Rating Classifications. Science to Achieve Results Program, EPA-G2016-STAR-A1/A2. https://www.epa.gov/research-grants/integrating-human-health-and-well-being-ecosystem-services-request-applications-rfa.

Course navigation