Reviewer FAQ

This paper was submitted in the Pre-Results Review track at the Journal of Development Economics. This submissions track involves a two-stage review process where prospective research projects are reviewed and accepted before the results are known. Throughout the publishing process, we refer to these articles as “Registered Reports”, in line with the terminology adopted by other academic journals and the existing Elsevier publishing infrastructure.

At Stage 1, authors submit a Proposal, which typically contains a literature review, research question(s), hypotheses, and detailed methodological framework. Following Stage 1 peer review, high quality studies are accepted based on pre-results review, which constitutes a commitment by the JDE to later publish the full paper regardless of the nature of its results. After Stage 1, authors collect and analyze data, and then submit a full paper (including results and discussion sections) for Stage 2 review. This paper is published as long as the data collection and analysis maintain standards of quality and the study was implemented in alignment with the research design accepted at Stage 1.

Learn more in the JDE Author Guidelines for Pre-Results Review.

Assessment of Stage 1 Proposals should be based on the following criteria:

1. The importance of the research question(s), in terms of its contribution or value added to the development economics literature;
2. The logic, rationale, and plausibility of the proposed hypotheses;
3. The soundness and feasibility of the methodology and statistical analysis plan (including statistical power analysis, where appropriate);
4. Whether the clarity and degree of methodological detail are sufficient to replicate the proposed experimental procedures and statistical analysis plan in line with the JDE Mandatory Replication Policy;
5. Whether the authors have pre-specified sufficient tests for ensuring that the results obtained can test the stated hypotheses, including data quality checks for data accuracy, consistency, bias, and completeness.

Proposals need to satisfy all of the criteria above. Note that Stage 1 Proposals should be evaluated based on the ability of the research design to effectively test the proposed hypotheses, and arguments that the study may end up with a null result will not be taken into consideration.

Based on the referee recommendations, Stage 1 Proposals will be 1) rejected; 2) returned to authors for the opportunity to revise and resubmit; or 3) accepted based on pre-results review. Learn more about Stage 1 of the review process in the JDE Author Guidelines for Pre-Results Review.

Stage 2 of peer review ensures that data collection and analysis maintain high standards of quality and the study was implemented in alignment with the research design accepted at Stage 1.

Assessment of Stage 2 Articles should be based on the following criteria:

• • Whether the study was implemented according to the pre-specified research design accepted at Stage 1. This involves an evaluation of whether the study implementation maintained high standard of quality, particularly whether the attrition rate was high in the given context, or there was a high differential attrition between treatment and control groups;
  • Whether the collected data is of sufficient quality to test the pre-specified hypotheses. In particular, whether the pre-specified data quality checks were satisfied and descriptive statistics provide convincing evidence that the data could credibly measure the impact of the intervention;
  • In cases of deviations from the pre-specified research design, whether the author(s) provided a convincing theoretical and/or methodological justification;
  • Whether any exploratory analyses added by the authors are justified, methodologically sound, and informative. Exploratory analyses must be clearly noted, and the results of all hypotheses specified in the Stage 1 submission must be reported.
  • Whether the author's’ interpretation of the research findings are consistent with the data and the overall evidence.

Referee reports should provide an evaluation of the submission strictly based on the Stage 2 criteria above. Editorial decisions will not be based on the statistical significance, perceived importance, novelty or conclusiveness of the results of the study. This is a key feature of pre-results review. Reviewers at Stage 2 may suggest that authors report additional tests that were not pre-specified; however, authors are not obliged to complete these tests unless such tests are necessary to satisfy one or more of the Stage 2 review criteria.

Learn more about Stage 2 of the review process in the JDE Author Guidelines for Pre-Results Review

Referees can assess the soundness and feasibility of the research design and make relevant suggestions only at Stage 1 of the peer review process. At Stage 2, the focus of the peer review process should switch to assessing the quality of implementation of the pre-specified research design based on the Stage 2 criteria.

Whereas you are free to make comments regarding the pre-specified research design in your Stage 2 referee report, these will be considered irrelevant at Stage 2, and will not influence editorial decisions.

Authors are expected to implement the study based on the research design accepted at Stage 1. However, in the case of a deviation from the pre-specified design, the Editors will strive to err in favor of the authors, as long as the deviation(s) are reported transparently and justified based on methodological and/or theoretical considerations. Referees will evaluate whether the deviations compromise the integrity of the pre-specified research design and the credibility of the findings, and ultimately whether they find the author’s justification acceptable.

Please reference the “Deviations from the Pre-specified Research Design” section in the JDE Author Guidelines for Pre-Results Review for a non-exhaustive list of deviations that may render a study ineligible to be published as part of this process. Referees are encouraged to make recommendations in this regard in their Stage 2 referee reports.

No. The results of all hypotheses pre-specified at Stage 1 must be included in the full manuscript submitted at Stage 2. In instances where a pre-specified hypothesis is subsequently shown to be logically flawed or unfounded, authors may include it in an Appendix (if particularly lengthy) or as a footnote, rather than in the “Results” section.