Author FAQ

Pre-specifying the details of a research design before data collection and analysis is central to the pre-results review (registered reports) format, pre-analysis plans (PAPs), and study (pre-)registration. What distinguishes pre-results review from the other two is that it also incorporates peer-review of the pre-specified research design at Stage 1. Following data collection and analysis, Stage 2 of peer review then ensures that research projects were implemented in alignment with their pre-specified research design, and the results of all pre-specified analyses are clearly reported.

The JDE does not require, but strongly encourages authors to pre-register their Stage 1 Proposals before the start of data collection. Authors can use the information from their Stage 1 Proposal submission to easily pre-register their study.

For more information, please consult the “Pre-registration Resources” section in the JDE Author Guidelines for Pre-Results Review.

Pre-results review does not restrict authors’ ability to conduct post hoc, exploratory analyses and report them in the full-length article submitted at Stage 2. However, such analyses must be clearly labeled as exploratory and distinguished from the confirmatory analyses that were specified prior to data collection and analyses.

Stage 1 Proposals will not be published as stand-alone items in the JDE following acceptance based on pre-results review. Instead, the JDE will post the title and abstract of Stage 1 Proposals accepted based on pre-results review in the Catalog of Prospective Articles.

Note that authors are required to submit the accepted Stage 1 Proposal as supporting material to their Stage 2 submissions. Stage 1 Proposals will be included in the appendix of the published article.

Yes. Authors should communicate such concerns to the Editors, and submit a version of the title and abstract where the necessary details have been masked.

During Stage 1 peer review and before data is collected, reviewers will assess whether authors have pre-specified sufficient data quality checks for accuracy, consistency, bias, and completeness. At Stage 2, reviewers will verify that pre-specified data quality checks have been met, and that data is of sufficient quality to test the pre-specified hypotheses.

In addition to pre-specified data quality checks, as part of the Stage 1 Proposal submission, authors are advised to consider potential challenges that may arise during implementation (e.g., attrition, non-compliance with treatment) and offer strategies to address them.

No, this format is not limited to any particular study design, but may be particularly well suited for RCTs. Any study design may be eligible if the data are yet to be collected/accessed and analyzed at the time of submission. This also includes quasi-experimental and observational studies (e.g., Neumark (2001) and Burlig (2018), and the 2016 Election Research Pre-acceptance Competition).

Submissions where follow-up data have been collected (outside of pilot and baseline data) are not eligible for this track.

Work should be submitted at a minimum of three months, but ideally six months to a year, before data collection of the key outcome data. All participants, authors, referees and editors, must be blind to the key outcome variables at each stage of revision of the Stage 1 submission. This is useful for two reasons: (a) to maximize the possibility that feedback can be used in shaping final data collection, and (b) to ensure enough time for a revise and resubmit round before the final data come in.

(Note: it is typically quite useful to have baseline data in hand when submitting the Stage 1 Proposal for pre-results review: baseline data provides useful information on the nature of measures, including their means and variances, which can help to understand the context and carry out meaningful statistical power calculations.)

No, however authors may include pilot data in their Stage 1 submissions to establish proof of concept, effect size estimations, and/or feasibility of proposed methods.

PAPs and/or pre-registrations can serve as a helpful base for your Stage 1 submission. Stage 1 Proposals go a step further and require that authors also outline the importance of the research question in terms of its contribution or value added to the development economics literature, as in a regular full-length submission (see criterion 1 of the Stage 1 review criteria here). Stage 1 Proposals are expected to mirror full-length conventional articles, with the exception of the “Results” and “Discussion”/”Conclusion” sections. Authors may refer to the optional Stage 1 Submission Template for more guidance.

Stage 1 Proposal submissions should follow the same structure and formatting rules as standard JDE article submissions, however need to provide enough details to allow for the project to be evaluated in the light of Stage 1 assessment criteria (learn more in the JDE Author Guidelines for Pre-Results Review). For items of the Stage 1 Proposal that are not commonly found in JDE papers (e.g. ex-ante plans for data processing, variations from intended sample size, pilot data, etc.), authors can include brief summaries in the body of the paper, and include the rest as an appendix to the Stage 1 Proposal submission.

In studies where parts of an analysis depend on the results of another, authors may consider constructing a decision tree as part of their Stage 1 Proposal (e.g., “If A is observed, then we will adopt analysis A1 but if B is observed then we will adopt analysis B1”). Alternatively, a single primary outcome (or a narrow set of outcomes) can be pre-specified at Stage 1, and the remainder of the analyses may be conducted as exploratory.

Yes, however exceptions may be granted on a case-to-case basis. The JDE will not publish the resulting full-length article unless it has successfully obtained IRB approval.

Publications in this track will look like any other article published in the JDE, with two exceptions:

• • the main text will contain a footnote stating that the paper was accepted based on pre-results review, and
  • the accepted Stage 1 Proposal will be included in the supplementary online appendix.

Here are some additional tips to consider:

• • Authors must report deviations from the pre-specified research design. These can be addressed throughout the body of the Stage 2 full-length article, or in a specific “deviations from the from pre-specified research design” section of the paper, along with details on the context. If particularly lengthy, they may be included in the appendix.
  • Authors are expected to report the results of all hypotheses from the Stage 1 Proposal in the Stage 2 Article, unless a hypothesis is proven to be logically flawed or unfounded. In such cases, it can be included in the appendix with a description and/or justification.
  • Analyses that were not part of the Stage 1 submission should be appropriately labeled (e.g. as exploratory, additional, post-hoc, etc.), and included in a separate section.

Once a Stage 1 Proposal is accepted based on a pre-results review, authors will receive an official notice from the Editors.

Note that due to functionality limitations of our manuscript handling platform, this will be paired with an auto-generated “revise and resubmit” notification, though this is not the actual decision. To submit a full paper for Stage 2 review, authors should  submit it by responding to the “revise and resubmit” notification, and choosing the article type from Registered Report Stage 1: Proposal to Registered Report Stage 2: Full Length Article.

No. The results of all hypotheses pre-specified at Stage 1 must be included in the full manuscript submitted at Stage 2. In instances where a pre-specified hypothesis is subsequently shown to be logically flawed or unfounded, authors may include it in an Appendix (if particularly lengthy) or as a footnote, rather than in the ‘Results’ section.

Yes, as long as such analyses are clearly caveatted (e.g. reported in a separate section) from the analyses based on the Proposal that was accepted by the JDE at Stage 1.

This may be relevant in instances where the author(s) submit a Stage 1 Proposal based on a pre-analysis plan (PAP) for a project in collaboration with an external partner (e.g., government, funding agency) that asks for particular empirical analyses (e.g., estimates over sub-sample A, given primary outcome(s), etc.), however underwent revision during Stage 1 of peer review. If partners require the original PAP as is, then those analyses can be presented in the final paper, but in addition to the analyses accepted by the JDE at Stage 1

Yes.

During Stage 1, submissions will be either 1) rejected; 2) returned to authors for the opportunity to revise and resubmit; or 3) accepted based on pre-results review. Pre-results acceptance constitutes a commitment by the JDE to later publish the resulting full paper regardless of the nature of its empirical findings, subject to a successful Stage 2 review

Information about the duration of peer review for full-length papers in the standard peer review track at the JDE is available here. Given the novelty of the pre-results review track, review times vary from 60 days (accounting for both rejected and accepted papers) to 115 days (accounting for accepted papers, including time to revise and resubmit) as of October, 2019. We therefore ask that authors submit a minimum of three months, but ideally six months to a year, before data collection of the key outcome data.

Once offered acceptance based on pre-results review, authors can include the article as an upcoming publication in their research portfolios with the label: “Accepted at the Journal of Development Economics based on pre-results review”, and the URL of Catalog of Prospective Articles