NSF Seeks Input on Proposed Revisions to Review Criteria

Rather than “enhancing” peer review like big brother NIH, the NSF has been working to clarify the principles and criteria by which the intellectual merit and (especially) broader impacts of proposed research are assessed. The problem is mainly with the broader impacts, which seem to keep growing broader and broader.

As currently laid out in the Grant Proposal Guide, reviewers are asked to consider:

What is the intellectual merit of the proposed activity?
How important is the proposed activity to advancing knowledge and understanding within its own field or across different fields? How well qualified is the proposer (individual or team) to conduct the project? (If appropriate, the reviewer will comment on the quality of prior work.) To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts? How well conceived and organized is the proposed activity? Is there sufficient access to resources?

What are the broader impacts of the proposed activity?How well does the activity advance discovery and understanding while promoting teaching, training, and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships? Will the results be disseminated broadly to enhance scientific and technological understanding? What may be the benefits of the proposed activity to society?

Now the NSF seeks input on the revised Merit Review principles and criteria:

Merit Review Principles and Criteria
The identification and description of the merit review criteria are firmly grounded in the following principles:

    1. All NSF projects should be of the highest intellectual merit with the potential to advance the frontiers of knowledge.
    2. Collectively, NSF projects should help to advance a broad set of important national goals, including:
    ◦ Increased economic competitiveness of the United States.
    ◦ Development of a globally competitive STEM workforce.
    ◦ Increased participation of women, persons with disabilities, and underrepresented minorities in STEM.
    ◦ Increased partnerships between academia and industry.
    ◦ Improved pre-K–12 STEM education and teacher development.
    ◦ Improved undergraduate STEM education.
    ◦ Increased public scientific literacy and public engagement with science and technology.
    ◦ Increased national security.
    ◦ Enhanced infrastructure for research and education, including facilities, instrumentation, networks and partnerships.
    3. Broader impacts may be achieved through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by the project but ancillary to the research. All are valuable approaches for advancing important national goals.
    4. Ongoing application of these criteria should be subject to appropriate assessment developed using reasonable metrics over a period of time.

Intellectual merit of the proposed activity
The goal of this review criterion is to assess the degree to which the proposed activities will advance the frontiers of knowledge. Elements to consider in the review are:

    1. What role does the proposed activity play in advancing knowledge and understanding within its own field or across different fields?
    2. To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
    3. How well conceived and organized is the proposed activity?
    4. How well qualified is the individual or team to conduct the proposed research?
    5. Is there sufficient access to resources?

Broader impacts of the proposed activity
The purpose of this review criterion is to ensure the consideration of how the proposed project advances a national goal(s). Elements to consider in the review are:

    1. Which national goal (or goals) is (or are) addressed in this proposal? Has the PI presented a compelling description of how the project or the PI will advance that goal(s)?
    2. Is there a well-reasoned plan for the proposed activities, including, if appropriate, department-level or institutional engagement?
    3. Is the rationale for choosing the approach well-justified? Have any innovations been incorporated?
    4. How well qualified is the individual, team, or institution to carry out the proposed broader impacts activities?
    5. Are there adequate resources available to the PI or institution to carry out the proposed activities?

You have until July 14th to e-mail your comments to meritreview@nsf.gov. You can also review the history of modifications in merit review over the years.

2 Comments »

  1. writedit said

    Thoughtful response in a letter to Science:

    NSF’s Struggle to Articulate RelevancePublic science today finds itself caught between competing demands: Researchers need autonomy to pursue questions wherever they lead, whereas funders demand that research meet societal needs. The National Science Foundation (NSF) offers a case study of the balancing of scientific autonomy and societal accountability. NSF is charged with funding basic (i.e., nonmission) research. Yet Congress funds basic science in the hope that societal benefits will result. In 1997, NSF added a “broader impacts” review criterion to address concerns about relevance: Justify research in terms of societal outcomes.

    Over the past decade our nation’s concern with accountability has increased, and in response, the NSF recently issued new draft criteria for the review of submitted proposals (1). The new plan will require researchers to identify the broader good of their research by selecting from a list of national priorities. No doubt, scientists who complained about the vagueness of the “benefits to society” clause in the former criterion will welcome the proposed changes as providing much-needed clarity and direction to the idea of “broader impacts.” But specifying impacts raises three potential problems.

    First, the list focuses on economics and national security, but excludes protecting the environment and addressing other social problems. Aside from the consequences of neglecting these areas, this new focus may undermine the attractiveness of STEM disciplines to more idealistic students who are interested in meeting human needs rather than fostering economic competitiveness. Second, under the proposed new criteria, applicants and reviewers are restricted to the provided list of national needs, which will complicate efforts to respond to new challenges as they develop. Third, addressing these national needs is now supposed to happen “collectively.” This reopens the question of whether each individual proposal must address broader impacts. The new criterion thus replaces vagueness regarding what counts as a broader impact with vagueness regarding who is responsible for addressing broader impacts.

    The new criteria are not without merit. For example, the guidelines on how to implement proposed broader impacts are improved. Once one identifies the national goal to be pursued, the new broader impacts criterion focuses on logistical questions that should be asked in peer review.

    The proposed changes in the merit review criteria move too far in the direction of accountability, at the cost of scientific creativity and autonomy. The set of principles (in terms of national goals) also suffers from excessive detail at the cost of flexibility. Of course, revising the criteria is a perennial process of renegotiation as cultural values change, and these are only draft criteria and principles. NSF invites comments until 14 July. Both the scientific community and the science policy community need to make their voices heard.

    Robert Frodeman*, J. Britt Holbrook

  2. […] summer, NSF sought input on merit review criteria for intellectual merit and broader impacts. Nature News now reports that […]

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: