This chapter establishes policy and procedures for (1) outside scientific review and evaluation of intramural research at NIH by Boards of Scientific Counselors (BSCs) and (2) review of the Scientific Directors' scientific and administrative leadership by ad hoc committees of the ICD Advisory Councils or Boards.

The BSCs are constituted to assist the Scientific Directors (SDs) in evaluating the quality of the intramural research programs for which they are responsible. To assure that the BSCs' evaluations will be most useful to the SDs in their decision making, the BSCs must be composed of individuals who themselves have outstanding scientific credentials and who are committed to providing rigorous, objective reviews. Although the principal purpose of these independent evaluations is to advise the SDs, the reports of the BSCs will be distributed to the Director, National Institutes of Health (NIH), the Deputy Director for Intramural Research (DDIR), the appropriate Institute, Center or Division (ICD) Director, and the Board of SDs. The BSC also reports annually to the National Advisory Council or Board of the ICD.

Intramural research at NIH has been reviewed by committees of scientists from outside the NIH since 1956. In that year the first committees, called BSCs, were appointed to review intramural research in the individual ICDs of the NIH. Board procedures have evolved over the years. On March 4, 1985, the original version of this Manual Chapter was issued to establish policy and procedures for the evaluation of intramural research by the Boards. Subsequently, on November 20, 1985, Public Law 99-158, the Health Research Extension Act of 1985, was enacted, including a provision making mandatory what was then standard policy on review of intramural research. The chapter was revised on August 4, 1986, to make explicit the rules concerning intramural research review, although none of the earlier provisions was changed by Public Law 99-158. On December 1, 1990, the chapter was revised again to incorporate the recommendations of the Consultant Panel to the Advisory Committee to the Director, NIH, as approved by that Committee on January 26, 1990. The present chapter is a revision based on recommendations from the April 11, 1994 Final Draft Report, "The Intramural Research Program," from the External Advisory Committee, a subcommittee of the Advisory Committee to the Director, NIH. Subsequent modifications to those recommendations were made following discussions with the Chairs of the BSCs, with the ICD Directors, and with the Board of SDs.

1. Section 301 of the Public Health Service (PHS) Act authorizes the conduct of research, investigations, experiments, demonstrations and studies relating to diseases and impairments of man.
2. Sections 402(b)(3), 406(a)(3) and 492(b) of the PHS Act.
3. Public Law 92-463, the Federal Advisory Committee Act, as amended, which established the system governing the creation and operation of Federal advisory committees.
4. Memorandum of July 18, 1980, from Dr. Robert Goldberger, Deputy Director for Science, entitled "Modus Operandi of Boards of Scientific Counselors."
5. Report by Dr. John C. Eberhart, dated November 3, 1982, entitled "Review and Evaluation of Intramural Research."
6. The NIH Committee Management Handbook contains instructions and sample notices concerning the meetings of BSCs.
7. Report of the Consultant Panel to the Advisory Committee to the Director, NIH, entitled "Strengthening the Scientific Review Procedures of the NIH Intramural Research Program," December 1989, including the working paper, "Review and Evaluation of Intramural Research-1989," by Dr. John C. Eberhart.
8. Report of the External Advisory Committee, a subcommittee of the Advisory Committee to the Director, NIH, entitled "The Intramural Research Program," April 11, 1994.
9. The Tenure Program of the National Institutes of Health, dated June 17, 1994.

1. Boards of Scientific Counselors (BSCs). Advisory committees, established under provisions of the Federal Advisory Committee Act, as amended (Public Law 92-463), which review, discuss, and evaluate research programs, projects and investigators in the intramural Laboratories and Branches of the NIH, and by FDA intramural programs that are integrated with the NIH intramural program. The reviews include evaluation of the work of individual scientists as well as of the quality of the research itself.
2. Outside review. The review of NIH research and of intramural scientists by highly qualified scientists from outside the NIH who are formally appointed to an advisory committee called a BSC. This process is referred to in the law as "technical and scientific peer review." Ad hoc external reviewers may supplement the review process, and on occasion, an ad hoc reviewer may be added from another NIH ICD.

It is the policy of the National Institutes of Health (and of the Center for Biologics Evaluation and Research, FDA, at NIH) that all research conducted intramurally must be reviewed at regular intervals by highly qualified outside scientists. The review process used by the BSCs will take into consideration the special nature of NIH intramural research made possible by stable funding, that high-risk research should be encouraged, and that the review process will emphasize past performance. The review will address the accomplishments of individual scientists and the quality and productivity of their research. Scientists within Institutes will be reviewed by their established BSCs. Scientists within Centers and Divisions (e.g., National Center for Research Resources, Clinical Center, Division of Computer Research and Technology, and National Center for Biotechnology Information, National Library of Medicine [NLM], etc.) will also be reviewed by a BSC or by a duly constituted Board, approved by the DDIR. Proceedings of the closed portions of the BSC meetings remain confidential to protect the privacy of individuals and of proprietary information.

This policy is reinforced by Section 492(b) of the PHS Act:

These reviews and any resulting recommendations should be included in a written report submitted to the SD, who will transmit it to the ICD Director, the DDIR, and the Director, NIH. As required by the PHS Act Section 492(b) above, each national research institute will provide annually to its National Advisory Council written descriptions of the research reviewed by the BSC, and the results of that review. This provision shall also apply to Centers and Divisions as well. As stated in the PHS Act Section 406(a)(3)(A)(i), the Advisory Council "may on the basis of the materials provided under Section 492(b)(2) respecting research conducted at the institute, make recommendations to the Director of the institute respecting such research."

1. Composition of BSCs A primary consideration of scientific qualification for BSC membership shall be international recognition as an authority in one of the fields of research under review. While the primary criterion for all appointments to the BSCs should be scientific excellence, each BSC should exhibit reasonable balance in membership in terms of points of view (scientific interests/disciplines) and with respect to gender, ethnicity, and geographical distribution of members' institutions.

   BSC members should serve for five-year terms, if possible, to allow them to be involved more than once in the regular quadrennial review of some programs. An effort should be made to have some BSC members (approximately one-third) who are not primarily funded by the ICD on whose BSC they serve.

   Nominations for new members of a BSC should be solicited by the Chair of the BSC from the current BSC members, the SD, the ICD Director, the DDIR, and the Director, NIH. The BSC Chair should then offer a proposed slate of new members to the ICD Director. The ICD Director will discuss the nominations with the DDIR. The proposed slate will be transmitted from the ICD Director to the NIH Director for approval and appointment of members in accordance with established committee management policy and procedure. Following approval by the Director, NIH, formal letters of invitation will be written by the DDIR.

   In general, the chair of a BSC will be selected from past or current BSC membership by the ICD Director after consultation with the out-going BSC chair, the SD, and the DDIR. The Chair shall serve for a two-year renewable term.

   All appointments of members and Chairs are made by the Director, NIH, except that under the PHS Act Section 413(b)(7), the Director, NCI, has authority, in consultation with the National Cancer Advisory Board, to appoint members of NCI BSCs. For NCI, nominations of BSC members and chairs should proceed as for other ICDs except that final appointment authority will reside with the Director, NCI.

   A BSC may make use of ad hoc reviewers when the Chair of the BSC, in consultation with the SD, deems it necessary. Such ad hoc reviewers should be selected by the BSC Chair, with the advice of the other BSC members, the SD, and the ICD Director. At least two appointed BSC members should be present for any site visit of a Laboratory/Branch in those ICDs that use site visits.

2. Frequency of Review Meetings The BSCs shall meet often enough (ordinarily two or three times each year) to assure that the work of each tenured and tenure-track intramural scientist and each Laboratory or Branch is reviewed at least once every four years.

3. Who is to be Reviewed

   1. Intramural Scientists Every independent intramural scientist on a tenured appointment must be reviewed and evaluated at least every four years. Advice of the BSC also must be obtained concerning tenure-track scientists at the mid-point of their tenure-track and prior to their being considered for conversion to tenure. The SD may, in addition, choose to ask the BSC to review the work of permanent staff scientists or other intramural scientists.

   2. Scientific Directors The BSCs are not expected to evaluate the scientific and administrative leadership of the SDs, since such an oversight role might interfere with the BSCs' function in assisting the SDs. [NOTE: If the SD, however, is directly responsible for his or her own research laboratory, the BSC shall review the work and personnel of that entity.] Therefore, such review of an SD should be conducted by an ad hoc committee, composed of at least four members, every four to six years, or as the need arises. This committee shall be established by and report to the ICD's Advisory Council or Board, which will, based on the report of the ad hoc committee, make recommendations to the ICD Director. One member may be a present or former BSC member while the other three or more may be drawn from the ICD Advisory Council or Board, former ICD intramural scientists, and other senior scientists and scientific administrators. [NOTE: Alternatively, this review may be conducted by an ICD Special Emphasis Panel, where one exists, that is directly advisory to the ICD Director.]

      The review process should include interviews with the SD, Laboratory/Branch Chiefs, ICD scientists at all levels, the ICD Director, and the DDIR. Letters of reference may be solicited as appropriate. The SD should submit a statement describing short and long-term goals and program achievements since the previous review or since assuming the position of SD. The committee report should comment on the overall performance of the SD as a shepherd of the Intramural Research Program of the ICD, including, but not limited to, the scientific vision and administrative skills of the SD, the quality of training and mentoring within the IRP, success in assuring diversity and fair treatment of scientists within the IRP, the administrative style of the SD, relationships with the Lab/Branch Chiefs and the offices of the ICD Director and the DDIR, and recommendations.

      This report should be submitted via the ICD Director to the ICD Advisory Council or Board that will make recommendations to the ICD Director and the DDIR. A response from the ICD Director should be made to the ICD Advisory Council or Board and the DDIR within six months.

4. Information Supplied to BSCs Prior to Meeting The following information shall be provided to each BSC reviewer in advance of the meeting.

   1. General Information

      Orientation guidelines that describe the review procedures, the goals of the process, conflict of interest policies and procedures, and the responsibilities of the Counselors, issued by the DDIR.

   2. Specific Information

      (1) For each Laboratory/ Branch being reviewed: A written description that presents the overall past accomplishments of the Laboratory/Branch and its investigators since the last review was conducted. Specifically, Board reviewers shall receive:

      (a) A summary of the organization of the laboratory to be reviewed, including the various sections.
      (b) All personnel, to include occupational specialty, type of appointment, grade, and salary.
      (c) Space usage.
      (d) Operating budget.
      (e) Outside contracts.
      (f) Cooperative Research and Development Agreements (CRADAs).
      (g) The previous BSC review and response of the SD.

      (2) For each scientist being reviewed:

      (a) A current CV and bibliography (either complete CV or 2 page abbreviated form), and copies of three most important recent manuscripts or publications.
      (b) Each project should be described in a progress report of ongoing research. The report should not be lengthy, but should be long enough to summarize the background concepts and contributions of the laboratory, and should also include some description of future plans, with emphasis on ideas and questions to be answered rather than research methods. For each project, a progress report of 3-5 single spaced pages (2,500 words) and a description of future plans of 1-2 single-spaced pages (1,000 words) should be sufficient in most, but not all, cases. If an investigator supervised multiple independent projects, then a separate write-up should be provided for each.
      (c) An analysis of the amount of support staff and space that the scientist uses, by project, in addition to information about budget, contracts, and CRADAs.

5. Review Format This policy addresses various components of the scientific review procedures to be used by the BSCs. Consistent with these stipulations, each ICD shall develop its own procedures for the organization and structure of BSC meetings. The BSC meetings shall be conducted under provisions of the Federal Advisory Committee Act. The DDIR shall also meet annually with the BSC chairs to discuss and evaluate the ICDs' procedures to ensure that the NIH standards are met. The experiences of the different processes shall be communicated by the DDIR to all of the SDs.

   Oral presentations at BSC meetings should be of sufficient length to allow for a formal presentation and a question/answer period. The duration of these presentations will vary with the complexity of the research program and seniority of the scientist.

   Some effort should be made to visit laboratories in which scientists work, to get a sense of the research environment and interact with junior level and non-scientific personnel. The review format may include 1-on-1 meetings with scientific staff, poster sessions for post-doctoral fellows, and meetings with the Laboratory/Branch Chiefs at the end of the scientific presentations, without the SD present. The latter is a way to learn about concerns and constraints of the Laboratories/Branches and to explore issues that may concern the BSC, directly with the scientist(s) under review, prior to preparing the written report.

   The BSCs shall provide evaluation and advice on scientific directions of the laboratory, tenure actions under consideration, resource allocation, specific projects including new areas of development, and other administrative matters.

6. Reporting of Results of Reviews

   1. Before the Board adjourns, an oral summary of the review and recommendations should be presented to the SD, the ICD Director, and the DDIR, or their designees. In addition, the BSC is encouraged to meet with the Laboratory/Branch Chief prior to adjournment.

   2. The written report of the BSC or site visit team is to be a narrative critique submitted by the Chair of the BSC, following the format preferred by the SD.

   3. Evaluations of individual investigators must address, at a minimum, the following questions: Are good questions being asked, are appropriate approaches being used to obtain answers, and are the resources available to this scientist appropriate to the accomplishments to date? Should projects be continued or modified? Recommendations regarding resources should be as explicit as possible, with a clear indication as to which resources (budget, space, personnel) should remain the same, be increased, or decreased. Recommendations to increase the resources of an investigator or project should recognize the limitations on total resources available to the Intramural Research Programs. These evaluations should be written by members of the BSC or consultants, and should reflect the collective views of the total BSC (or at least of a majority).

   4. The written report is to be sent within two months of the review to all BSC members, the SD, and the ICD Director. Those portions of the report that pertain to particular Laboratory/Branch Chiefs should be provided to them by the SD. The SD and the Laboratory/Branch Chief should discuss the evaluations and recommendations in the report. Those sections appropriate to a particular scientist will be forwarded to that individual for comment.

7. Follow-up

   1. At the next meeting of the BSC, the SD should respond to the Board indicating areas of agreement and disagreement and an outline of actions already undertaken or planned for the future.

   2. Not later than six months after the report is received, a written response from the SD via the ICD Director should be provided to the BSC, the DDIR, and the NIH Director, along with a copy of the report.

   3. The BSC should also report annually to the National Advisory Council of the ICD, either by endorsing a written report of the SD, by providing the BSC report and SD response, or by providing an independent report to be presented to the Council

8. Schedule of Reviews Each SD must submit to the DDIR a schedule of the proposed dates of review of each Laboratory/Branch in the Institute, Center or Division covering the next four-year review cycle. This schedule must be updated annually and submitted to the DDIR by September 1 of each year.

9. Survey of BSC Members Completing Terms Upon completion of their terms, members of the BSC shall be surveyed by the Chair on issues of program quality and the scientific review process.