![[NASA Logo]](../Images/nasaball.gif) |
NASA Procedures and Guidelines |
||||
This Document is Obsolete and Is No Longer Used.
|
|||||
|
|
|||||
|
|
||||
| | TOC | Preface | Chapter1 | Chapter2 | Chapter3 | Chapter4 | Chapter5 | Chapter6 | AppendixA | AppendixB | AppendixC | AppendixD | AppendixE | AppendixF | AppendixG | ALL | | |||||
This appendix describes the recommended best practices for technical reviews conducted as a part of information technology projects. During creation of the project plan, the project manager works with the program manager to appropriately tailor the project-level review entrance criteria and success criteria. The final set of required entrance and success criteria should reflect the nature of the system under development; be appropriate for the project's size, risk, and importance; and be achievable with approved project resources and on an acceptable schedule. The review plan section of the project plan documents the tailoring of the reviews.
The SCR evaluates the scope, cost benefit analysis, and a recommended solution/concept for the product or service to be delivered for the purpose of receiving approval, formalized via the Formulation Authorization Document, to proceed to the Formulation Phase. Assesses the effect on the "as-is" and "to-be" enterprise architecture. Ensures applicable security controls have been considered.
Table G-1 - SCR Entrance and Success Criteria
| System Concept Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. System goals and objectives. 2. Analysis of alternative concepts to show at least one is feasible. 3. Concept of operations. 4. Preliminary risk assessment, including technologies and associated risk management/mitigation strategies and options. | 1.System objectives are clearly defined and stated and are unambiguous and internally consistent. 2. The preliminary set of requirements satisfactorily provides a system which will meet the system objectives. 3. The system is feasible. A solution has been identified which is technically feasible. A rough cost estimate is within an acceptable cost range. 4. The concept evaluation criteria to be used in candidate systems evaluation have been identified and prioritized. 5. The need for the system has been clearly identified. 6. The cost and schedule estimates are credible. 7. A technical search was done to identify existing assets or products that could satisfy the mission or parts of the system. 8. Technical planning is sufficient to proceed to the next phase. 9. Risk and mitigation strategies have been identified and are acceptable. |
The SRR examines the functional, technical, performance, and security requirements for the system and the preliminary project plan and ensures that the requirements and the selected concept will satisfy the system objectives.
Table G-2 - SRR Entrance and Success Criteria
| System Requirements Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. A preliminary SRR agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair prior to the SRR. 2. The following technical products for hardware and software system elements are available to the cognizant participants prior to the review: a. System requirements document. b. System software functionality description. c. Concept of operations. d. Preliminary system requirements allocation to the next lower-level system. e. Updated cost estimate. f. Risk assessment and mitigations. g. Configuration management plan. h. Initial document tree. i. Verification and validation approach. j. Information/system security categorization. k. Identification of personally identifiable information. l. Identification of records retention requirements. m. Identification of required system security controls. n. Preliminary software development/management plan. o. Other specialty disciplines, as required. | 1. The project utilizes a sound process for the allocation and control of requirements throughout all levels, and a plan has been defined to complete the definition activity within schedule and cost constraints. 2. Top-level requirements definition is complete, and interfaces with external entities and between major internal elements have been defined. 3. Requirements allocation and flow down of key driving requirements have been defined down to subsystems. 4. Preliminary approaches have been determined for how requirements will be verified and validated down to the subsystem level. 5. Major risks have been identified, and viable mitigation strategies have been defined. 6. IT security, privacy, and records retention requirements are complete and have been incorporated into project requirements documentation. 7. The preliminary software development/management plan meets the requirements of NPR 7150.2. |
The PDR demonstrates that the preliminary design meets all system requirements with acceptable risk and within the cost and schedule constraints and establishes the basis for proceeding with detailed design. It will show that the correct design options have been selected, interfaces have been identified, and verification methods have been described.
Table G-3 - PDR Entrance and Success Criteria
| Preliminary Design Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. Successful completion of the SRR and responses have been made to all SRR Requests for Action (RFAs), or a timely closure plan exists for those remaining open. 2. A preliminary PDR agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair prior to the PDR. 3. PDR technical products listed below for both hardware and software system elements have been made available to the cognizant participants prior to the review: a. Updated baselined documentation, as required. b. Preliminary subsystem design specifications for each configuration item (hardware and software) with supporting tradeoff analyses and data, as required. The preliminary software design specification needs to include a completed definition of the software architecture and a preliminary database design description as applicable. c. Updated risk assessment and mitigation. d. Updated cost and schedule data. e. Updated logistics documentation, as required. f. Applicable technical plans (e.g., technical performance measurement plan, reliability program plan, quality assurance plan). g. Operational concept. h. Applicable standards. i. Engineering drawing tree. j. Interface control documents. k. Verification/validation plan. l. Plans to respond to regulatory requirements (e.g., Section 508), as required. m. Requirements traceability matrix. n. Disposal plan. o. Technical resource utilization estimates and margins. | 1. Agreement exists for the top-level requirements, including success criteria and any sponsor-imposed constraints, and ensures that these are finalized, stated clearly, and are consistent with the prelim. design. 2. The flow down of verifiable requirements is complete and proper or, if not, an adequate plan exists for timely resolution of open items. Requirements are traceable to system goals and objectives. 3. The preliminary design is expected to meet the requirements at an acceptable level of risk. 4. Definition of the technical interfaces is consistent with the overall technical maturity and provides an acceptable level of risk. 5. Adequate technical margins exist with respect to performance requirements. 6. Any required new technology has been developed to an adequate state of readiness, or back-up options exist and are supported to make them a viable alternative. 7. The project risks are understood, and plans and a process and resources exist to effectively manage them. 8. The operational concept is technically sound. It includes (where appropriate) human factors that apply, and requirements for its execution flow down. |
The CDR confirms that the maturity of the design is appropriate to support proceeding and that it was developed in conjunction with stakeholders, demonstrates that the design meets detailed requirements, and identifies open design issues for the purpose of obtaining a decision to proceed with development and deployment. It reviews the technical architecture to ascertain the effect on the enterprise architecture and reviews the application security design and the inclusion of security controls.
Table G-4 - CDR Entrance and Success Criteria
| Critical Design Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. Successful completion of the PDR and responses has been made to all PDR RFAs, or a timely closure plan exists for those remaining open. 2. A preliminary CDR agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair prior to the CDR. 3. CDR technical work products listed below for both hardware and software system elements have been made available to the cognizant participants prior to the review: a. Updated baselined documents, as required. b. Product build-to specifications for each hardware and software configuration item, along with supporting trade-off analyses and data. c. Development, integration, and test plans and procedures. d. Technical Data Package (e.g., Integrated Schematics, Spares Provisioning List, Interface Control Documents, engineering analyses, specifications). e. Operational limits and constraints. f. Preliminary operations handbook. g. Technical resource utilization estimates and margins. h. Acceptance criteria. i. Verification plan (including requirements and specification). j. Validation plan. k. Disposal plan (including decommissioning or termination). l. Updated risk assessment and mitigation. m. Updated cost and schedule data. n. Updated logistics documentation. o. Software design document(s) (including interface design documents). p. Systems and subsystem certification plans and requirements (as needed). | 1. The detailed design is expected to meet the requirements with adequate margins at an acceptable level of risk. 2. Interface control documents are appropriately matured to proceed with development, integration, and test, and plans are in place to manage any open items. 3. High confidence exists in the product baseline, and adequate documentation exists and/or will exist in a timely manner to allow proceeding with development, integration, and test. 4. The product verification and product validation requirements and plans are complete. 5. The testing approach is comprehensive, and the planning for system development, integration, test, and operations is sufficient to progress into the next phase. 6. Adequate technical and programmatic margins and resources exist to complete the development within budget, schedule, and risk constraints. 7. Risks to system success are understood, and plans and resources exist to effectively manage them. |
The TRR evaluates the project's readiness to proceed with testing, ensuring adequate schedule, resources, and management processes are in place. It ensures the completion of an integration test plan and the system's readiness for execution of integration testing.
Table G-5 - TRR Entrance and Success Criteria
| Test Readiness Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. The objectives of the testing have been clearly defined and documented, and all of the test plans, procedures, environment, and the configuration of the test system support those objectives. 2. Configuration of system under test has been defined and agreed to. All interfaces have been placed under configuration management or have been defined in accordance with an agreed-to plan, and a version description document has been made available to TRR participants prior to the review. 3. All applicable functional, unit level, subsystem, system, and qualification testing has been conducted successfully. 4. All TRR specific materials such as test plans, test cases, and procedures have been available to all participants prior to conducting the review. 5. All known system discrepancies have been identified and dispositioned in accordance with an agreed-upon plan. 6. All previous design review success criteria and key issues have been satisfied in accordance with an agreed-upon plan. 7. All required test resources--people (including a designated test director), facilities, test articles, test instrumentation, and other test enabling products--have been identified and are available to support required tests. 8. Roles and responsibilities of all test participants are defined and agreed to. 9. Test contingency planning has been accomplished, and all personnel have been trained. | 1. Adequate test plans are completed and approved for the system under test. 2. Adequate identification and coordination of required test resources are complete. 3. Previous component, subsystem, and system test results form a satisfactory basis for proceeding into planned tests. 4. Risk level is identified and accepted by program/competency leadership, as required. 5. Plan to capture any lessons learned from the test program is established. 6. The objectives of the testing have been clearly defined and documented, and the review of all the test plans, as well as the procedures, environment, and the configuration of the test item, provide a reasonable expectation that the objectives will be met. 7. The test cases have been reviewed and analyzed for expected results, and the results are consistent with the test plans and objectives. 8. Test personnel have received appropriate training in test operation. |
The ORR determines that the project is ready to go-live with the system or service: requirements have been met; the functionality, performance, and security controls have been thoroughly tested; procedures are in place for operations; and that the organization responsible for operations and sustaining engineering is ready to assume responsibility. It ensures a security plan is in place and that system authorization has been received.
Table G-6 - ORR Entrance and Success Criteria
| Operational Readiness Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. All validation testing has been completed. 2. Test failures and anomalies from validation testing have been resolved and the results incorporated into all supporting and enabling operational products. 3. All operational supporting and enabling products (e.g., facilities, equipment, documents, maintenance, and updated databases) that are necessary for the nominal and contingency operations have been tested and delivered/installed at the site(s) necessary to support operations. 4. Approved operations handbook. 5. Training has been provided to the users and operators on the correct operational procedures for the system. 6. Operational contingency planning has been accomplished, and all personnel have been trained. | 1. The system, including any enabling products, is determined to be ready to be placed in an operational status. 2. All applicable lessons learned for organizational improvement and systems operations have been captured. 3. All waivers and anomalies have been closed. 4. Systems hardware, software, personnel, and procedures are in place to support operations. |
The PCR provides assurance that the implemented system is performing as expected and that all necessary support requirements are in place and functioning properly. It confirms that the system is operating properly in its production environment and primary responsibility for the system is turned over to the operations and sustaining engineering teams. It is the official closeout of the project and project team. The final project schedule is published, and remaining open risks are transferred, closed, or accepted. At the conclusion of the PCR, the system is considered fully operational.
Table G-7 -Project Completion Review
| Project Completion Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. A preliminary agenda has been coordinated prior to the PCR. 2. The following PCR technical products have been made available in a final project report to the cognizant participants prior to the review: a. System verification results. b. System validation results. c. Documentation that the delivered system complies with the established acceptance criteria. d. Documentation that the system performs properly in the expected operational environment. e. Technical data package as updated to include all test results. f. Certification package. g. Updated risk assessment and mitigation. h. Previous milestone reviews have been successfully completed. i. Remaining liens or unclosed actions and plans for closure. | 1. Required tests and analyses are complete and indicate that the system performs properly in the operational environment. 2. Risks are known and manageable. 3. System meets the established acceptance criteria. 4. All operations and training documentation is complete and sufficient for production use of the system. 5. Technical data package is complete and reflects the delivered system. 6. All applicable lessons learned for organizational improvement and system operations are captured. |
The DR confirms the decision to terminate or decommission the system and assesses the readiness of the system for the safe decommissioning and disposal of system assets.
Table G-8 - DR Entrance and Success Criteria
| Decommissioning Review | |
|---|---|
| Entrance Criteria | Success Criteria |
| 1. Requirements associated with decommissioning and disposal are defined. 2. Plans are in place for decommissioning, disposal, and any other removal from service activities. 3. Resources are in place to support decommissioning and disposal activities, plans for disposition of project assets, and archival of essential data and official records. 4. Current system capabilities are described. 5. The NASA Enterprise Architecture team has been notified that a DR is planned for the system and will be notified of the results of the DR. | 1. The reasons for decommissioning disposal are documented. 2. The decommissioning and disposal plan is complete, approved by appropriate management, and compliant with applicable Agency policies. 3. All personnel have been properly trained for the nominal and contingency procedures. 4. Risks associated with the disposal have been identified and adequately mitigated. Residual risks have been accepted by the required management. 5. Plans for disposition of system-related assets (i.e., hardware, software, facilities) have been defined and approved. 6. Plans for archival and subsequent analysis of system data have been defined and approved. Arrangements have been finalized for the execution of such plans. Plans for the capture and dissemination of appropriate lessons learned during system operation have been defined and approved. Adequate resources (schedule, budget, and staffing) have been identified and are available to successfully complete all decommissioning, disposal, and disposition activities, including the dispositioning of official records as required by NPR 1441.1. 7. IT infrastructure, including documentation and systems that interface with the decommissioned system, are planned to be modified to reflect the system termination. |
| TOC | Preface | Chapter1 | Chapter2 | Chapter3 | Chapter4 | Chapter5 | Chapter6 | AppendixA | AppendixB | AppendixC | AppendixD | AppendixE | AppendixF | AppendixG | ALL | |
| | NODIS Library | Program Formulation(7000s) | Search | |