|
Workshop on Software Tools for HPC Systems
Software Selection and Vendor Involvement |
 
|
|
|
|
|||
 |
|
||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||
Working Group Members
|
|
||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||
|
Introduction
|
A Software Tools Testing and Evaluation center (STTEC) must facilitate the use of high quality and innovative tools by the developers of scalable, parallel scientific applications. Such tools are not available. Tools developed by researchers in the academic sector, although innovative, usually do not evolve into usable tools that could be adopted as or integrated into products by vendors. Tools developed as part of commercial efforts usually lack the innovation and investment necessary to solve the hard problems related to application development. This working group was asked to focus on two key questions related to the structure of a STTEC, which we will call the center: (1) What selection criteria should be used to identify the software to be developed at the center? (2) How might vendors participate in the center's activities and benefit from the center's accomplishments? The discussions and conclusions from this working group covered a broad range of possible mission statements for a center and was influenced by the desired degree of vendor involvement. A number of important points surfaced regarding difficulties that various types of vendors might encounter in taking advantage of the testing, evaluation, and hardening efforts that the center might undertake. It appears that while a center should encourage the participation of vendors as much as possible, the primary customers should probably be the application developers that use high-end systems. Using this focus, the working group elaborated on a possible mission/charter for the center. This charter included evaluation, testing, and hardening activities for various software tools. Based on these activities, the group developed selection criteria for choosing software that a center would handle. These criteria included factors related to the state of the tool (current and proposed), the viability of the technical plans for advancing the tool, the resources needed to carry out the plan, and various requirements upon the center itself. In addition to these criteria, this group recognized that the Working Group on Intellectual Property Issues raised some significant points that needed to be addressed as part of the selection criteria (which are not replicated here). The remainder of this working group report contains four sections. Section 1 summarizes the key issues and conclusions regarding the potential for vendor involvement in a center. Section 2 gives a detailed description of the assumptions made about the structure and charter of a center. Section 3 contains the description of the selection criteria for transitioning software tools into and out of the center's responsibility. Section 4 summarizes the key conclusions of the group's deliberations. |
||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||
|
Vendor Involvement Comments and Analysis
|
Any long-term solution to the inadequacy of software tools for high performance computing (HPC) must involve vendors because only they can provide the necessary infrastructure support and migration of these tools to new platforms. In this context, "vendor" refers to a computer manufacturer or independent software vendor. If a center is proposed to help solve the software tool inadequacy, then seeking vendors' acceptance and use of the center is critical. Not surprisingly, vendors have a diversity of positions and concerns regarding what a center might contribute and what type of vendor involvement would be most beneficial and desired. This section highlights the Working Group's discussion of vendors' perspective on software tools for HPC, evaluation vs. hardening options for a center charter, and the potential roles vendors could play. Conclusions arising from these discussions make it clear that vendor involvement is a challenging problem that will require significant care and attention in any center proposal. For hardware vendors, revenue comes from selling hardware. Software tools are only critical when they directly affect the purchase of hardware. In such an environment, the center must make the transfer of technology to vendors very simple. Quantifying exactly what users need will be key. Once identified, passing these solutions to vendors must be straightforward. Sometimes vendors will be willing to take source code supplied by the center directly. Other times the basic idea can be incorporated into an existing tool (using all, some or none of the original source). Some vendors will like an idea but will be unable to use the supplied code and will need to rewrite the code to meet their internal standards. These standards cover issues such as internationalization, service tools, compatibility with vendors' related products and even look and feel. In all these cases, the center staff will need to supply expertise to make sure the transfer of technology succeeds. The assistance will range from consulting on the tool design to supplying users for evaluating the vendor's implementation. The test and evaluation functions of the center were perceived to be most universally attractive to vendors, particularly if the testing were to provide favorable results that could be viewed as an "endorsement" of the marketability of the software. In addition, any feedback from the evaluation could provide independent information for planning and further commercial investment. However, if the software is proprietary, this test and evaluation function and implied endorsement likely would be incompatible with the nature of a center as an independent, government-funded organization. Thus, the center might only accept non-proprietary tools. Even so, companies may be willing to provide tools if improvements (e.g., making them interoperable over many platforms and more robust) would enhance the value of their hardware. Universities might also be willing to provide software that they would not otherwise commercialize and for which they seek broader acceptance and use. The center could begin with "donated" software tools meeting certain initial evaluation criteria. The center's test and evaluation functions would identify both the potential value of the tool and what kinds of improvements would be needed to transform it from a research state into a commercial prototype. The tool originator and vendors would benefit from the information obtained during the test and evaluation. If the center would then proceed to further develop the tool, the entire community would benefit from having a more robust, better documented, commercial prototype. This prototype could then be distributed through NHSE and be commercialized by any interested vendor. A significant group of vendors not mentioned so far, are Independent Software Vendors (ISV). ISVs are, if anything, a more delicate problem. Such tool vendors' revenue is often closely tied to the uniqueness of their products and the breadth of platforms on which they are supported. The center must take particular care not to "compete" with their products. At the same time, ISVs have experience at porting between machines, which could be an excellent resource for the center. If a center could find ways to work with ISVs to help "harden'' programs, it would be beneficial (e.g., hiring an ISV to consult on a project or contracting with an ISV to perform the hardening and documentation). These types of arrangements lead to a natural migration path out of a center and to the ISV to create a supported product. By finding and working with ISVs, a center might be able to satisfy the trickiest legal problem (not overlapping ISVs) while providing an exit process for successful programs. The big question in joint development between a center and ISVs will be ownership. This issue needs to be resolved early in the operation of any successful center. A single policy on ownership needs to be adopted and very few, if any exceptions, need to be made to it. Consider the life-cycle of a typical tool which starts in a university setting with many students contributing which is then transferred to the center. The center applies its staff to evaluate, harden and test with input from some set of users to enhance the tool. By the time a tool is ready for commercial adoption, many individuals will have contributed to it. Corporate attorneys will seek legal assurance that the technology from the center can be used in a commercial product without becoming involved in litigation over ownership. For a more comprehensive discussion of these issues, refer to the Working Group on Intellectual Property Issues. A second question will be whether a vendor (ISV or hardware) can supply code to a center for evaluation. The group considered it unlikely that any vendor would submit any software "critical" to the company's success. However, the general feeling (with dissent) was that vendors should be able to submit potential codes just like any research center, applying the same qualifications and ownership rights. |
||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||
|
Assumed center Structure
|
Given all of the options and uncertainty to arise out of the vendor involvement discussions, the working group decided it had to focus on a particular vision of a center and its charter. Such a focus suggests one detailed approach to the direction a center might take and it provides a more concrete platform for discussions about selection criteria for tools entering and leaving the center's responsibility. This section describes the working group's vision for a Software Tools Testing and Evaluation center. Figure 1 shows a simple model of the participants and expected products. In this model, the primary customers of the center are the people who develop applications for HPC systems. The center's activities would include testing, evaluating, hardening, documenting, and/or consulting for selected software tools. An Evaluation Board would provide direction for handling proposals for the center's involvement in any software tools. The working group believes that such a center might naturally fill a gap between the current Parallel Tools Consortium and the National HPCC Software Exchange. 
Figure 1: Working Group Model of Participation and Products for a center. The working group concluded that the primary customer focus for the envisioned center needed to be the HPC application developers. In this model, they will present requests for tools or tool improvements to the center and will offer their time as volunteers to assist in tool evaluation and testing. The rationale for putting application developers first reinforces the point that they are the people who need the products of the center; it is their time and energy that is being leveraged to produce more effective software. Their input on the selection of software to be handled by the center and their recommendations on how the center's activities can enhance the use of the software will be of critical importance. The working group felt vendors could not be viewed as the primary customer of the center, due to the risk involved in investing in speculative tools. If HPC computer systems are viewed as a relatively low volume market, then the software tools that support those platforms must capture a large market share to be profitable. The Federal Government is an important "customer" of the center because it is expected to provide the bulk of the funding needed to make the center a reality and because many of its critical missions can only be met through effective use of HPC systems. As such, the government tie adds some important constraints (e.g., public access to results and non-competition with the private sector) and responsibilities (e.g., an effective portfolio of software tool investments that span the needs of the entire HPC community). Proposals to the center could include the enhancement of an existing prototype tool, the development of a proposed tool, and the testing and evaluation of existing prototype tools. Potential sources of proposals span the entire community. Software researchers (from the academic, government, and commercial sectors) often develop innovative prototype tools. Such tools might provide new capabilities or use new solution techniques. However, due to funding constraints or other practical reasons, these tools are not hardened products. Proposals from these researchers would further efforts to make the tools usable by the user community. In contrast, members of the user community might propose desirable tools that currently do not exist at all. Vendors could propose testing and evaluation of their prototype tools or tool "kernels." They could also propose to contribute non-proprietary tools that could be made available to the community on an "as is" basis. Of course, ideas or tool elements produced by the PTOOLS consortium can be incorporated into a proposal to the center. Members of all three potential sources of proposals can benefit from the efforts of the center. The research of software researchers can be enhanced by their building on top of hardened, state-of-the-art software instead of research prototypes. The user community as a whole can access state-of-the-art software development tools. Vendors can have prototype tools tested and evaluated by the center, thus, facilitating the software development process, and can direct their user community to center tools. Figure 2 illustrates the envisioned center activities and work flow. Different tools will have different activity paths through the center. An Evaluation Board will manage the path of each tool based on proposals received for consideration. To facilitate the job of the Evaluation Board, any existing tool that is part of a proposal will receive an initial test and evaluation by center staff. This evaluation will be made public at some point in time. The time at which the evaluation will be made public may depend upon vendor considerations. In this way, vendors can propose that the center periodically review a tool throughout its development, without breaching confidentiality before the tool is released. The center will create proper documentation for tools that it "hardens" to meet real-world usage and for tools that it develops in their entirety. In either case, the center will be responsible for supporting these tools for some reasonable time period and will provide related additional documentation, maintenance, and consulting as needed. However at some point, every tool is expected to exit the center. At that time, the center relinquishes all responsibility and ownership for the tool. 
Figure 2. center Activities and Workflow Structure An initial solicitation for proposals would be generated to initiate the center's activities. After this initial solicitation, proposals could be submitted anytime. All proposals would be evaluated by the center's Evaluation Board (which we will call the Board) on a regular basis. As mentioned above, a proposal could be: (1) a request for test and evaluation only, (2) a plan for "hardening" a prototype tool by the center, or (3) a plan for developing a new tool from scratch by the center. For each proposal, the Board would assess the anticipated costs and benefits, as well as the availability of required resources. Given this information, the Board would rank proposals according to a defined selection criteria, which is discussed in the next section. The center's Evaluation Board would be comprised of a mixture of researchers, vendors, users, and center staff. The board, as a whole, would be required to have expertise in the field of high-performance computing, and sufficient knowledge and experience to be able to evaluate and estimate the cost of center activities that would be associated with implementing software tool proposals. Researchers will be able to evaluate the difficulty associated with enhancing or developing a tool. Vendors will be able to identify tools that would be useful to their user base or applicable to upcoming architectures. Users will be able to identify tools that would be adopted by the user community. center staff will be able assess the cost of processing (be it test, evaluation, enhancement, development, or other support). Board members should serve on a staggered, rotating basis. This approach would insure continuity of decisions and participation of different vendors, universities, and government laboratories and agencies. Travel expenses for academic Board members would be reimbursed. The working group's vision of the center fits comfortably with other existing organizations that are actively involved with software tools efforts, including the Parallel Tools (PTOOLS) Consortium and the National HPCC Software Exchange (NHSE). The NHSE acts as a distribution service for software, documents, data, and information of interest to the high performance and parallel computing community. As such, it promotes software sharing and reuse within and across HPCC agencies and facilitates the development of discipline-oriented software and document repositories. The NHSE is the logical place to distribute software "produced" by the center. The mission of the PTOOLS Consortium is to take a leadership role in defining, developing, and promoting parallel tools that meet the specific requirements of users who develop scalable applications on a variety of platforms. PTOOLS focuses on new tools, guiding the development, testing, and evaluation of them from inception through reference implementation. All tools must meet a demonstrable needs in the user community and the PTOOLS processes involve users throughout the development. Funding to support the tool development work is the responsibility the proposer of a PTOOLS effort. The Consortium provides a forum for potential tool users to guide the tool implementors toward products that will have the features and portability to have broad acceptance in the community. The working group viewed the activities of PTOOLS and the center as complimentary. Both share a common mission and stress the importance of user involvement toward the development of hardened tools. The primary differences focus on the types of activities to be undertaken and the means for financially supporting them. The center is expected to have immediate resources (money and personnel) to dedicate to carrying out its activities and harden successful research prototypes developed elsewhere. The center can also test and evaluate software developed elsewhere. Hopefully, the combination of the two approaches to tool development will increase dramatically the availability of high-quality tools and expand their use in the community. |
||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||
|
Selection Criteria
|
Having come to agreement on the structure and charter for a proposed center, the Working Group encountered little difficulty identifying appropriate selection criteria for software to be brought into the center. The criteria seemed to divide naturally into five categories: current state of the tool, proposed state of the tool, the plan for advancing the tool, resource considerations, and "center" considerations. All of the criteria include a high level of subjectivity. This section elaborates on the working group's vision and understanding of these criteria and how they could be effectively applied in practice. Clearly, a tool may not be able to "score'' well in all of the criteria, however any tool submission should address these issues. The Evaluation Board for the center must be responsible for carefully weighting these different criteria to best support the HPC user community's needs. The state of a tool upon entry into the center determines the fundamental value of the tool as it currently stands and indicates the potential for useful center improvements. Likewise, the proposed state of the tool provides an evaluation of whether it would be a worthwhile endeavor for the center to work on the tool, or whether the end result itself would be useful. The qualities evaluated for the (current and planned) state of the tool include:
center activities are meant to realize state-of-the-art hardened tools and make them available to the user community. To attain this goal in a timely manner, center activities associated with any one tool must be limited to a reasonable number of person-hours. The plan for handling each tool must be clearly delineated and meet the following selection criteria.
The center makes a resource commitment to any tool that it accepts. This investment is made in expectation of returns on the investment accrued by later users of the tool. Hence, the type and level of resource requirement for the tool project within the center should be considered as part of the selection criteria. Resource needs include the "expenditures" of personnel (center staff) over the lifetime of the project and the money needed to support their working environment. The tool might also require certain hardware and software infrastructure that the center may need to buy and then maintain. Clearly, some tool projects will be larger than others, requiring greater assignment of resources for their successful completion. It is therefore important for the center not only to have a clear sense of resource needs as they relate to the project plan (different project stages may have different needs), but also a method to account for the cost of those resources with respect to the center's operating environment. Interestingly, the working group considered the possibility of a tool project being proposed with supplemented resources. Resources brought in with a project could come from several places and take various forms. For instance, an interest group of industry and/or user participants might decide that a tool is important enough to sponsor its testing and evaluation in the center through money, the donation of machines, or the allocation of personnel. Such direct support goes to offsetting the costs of the tool project. It is also conceivable that there may be other forms of indirect support proposed such as granting access temporarily to computing systems, providing consultant services, certain "after tool completion" service, etc. Any form of supplemented resources should be weighed in consideration of tool project acceptance. The selection criteria identified so far focus on attributes of each individual proposal. In addition, the center must consider broader criteria that examine each proposal in the context of the overall effectiveness of the center and its ability to serve all of its customers. It makes no sense for the center to support five different flavors of the same type of tool, even if they all have their own important user communities. Through careful selection criteria, the center can develop a strong "portfolio" of tools that covers the spectrum from writing code to tuning performance. The center also needs to consider each decision in the light of how it will affect the long-term survivability of the center itself. For the short-term, the center will certainly need some quick wins to show it can deliver. However, for the long-run, the center must take on the hard problems and show some big wins. Such strategic planning must be part of the selection criteria. Probably one of the trickiest issues for the selection criteria is avoiding competition with industry. If a computer vendor or ISV decides to develop a particular tool that is closely related to a tool proposed to the center (or worse, already under some level of hardening by the center) tough decisions must be made to avoid running afoul of federal law. This situation likely requires the center to be very public about projects even at the proposal stage. Detecting potential conflicts early may avoid having to cancel a project already in progress and may actually support the goal of helping move new technology into commercial products. |
||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||
|
Conclusions |
The discussions in this working group led to three specific types of conclusions. Much of the actual time in the meetings centered on the basic question of what kind of role and charter for a Software Tools Testing and Evaluation center might make sense and prove most valuable. Significant skepticism was expressed about the value likely to be returned for expected high cost of such a center. By the end of discussions, some of these concerns appeared to diminish; however, the diversity of visions for a center produced by the different groups demonstrates the need for a more refined and carefully defined vision for such a center. This working group's first conclusion affirms the needs for this more precisely stated vision and charter. Based on its deliberations, the group favors a broad charter that includes testing, evaluation, hardening, documenting, and consulting (where specific decisions must be made as to which of these activities is applied to each tool in the center). Vendor involvement in the center turned out to be a very challenging problem. Ideally, one would hope that tools produced by the center would be easily adopted by all of the vendors, based on the center's ability to move valuable research ideas to hardened tools that everyone would want to use. In reality, vendors cannot be viewed as a homogeneous group. They have different policies, procedures, and driving forces that significantly influence their desire and ability to support this ideal. This group's second conclusion is that vendor "involvement" for the center needs to be encouraged throughout the process, but that expectations for results need to be framed carefully. While vendors can and should participate, they should not be viewed as the primary customer of the center. Commercial success of tools handled by the center should not be the metric for project success. Instead, the group recommends that HPC application developers be the primary customers of the center and their use of the tools be the key metric for success. Moreover, if a tool is enthusiastically embraced by HPC application developers, it may naturally find its way into commercial products rapidly. Selection criteria for determining what activities the center might choose to undertake for specific software tools generated the least debate in the working group. The criteria stressed meeting anticipated users' needs, maximizing the likelihood of delivering the expected results, and moving the state of the tools toward a more interoperable and effective software platform for doing high performance computing. The working group's third key conclusion was that the selection process involve all of the key players (users, researchers, vendors, and center staff) and that individual tool decisions be made in the broader context of a long-range strategic plan for software tool development and use. Once these high-level understandings become clear, the selection criteria appear to fall out in a reasonable and natural fashion. |
||||||||||||||||||||||||
|
|
|
||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||
