Workshop on Software Tools for HPC Systems Technical Issues -- Debugging

	Working Group Members General Discussion Recommendations High Performance Debugging Standards Forum
Working Group Members	Technical Issues -- Debugging   Gail Alverson     Tera   Jeffrey Brown   LANL   Karla Callahan   Intel   Suresh Damodaran-Kamal   Hewlett-Packard   Erica Dorenkamp   Thinking Machines   Joan Francioni (Chair)     SW Louisiana   Mike Gittings   LANL   Kenneth Koch   LANL   David Metcalfe   SGI/Cray   Juan Meza   SNL   Lauren Smith   NSA   Rich Title   Hewlett-Packard   Abduhl Waheed   Michigan State
Introduction	This working group was given the task of discussing issues related to debugging, testing and verifying the correctness of paral lel programs used in the high performance computing arena, and making recommendations along these lines with respect to the role of a national tools center. In particular, the group was asked "What should be the role of a tools center to encourage the development and accelerate the deployment of high quality parallel debuggers for high performance computing?" This document summarizes the group's responses to that question.
General Discussion	The goals of an effective debugging tool for high performance computing are for the tool to be useful in helping users find out what they need to know to understand and debug their programs, and also for it to be portable, extensible, scalable, and easy to use. The debugging community has yet to come up with one debugger that meets all of these goals. This is not because the goals are not understood. Rather, it is because debugging tools are very complex pieces of code that must be written to run on a specific machine with a specific compiler and under a specific operating system. As these three components are constantly changing in the high performance computing arena, it has been difficult for researchers and vendors to have access to stable platforms long enough to be able to develop appropriate tools. Also, there are no accepted standards about what debuggers should do nor about what compilers/operating systems should provide to the debugging program. This greatly increases the complexity of writing a debugger from scratch for each new system.   The problems users are experiencing with current tools range from nonstandard semantics of the commands (e.g., breakpoint and next) - to confusing screens full of too many windows - to an inability to do the things they need done. The effort required to learn a new tool has also frequently turned out to be a problem for users. It is not cost effective for a user to spend a large amount of time learning a new tool for a platform that may not be in existence for very long or for a tool that is not very useful. In addition to these types of problems, both users and developers cite the lack of user experimentation and input for tools under development as a serious impediment to building useful debugging tools.
Recommendations	The working group considered the overall purpose of a tools center to be two-fold: (1) to support the development of useful technologies and research ideas; and (2) to support the promotion of effective tools back to the user community. To this end, five goals for the center were defined. These goals are listed below along with possible functions a tools center could facilitate. In some cases, functions appear listed under more than one goal.   1. Increase understanding of the needs of both users and developers from each other.   There is a definite lack of communication between users and tool developers that has been hard to overcome. The need for this communication is well accepted, but many strategies used in the past have been less than successful for a large number of tool development efforts. The PTOOLS consortium is definitely attacking this problem, but more can be done. Possible functions for a center related to this goal include education programs/workshops, usability testing, developing and promoting standards, and establishing mechanisms for user feedback of tools throughout their development.   2. Support proof of concept for promising but immature research software and ideas.   A good debugging research idea can easily be prototyped, written up and published, but that doesn't tell us enough about how the tool/idea will work "in the field." Conversely, most academic researchers do not have the resources to develop full-blown, robust software to test out their theories on a large slate of real applications and/or with a group of real users. Thus, achieving this goal requires identifying "promising" software via early evaluation of research ideas and then providing support to further develop the software so that it is robust enough to go through usability testing. Related functions would include porting the code to other platforms and developing a base of code building blocks for constructing prototypes quickly.   3. Facilitate testing and evaluation of tools.   For both vendors and academic researchers, having access to the large/realistic systems that the users actually use has been a problem, particularly in the testing and evaluation phases of tool development. In addition, there is no accepted suite of programs that can be used for testing the functionality and performance of debugging tools. Functions which could be provided by a center related to this goal include usability testing, developing and promoting standards, and scalability testing. It would also be appropriate for the center to develop a test suite and a set of benchmark programs for evaluating new tools.   4. Reduce effort needed for academic and vendor developers to create usable tools.   This goal is about establishing an effective mechanism for getting user feedback and input to developers at appropriate stages of a tool's development. It includes the functions of developing and promoting standards, usability testing, scalability testing, developing benchmark programs, and establishing a code base.   5. Support useful tool base for user community.   The intent is not for a center to compete with system vendors or independent software vendors. Rather the goal is to help establish a critical mass of users for effective tools. Once a useful tool has enough dedicated users, it can then be supported by the vendors. (We consider a recent example of this model to be the development of PVM.) In particular, a tool would only be supported by the center for a finite amount of time. If it fails to be picked up by vendors by that time, it would be dropped. Possible functions in this category include usability testing, benchmark programs making tools more robust, and porting code to multiple platforms.   A common thread during our discussions, and reflected above in the function lists of each of the goals, is the need for usability testing by a representative group of real users. This is considered a critical function for making any headway in developing useful and cost effective debugging tools. In addition, user testing should be done early and repeated throughout the entire development of a tool. Although this is a critical function, it is not an easy thing to accomplish, especially in an ad hoc manner. Users are busy working on their own problems and it is not considered part of their job to spend a large amount of time testing out new debugging tools. Tool developers frequently do not have enough connections with a wide audience of users to get appropriate input and feedback throughout the development of a tool. A nationally funded center, however, should be able to set up and support a mechanism that will allow real users to try out different tools and provide expert guidance back to the developers so that the developers can spend their efforts on designing effective and useful tools.   The tools center that is developed may address some subset of the five goals listed above. Within our working group, the users were interested in all goals being met, the vendors primarily supported 1 through 4, and the academics were most interested in 1 through 3.   After defining the above set of goals for a center, the working group discussed how these goals might be met outside of a tools center. A number of alternatives were suggested that included the following. Workshops, such as this one, certainly support goal 1 and could be increased. A High Performance Debugging Standards Forum could be established (see below) to support goals 1, 3, and 4. More support could be given to facilitate academic researchers spending summer or sabbatical terms at vendor or user sites. This would support goals 1 and 3. The mechanisms in place by funding agencies partially support goal 2, but research funding stops short of the work needed to make tools ready for usability testing. This could be changed somewhat, but it is not necessarily of interest or even appropriate for researchers to do all the detail work necessary to bring a tool far enough along for wide usability testing (i.e., different programming models, architectures, applications, etc.). Finally, it was suggested that a vendor consortium could be established to share ideas and common components, thereby supporting goal 4.
High Performance Debugging Standards Forum	As a direct result of the discussions of this working group, a Birds-of-a-Feather session was held at Supercomputing '96 to explore the community interest in beginning a formal debugging standards effort. It was decided at that session to go ahead with this effort and plans are underway to have a first formal meeting in March in conjunction with the SIAM conference. A steering committee for the forum has been created and consists of Jeffery Brown, Joan Francioni, and Cherri Pancake. In addition, funding for the initial efforts of the forum is close to being secured.