View Full Text: Convergence-HPC-BD-ML-JointWSreport-2019.pdf
The high performance computing (HPC) and big data (BD) communities are evolving in response to changing user needs and technological landscapes. Researchers are increasingly using machine learning (ML) not only for data analytics but also for modeling and simulation; science-based simulations are increasingly relying on embedded ML models not only to interpret results from massive data outputs but also to steer computations. Science-based models are being combined with data-driven models to represent complex systems and phenomena. There also is an increasing need for real-time data analytics, which requires large-scale computations to be performed closer to the data and data infrastructures, to adapt to HPC-like modes of operation. These new use cases create a vital need for HPC and BD systems to deal with simulations and data analytics in a more unified fashion.
To explore this need, the NITRD Big Data and High-End Computing Interagency Working Groups held a workshop, The Convergence of High-Performance Computing, Big Data, and Machine Learning, on October 29-30, 2018, in Bethesda, Maryland. The purposes of the workshop were to bring together representatives from the public, private, and academic sectors to share their knowledge and insights on integrating HPC, BD, and ML systems and approaches and to identify key research challenges and opportunities. The 58 workshop participants represented a balanced cross-section of stakeholders involved in or impacted by this area of research. Additional workshop information, including a webcast, is available at The Convergence of High Performance Computing, Big Data, and Machine Learning.
From the presentations and discussions, a vision emerged of a rich computational ecosystem consisting of heterogeneous combinations of edge, cloud, and high performance computing systems. This ecosystem would be flexible and be able to receive data from a variety of sources such as scientific and medical instruments, sensor networks, and security and infrastructure monitoring systems. It would have edge Internet of Things devices that would extract important features and convert data into forms suitable for ingesting and storing in the cloud. Large-scale data analytics would run in the cloud, combining ingested data with stored databases. HPC systems would perform more computationally intensive forms of analysis and optimization, as well as run simulations for predictive modeling.
Such a rich computing environment could provide capabilities well beyond those of today’s isolated systems. For biomedical and clinical research and healthcare, it would enable the use of clinical, laboratory, and even molecular data for patients and for researchers. Data sources could include smart health applications where patient outcomes are connected to an evidence-based computed model, thereby putting data as a “digital first” asset in the healthcare system. The computing environment would allow scientific and medical researchers to solve problems with many degrees of freedom in ways that allow data to inform models and simulations to build better models.
Achieving this vision of a rich computing ecosystem will require new capabilities in hardware (computing, network, and storage); management modes of operation; and software. Providing true convergence among current and future computing environments presents many technical and organization challenges, but it could provide capabilities in scientific research, national security, healthcare, and industry well beyond what is possible today.