Meso- to Macro-Scale Environmental Modeling
NOAA, EPA, NASA, DOE, and NSF are involved in these Grand Challenges and those in the next section on Ecosystem Simulations. These agencies collaborate on developing shared models of the Earth's atmosphere and oceans that the individual agencies use for global climate modeling and agency-specific applications. NOAA uses them for weather modeling (for use in weather forecasting), EPA for pollution modeling (using meteorological models that are modified weather models), DOE for use in groundwater management and environmental remediation, and NSF for basic research in subjects such as earthquake motion and land cover dynamics. The global climate modeling being conducted by these agencies builds on these basic models, mission-specific applications, and data. This work is now described in detail.
Massively Parallel Atmospheric Modeling Projects
Atmospheric models are used in weather modeling (for example, in understanding the mechanisms that control the development of severe storms) and in environmental modeling (for example, in understanding the impact of emissions controls on air quality). They have been optimized for vector supercomputers, where they have been run for years. These computers have several limitations, notably limited memory, high cost, and uncertain future scalability. Memory limits restrict the geographical extent and resolution of the simulations, and their cost prohibits the regional deployment of the models that will be required for planning future air quality measures.
Parallel computing offers significant improvements in raw speed and cost performance. Speed improvements allow scientists to address larger problems, more complex model processes, and finer resolutions. Cost performance improvements allow wider deployment -- a local airport can run its own weather model, a state government its own air quality model. Such a cost-effective portable scalable parallel atmospheric circulation model has been developed by researchers at Argonne and NCAR. The model is a parallel version of NCAR's Mesoscale Model 5 (MM5) that can resolve distance scales ranging from continental to one kilometer or less through the use of specialized numerical solvers and support for nested grids.
The model incorporates new algorithms that scale to the number of processors and address issues such as load balancing that are unique to massively parallel systems. New programming techniques were developed to simplify the production of portable efficient parallel models. The model has been run on systems with hundreds to thousands of processors that provide the large memory and computational power needed for high resolution long duration simulations; these include the Cray Research T3D, IBM SP, and Intel Delta and Paragon. It can be run on small clusters of workstations connected by high speed networks, providing cost effective simulations when high resolution is not required.
MM5 is a building block for EPA's next-generation air quality modeling and decision support system, Models-3, that is being developed to improve both the scientific accuracy and accessibility of modeling tools and the data used in air quality management. By using distributed computing techniques, a model can be run on high- end massively parallel computing systems in research labs as well as on small workstation clusters in local government offices. Deployment is expected to result in improved aircraft safety, air quality control measures, and understanding of meteorological processes.
HPCC funding comes from EPA and DOE; the U.S. Air Force provides additional support. This project complements the DOE CHAMMP (Computer Hardware, Advanced Mathematics, and Model Physics) Program that supports development of the Parallel Community Climate Model, a scalable global atmospheric model, at Argonne, Los Alamos, and Oak Ridge.
http://www.mcs.anl.gov/home/itf/epa.html
The SKYHI general circulation model developed by NOAA/GFDL has been used for a decade to investigate the dynamical behavior of the stratosphere and mesosphere. The new version is scalable and is designed to run efficiently on both shared and distributed memory systems under either data parallel or message passing programming models. It allows scientists to incorporate new physics software modules of atmospheric features such as radiation and convection.
A new scientific experiment is investigating how waves generated by tropospheric disturbances, such as thunderstorms and flow over mountains, affect winds in the stratosphere and mesosphere. This experiment will be run on the 1,024-node Thinking Machines CM-5 at DOE/LANL and will use a grid resolution greater than can be achieved on the supercomputer at NOAA/GFDL. This collaboration is supported by HPCC funds through NOAA and DOE/CHAMMP.
The GFDL Modular Ocean Model (MOM) is public-domain software that is being used by more than 200 scientists in more than 20 countries. Research topics include climate studies (both present day and paleoclimate), data assimilation, predictability, and basic research into understanding world ocean processes on a number of space and time scales. A new version, MOM-2, is being released in FY 1995. One feature of the new version is a flexible memory window that allows solution for a group of latitude rows at one time; as a result MOM-2 can be run on a range of computer architectures with the number of latitude rows chosen to fit the particular system. The modular design of the model also permits alternate ways to treat the atmosphere-ocean boundary at the top of the model and provides an improved way to represent the effects of rough bottom terrain.
http://www.hpcc.noaa.gov/Ocean94
The Parallel Ocean Program (POP) at DOE/LANL was developed to perform high resolution ocean simulations. Recent POP results are shown in the lower image in the section "Advances in Ocean Modeling due to HPCC", where the significance of the calculations is also described.
http://www.acl.lanl.gov/
Mathematical Modeling of Air Pollution Dynamics
Massively parallel computing systems provide an avenue for overcoming the computational requirements in the study of atmospheric chemical dynamics. Implementation issues include domain decomposition strategies, algorithm design and evaluation, portability, modularity, and buffering techniques used in I/O operations. The Caltech urban air pollution model has been implemented on distributed memory MIMD systems including a 512- node Intel Paragon and a workstation cluster.
The central challenge in developing a parallel air pollution model is implementing the chemistry and transport operators used to solve the atmospheric reaction-diffusion equation. The chemistry operator is generally the most computationally intensive step in atmospheric air quality models, and a new method based on Richardson extrapolation to solve the chemical kinetics has been developed. The transport operator (advection equation) is the most challenging to solve numerically. Because of its hyperbolic nature, non-physical oscillations and/or negative concentrations appear near steep gradient regions of the solution. Six algorithms for solving the advection equation have been evaluated for their suitability for use in parallel photochemical air quality models. A speedup factor of 94.9 has been measured when chemistry, transport, and I/O are done in parallel. This work provides the computational infrastructure needed to incorporate new physico- chemical phenomena in the next generation of urban- or regional- scale air quality models. HPCC provides the tools essential to develop our understanding of air pollution further.
Funding for this work came from EPA and the IBM Environmental Research Program. The research was performed at Caltech using the Intel Delta and Paragon systems at Caltech's Concurrent Supercomputing Consortium.
http://nicarao1.che.caltech.edu/~dabdub
Ozone concentrations for the California South Coast Air Basin predicted by the Caltech research model show a large region in which the national ozone standard of 120 parts per billion (ppb) are exceeded. Measurement data corroborate these predictions. Scientific studies have shown that human exposure to ozone concentrations at or above the standard can impair lung functions in people with respiratory problems and can cause chest pain and shortness of breath even in the healthy population. This problem raises concern since more than 30 urban areas across the country still do not meet the national standard.
Additional EPA-funded research on optimizing the performance of atmospheric chemistry solvers conducted at North Carolina State University demonstrated superlinear speedup, that is, the ratio of execution time on a single processor to that of multiple processors is greater than the number of processors. The explanation for this anomalous result is that the data size of the problem exceeds the memory of a single processor, and thus incurs a data-movement penalty in the single-processor case. Using multiple processors allows the data to be spread over the system, accommodating much larger problem sizes. The research indicates excellent scalability for the Cray Research T3D from 8 to 128 processors.
http://www.csc.ncsu.edu/departmental/proposals/
("A Case Study Combining Grand Challenge and National Challenge Technologies: Air Quality Management" describes efforts to model ozone control strategies in Charlotte, North Carolina.)
A Distributed Computational System for Large Scale Environmental Modeling
In the past, the only way to determine the efficacy of solutions to various environmental problems was to implement a set of control strategies and measure the results. When the problem was air quality, this meant using the only experimental laboratory available, namely the atmosphere. More recently, computational models of the physical and chemical processes that take place in the atmosphere allow trials with different control strategies without the expense and difficulty of real-world experiments. These computationally intensive mathematical models are benefiting from the use of parallel and distributed algorithms and mass storage resources.
An NSF-funded Grand Challenge team is studying the effects of six different alternative fuels in each of 12 scenarios. While the study will require over 1,000 CPU hours (45 CPU days), it will be done in parallel on a collection of high performance workstations and will be completed in a week.
This work is conducted at Carnegie Mellon University and MIT in coordination with EPA's research activities. Computing platforms available include a DEC Alpha Supercluster and the Cray Research T3D and C-90; in addition, parts of the simulation are available on the Intel Paragon and iWarp systems.
http://www.cs.cmu.edu:8001/afs/cs.cmu.edu/project/gems/www/current.html
Ozone concentrations for the California South Coast Air Basin predicted by the Caltech research model show a large region in which the national ozone standard of 120 parts per billion (ppb) are exceeded. Measurement data corroborate these predictions. Scientific studies have shown that human exposure to ozone concentrations at or above the standard can impair lung functions in people with respiratory problems and can cause chest pain and shortness of breath even in the healthy population. This problem raises concern since more than 30 urban areas across the country still do not meet the national standard.
Nitrogen is a major cause of eutrophication in coastal estuaries. It is the controlling nutrient in the Chesapeake Bay. Atmospheric nitrogen accounts for 25 to 35 percent of the nitrogen input (or "loading") to the Bay. Control of this atmospheric nitrogen loading may be crucial to efforts to restore coastal estuaries and appears to be critical to Bay restoration. A Bay Agreement that was signed in 1987 and renewed in 1992 calls for a 40 percent reduction in the amount of controllable nutrients reaching the Bay by the year 2000.
EPA is linking together air and water simulation models in order to (1) assess the impact of air pollution controls on nitrogen loading to the Bay, (2) assess the benefit of these controls on Bay restoration, and (3) link the models more effectively by reducing the temporal and spatial mismatches between the individual air and water models.
The three models that are being linked are (1) the Regional Acid Deposition Model (RADM) for atmospheric deposition, (2) the Hydrologic Simulation Program Fortran model for nutrient flow in the watershed to the Chesapeake Bay, and (3) the three-dimensional Chesapeake Bay Water Quality Model (CBWQM) of response to nutrient loading. In addition a weather model is used to drive the atmospheric model, and a separate three-dimensional hydrology model is used to simulate flows in the Bay. RADM and CBWQM were developed on Cray Research systems. The atmospheric model is the most computationally intensive and has the greatest temporal/spatial mismatch. It is being moved to a scalable parallel system.
The grid resolution of the atmospheric model was reduced from 80 kilometers to 20 kilometers over the northeastern U.S. airshed. This will result in more accurate spatial linkage with the water quality models and in improved resolution of urban and point-source emission influences. This increased resolution quadruples computing time.
The airsheds for the Chesapeake Bay watershed and the Bay itself were determined by two sets of simulations. The first estimated average annual deposition of nitrogen oxide emissions for representative meteorology from ten subregions. The second looked at sulfur dioxide emissions. The simulations took 1,400 hours on a Cray Research Y-MP. The airshed was found to be four times the size of the watershed.
This work has been nominated for the annual Smithsonian Computer World Award in the environmental category.
http://www.epa.gov/docs/HPCC/homep.html
(1) Dissolved oxygen in Chesapeake Bay, (2) nitrate loading in the Potomac Basin, and (3) atmospheric nitric acid and wet deposition across the Eastern U.S. Three air and water models are linked together for cross-media modeling of the Chesapeake Bay. Atmospheric nitrogen deposition predicted by the atmospheric model (right) is the input load to the watershed model and the three- dimensional Bay model. The watershed model (lower left) delivers nitrate loads from each of the water basins to the three- dimensional Bay model (upper left).
This NSF-supported Grand Challenge, centered at Stanford University, addresses several issues related to geophysical simulation and prediction, the use of observing systems for data acquisition, and the assimilation of the measured data into numerical simulations. The basic idea is that a predictive computer model, carrying out a simulation faster than real time, can be used to estimate what data need be gathered, as well as the location and resolution of these data, to enable accurate prediction of the future behavior of a complex nonlinear fluid system, such as the atmosphere or the ocean. The data can then be acquired at different resolutions in each region in accord with the predictions of the computer model. Because the density of data needed to accurately describe flows varies greatly over the flow domain, the interactive use of adaptive models with error estimation and control together with the automated observing systems will allow significant reduction in the amount of measured data required to achieve this goal.
A laboratory-testbed interactive system similar to the Earth Observing System that is being developed for use with atmospheric and oceanic models is under construction. It is impossible to study the intended applications directly at present because the observing systems and necessary computing power do not exist. Accordingly, the laboratory experiment will involve a rotating annular tank, with a sloping outer wall and filled with a stratified saline solution, which generates fluid motions very similar to those of geophysical fluid dynamics.
The application of the powerful adaptive composite grid method to this problem and its implementation in a parallel algorithm is essential to the successful completion of the project. The simulations will be carried out on parallel systems at NASA Ames Research Center, and the experimental data acquisition and computer systems will communicate via the Bay Area Regional Research Network.
http://www-cs.stanford.edu/hpcc.html
A coupled atmosphere-ocean model is used to simulate the entire atmosphere-ocean system for integration time scales ranging from months to tens or hundreds of years. The model being used at GFDL today consists of a spectral atmosphere general circulation model and the Modular Ocean Model (MOM) (described earlier in this section). Both models are being redesigned, and the spectral model is being modularized for consistency with the SKYHI grid-point model (also described earlier in this section). These two atmospheric models will more easily share physics modules. The new model's dynamic core is designed to run efficiently on both shared memory and distributed memory systems. The model contains a more flexible vertical coordinate (for handling mountain terrain) and more efficient algorithms for solving the model's equations of motion. This work is illustrated below.
http://www.hpcc.noaa.gov/Climate94
The colored plane floating above the block represents the simulated atmospheric temperature change at the earth's surface, assuming a steady one percent per year increase in atmospheric carbon dioxide to the time of doubled carbon dioxide. The surfaces in the ocean show the depths of the 1.0 and 0.2 degree (Celsius) temperature changes. The Southern Hemisphere shows much less surface warming than the Northern Hemisphere. This is caused primarily by the cooling effects of deep vertical mixing in the oceans south of 45 degrees South latitude. Coupled ocean-atmosphere climate models such as this one from NOAA/GFDL help improve scientific understanding of potential climate change.
Scientists are often faced with the task of sifting through voluminous data that come from observations or computational experiments. Virtual reality promises a means to search and manipulate such data rapidly by taking advantage of human vision and motor capabilities.
NASA is developing methods for analyzing this type of data generated in its Earth and Space Sciences (ESS) Grand Challenges. To do so it adapted the Flow Analysis Software Toolkit (FAST) developed at NASA Ames Research Center. Its virtual reality adaptation VR-FAST is being used by ESS researchers in atmospherics and in hydrospheric processes. Plans include incorporating additional data exploration capabilities and providing feedback from ESS researchers.
NASA is analyzing the applicability of the Vis5D software to virtual reality.
http://cesdis.gsfc.nasa.gov/hpccm/accomp/94accomp/ess94.accomps/ess2.html
A scientist uses NASA's virtual reality modeling resources to explore the Earth's atmosphere as part of the Earth and Space Science Grand Challenge.