Statistics/Metrics:
Implications for Federal Agencies
Presented by Phillip Dykstra, Co-Chair FNC/EOWG
at NSF's ISMA Workshop
San Diego, February 19-20, 1996
1. Background
Federal R&E networks currently collect basic statistics on their own network's performance and traffic flows. Typically this includes measurement of throughput, delay, and availability.1/ In the era of the post-NSFnet Backbone Service, the only baseline against which networks evaluate performance is their past performance metrics. There are no data available against which national level comparisons or comparisons with other networks' performance can be made. Current Federal concerns about statistics/metrics are centered on end-to-end performance -- beyond the realm of what is controllable by individual networks/agencies.
Tracking of trouble tickets, for example, is performed by most Federal R&E networks. However, these tracking systems are network-centric, focused on problems which originate and can be resolved within the context of an individual network or Internet Service Provider (ISP). Trouble ticket
resolution and route instabilities were once the responsibility of Merit (under its cooperative agreement with NSF), however, in the current environment there are no entities that claim or share responsibility for national level management of the Internet. As a result, there are no mechanisms available today for resolving or tracking problems originating or extending beyond the control of individual networks or ISPs.
Route instabilities can have a direct, sometimes profound, affect upon the performance of individual networks. Some Federal R&E networks are seeking to improve the stability of their routing by peering directly with the routing arbiter (RA) at network access points (e.g., SprintNAP and
FIX-West/MAE-West). Merit/ISI are also developing Route Flapping Reports which will assist in gauging the overall routing performance and identifying trouble areas. These efforts are still in their nascent stages and do not yet have sufficient buy-in or support from commercial players to make them a fundamental component of the Internet architecture.
The vacuum created in national-level statistics/metrics collection which followed the transition to the commercial architecture has also complicated Federal planning. While detailed traffic and performance measurements are essential to identifying the causes of network problems and formulating corrective actions, it is trend analysis and accurate network/systems monitoring which permit network managers to identify "hot spots" (overloaded paths), predict problems BEFORE they occur and identify ways to avoid them by efficient deployment of resources and optimization of network configuration.
In addition, R&D agencies, particularly ARPA, DOE, NASA, and NSF, have a long and impressive history of promoting cutting-edge networking technologies and furthering development of the national information infrastructure (NII) and global information infrastructure (GII). By evaluating Internet usage and performance limitations, these agencies have been able to push the technology envelope, investing in research and development efforts to continually enhance the overall capabilities of the NII/GII. While networking R&D will continue to be funded by these Federal agencies, the ability of R&D managers to evaluate networking trends as inputs to their decision making is diminished by the lack of broad, Internet-wide metrics on traffic flows and performance. 2/
2. FNC / FNCAC
The Federal Networking Council (FNC) and its Advisory Committee (FNCAC) participated in the planning of the transition of the NSFnet Backbone
Service to the commercial Internet architecture and in working to promote a scalable, reliable national information infrastructure. At the October 1995 FNCAC meeting, there were calls upon the FNC to promote...
a) a better understanding of the Internet internetworkings and the requirements for service models -- both at the component and a systemic levels;
b) an accounting and pricing architecture with multiple service qualities to promote incentives for users to place a value on their resource consumption;
c) broad integration of security and privacy measures;
d) the leveraging across the many networking activities, to help address the complexities; an assessment of the cost/benefit tradeoffs of "doing" or "not doing" certain things.
Inherent in these items is an assumption about the availability of reliable metrics upon which to base decisions. In reality, performance metrics for ISPs and national services providers (NSPs) are deemed proprietary. Data on performance at the NAPs are collected and selectively made available as part of the NAPs' cooperative agreements with NSF. The provisions of these cooperative agreements, however, do not specify which statistics need to be collected or made public. FNCAC meeting participants have urged NSF to revisit the statistics collection provisions following the SDSC/Bellcore statistics workshop this February 1996.
FNCAC meeting participants also agreed that technology facilitating statistics acquisition should be the focus of a basic research effort (possibly by ARPA or NSF). The desired product would be dedicated hardware capable of gathering and aggregating packets for critical data collection on ATM networks. (Currently Bellcore is supporting ARPA on a related effort.) It was also suggested that the FNC agencies should cooperate and jointly agree on how their networks will instrumented and that this topic be placed on the High Performance Computing and Communications (HPCC) research agenda. Furthermore, participants discussed the overall importance of Federal networking agencies becoming a role model for implementing "service models", including, among other things, the articulation of goals, objectives and requirements for statistics collection and analysis.
FNCAC members also noted the importance of encouraging an industry group to be constituted which can sort out how industry can best address the need for statistics acquisition, and the user communities' requirements for certain traffic details. Participants also noted that statistics acquisition is inseparably linked to the industry "getting its act together" to better manage the commercial Internet.
3. Privacy
Privacy is an important concern to both the FNC and the FNCAC, and the Federal R&E networks are committed to maintaining the integrity and privacy of network data flows. Representatives from Federal R&E networks have expressed their concern about data collection efforts which enter the data stream to collect information. On the other hand, some data can only be collected by going beyond the packet header. For example, while WWW requests can mostly be identified by the UDP/TCP port number specified in the header, this same header will not identify whether the traffic is using PGP or PEM encryption or authentication. Policies therefore need to be developed related to aggregation and ensuring the privacy of data flows.
4. Service Models
The FNCAC has recommended that the FNC define the model or models for service for its R&E networks. Currently, each network individually defines its quality of service (QOS) specifications with respect to bandwidth, performance, security, etc, as well as its policies for acceptable use, reselling of connectivity, etc. The FNC is initiating an effort to explore and define areas of commonalities among these R&E networks and to detail baseline criteria for minimally acceptable service and practice across Federal R&E networks. By communicating these evolving Federal requirements to other user communities and to service providers, the FNC hopes that it can further the overall quality of service available on the Internet.
5. Outstanding Issues
As the Federal government increasingly transitions into a consumer of commercial networking services, the dynamics of these statistics / metrics issues and the Federal government's ability to influence their resolution will change. Key issues include...
- How can one measure "the Net", when no one owns it or controls it? What is the Federal government's role in promoting cooperation among service providers and major users?
- Who should be responsible for collecting and publicizing statistics / metrics related to performance and traffic flows across the NII, e.g. the Bureau of Census, a neutral private-sector organization, etc.?
- How can privacy of Internet traffic be ensured? How can organizations engaged in statistics acquisition delve beyond the packet header, while still ensuring privacy and confidentiality of data?
---- Footnotes:
- Federal networks measure throughput either directly (bytes and packets) or by inference -- from measurements of offered load, resource utilizations, or protocol overhead and performance. Delay is estimated from the effects of routing and resource utilization, protocol interactions, and end-system (e.g., host) performance. Availability is frequently derived from the availability/reliability of individual network or system components.
- As an example, in the Internet security field, trend analysis related to the volume and type of encrypted traffic across the Net could have significant implications for the allocation of research funds. In addition, such data could have a direct affect on Federal policies, encouraging awareness of commercial practices. Acquisition of these data could be handled by identifying the attributes of specific software / encryption hardware, e.g. SSH uses Port 22.
Last modified on March 1, 1996
Questions or Systems problems should be reported to Webmaster@fnc.gov