Advanced Networking

At the outset, it is important to note that advanced networking, almost by definition, is a volatile area. This seminar will focus on the fundamental issues involved in building and managing advanced networks, but viewers are encouraged to review current publications and the Web to enhance their understanding.

We'll begin by looking at some of the issues associated with advanced networking. What is an advanced network? What are the application drivers? What are some of the fundamentally difficult issues that one needs to resolve in order to build an advanced network? From there, we'll move into the core technologies that will be used to support advanced networking; technologies such as ATM and RSVP. We'll then move into consideration of the deployment issues associated with building advanced networks, and we'll begin from the desktop and move out. So, we'll look first at connectivity between the desktop and the campus backbone. We'll then look at construction techniques for advanced campus backbones and lastly, we'll start to look at the issues associated with connecting that campus advanced network into a national fabric through a gigapop into the national backbones. That last topic will lead to the second part of this seminar, on what are the external networks that we want to connect to, and how do we go about that?

Looking at the application drivers, perhaps the most potent of them is video-over-IP. We have adequate solutions at this point for one-to-many broadcast technologies. Satellite and broadcast television, for example, work quite well in these arenas. However, desktop video conferencing, one-to-one or several-to-several, will be a fundamental tool for educational development over the next several years, and advanced networks will be necessary to develop that service. Virtual reality will give educators an opportunity to convey information in a much more fluid and valuable way to students. It's important to remember that the students entering our colleges now were raised on visual media, and we need to make sure that our presentation of information is cognizant of that fact. For scientists dealing with huge data sets, the only tractable way of understanding that information is through scientific visualization, where researchers can fly through the data and have it presented in visual format for greater understanding. Digital libraries are a fourth application driver. This will include not only traditional print materials, but enhanced video and auditory communications as well as some elements of virtual reality. Scientists sitting in a site at one side of the country may want to control instruments at another part of the country, and that remote instrument control may require a Quality of Service in the network that would insure that the instruments are properly managed. Lastly, there is a concept called "collaboratories," where scientists around the world can get together in enhanced communication environments, including shared whiteboards and shared text screens, for a collaboration on their science.

Advance networking is a moving target. Right now, two clear elements of advanced networking are the speed of the transmission media and the tuning of the workstations to utilize that speed. We're talking here of speed on the order of 155 mbps or greater. Very soon, advanced networking will have an element of Quality of Service. That Quality of Service will provide you with guaranteed bandwidth or guaranteed maximum packet loss, and will allow smooth conveyance of information across what is by nature a very "bursty" media. In a while, there will be more speed issues, more Quality of Service issues -- it's hard to forecast what will be the leading edge in three to five years. One other consideration is that the advanced network will likely involve embedding storage and computing elements into the network architecture. For example, a streaming video server that is serving distance education needs should not be located on a campus and congest the campus external network link. It might be better to have that video server out there, in the network itself, to serve the distance education community.

There are, of course, other definitions for advanced networking. One of the major drivers today is the Very High Speed Backbone Service that has been launched by NSF over the last two years. That vBNS has a campus extension that will reach to the desktops. Another activity that could be considered advance networking is Internet 2, or the Next Generation Internet. At this time, those projects are just beginning. Over the next year or two, they will help to define what advanced networking is. Of course, advanced networking can also be considered to be whatever the physics and engineering researchers on your campus want, and it's pretty clear that advanced networking, unlike the first generation of networking, is not about mere connectivity.

In fact, we should take a moment to look at what will be different about advanced networking from "Networking the First Generation." I said earlier, it is not just for connectivity, and, indeed, advanced networking may not reach to every desktop on the campus any time soon. Secondly, we now have an embedded base of network that we have to support. And so, this second, advanced networking activity is going to be overlaid on that existing base, and we will need to continue our support of that and integrate the advanced network into it. Networking is "hot stuff" these days, and so there's much greater volatility, not only in the technology, but also in the politics. There are more people looking over our shoulder, and this suggests that the numbers are a lot bigger -- not only the speeds, not only the costs, but the importance of the network to the institutional mission.

A number of external influences are going to constrain our development of advanced networks. First and foremost, the Commodity Internet is out there. We will continue to need to direct our commodity traffic, our day-to-day traffic, our traffic to commercial sites across that Commodity Internet, and so one of the key issues that we'll be wrestling with is learning how to route our traffic to multiple locations and multiple external networks, depending on the nature of the traffic. We're also going to be driven by the kinds of cards that are in the desktops of the devices that connect to the advanced network. If researchers are bringing in work stations with ATM cards embedded in that equipment, then we need to provide ATM service. If those desktops have FDDI cards, we'll need to consider FDDI as a possible connectivity tool for that last mile from the point of presence in a building to a desktop. We're also carrying some baggage from the first Internet that we have to confess to. We've never developed the diagnostic tools sufficient for managing our complex network environments. As we introduce new technologies, most particularly Quality of Service, this lack of diagnostic tools will be conspicuous. Similarly, one of the Holy Grails of that first networking effort was to develop trouble tickets that would assist in the diagnosis and remedy of network problems. Those trouble tickets never advanced particularly far. We're going to feel the lack of adequate trouble tickets in the months ahead. One of the other key influences in the development of advanced networking will be how it's packaged to the campus. What is the economic model? How are we going to pay for this? Will we be paying for each high bandwidth or Quality of Service application with a nickel at that moment, or will the marketplace provide us with some kind of flat rate pricing that, in turn, we will allocate around our campus? Lastly, we have limited tools and limited standards for monitoring high performance networks. We'll need those tools and statistical standards to evolve rapidly so that we can talk meaningfully about our network problems.

You'll hear repeatedly during this seminar that Quality of Service is one of the key drivers of advanced networks. Unfortunately, Quality of Service has a number of definitions, and it's unclear which one will dominate the packaging of advanced networks. One definition is guaranteed bandwidth, and that can be measured in either bits per second -- in this case, megabits per second -- or in packets per second. But another definition would say that an advanced network and good Quality of Service certifies that I will not be losing more than x% of the packets across my transmission media. A third definition will be that the maximum end to end delay of any packet does not exceed a given quantity. We may not be concerned as much with the end-to-end delay. We may worry more about what's the variation in that end-to-end delay. That's called jitter, and we may seek to guarantee that the jitter is at a fixed amount. Indeed, the definition that we pick for advanced networking and Quality of Service will depend upon the application that we are using. If we are using an interactive video application, then issues such as end-to-end delay and jitter become significant, and a packet loss here and there does not matter in a video screening. On the other hand, if we're considering transferring large data sets in real time, then packet loss is going to be a key issue as well as bandwidth, but we don't care as much about what the end-to-end delay is.

There are a number of key issues that will dominate this discussion today. One of the most important is "filling the pipe." We're talking about pipes that have very high bandwidth and relatively long latency, if only because the distances for these pipes may well be national or international. The challenge, then, is to make full use of a network connection that has a high latency times bandwidth product. I'll give you some illustrations in a moment.

I said earlier how the marketplace chooses to package QoS will be very important, and how that QoS is implemented on top of the basic switching fabric of the network will be very important. QoS can be implemented at the IP level through tools such as RSVP, or it can be implemented at the ATM level through PVCs and switched virtual circuits. Indeed, what will be the presentation of an advanced network to the campus will be critical. Will that common bearer service, the connectivity option, be IP? Will it be ATM? Perhaps we will see both and we'll be able to pick the bearer service that most fits our application needs. In any case, it is likely that when we begin to talk about applications running between academic institutions, the networking piece may become secondary to the middleware. If we're going to be paying real dollars for real bandwidth, then we're going to want to insure that we have authentication between users that may be 3,000 miles apart at separate institutions, to guarantee that the billing is consistent with the requirements. Another major issue here will be management and diagnostic tools. If we learned something from the history of the Internet to date, it is that deployment and implementation always run far ahead of the tools to manage that implementation. It would be useful if, this time around, we had some diagnostic and management tools available to us as we begin the deployment. Lastly, one of the interesting issues of the first Internet and one of the reasons that higher education was so involved in that first Internet was that the technologies that were being used nationally were the same technologies that we were using on our campus. This gave us the opportunity to participate nationally because we understood the technologies. It also gave us a rationale for participating nationally in that what we learned in that national networking arena helped to inform our approaches on the campus.

One of the greatest challenges in implementing advanced networks will be tuning operating systems and machines to fully utilize the bandwidth that we're presenting to them. If you consider a cross-country T1 link, there are only 18 KB of data in flight at any moment. When we start to look at an OC3 link running at 155 mb, suddenly there's 1.8 mb of data in that pipe. At OC48, we've suddenly got almost 30 mb of data. This has huge implications on how we manage buffers in the desktop machines, how we manage time-out mechanisms at the TCP level, how we operate flow control for our network. We need to learn a great deal about tuning machines to handle this type of pipe.

One useful construct to keep in mind as we move through these issues is to distinguish those topics which are end-to-end concerns from those which apply to the per-hop interactions. For example, we will need a comprehensive and interoperable trouble ticket system so that when our networks don't perform properly, we can diagnose the individual network that is having problems and apply remedy. Similarly, we will need a reservation policy that will allow the user to request bandwidth and Quality of Service from their desktop to the far resource that they're seeking to access. Recovery from failure should also be transparent to the user. The individual point of failure should be isolated and remedied without any intervention from the user. In these network environments, we will not have resources to consistently fragment and reassemble packets in their delivery across these multiple networks. Rather, it will be useful for us to launch a Maximum Transmission Unit discovery packet at the beginning of our connection so that we can discover what is the largest packet that can be passed end-to-end without fragmentation. We will need flow control end-to-end so that the fast sender does not overwhelm the slow receiver. Since resources will be connected to multiple networks, high performance and commodity, we will need addressing schemes that allow us to identify which entry point into the resource we wish to access.

There are a set of issues which apply to the per-hop development of connectivity. Routing will be done repeatedly throughout the network as we move from one subnet to another. Resource reservation consistent with the Quality of Service that we've selected will also be done on a per-hop basis. While we have flow control that governs end-to-end interactions, it is still possible and even likely that there will be congestion points at major crossroads within our advanced networks. We will need mechanisms to identify those congestion points and throttle traffic into them. Lastly, since we will be traversing multiple networks, we need these networks to inter-operate across a variety of vendors and a variety of equipment so that our reservation schemes can work.

Let's move now into some of the core technologies that will be used to implement advanced networking. We'll consider ATM, which is a cell-based relay service; RSVP, which is an emergent protocol for reserving bandwidth and other Quality of Service issues at the IP level; we'll look a little bit at the issues associated with sharing networks versus switching networks, and then we'll try to see if the layering approaches that we've used to date to build modular networks will scale onto high performance arenas.

Looking at ATM first, we want to consider some of the physics associated with ATM. We need to look as well at this packaging of ATM into permanent virtual circuits and switched virtual circuits. We need to look at how applications will be able to tie into the underlying ATM fabric. It's important to consider exactly what kind of performance that an end user will realize at the application level for an ATM network and then we'll make a brief survey of what's out there in the marketplace, both in terms of equipment (should you want to build your own ATM network) or commercial offerings for service and circuits of ATM.

ATM was originally developed to carry voice and, ironically, voice is one of the applications which still does not have a standard for ATM. But because it was designed to handle voice, the cell structure is very small. There are 48 byte payloads in 53 byte cells. One of the beauties of ATM, however, is while the cells are small, the speed for signaling and transmitting those cells is independent of the ATM specification itself. Unlike Ethernet, which is locked into 10 mbps, or FDDI, which is locked into 100 mbps, ATM can be run at a variety of speeds from running ATM on a packet radio at below T1 rates to very high OC48 speeds. Similarly, ATM has been adapted to transmit across multiple media. It can run on fiber, it can run on wireless transmission technologies, and there's even some work going on now on porting ATM transmission protocols onto satellite communications. ATM, having been developed largely in the commercial sector has faced some of the complexity that's associated with developing standards in the commercial sector. In particular, some of the adaptation layers and some of the mechanisms for taking specific applications, such as voice and video, and implementing them on ATM have been very slow to standardize.

ATM circuitry is composed of permanent virtual paths that are established across an ATM cloud from one site to another. Inside those permanent virtual paths, we are able to implement connections between hosts. Those connections can be done on a permanent virtual circuit, which is established by manual operations, or on a switched virtual circuit, where the user at the desktop signals that they want a brief connection through the permanent virtual path to a host on the other end of that pipe and the connection is made on an instantaneous basis.

Applications do not want to see the ATM level. They want to be running across a network where the underlying technology of that network is not visible to the application. The ATM adaptation layer, or AAL, is responsible for converting standard packet formats into ATM structure. This is a two-step process where, in the first step, an IP packet or perhaps an Ethernet packet is converted into a standard protocol data unit for use with ATM. That protocol data unit is still fairly large and now it, in turn, needs to be chopped into the small cells that constitute the ATM switching structure. There are two ways of going about that; adaptation layer 3/4 uses four bytes of payload to sequence the small cells into the larger protocol data unit. AAL 5 instead uses one bit of header to indicate whether there are more cells to follow. This places a burden on the switching fabric because we've stolen a bit of header to do this, and the headers are generally the prerogative of the switching fabric. On the other hand, we've saved four bytes of payload, and when you're beginning with only 48 bytes of payload, saving those four bytes really improves performance. The world is moving toward AAL 5, but it's not uniformly there.

There are a number of real performance issues that change how ATM is perceived from how it's marketed. The protocol overhead can be extreme, 25% or more. Again, we began with small cells. We may be stealing some of the payload of those cells for signaling information. There is precious little information per cell that is the user's compared to the network's. Secondly, since we're early in the marketplace, the throughput of desktop cards is very inconsistent. We have seen desktop cards that, while billed to run at 155 mbps, in fact can handle no more than ten to 15 mbps. The newer generations of ATM cards seem to be able to keep up with the current maximum speeds that are being marketed, of 155, but it is very likely that those desktop cards will not perform adequately as we move to higher speeds of ATM. Once the data has reached the desktop card, it needs to climb its way through the protocol stack to the application, and the middle layers of that protocol stack, such as TCP or RPC for remote procedure calls, have their own overhead associated with it. By the time the application sees the data stream, the data stream that was billed at 155 mbps is likely coming into the application at no more than 110 mbps. Furthermore, the marketplace is packaging ATM in ways that don't accommodate burst traffic. So, for example, you may sign a contract for service with an ATM vendor and they'll allow you to burst over your guaranteed bandwidth, but that burst may be limited to one or two cells and, as I indicated, one or two cells will not convey any significant part of the upper layer packets that are driving the burst. Lastly, since we do not have a great deal of experience in running large-scale ATM networks, we're not sure how to manage congestion in these environments. One tool that appears particularly promising is called Random Early Drop, where machines and switches in the network, when faced with congestion, will drop cells on a random basis; but then, having dropped one cell from a packet, will be instructed to drop all other cells from that packet, since in fact that packet cannot be reassembled at the far end. That will help us a great deal in terms of congestion control, but the user may see delays as a result.

Vendors are now bringing ATM equipment into the marketplace. The ATM equipment tends to take two formats; switches and routers. Switches can be obtained with four to eight ports per card, and those cards are slid into backplanes that can accommodate from two to 32 cards per switch. Switches come in two formats; blocking and non-blocking. In a blocking switch, there is no guarantee that the switch can handle full capacity transmissions from all ports on all cards at the same time. In the non-blocking switch, you're guaranteed that full traffic load can be handled by the backplane on that switch. Clearly, to handle that full capacity requires a faster backplane and so non-blocking switches are generally more expensive than blocking switches. The second format for ATM equipment these days is routers that want to carry their IP traffic, not across a T1 or a T3 link, but across an ATM switching fabric. In this case, we need a card inside the router that will take the IP packets, segment them into ATM cells, and then reassemble those cells back into IP packets at the router at the far end. Since segmentation and reassembly are complicated operations, requiring a significant amount of buffer space, the ATM cards in routers are fairly expensive, but relieve the CPU of the router from this overhead.

Vendors are also bringing ATM services to the marketplace. The most ubiquitous type of service is guaranteed bit rate, where one pays a flat rate to get a set bandwidth. One of the attractive aspects of this type of service is that the pricing is generally distance-insensitive, and so is very attractive if you had a number of remote sites spread out over a large area. We'd also like to have an available bit rate service, one that gives us the best possible bandwidth currently available. It is likely that available bit rate services will be priced at a much lower level than guaranteed bit rates, since vendors will be able to multiplex these available bit rate requests into their pipes. We should see those available bit rates emerge on the marketplace fairly soon. A variable bit rate, one where we pay per use and can get guaranteed bandwidth for a short period of time, is also a likely development in the marketplace, but we don't see that there yet. Lastly, ATM service tends to have a number of fixed charges per month associated with it. For example, paying a flat fee to get a port on an ATM switch, paying fees per virtual circuit, paying fees per virtual path.

ATM is not the only transmission media that we'll see out there for advanced networking. There is a great deal of interest in running IP packets directly over fiber without the intervening layer of ATM. To develop Quality of Service in such an environment, without depending upon an ATM infrastructure for Quality of Service, requires a new protocol, and there is one that's been introduced in the last year called RSVP. It includes several major components that manage the Quality of Service. A flow spec is provided by the user to shape the kinds of demands for service, the Quality of Service that the user or his application wants to see. Admission control determines whether or not that Quality of Service can be factored into the current load on the network. If that service level can be accommodated, then a reservation protocol reserves adequate bandwidth for that Quality of Service at each individual router along the way. It is instructive to compare the mechanisms for reservation of Quality of Service between RSVP and ATM. In ATM, a request for Quality of Service is initiated by the sender and is established on a per-hop basis through programming of the intermediate switches and routers. In RSVP, on the other hand, the request for Quality of Service is initiated by the receiver, and the reservation of resources in the routers is done via a soft reservation scheme that maintains those resources for brief periods of time.

One of the other major technologies being deployed now to service advanced networking requirements is moving to switched media. In shared media, we have a common bandwidth allocated by contention; that is, the various hosts requiring bandwidth signal their needs dynamically onto that media and get what they can. That's typical of traditional coaxial Ethernets and even some 10baseT. Switched environments provide dedicated bandwidth to each workstation. A switched Ethernet, for example, will give 10 mb to each workstation, not to have those 10 mb shared among multiple workstations. One other major benefit of switching is that it provides security in your environment. Most Ethernet switches will only broadcast traffic onto that switched port for machines at the end of that port, rather than all traffic going across the switch. Switches, however, have a number of drawbacks. They're significantly harder to manage. They cost more per port. You may see that the cost per Ethernet port for a shared Ethernet hub is on the order of $200 and the cost per port for switched Ethernet may be three or four times higher.

Well, there's a number of options available to us, and it's worth thinking what's going to be out there in the marketplace in two years for advanced networking. Our choices are that we can send IP packets or MAC packets across a SONET infrastructure, avoiding ATM and having that SONET framed with PPP, the point-to-point protocol, in order to provide some minimalist structure to the SONET. Alternatively, we can carry our IP traffic on an ATM fabric, and that ATM in turn will be riding on SONET, but we won't see that level. One of the attractive aspects of using the ATM fabric is that it is likely, over the next several years, that we will be able to run our voice traffic and our video traffic as native ATM applications, vs. having to deal with them with the intermediate layer of IP. ATM, as we've talked about, has very high overhead associated with it, but it is a mature technology available today. The price of the switches, the price of the desktop cards, have already started to descend into reasonable price ranges, where some of these alternative technologies, such as framed SONET have yet to be introduced into the marketplace.

You can see, then, that we can expect that we will have all of the above options in our environment. We will be running IP traffic across some type of medium access network. The frames on that medium access network will be chopped into ATM and conveyed across SONET. In other instances, we will avoid the medium access layer and take our IP packets directly into ATM using those AAL 3/4 layers or AAL 5. Alternatively, we may be running IP directly upon framed SONET, and, more than likely, we'll wind up with a very heterogeneous environment where we will be transmitting IP packets on top of ATM as well as transmitting IP packets directly onto the framed SONET layer.

Let us now move into considerations of how we will take these core technologies and link them together into an overall connectivity architecture. We'll begin by looking at the issues associated with running advanced networks from the desktop to a single point of presence within the building. That single point of presence will be on the campus backbone, and we'll start to look at issues associated with engineering high performance campus backbones. That campus backbone, in turn, will want to connect to a regional high performance switching node. Those have come to be called gigapops. We'll look at the gigapop sets of issues, and then we'll see how those gigapops relate to some of the national networking issues such as the commodity Internet, the Next Generation Internet, and a number of the mission-specific networks that are out there today.

In looking at the desktop to closet sets of issues, we have a variety of technologies available to us. We've described ATM. We've talked a bit about Switched Ethernet, where each desktop has 10 mbps to work with. It is likely that 10 mb will be sufficient for accommodating a combined video stream, a voice stream, and a data stream to the desktop simultaneously. In some instances, we will need greater bandwidth and we will likely move to FDDI and Switched FDDI, as well as running that same Token Ring FDDI protocol over a copper network, and that's called CDDI. Yet another alternative is Fast Ethernet, an Ethernet that has had the clock turned up to run at 100 mbps. There are at least two standards at this level, though the marketplace is beginning to converge on a 100bT alternative. Finally, a new product is entering the marketplace called Gigabit Ethernet, an Ethernet that will be running at gigabit per second speeds. Clearly, whenever one turns up the clock on a signaling technology, one begins to reduce the distance that that signal can be carried without noise entering into the environment.

Given this variety of technological options for connecting the desktop to the closet, how does one decide which technology to deploy? Perhaps the most significant factor is the driving distance from the closet to the desktop. Some of these technologies can be signaled at high speed for only short distances, several meters. Other technologies, such as FDDI, may have somewhat longer driving distances. The type of wiring one has between the closet and the desktop is also a significant factor. Copper can only signal at fairly low rates, whereas fiber to the desktop permits a greater variety of technological options. At the same time, the type of card that is in the user's machine is going to be a significant factor in the deployment of technology. If users order their high speed workstation and it comes equipped with an ATM card, you need to probably provide ATM service. Lastly, in trying to manage a fairly complex environment like advanced networking, it's important to consider the other technologies that are already in place, in order to insure that we have as few technologies as possible so that our support services can be focused.

Regardless of the technology that you deploy between the desktop and the closet, there will be some point of stress. Few of us have adequate vertical fiber to accommodate the high performance networking needs. Since driving distance is a factor, we often move our hubs closer to the desktop than we had previously. In this case, we need more vertical fiber to extend that campus point of presence up through the floors of the building to a closet close to the desktop. Closets themselves will have to accommodate now multiple types of equipment --the regular commodity networking on your campus, as well as the advanced networking equipment. Many closets have been constructed ad hoc, with limited power and air conditioning or cooling. It could be tight in there. Yet another factor is technical support. Your technical staff will have to maintain multiple networks. It's going to be important to make sure that those technologies between those networks are as consistent as possible. Lastly, as we alluded to earlier in terms of filling the pipe, the desktop operating system will generally not be tuned to accommodate high performance networking. You're going to have to change the buffer space allocated to TCP. You may need to change some of those transmission timers associated with that. You may need, as well, to include new parameters at the user interface level for them to be able to select the tuning appropriate to the network connections that they have.

From the closet, we want to connect the high performance network to the campus backbone. There are several options for running that campus backbone in an advanced format. Perhaps the most common is to merely over-provision the traditional technology. Instead of providing Quality of Service at that point, what we are providing is so much capacity that you can be guaranteed adequate bandwidth for your applications. Alternatively, one can take the lines that currently exist between your routers and upgrade their performance to ATM levels and build an advanced network architecture for the campus out of ATM circuits between IP routers. One can replace those routers with ATM switches and run ATM in that environment. Perhaps one is interested in keeping IP as the common bearer layer, in which case, the likely method for developing high performance on the campus will be to install RSVP in those routers so that they can accommodate Quality of Service. There are other choices that will likely be available over the next several years; for example, going to a Gigabit Ether collapsed backbone and over-provisioning service in that model.

Again, in choosing the technology to deploy for the campus advanced network backbone, several considerations come to the forefront; the embedded base, we want to try to stay consistent with the technologies that we've used to date. When it comes time to over-provision, you may be limited in what you can do. For example, if you want to run multiple backbones, you may need additional fiber between buildings in order to provide that level of service. Indeed, running dual backbones could be a challenge in terms of routing considerations as well as expense. These technologies are immature. We're at a point now where we can only speculate about what the marketplace will use in campus backbones. Lastly, money, as always, is a key determining factor in how one goes about providing an advanced networking environment on campus. In terms of the technologies we've looked at, over-provisioning may be less expensive than installing a whole second backbone, based upon ATM technologies.

Regardless of the technology utilized to build the campus advanced networking backbone, there will be common points of stress. If we take the dual backbone approach, we may well run out of fiber between our buildings. Again, with dual backbones, one needs to carefully monitor and control routing to avoid sending inappropriate traffic across one of the backbones. For those institutions that have remote campuses, providing advanced networking capability to those remote campuses will necessitate interactions with local telco providers and corresponding expenses. The single most significant point of congestion and stress in this environment is likely to be the backplanes of the routers. It is those backplanes which exchange traffic between the various cards connected to subnets. Because of that congestion point, because of the inabilities of those backplanes to handle large numbers of packets, there is a premium on engineering the network so that subnets which are likely to exchange traffic exchange, keeping that traffic on a single card, versus having to go across the congested backplane.

In any case, we have now built a high performance campus network. It is likely that the purpose in building that advanced network was to connect to external networks for collegial work between campuses. Let us now turn our attention to how do we connect this campus advanced network into a larger mesh.

Today, there are several high performance national backbones out there. Most are run by federal agencies to serve a particular mission. The vBNS is operated by the National Science Foundation to support general scientific and scholarly work. Internet 2 is an endeavor being developed by numerous universities in this country to provide a national fabric that accommodates Quality of Service. The Next Generation Internet, or NGI, is a larger entity that will likely embrace Internet 2. The NGI has been commissioned by the current administration to provide investigation and development of the next generation of networking technologies so that the United States can keep its competitive advantage in this very important area. Two particular mission networks that are beginning to extend themselves are the NASA Science Internet and the ESNet, which is run by the Department of Energy.

Most of these national networks are being run utilizing ATM technology. There is very little pure SONET out there today. The national backbones are being provided by MCI and by Sprint, and, as a result of ongoing negotiations between the mission agencies and higher ed, it is likely that the mission networks will not touch down directly onto campuses; but instead will connect to the gigapops that represent a regional drainage point for advanced networking. What will dominate out there in the long term for national backbone architectures of advanced networking? It's hard to say. The key parameters are going to be driven by the marketplace. Many of the vendors are looking to make sure that the advanced networking technologies will segue smoothly into the commercial Internet over the next few years. Similarly, the direction that federal agencies take in regards to how they connect researchers on the campuses to agency laboratories will set a direction for national backbone architectures.

These national backbones will not connect directly to the campuses. Rather, they will connect to regional aggregation points called gigapops. These gigapops will also serve as Internet malls where commodity Internet service providers will connect and, in turn, drain the commodity traffic from the campus. Gigapops are likely to offer both IP services and ATM connectivity so that we can begin to conduct end-to-end ATM experiments across these national backbones. Lastly, gigapops may include storage options; options that would allow us to locate video servers and caching information servers within the gigapop. Typically, a video server for distance education needs to broadcast its information, not back to the campus, but out to the larger community. In that case, it makes less sense to congest the campus drainage point to the gigapop, and instead locate the video server at the gigapop for dissemination outward from there.

Today, the national advanced network for higher education is the vBNS. To connect to the vBNS, one needs to provide a proposal to the National Science Foundation. Guidelines for that proposal and time frames for submission of those proposals can be found on the NSF website. NSF will provide fixed funding to a campus to connect to the vBNS. Today, those funding levels are on the order of $350,000, spread out over two years. That leaves a significant financial responsibility to the campus. In some instances, campuses that are fairly close to the gigapop may find that the cost of connectivity between the campus and the gigapop are on the order of $50,000 to $100,000 per year. Campuses that are distant from gigapops may face funding requirements of more than one and a half million dollars per year to run an OC3 link between their campus and the nearest gigapop.

Once one has connected to the vBNS, there are a number of real issues that are still somewhat unresolved. We want to run our traffic on a single link between our campus and our gigapop, and yet that single link is going to accommodate multiple services; vBNS connectivity, mission network connectivity, commodity Internet connectivity, among others. How one chooses to allocate that bandwidth on the link between the campus and the gigapop is an open issue. Routing is also a challenge today, that the routing on the vBNS -- the routing from the campus to the vBNS is far more of an art than a science. And there are instances today where the end-to-end desktop connectivity between workstations separated by a continent, running across 155 mb, is still at the Ethernet speed levels. We suspect that this is due to routing problems more than anything else. We do not have a scalable mechanism today for adding PVCs, and a permanent virtual circuit may take upwards of two weeks to engineer into the national fabric. One last challenge that we have to consider in this environment is tuning that desktop workstation to adequately utilize the bandwidth that's available. Researchers who connect to the vBNS anticipate high performance. Delivering that high performance is not easy.

One last function for the gigapops will be to store information that is either destined for external constituencies or information that is frequently accessed and whose network delay in accessing it at the original site is prohibitive. In such instances, we will want to locate servers inside the gigapop, and we will want those servers to be both providing video streams to distance education locations around the country; and the same servers should be responsible for caching information which is frequently accessed by on campus users.

Advanced networking today has a number of significant open issues. It is not clear that routing can perform at high levels of packets per second, and so other technologies are being investigated. One such technology is Tag Switching, which identifies a stream of data entering the advanced network, removes the headers and replaces it with a tag that will allow those packets to be switched at high speed. Alternatively, fast IP switching will look at the source and destination addresses inside each packet, much as routing does today, but will not wait for error correction to determine if the packet has arrived fully intact and correct before making the decision to route that packet forward into the mesh. One of the biggest open issues is how will Quality of Service be implemented? Will it be done on ATM or will it be done at a higher level at the IP level, using RSVP? We are still quite uncertain about what the bitpath technologies will be in the campus environment and from the campus point of presence within a building to the desktop. Can we build routers that work at high performance? The code name for such routers are BFRs, standing for Big Fast Routers -- or something else. Can those be constructed? Will we develop trouble ticket systems that can inter-operate so that if a Quality of Service request is refused by the network, we will know which hop did not have adequate capacity and be able to engineer around that. Another major issue, for campuses in particular, will be managing Quality of Service. Whereas in the residential marketplace, Quality of Service will be introduced on a pay-per-service level, for universities, Quality of Service will likely be a fixed commodity purchased at flat rate from the external networking environment; and then Quality of Service coupons will be distributed around campus. How will we do that? It has been often commented that Quality of Service on a campus looks a lot like time sharing from 25 years ago, only spread out on a distributed fabric. Lastly, how will we build the hooks into the operating systems so that the tuning parameters can be adjusted depending on whether one is going on a high performance link to another research site, or the timers and buffers changed to accommodate low performance networking going out across the Commodity Internet.

In closing, it is apparent that the fusion of computing and data communications has created a particularly powerful technology. That technology is still in its early stages, and faces several more stages of maturation. Advanced networking represents the leading edge of that maturation process. It is clear that we face great uncertainty and great excitement in the years ahead.

About CREN

© CREN, 1999

Contact Us