NRS: Documentation

Richard Smith

Table of Contents

1. Introduction

2. Features

3. Compatibility

3.1. Hardware
3.2. Software

4. Requirements

5. Tutorial

5.1. Prerequisites
5.2. Configuration
5.3. FreeBSD/ALTQ
5.4. Launching with Ant
5.5. Launching without Ant

6. User authentication

7. Writing a router control module

7.1. activateSLS
7.2. deactivateSLS
7.3. getInterface

8. Using the client APIs

9. Demonstration

10. Bugs

11. Planned features (TODO list)

12. Components

Network sites A and B, located on opposite sides of the world, are connected by a high speed backbone core network. They use this network for transferring large files (or perhaps streaming real-time video). The admins of both sites have configured their edge routers to mark a certain percentage of inter-site traffic with DIFFSERV EF tags. DIFFSERV aware routers will give priority to these packets. Assuming the core network is over-provisioned (at least with respect to DIFFSERV), the sites can be assured their EF packets will not be dropped when there is congestion. They have a virtual pipe of a guaranteed capacity.

But how do they manage this capacity?

The size of the EF pipe is set to 10 Mbps. Two users at site A both wish to transfer large files to site B. But they are not aware of these other. They both transfer the files at 10 Mbps. The router cannot mark 20 Mbps of traffic as EF because this would violate the agreement with site B. Therefore half the traffic does not get marked. The users do not get any guarantee of how much bandwidth they will receive, or even that they will receive half each.

How can we avoid this situation? The admin at site A could chose not to enable EF marking automatically. Instead, users would be required to contact him and make a request to use the EF capacity. They would send him a specification of the flow and the amount of bandwidth they require and he would check that the bandwidth was available. He would then telephone his counterpart at site B and confirm that the bandwidth is available at the other end also. Finally he would configure the router to identify packets from this flow and mark them as EF.

This solution does not scale to large numbers of users, and does not allow bandwidth to be easily reserved ahead of time. So, we automate it. The job of taking bookings, checking bandwidth availability at both sites, and configuring the router at the appropriate time is done by NRS.

2. Features

Back-end supports Cisco routers, but it is not difficult to write your own router control module to configure a different brand of router. Linux routers are no longer supported. We have just added support for FreeBSD ALTQ routers but this has not yet been extensively tested.

3. Compatibility

NRS is only supported if you run it with the hardware and software listed here.

3.1. Hardware

The only supported dedicated router is currently Cisco 7609. But you can use any machine that has a Java VM to run the NRSE.

We have now added support for x86 PCs running FreeBSD acting as routers using ALTQ.

3.2. Software

Java 1.4.1
Apache Ant 1.6.1
PostgreSQL 7.3.5
JDOM Beta 9 (included in distribution)
The NRS library bundle version 2 (included in distribution)

4. Requirements

NRS requires several libraries, but we have included them all in the distribution. You need only install the following packages:

Java 1.4
Ant 1.6
PostgreSQL 7.3

If you have JUnit installed and configured to work with Ant then you may run the unit tests (Ant target 'test') which may help to diagnose problems. From version 2.1, if you only wish to run pre-compiled binaries then you no longer need Ant.

5. Tutorial

Figure 2. Simplified MB-NG network

NRS will scale to many sites with many users, but we will begin with a simple two site set-up similar to the one described in the introduction. This diagram shows a simplified version of the MB-NG network we used during development. This tutorial explains how we configured NRS for MB-NG; you should be able to adapt it to work on your network.

We will use one Cisco router and one Unix PC. You may of course run the database, the NRSE and the Activator daemons on different machines, and you will probably have a number of machines running the client software, but for this tutorial will run them all on the same machine.

5.1. Prerequisites

First, download and install Java, Ant and PostgreSQL. How you do this depends on your operating system. For example, on Redhat Linux 8.0:

rpm -Uvh ftp://ftp2.uk.postgresql.org/sites/ftp.postgresql.org/\
binary/v7.3.4/RPMS/redhat-8.0/postgresql-7.3.4-1PGDG.i386.rpm
rpm -Uvh ftp://ftp2.uk.postgresql.org/sites/ftp.postgresql.org/\
binary/v7.3.4/RPMS/redhat-8.0/postgresql-server-7.3.4-1PGDG.i386.rpm
wget http://apache.rmplc.co.uk/dist/ant/binaries/apache-ant-1.6.1-bin.tar.bz2
tar apache-ant-1.6.1-bin.tar.bz2
export PATH=$PATH:$HOME/apache-ant-1.6.1/bin

Initialise the PostgreSQL database:

mkdir /var/lib/pgsql
chown postgres /var/lib/pgsql
su postgres -c 'initdb -D /var/lib/pgsql/data/'

Now we configure the PostgreSQL server to accept TCP connections. Edit /var/lib/pgsql/data/postgresql.conf and add this line:

tcpip_socket = true

Then we start PostgreSQL running:

/etc/init.d/postgresql start

And create a PostgreSQL user and database.

su postgres -c 'createuser root'
y
y
createdb

5.2. Configuration

Now we must configure NRS. There are three configuration files. All components (including client software) read their configuration from nrse.properties and log4j.properties. The file ciscorouter.properties contains options that are specific to the Cisco router configuration. We store these files in the data directory, which Ant automatically includes in the classpath. You can move the files to any location in your Java classpath if you want. Options that are not mentioned can be left at their default setting. For full descriptions of all the options refer to the example configuration files supplied.

If you are familiar with Log4j you may configure advanced logging options in log4j.properties. You will probably want to change the first line to simply

log4j.rootLogger=info, stdout

which hides the debugging output and displays informative messages on the console.

The default nrse.properties contains our unit test configuration. This tutorial assumes you are starting with that. There are some other example properties files in the data directory that will show you real world configurations.

The server option is currently only used to specify the address of the site's NRSE to give the client program a default NRSE to use. For convenience, we set it to the address of the UCL NRSE, which is 128.40.89.250

localNet is the subnet of our site. As you can see, UCL has multiple subnets, but all the machines that will be using NRS service are on 195.194.14.0/25, so this is the value we use.

server = 128.40.89.250
localNet = 195.194.14.0/25

We are using a PostgreSQL database that is running on the same machine (i.e. localhost) as the NRSE. We aren't concerned about security here (we have a firewall preventing any other hosts from connecting to the database) so we use the root PostgreSQL account that we created earlier with no password. The database itself it also called root. (The reason for this naming scheme is that we sometimes use the pgsql command to examine the database, and this command defaults to using the name of the logged in user for both the PostgreSQL username and PostgreSQL database name. If you spend most of your time logged in as root, then it makes sense to name the PostgreSQL user and database 'root' also.)

JDBCconnection = jdbc:postgresql://localhost/root
JDBCuser = root
JDBCpassword =
DBMS = PostgreSQL

Now we configure the virtual interfaces. The NRSE maps a number of 'virtual' interfaces onto each real NIC in the router. Typically there is one 'inbound' VI and one 'outbound' VI for each real interface. However, if you wish to offer multiple DIFFSERV code-points, then each code-point will need its own VI. So a router with two interfaces, offering EF and AF reservations will need 2*2*2=8 VIs.

Our router only offers EF. It has two real interfaces, one to the LAN and one to the WAN. Therefore we need to configure four VIs: two inbound, two outbound. Each VI will support 1000 000 Kbps of EF traffic. The names must match the names used by the router.

noOfIface = 4

bandwidth0 = 1000000
ifaceName0 = GigabitEthernet4/1
ifaceDirection0 = in

bandwidth1 = 1000000
ifaceName1 = GigabitEthernet4/1
ifaceDirection1 = out

bandwidth2 = 1000000
ifaceName2 = GigabitEthernet4/2
ifaceDirection2 = in

bandwidth3 = 1000000
ifaceName3 = GigabitEthernet4/2
ifaceDirection3 = out

Next, the remote NRSEs table. The NRSE maintains a table listing the other NRSEs it may communicate with. Currently this information cannot be auto-discovered and must be hand configured here. (We sometimes include an entry for the local NRSE, as that allows us to simply duplicate the same table on every site, but doing that is officially not-guaranteed-to-work.)

The network diagram only shows one remote NRSE, at Manchester, so we configure our (UCL) NRSE as follows, telling it the subnet which the Manchester NRSE is managing:

noOfNRSEs = 1
remoteNRSEserver0 = 195.194.15.18
remoteNRSEnetwork0 = 195.194.15.0/25

Finally there are a couple of options that are only for use in our tests. For a Cisco system you must set them thus:

useRemote = true;
operatingSystem = Cisco
routerConfigFile = data/ciscorouter.properties

The file data/ciscorouter.properties should look like this:

ADDRESS=195.194.14.1
PROMPT=ucl-bndry
LOG-PWD=password1
EN_PWD=password2

As you can see, it contains the IP address of your router, the IOS prompt expected from the router, and two passwords.

If you wish to use a differently named config file, you can edit build.xml to supply the name as a command line argument to NRSE. Do not specify a path - the Java classpath will be searched. On our systems we leave the default nrse.properties configuration as the unit tests expect it to be (if you alter the default config you may find unit tests will not work), and we create differently named configuration files for each site, to avoid confusion.

To actually use NRS will require you to perform a similar configuration at a remote site. (On the MB-NG network we used Manchester.)

5.3. FreeBSD/ALTQ

You need to be running a FreeBSD router with ALTQ. This means you patch your kernel source code with the ALTQ patch and recompile your kernel. Consult the FreeBSD handbook for instructions. You must have the ALTQ userland tools (i.e. the daemon) installed. However, the ALTQ daemon has a small bug. We have included a fix for this in the altq.patch file in the NRS distribution. Apply this patch before compiling and installing the daemon.

Set your NRS config with this option:

operatingSystem = FreeBSD

When you run the NRSE, the ALTQ daemon will be automatically launched. You must not run it manually. You should create an altq.conf file as explained in the ALTQ docs. You should use class based queing to create a premium class on each interface and a filter to automatically put all outbound EF marked packets into this class. It might look something like this:

interface sis2 bandwidth 100M cbq
class cbq sis2 root_class NULL pbandwidth 100
class cbq sis2 def_class root_class borrow pbandwidth 90 default
class cbq sis2 ef_class root_class pbandwidth 10 priority 7
        filter sis2 ef_class 0 0 0 0 0 tos 0xb8

This example is for a 100mbit link and it allocates 10mbit to EF traffic. NRS will take care of actually marking the traffic. When a reservation begins, NRS will create a filter to pass packets in the reserved stream to a conditioner (which sits on the inbound interface) which will police them, either marking them as EF or dropping them. Those marked as EF will then get priority in the outbound queue.

Our ALTQ module has not been extensively tested so please report if it works and what ALTQ configuration you used. We hope to provide more detailed examples of configuring ALTQ after we have completed more testing.

5.4. Launching with Ant

Ensure PostgreSQL is already running. Before launching the NRSE for the first time you must initialise the database.

ant wipe

Ignore any errors the first time you run the command. You may repeat this command any time you wish to erase all the reservations.

Then launch the NRSE.

ant nrse

In another terminal, launch the Router Controller. If you just wish to play with NRS without actually reconfiguring your router, then skip this step. You may run this application on a different machine from the NRSE if you wish (be sure to use the same configuration files).

ant activator

Finally launch the client application which will allow you to make reservations. You may run this application on any machine.

ant client

To see how you might use this application, watch the demo movie.

5.5. Launching without Ant

NRS now includes a pre-compiled binary (NRS.jar) which can be launched without Ant. These are the launch scripts:

bin/nrswipe
bin/nrse
bin/nrsrc
bin/nrsclient

6. User authentication

NRS users can now be authenticated using RSA keys in X.509 certificates. If you enter the credentials 'testcredentials' then authentication will be bypassed. This tutorial explains how to use authentication.

First you must use Sun's keytool to generate an X.509 certificate and add it to your keyring (the .keystore file in your home directory). It must use the RSA algorithm. If you do not specify an alias it will default to the alias mykey.

keytool -keyalg RSA -storepass testtest -genkey -alias mykey

Now we must add a user to the database. Run the graphical client program (and be sure your nrse.properties specifies the database that your NRSE is using, because this operation access the database directory). Press Add new user account. A new window opens. Type the desired username into the User. Press Import X.509 RSA cert. You will be asked for keystore filename, password and the alias of the key you wish to import. The defaults are often correct. Press OK and the key will be imported. Finally press Add user to add the username and key to the NRSE's database.

You will also want to configure the client to use your key. Do not close the Add User window yet. First, copy and past the contents of the RSA private key and the User fields into the corresponding fields of the main window. Your key will now be used to authenticate all operations. If you have problems remember to check the username and key details, and if all else fails return to using testcredentials to disable authentication.

7. Writing a router control module

NRS users are encouraged to extend the software by writing their own router control modules. Assuming the user knows how to manually configure his edge router's scheduling mechanism to perform Diffserv marking and policing then it is very easy to automate the configuration by writing an NRS module.

A router control module is simply a Java class which implements the simple Router interface, as follows.

public interface Router{
  public void activateSLS(SLS s)
              throws IOException;
  public void deactivateSLS(SLS s)
              throws IOException;
  public int getInterface(
                String dest,
                int direction,
                String src,
                String qualityClass
              ) throws IOException;
}

The best way to learn how to implement this is to read the code of CiscoRouter.java, but the following comments should help.

7.1. activateSLS

The NRSE calls activateSLS() whenever it is time to put in place a reservation. The details of the reservation are passed to your method in the SLS object. You then use methods such as SLS.getPeakRate() to find the details of the traffic flow. You configure your router to begin marking packets that match this flow specification as Diffserv EF (or whatever code point you have selected). The SLS object contains enough parameters to presicely describe a token bucket, but you may not be able to use all of them, depending on the capabilities of your router.

There are two basic approaches of how to configure the router. The simplest is to run the router controller process on the router itself. We used this approach when experimenting with Linux routers. Our Linux router module runs Linux's tc command to configure the scheduler of the local kernel on the fly, using parameters extracted from the SLS. This was easy to implement, but when we later deployed on Cisco routers we could not run a process on the router because Cisco routers do not have a Java virtual machine available. So instead we run the controller on a Linux PC and use a Java implementation of the telnet protocol to connect to the router and enter the Cisco IOS commands required.

7.2. deactivateSLS

deactivateSLS() is called when a reservation expires. It undoes all the router configuration done by activateSLS(). If there are a lot of commands to be generated, it may be simpler to generate the removal commands at the same time as the activation commands and store them in the database, then simply pull them out when deactivateSLS is called, rather than generate them again.

7.3. getInterface

The NRSE manages an edge router, which is multihomed, i.e. has two or more network interfaces. Each SLS processed by the NRSE describes a flow of traffic, of which member packets are identified by source IP address, a sink IP address, a protocol and port number. The NRSE is configured by the administrator with details of all the network interfaces of the router which is it managing, including how much bandwidth may be allocated to each class of traffic on each interface. Therefore to perform admission control the NRSE needs to know which network interface will be used by each flow.

The network interfaces should all be on the border of the NRSE's domain, but do not necessarily have to be on the same edge router. In our implementation the NRSE only manages a single router, but a module could be written that supports multiple routers. The NRSE does not need to know how its orders are translated into hardware instructions, or whether they are going to a single hardware device or several.

getInterface() is called by the NRSE to determine which network interface a flow will use. It is supplied with a source IP, a destination IP and the name of the traffic class to be used. It returns a reference to one of the NRSE 'virtual interfaces' as specified in the NRSE's configuration file. Most implementations of this can copy the code found in CiscoRouter class, provided they re-implement the private method getInterfaceName(). This internal method must be implemented to query the router's routing table to return the name of the interface that will be used for packets forwarded to a given destination IP. This name is entirely dependent on the conventions used by the routing hardware. NRS will map this internal name onto its own 'virtual interface' using the mapping given in the configuration file.

8. Using the client APIs

Here we explain how to write an NRS client, or add NRS client functionality to an existing application. We assume that you have all NRS classes and support libraries in your class path. (The easiest way to do this is probably to run

ant dist

to package up the NRS classes into lib/NRS.jar, and then add NRS.jar and everything else you find in the lib directory to your classpath.) You also need a config file somewhere in your classpath. Just copy and edit data/nrse.properties. You can ignore most of its parameters (which are only used by NRS servers, not clients) but you should set port, server and perhaps localIP parameters. The comments in the file explain these.

There are currently two APIs. First, the very high level, easy to use one. This makes a simple reservation suitable for an FTP file transfer. You specifiy the direction of the transfer (i.e. downloading with GET or uploading with PUT), the name of the server, the size of the file being transfered (bytes), the amount of bandwidth to reserve (kbps) and the port number being used at the local end of the TCP connection. The reservation will begin immediately and the duration is automatically calculated. It also automatically makes a reservation in the reverse direction for ack packets. If anything goes wrong, it will throw a ReservationNotMetException

Here is a simple example:

import client.ClientLib;
...
ClientLib.reserveFTP(ClientLib.GET,"ftp.example.com",1274246,100000,35466);

Alternatively, using the Client class gives you much more control over the reservation. Here is an example of how to make a reservation:

import client.*;
import nrse.*;
import util.*;
...
Config config = new Config();
Client client = new Client("127.0.0.1", config);
Request request = new SLS( ... );
Response response = client.sendRequest(request);
if(!(response instanceof SuccessResponse)
    System.out.println("Error:"+"response.getMessage());

The IP address should be that of your local NRSE. For details of the available Response classes and so forth you should consult the JavaDoc. The only complicated part, which has been omitted from the preceeding listing, is the parameters used to create the SLS. You can give the constructor a long list of parameters, but instead we usually construct our SLS objects from XML documents. For example, take a look at the file data/test.xml. It has comments to explain the fields. To construct an SLS object based on this data, we would load the file as an InputSource Stream, like this:

new SLS(new org.xml.sax.InputSource(MyClass.class.getResourceAsStream("/test.xml")))

If you would rather supply the list of parameters, you will find the parameters and their format are very similiar to the XML version.

Removing a reservation is easy:

Request request = new DeletionRequest(14);
client.sendRequest(request);

However, this assumes you already know the ID number of the reservation you wish to remove (in case, reservation ID 14). To find this, you need to query the NRSE.

Request request = new QueryRequest("testuser","testcredentials");
Response response = client.sendRequest(request);
QueryResults results;
if(response instanceof QueryResults) results = (QueryResults)response
Iterator iterator = results.getIterator();
while(iterator.hasNext()){
   SLS sls = iterator.next();
   System.out.println(sls);
}

Currently we only support simple queries. This query will return all reservations made by testuser. If the username is admin it will return reservations made by all users.

If you are using security, then it will not be sufficient to enter the credentials as "testcredentials". Instead, you must create a unique signature for each request before you send it, like this:

request.sign("Put your BASE64 encoded X.509 RSA private key here!")

Modifying requests is simpler than making them, because you only need to specify the fields you want to be altered:

ModificationRequest mod = new ModificationRequest(14);
mod.setSourcePort(15789);
client.sendRequest(mod);

This example modifies request 14 to have a source port of 15789. You may specify more than one paramater to be modified simulataneously. Not all paramaters are modifable - see the source code or JavaDoc for the full list.

9. Demonstration

Here is a 'script' to explain in detail what is going on in the video clip DEMO-clip5-mac-mpeg4.mov. Also refer to the diagram 'mbng1.png'.

Time indices refer to the clock-time displayed in the video.

14:53:32: Initially we see two flows of traffic from London to Manchester, one TCP, one UDP.
14:53:52: Then we introduce background traffic which swamps them both (although it takes a few seconds for all the background flows to come on line), with the TCP being reduced to almost nothing.
14:54:11: To remedy this, we load the NRS client application and make a new reservation of 100Mbps for the TCP flow. We also ask for 10% of this (10Mbps) to be reserved in the opposite direction for the TCP acknowledgments.
14:54:30: The client sends these requests to the London NRSE which confirms with the Manchester NRSE that the bandwidth is available
14:54:50: We see in the terminal windows that the NRSEs are issuing notifications 10 seconds in advance of the pending reservation.
14:55:00: Then, at the requested time (watch the clock), the NRSEs activate the reservation and configure the routers to mark the TCP flow with DIFFSERV EF priority. (The first shell window shows the debug output from the London NRSE, the second from the Manchester NRSE.) Immediately the graph shows that the TCP flow has gone from zero to 100Mbps.
14:55:18: Next we attempt to make a 950Mbps reservation, this time for the UDP flow.
14:55:55: However, the NRSEs are only configured to allow 1000Mbps total of EF traffic. This request would bring the total to 1050Mbps, so the request is denied.
14:56:11: We reduce the request to 900Mbps and it succeeds. The clock has already passed the requested activation time, so the reservation is activated immediately. As expected, the graph shows the UDP flow increase to 900Mbps. Note that this does not affect the TCP flow which remains constant - only the background traffic is dropped.
14:56:37: We then modify the reservations while they are in progress. We reduce the UDP reservation to 500Mbps, which gives us enough free EF capacity to increase the TCP reservation to 500Mbps.
14:57:12: We do increase the TCP reservation to 500Mbps. As expected, the graph shows the two flows equalise to 500Mbps each. The TCP rate zig-zags because of TCP's flow control (we are not sampling at a rate high enough to see the actual slow-start recoveries) but the UDP flow is a constant 500Mbps.
14:58:00: Finally, the reservations expire, and the NRSEs deactivate them. Traffic flow returns to how it was before the reservations were made.
14:58:19: Demo ends

10. Bugs

If you make a reservation from site A to site B using the NRSE at site C you can only delete or modify it from site C. This will be fixed when we implement global ID numbers.
Don't try to delete a reservation that is currently in progress. Behaviour is not yet defined.
Sometimes there are warnings from PostgreSQL about dropped connections. This could be because we are using a different version of the JDBC driver. It could also be that 'Router' objects are being constructed and deleted needlessly - we should fix RouterFactory so it always returns the same object.
Cryptix library sometimes does not work if you have Sun's jce.jar in your java libs directory. If you get SecurityExceptions, remove jce.jar. (This *seems* to be because we have not signed our library jar. So you could also fix it by using the original signed jar files from cryptix.org rather than our library bundle.)

11. Planned features (TODO list)

Use Sourceforge or setup Bugzilla so we can easily track bugs and feature plans.

12. Components

NRSE (Network Resource Scheduling Entity): The daemon which performs admission control for a DIFFSERV enabled network. Supports advance reservations, two-tier authentication and peer-to-peer operation with daemons at remote sites.
Router Controller: The daemon which queries the database for reservations and configures the router using the SLS of the reservation at the appropriate time, and unconfigures it when the reservation is over.
Cisco Router Control Module: The module used by the Router Controller to translate the XML SLS into Cisco IOS commands.Will telnet into router and run necessary commands to add/modify/remove reservations. Written by Nicola Pezzi.
Linux TC Control Module: An alternative router control module for use with Linux routers. Partially completed but not currently being developed.
FreeBSD ALTQ Control Module: An alternative router control module for use with FreeBSD routers. Not extensively tested.
Client Library: General and FTP-specific APIs are provided. Applications can use these to automatically book reservations as needed. The Booking Client and the FTP Client are examples of such applications.
Booking Client: General purpose client can be used for booking, modifying and deleting reservations. Use this to book a reservation if the application you are using does not support booking reservations.
FTP Client: A NRS-enabled command-line FTP client. Can automatically make reservations on demand to guarantee file transfer rate.