MBONE: Multicasting Tomorrow's Internet

Chapter 11: Accounting and Debugging Issues

The MBONE has many users, and more and more people are Tbeginning to use it every day. As the number of MBONE users increases, an important question comes to mind -- in the end, who pays for all of this new technology?

To answer that question, you first have to understand what the MBONE is. The MBONE is a research project. All research is funded by entities like governments, universities, and private corporations, and the MBONE is no exception. This research includes the protocols that are used on the MBONE, the tools and applications that have been developed for the MBONE, and everything in the global MBONE virtual network (multicasting, encapsulation, and so on).

The funding also comes from the MBONE end users in one form or another. Users (which may consist of an entire site full of computer operators) pay an Internet provider for access to a number of services, which may include an MBONE tunnel or multicast routing. This Internet provider, in turn, pays a network provider for the use of its physical links. These physical links most often belong to telecommunications companies such as Sprint or MCI. A very small number of Internet providers own their physical links, but in the end, they still have to pay to have access to other sites on physical links that belong to the telecommunications companies.

Telecommunications companies seem to be the big winners in this scenario, but these companies have to spend a fortune to install all the cabling that goes from one end of the country to the other end. They certainly make a profit by owning this equipment, but because the Internet is constantly expanding, they also have to keep upgrading these links, and the maintenance it takes to perform this task is very expensive. (So no one really wins, except perhaps the governments who tax a packet of data traveling the networks several times before it reaches its destination. Such is the case in Canada, because the government taxes these services there.)

So, if you are paying for the Internet and MBONE, you may want to know how much your users access the MBONE so that you can charge the costs back to them and become part of this virtual food chain.

There are two ways that your users can conveniently use the MBONE:

  1. You can give them a multicast feed to enable them to participate in the sessions they choose.
  2. Using a feed that enters your site, all your internal users or departments participate in the sessions they want.
With a multicast feed, you receive a feed from somewhere and provide feeds to other entities, such as departments, faculties, and so on.

People in these other entities use the feed that you provide to participate in all the sessions that they want. However, the number of sessions in which they are able to participate depends heavily on the bandwidth constraints that you impose on the feed. For example, in a tunnel line in the mrouted.conf file, the maximum bandwidth that a link is restricted to, by default, is 512 Kbps (see the rate column in Table 11-1). This bandwidth can be changed with the rate_limit option on the tunnel line, so you can use that option to implement various levels of service.

How do you know what sessions users participated in or how much data they sent? A partial answer can be obtained by using UNIX utilities. UNIX comes with a great deal of networking utilities, and some of them are useful for getting networking statistics on multicast traffic. First, take a look at netstat.

netstat is a utility that can provide you with unicast statistics. It can show you the routing tables on your machine, for example, or it can show you the usage of your network memory buffers. On some platforms ( FreeBSD is one of them), netstat can give you multicast statistics as well. The netstat -g command (as shown in Table 11-1) gives you all kind of useful information.

Table 11-1: Output of the netstat-g Command

Virtual Interface Table
Vif	Thresh	Rate	Local-Address	Remote-Address	Pkts-In	Pkts-Out
0	1	0	123.456.78.90	0	0
1	16	512	123.456.78.90	123.456.210.1	12406	112515353
2	8	512	123.456.78.90	123.456.3.4	5596	44642609
3	8	512	123.456.78.90	123.456.141.16	6	56865205
4	8	512	123.456.78.90	123.456.186.10	0	9664355
5	16	512	123.456.78.90	123.457.90.90	0	0
6	64	512	123.456.78.90	130.130.130.130	117148123	183497
7	8	512	123.456.78.90	123.456.101.182	0	0
8	32	512	123.456.78.90	124.222.4.3	149934	15749310
9	32	512	123.456.78.90	129.111.3.18	23821	5056246

Multicast Forwarding Cache
Origin		Group		Packets	In-Vif	Out-Vifs:Ttls
130.240.3.81	224.2.195.52	39771	6	1:16 4:8
130.240.3.76	224.2.213.97	32600	6	1:16 3:8 4:8
192.153.117.35	224.2.252.231	357	6	1:16 4:8
128.138.213.1	224.2.0.1	75657	6	1:16 4:8
198.93.139.41	224.2.252.231	50856	6	1:16 4:8
128.55.192.227	224.2.143.24	56682	6	1:16 3:8 4:8
128.232.0.49	224.0.1.1	7747	6	1:16 3:8 4:8 9:32
146.88.1.106	224.2.245.77	716	6	1:16 4:8
128.40.64.5	224.2.244.129	72840	6	1:16 4:8
130.240.12.79	224.2.187.164	8381	6	1:16 4:8
130.199.130.175	224.2.0.1	11	6	1:16 4:8
192.16.123.88	224.2.213.97	98957	6	1:16 3:8 4:8
128.100.197.178	224.0.1.1	1	1	4:8 6:64
171.69.58.88	224.2.231.173	45701	6	1:16 3:8 4:8
131.188.34.89	224.2.0.1	81533	6	1:16 4:8
171.69.58.92	224.2.143.24	14998	6	1:16 3:8 4:8
204.188.121.18	224.2.127.255	11	6	1:16 4:8
128.3.112.2	224.2.0.1	63309	6	1:16 4:8
129.89.9.110	224.2.187.164	14420	6	1:16 2:8 3:8 4:8 9:32
129.240.200.196	224.2.236.244	80710	6	1:16 4:8
128.4.1.1	224.0.1.1	5513	6	1:16 3:8 4:8
204.62.246.98	224.2.248.110	131170	6	1:16 4:8
193.60.11.44	224.2.143.24	191421	6	1:16 3:8 4:8
128.84.247.156	224.2.0.1	58315	6	1:16 3:8 4:8
128.3.112.2	224.2.127.255	15	6	1:16 4:8
204.62.246.73	224.2.143.24	6814956	6	1:16 3:8 4:8
131.225.220.147	224.2.0.1	98359	6	1:16 3:8 4:8
131.225.220.147	224.2.231.173	299682	6	1:16 3:8 4:8
130.199.130.175	224.2.1.1	10651	6	1:16 4:8
130.240.3.81	224.2.219.172	66366	6	1:16 3:8 4:8
130.240.3.81	224.2.177.132	16363	6	1:16 4:8
199.104.80.5	224.2.248.110	1199	6	1:16 4:8
128.232.0.56	224.0.1.1	8709	6	1:16 3:8 4:8 9:32
18.77.1.243	224.2.251.237	197837	6	1:16 4:8
192.6.28.26	224.2.0.1	26537	6	1:16 4:8
129.242.6.80	224.2.189.16	73074	6	1:16 3:8 4:8
171.69.60.189	224.2.231.173	30666	6	1:16 3:8 4:8
13.2.116.196	224.2.127.255	153	6	1:16 3:8 4:8 8:32 9:32
130.199.130.175	224.2.187.164	4	6	1:16 4:8
130.240.3.38	224.2.213.97	32596	6	1:16 3:8 4:8
163.221.196.12	224.2.0.1	98133	6	1:16 4:8
171.68.225.139	224.2.0.1	120	6	1:16 4:8
150.83.68.41	224.2.143.24	75111	6	1:16 4:8
130.240.3.81	224.2.236.244	39975	6	1:16 3:8 4:8
171.69.129.220	224.2.0.1	14937	6	1:16 3:8 4:8
130.240.3.81	224.2.189.16	25754	6	1:16 4:8
171.69.60.152	224.2.231.173	45353	6	1:16 3:8 4:8
156.40.113.11	224.2.143.24	210	6	1:16 2:8 3:8 4:8 8:32 9:32
171.69.129.220	224.2.231.173	45548	6	1:16 3:8 4:8
129.242.6.80	224.2.177.132	37541	6	1:16 3:8 4:8
171.69.58.81	224.2.231.173	45643	6	1:16 3:8 4:8
171.69.58.81	224.2.0.1	15031	6	1:16 3:8 4:8
204.62.246.68	224.2.127.255	3787	6	1:16 4:8
198.93.139.41	224.2.187.164	50838	6	1:16 4:8
192.80.13.56	224.2.0.1	1880	6	1:16 3:84:8
128.182.61.159	224.2.0.1	138180	6	1:16 4:8
204.188.121.18	224.2.187.164	7050	6	1:16 4:8
128.4.1.24	224.0.1.1	21169	6	1:16 3:8 4:8
192.58.206.100	224.2.0.1	112335	6	1:16 3:8 4:8
204.62.246.76	224.2.245.77	1142781	6	1:16 4:8
204.160.73.10	224.2.231.173	2	6	1:16 4:8 9:32
In Table 11-1, the first part of the output titled ìVirtual Interface Table,î shows multicast traffic statistics for each virtual interface on your multicast router, where a virtual interface can be a physical interface or a tunnel. You can use this output to find out how much traffic was received and sent by each of your down feeds. The Virtual Interface (VIF) 0 in Table 11-1 is the physical interface of the machine; because it doesn't send any multicast traffic to the local network, the number of packets that go in and out on it are zero.

The number of packets in for the VIF 6 is much higher than for the other VIFs, because VIF 6 is the feed that I use to receive all MBONE traffic from the rest of the world. You can also see that the number of packets out to VIF 6 is pretty low, at approximately 183 K-packets. What this means is that not very much traffic has been sent out of Canada; rather, the biggest part of the traffic has been sent within the Montreal area.

You will notice that for VIF 1, the number of packets out is almost as high as the number of packets in for VIF 6. The reason behind this is that the other end of this tunnel feeds a number of sites, of which one is an mrouted version 2.2 that doesn't support pruning (see Chapter 10 for a discussion on the importance of pruning).

As you can see, netstat can give you a fairly accurate picture of the MBONE usage of each of your feeds. There are drawbacks to using this method to check your feed stats though:

These drawbacks are not major ones, but they are significant enough to make the accounting tasks a little more difficult to perform because they require that you employ human resources to develop the accounting tools that will fit your needs.

The fourth drawback is important here because it means that you cannot monitor how MBONE is used on an individual basis. For example, suppose that you are willing to let your customers use the MBONE for free except when they use it for events that are not related to work (such as musical events, radio, world news, and so on). netstat does not provide you with the level of granularity needed to discern between work-related uses and personal uses. You could use the second part of the netstat output shown in Table 11-1 for this purpose (the part titled ìMulticast Forwarding Cacheî), but this method also has major drawbacks:

The multicast forwarding cache is simply a who's who list of who sends packets to what multicast group. It can give you an indication of where the traffic originated and to which VIF location your multicast router should send it. Only the most active senders appear in this list, however, which makes the cache ineffective for monitoring all MBONE activity.

Netstat is also an inadequate tool to use if you receive a single MBONE feed that will be used by all your internal users. Imagine a corporation placing the MBONE that is available to its employees in various departments. The problem with such a setup is that although you will be able to monitor the amount of data flowing in or out of your site, you will not be able to determine the individual sessions in which the traffic was sent or received. Again, this restriction presents a problem if you want to charge for MBONE usage.

The tool that comes to the rescue in these situations is called mlisten. The mlisten tool is part of a package that also includes mprocess.

mlisten listens to the control port of a specific multicast address and records the IP addresses of all the participants for that session. This tool uses a parameter file containing a few settings that influence the way that mlisten collects data.

mprocess takes the output produced by mlisten and creates a new output file in another form. This new output file contains four columns of summary information about the session that mlisten monitored. These four columns contain the following information:

This mprocess output file can be fed to a plot-generation program so that you can get a graphic representation of your audience.

Another tool that may be of use to you is traceroute (and its multicast equivalent, mtrace). You can't use this utility for doing accounting tasks per se, but it will help you establish policies at your site regarding who gets a feed from you and who doesn't, or who you get a feed from. This task is more important than you might think, because if you set up your network in a chaotic way, multiple MBONE feeds may end up crossing your network provider's physical links, costing you money. traceroute shows the route that a packet takes from your host to a destination host that you specified (see Figure 11-1). This trace will be useful to you if, for example, you are a network provider and a site requests an MBONE feed from you. By doing a traceroute to the site, you can determine whether you would be the best person to provide them with a feed.

Figure 11-1 and 11-2 show two examples of traceroute in action. Suppose that both sites in the examples requested an MBONE link from me, an Internet service provider. traceroute clearly shows me that I could give the first site a feed if they want one, but giving a feed to the second site would be out of the question.

traceroute to mix.net (198.168.73.2), 30 hops max, 40 byte packets
 1  Therouter.CC.McGill.CA (123.456.78.9)  1.566 ms  1.367 ms  1.359 ms
 2  McGill-gate.Provider.Net (123.456.79.1)  2.343 ms  1.747 ms  1.700 ms
 3  192.77.58.10 (192.77.58.10)  1.601 ms  1.443 ms  1.507 ms
 4  MIX.NET (198.168.73.2)  38.857 ms  43.820 ms  39.798 ms
Figure 11-1: The traceroute output to the first site.

traceroute to cam.org (198.168.100.5), 30 hops max, 40 byte packets
 1  Therouter.CC.McGill.CA (123.456.78.9)  1.742 ms  1.456 ms  1.694 ms
 2  McGill-gate.RISQ.Net (123.456.79.1)  1.628 ms  2.258 ms  2.140 ms
 3  psp.qc.canet.ca (192.68.56.5)  57.032 ms  19.398 ms  5.093 ms
 4  psp.ny.canet.ca (205.207.238.154)  8.557 ms  11.782 ms  11.525 ms
 5  border4-hssi1-0.Boston.mci.net (204.70.23.5)  13.494 ms  14.662 ms 20.854 ms
 6  core-fddi-1.Boston.mci.net (204.70.3.33)  12.987 ms  13.102 ms  12.707 ms
 7  core-hssi-2.NewYork.mci.net (204.70.1.1)  21.413 ms  17.191 ms  16.772 ms
 8  border2-fddi-0.NewYork.mci.net (204.70.3.18)  16.663 ms  15.848 ms 16.102 ms
 9  sprint-nap.NewYork.mci.net (204.70.45.6)  19.688 ms  22.025 ms  21.864 ms
10  fd-0.enss218.t3.ans.net (192.157.69.4)  24.534 ms  23.664 ms  20.710 ms
11  t3-3.cnss32.New-York.t3.ans.net (140.222.32.4)  26.026 ms  25.305 ms 24.435 ms
12  cnss40.New-York.t3.ans.net (204.149.4.8)  27.959 ms  28.373 ms  29.114 ms
13  t3.New-York-Mtl.t3.ans.net (198.168.57.25)  38.749 ms  37.864 ms  36.716 ms
14  Hydro.CAM.ORG (198.168.73.132)  53.998 ms  40.718 ms  44.047 ms
15  * Ocean.CAM.ORG (198.168.100.5)  64.561 ms *
Figure 11-2: A traceroute output for the second site.

The second site in Figure 11-2 should probably request a link either from ANS or MCI.

The multicast equivalent of traceroute is mtrace. It will show you the multicast route that a packet will use to go from point A to point B. This information not only enables you to determine whether giving a feed to a requesting party is appropriate for you, but also enables you to direct them to a more appropriate site for requesting a feed if you are unable to provide the feed.

Figure 11-3 shows an example of an mtrace output. I had to get some help from friends for this example because mtrace has to be supported by multicast routers for it to work. Because no mtrace path exists in Canada that is longer than 2 hops, and 2 hops is inadequate for providing a good example, here is an mtrace from MFSDatanet.com to uunet.net .

Mtrace from 137.39.246.98 to 150.225.14.20 via group 224.2.0.1 Querying full reverse path... * switching to hop-by-hop:
 0 tacostand.MFSDatanet.COM (150.225.14.20)
 -1 comet.MFSDatanet.COM (150.225.14.11) DVMRP thresh^ 1
 -2 wdcmon.MFSDatanet.COM (150.225.20.36) DVMRP thresh^ 1
 -3 * mae-bone.psi.net (192.41.177.247) DVMRP thresh^ 64
 -4 * MBONE1.UU.NET (137.39.43.34) DVMRP thresh^ 64
 -5 * MBONE2.UU.NET (137.39.246.98) DVMRP thresh^ 64
 -6 MBONE2.UU.NET (137.39.246.98)
Round trip time 156 ms
Waiting to accumulate statistics... Results after 10 seconds:
 Source Response Dest Packet Statistics For Only For Traffic
137.39.246.98 224.0.1.32 All Multicast Traffic From 137.39.246.98
 v __/ rtt 152 ms Lost/Sent = Pct Rate To 224.2.0.1
137.39.246.98 MBONE2.UU.NET
 v ^ ttl 64 -1/62 = -1% 6 pps 0/0 = --% 0
pps
137.39.43.34 MBONE1.UU.NET
 v ^ ttl 65 -1/381 = 0% 38 pps 0/0 = --% 0
pps
192.41.177.247 mae-bone.psi.net
 v ^ ttl 66 2/469 = 0% 46 pps 0/0 = --% 0
pps
192.41.177.25
150.225.20.36 wdcmon.MFSDatanet.COM
 v ^ ttl 67 0/21 = 0% 2 pps 0/0 = --% 0
pps
150.225.14.11 comet.MFSDatanet.COM
 v \__ ttl 68 2 0 pps 0 0
pps
150.225.14.20 150.225.14.20
 Receiver Query Source
Figure 11-3: An example of an mtrace output.

The first part of the mtrace output shows the multicast route used by the packets to get from the source host to the destination. Here we can see the thresholds for each hop on the path; this information is useful for determining the TTL value that you should use when you want to create an MBONE event and make sure that it reaches every location to which you want it to go.

The second part provides statistics about the path itself. This data includes the percentage of packet loss incurred along the routing path. You can use this information to locate the source of the problem when you are receiving traffic from an MBONE event and experiencing a high percentage of data loss.

I said earlier that all the multicast routers between the source host and the destination host have to support mtrace for mtrace to work. You can verify if a particular router supports mtrace by using the mrinfo utility. Figure 11-4 shows an example of an mrinfo output.

127.0.0.1 (localhost.CC.McGill.CA) [version 3.8,prune,genid,mtrace]:
  123.456.78.90 -> 0.0.0.0 (local) [1/1/querier/leaf]
  123.456.78.90 -> 123.456.210.1 (MBONE.Collaboration.CA) [1/16/tunnel/leaf]
  123.456.78.90 -> 123.456.3.4 (deadbeef.Algo.McGill.CA) [1/8/tunnel/leaf]
  123.456.78.90 -> 123.456.141.16 (LoveNotes.Harmo.McGill.CA) [1/8/tunnel/leaf]
  123.456.78.90 -> 123.456.186.10 (crash.WaysToFly.mcgill.ca) [1/8/tunnel/leaf]
  123.456.78.90 -> 123.457.90.90 (MBONE.UOtherSideCity.CA) [1/16/tunnel/down/leaf]
  123.456.78.90 -> 130.130.130.130 (mbonehost.jvnc.net) [1/64/tunnel/leaf]
  123.456.78.90 -> 123.456.101.182 (123.456.101.182) [1/8/tunnel/down/leaf]
  123.456.78.90 -> 124.222.4.3 (MBONE.cs.FarEast.ca) [1/32/tunnel/leaf]
  123.456.78.90 -> 129.111.3.18 (cythera.GOE.ca) [1/32/tunnel]
Figure 11-4: An example of an mrinfo output.

To generate the mrinfo output in Figure 11-4, I simply issued the mrinfo command without arguments; this gave me the mrinfo for my localhost. Issuing the mrinfo command with the name of the multicast router as an argument would give you the mrinfo for that router. In Figure 11-4, the first line shows that my localhost supports mtrace and also supports pruning. mrinfo also shows which tunnel is down, and it (the remote host) is a leaf in the tree for mtrace. The output also shows every link with their metric and threshold. Doing an mrinfo on each of these nodes would tell me whether or not the thresholds are symmetric; if they aren't, I could then contact the remote site and have them change the thresholds.

These various tools can help you perform accounting tasks on your MBONE usage, as well as assist you in debugging your MBONE setup. These tools comprise a limited toolkit, however, and some people may say that the tools are not very user friendly, either. If you are willing to put in some supplementary programming effort, however, you'll find these tools to be very flexible in how you can use them and what they can provide for you.

The future will undoubtedly provide us with even more of these tools for working with the MBONE. And keep in mind that the MBONE is still a research project; as such, the structure of the MBONE itself will continue to change, and so too will the tools for managing it.


Jump to Chapter 12
Return to the Table of Contents

Table of Contents | Previous Section | Next Section