Minutes of the Reliable Multicast IRTF Research Group Meeting Memphis Tn, April 5,6 1997 Chair: Allison Mankin Co-Chairs: Mark Handley (ISI/USC), Ran Atkinson (@Home) Mail list: rm@mash.cs.berkeley.edu Archive:http://mash.cs.berkeley.edu/irtf/rm.html Web Page: TBA these minutes taken by - jon crowcroft with help from colin perkins. --------------------------------------------------------- Attendees - with introductions Name Affiliation Email Mark Handley ISI mjh@isi.edu works on mbone tools, sdr, and nte Jon Crowcroft UCL jon@cs.ucl.ac.uk IAB person, also works on mcast at transport and routing level Van Jacobson LBNL van@ee.lbl.gov has ideas of merit on everything Carsten Bormann Uni Bremen TZI cabo@tzi.uni-bremen.de has MTP SO, as well as work on RTP based applications Joerg Ort Teles/TU Berlin jo@cs.tu-berlin.de same Lixia Zhang UCLA lixia@cs.ucla.edu key contributions on ideas on traffic service models Raj Yavatkar Intel yavatkar@ibeam.intel.com streaming applications Mostafa Ammar georgia tech ammar@cc.gatech.erdu many multicast aspects Christophe Diot INRIA S_A Christophe.diot@sophia.inria.fr many aspects (including DIS related) D Towsley U Mass towsley@cs.umass.edu interest in network support for multiparticipant applications Brian De Cleene TASC btdecleene@tasc.com analysis and taxonomy of multicast Jim Kurose UMass kurose@cs.umass.edu analysis and taxonomy of multicast Sanjoy Paul Bell Labs sanjoy@research.bell-labs.com has worked extensively on a reliable multicast protocol Vach Kompella IBM kompella@raleigh.ibm.com general interest Rajesh Talpade georgia tech taddy@cc.gatech.edu reliable multicast (SCE designer) Roger Kermode MIT Media Lab woja@media.mit.edu media (very large realtime object) semi-reliable streaming Vince Laviano George Mason Uni vlavliano@cne.gmu.edu Ross Finlayson Live Networks finlayson@lvn.com frameworks and toolkits for mcast Dallas Wrege IBM wrege@raleigh.ibm.com ?? Matthew Lucas UVA matt@virginia.edu Nagastugu Yamanouchi IBM Japan yamanous@trl.ibm.co.jp Terji Shiroshita NTT siro@isl.ntt.co.jp have the designed and implemented the "OTHER" RMTP protocol Bob Briscoe BT rbriscoe@jungle.bt.co.uk interest in applications and services Tony Speakman Cisco speakman@cisco.com router support Ken Miller Starburst miller@starburstcom.com multicast file transfer product Kary Robertson starburst krobertson@starburstcom.com multicast file transfer product Michael Myjak Inst. of Simul and training mmyjak@ist.ucf.edu DIS Werner Vogels Cornell vogels@cs.cornell.edu Virtual Synchrony expert. Mark Green @Home Network markg@home.net Asymmetric Network capacity requirements Scott Bradner Harvard sob@harvard.edu Router Expert (and transport area person to be) Randall Atkinson self rja@inet.org formerly worked in security; interests in asymmetric network support and in traffic management/congestion control mechanisms. Isidor Kouvelas UCL i.kouvelas@cs.ucl.ac.uk conference control (has worked on media tools) Colin Perkins UCL c.perkins@cs.ucl.ac.uk shareware apps, conference control (has worked on media tools) John Du Intel john-du@ccm.jf.intel.com Donald Newell Intel donald_newell@ccm.jf.intel.com ??? Robert Mckenzie Smith Micro bmckenzi@smithmicro.com ??? Vladimir Ksinant Dassault Elec ksinant@dassault-elec.fr aviation related Nils Seifert Condat Berlin se@condat.de ? Barbara Denny 3Com denny@nrd.3Com.com router support Sally Floyd LBL floyd@ee.lbl.gov one of the authors of SRM Miriam Kadansky Sun miriam.kadansky@sun.com Steve Hanna Sun shanna@sun.com Srinvasan Seshan IBM srini@watsn.ibm.com Katia Obraczka ISI katia@isi.edu congestion control and multicast Brian Whetten Global Cast brian@gcast.com has another multicast transport protocol Scott Corson UMaryland corson@isr.umd.edu Works on mobile and asymmetric (e.g. SATCOM) networking; Joe Macker NRL macker@itd.nrl.navy.mil Has been extending IMM for reliable multicast file transfer applications, experienced with asymmetric networks and FEC; Jagan Shantigram Bay jagan@baynetworks.com router person Allison Mankin ISI mankin@isi.edu chair... Steve McCanne UC Berkeley mccanne@mlk.cs.berkeley.edu one of SRM authors and mbone tools person, interest in RM framework and applications Sunday only ----------- Steve Casner Precept casner@precept.com have IP/TV (includes reliable aspects...) Frank Solensky FTP Software solensky@ftp.com PC stack provider of quality... Karen o'Donoghue US Naval Surface Warfare Centre (NSWC) kodonog@nswc.navy.mil Tim Plunkett Navy tplunke@nswc.navy.mil DIS Patrick Cocquet DE cocquet@dassault-elec.fr DIS and aviation related things Don Brutzman NPS brutzman@nps.navy.mil DIS Eric Mannie Helios mannie@helios.iihe.ac.be ? Mark Green @Home markg@home.net asymmetry? Steve Seidensticker seiden@netcom.com DIS Bob Quinn TIAC rcq@tiac.net ? Ramon Cacares ATT Ramon@research.att.com ? Allyn Romanow Sun romanow@sun.com --------------------------------------------------------- 0.Agenda Bashing:/Introductions 1. Outline of Research Group Strawman Goals - Allison gave a brief introduction to what was intended as the outcome for the two day meeting - the idea is to get input for framework, and specific workitems for now, future meetings, and in between, as well as to establish a working method - some examples (that had been presented at the San Jose IETF) were given of why we need to work on reliable multicast, but why it is not quite ready for an IETF WG (or several) were given. A Web site will be set up and managed, and slides from talks at this meeting, plus links to other web sites/documents of relevance will be put up there - this will include a list of "required reading" for members of this group. --------------------------------------------------------- 2. This session was about frameworks, for reliable multicast - both for protocols and for considering systems, as well as to motivate (structure_) comparisons and design work. 2.1 Brian de Cleen from TASC gave a presentation of the RMF (reliable Multicast Framework) from TASC and UMass. This could be viewed as a "Union" of all the functions gleaned from examination of all the multicast protocols (transport mainly, although the framework includes an abstraction for considering a "session layer" which is used to deal with problems not appropriate in the end2end layer). Diot noted that RMF is not very ALF oriented. slides at: http://www.tascnets.com/mist/RMF Key concepts includes: Self Identifying Packets Session State Information protocol [question was whether this was dynamic (per packet, change of protocol style etc) - A\ A - an Open area for research The selection of multicast protocol functions of interest was motivated around the idea of Universal Receivers - these are configured (through templates, or through the session state protocol, or through prior agreement Don Towsley (templates etc) There was a brief discussion of the ideas in RMF of retransmission (the range of methods (who retransmits, how it is requested, whether it is to uni cast or multicast etc) went on - note that RMF is not prescriptive here - it is descriptive. Steve McCanne asked "Can you think of applications and how they will use these different features@ Answer: the RMF is derived from a set of protocols, each of which are derived from applications, so its kind of implicit..... Sally Floyd asked about the use of "union" frameworks as part of a design process, rather than application requirements capture as an alternate approach to deriving a set of useful protocols ... McCanne pointed out the dangers of the complexity of this union approach. Allison Mankin remarked that we can use this to critique the development of ideas maybe... Briscoe asked "What is a framework anyhow?". de clean said that this one (RMF) is an "a la carte" for choosing pieces of protocol. Allison Mankin said that this is a key discussion.... Q: isn't this framework just a "thin layer" A: Maybe not so thin...... Katia asked - is this per packet reliability: A: we can have a change of mode (c.f. DIS work) but this is very receiver oriented in choice of mechanism Mark Handley asked - if we find there are only two protocol structures that are useful, maybe we will discard the framework later - perhaps if there are 20 (and more) we keep it.... Sally Floyd asked then "how to apply this framework" Sanjoy Paul asked about using the framework as a policy.... - Answer: no, its an infrastructure Mark asked about "lego blocks for protocols" Allison Mankin said we could maybe find we get 20 protocols of interest, but what is the killer application? Briscoe asked if this is a design methodology or a modelling technique Working: it's design.... The next slide (phew) was about a Session API On a slower timescale Address/id Join/leave/ Connection management Timer management Control packets etc etc Next slide was about the packet level (i.e. a specific framing protocol......) There was some discussion about some if the fields here - particular, if the protocol is configured, and is receiver driven then why are there some reliability function bits (Crowcroft asked) and is multiplexing in the manner of SSRC in RTP sensible (SSRCs are random -= actually for source identifiers, and data naming, we need some general architecuture - see later...) McCanne asked why there was a checksum.... Handley asked what about the "what is in RTP and why approach"? Allison Mankin said: Maybe RTP went too far down that road - perhaps we should Consider some things they did though (ALF + daisy chained headers a la ipv6 maybe?) Lixia Zhang asked about how Universal Receiver and Interoperability fitted together..... Brian de Cleene said: we define this in terms of the receiver responses (that is what is profiled/configures) not in terms of the senders usage. (c.f. (Crowcroft) Internet Design Rule #1!) Raj Yavataka asked: RMF d/c doesn't preclude interop (e.g. RTP profiles fore H.261 v. H.263...) Q: Why not use UDP and UDP checksum? Q: Security: Allison asked about security in general - is this a a big question for this group? it was unanimous that this was an important part of the problem, s[pace. Brian went on to cover the packet formats for RMF control and data, and said that they have an implementation in ++ with profiles for SRM, RMTP and are doing RAMP... 2.2 Jon Crowcroft outlined the UCL/INRIA work on RMFP - a Reliable Multicast Framing Protocol since this has a lot of synergy with the TASC/UMASS RMF work, we will probably merge ideas here. Two key discussions emerged on this (whether RMF or RMFP are a good idea or not): Source identifiers Data naming (see more below....)_ 2.3 Steve McCanne outlined the Berkeley ideas on a class structure driven design fr [slides to be scanned for web site - see also http://www.cs.ucl.ac.uk/staff/jon/ XXX for sigcomm multicast discussion this is rooted at the idea of SRM (the framework in SRM, not the protocol specifically, which (at least in the sigcomm paper) is mostly the reliability aspects of the framework idea) below the root at a collection of receiver based protocols - reliability congestion control then a split to persistent applications and control applications persistence (e.g. wb/mediaboard, or m-nam, webcast, or newscast and mmd) as well as control (floor controllers etc) outside the scope of this are sender based approaches (regarded mainly by this group as anathema to working the big I Internet). at the root are also some important ideas on naming, authentication and cryptographic techniques... sources are named using RTP CNAME like (but with more persistence) so that the creator (sender) of an object to a persistent multicast applications is permanently identified... data naming is complex, and has a tree structure - the object names can be flattened efficiently (although Steve said there may be some problems here ...) - the use of these is i nthe ALF approach to a repair (of course) they currently use a generic ASCII name optimised into a two level (oid +seq?) - the hierarchy can have dates at the route, and moves down to sequence numbers at the leaves..... Q: isn't this very heavy weight: A: it can be optimised... To scale Session massages: i/ advertise to group recursively apply SRM to recursively let users lookup directory entries (very like the way currently Wb boots a new user and they find out what data (draw operations and so on) are on which pages in a whiteboard session.. ii) scaling: of state announcements (e.g. re-synchronisation a user) - they use MD5 like hash summaries of a node i nthe tree, and can recurse down a directory structure sending more state about the areas that have changed, and less about ones tat haven't ("much"?>) Building bridges between application constraints (e.g. nte and wb have similar ish constraints) - he is looking at this as another piece of the framework. 2.4 Mark Handley summarised this session and said: RMF is a union of protocols RMFP is the intersection approach SRM + Applications is more of a design framework/architecture.... [Diot noted that there are (at least) three different interpration of "framework": RMF is a framework (independant generic/universal layer with functionalities) RMFP is a framing protocol (not a framework, but an encapsulation scheme such as RTP) For SRM, there are problems formulating what is their understanding of framework, except that I see SRM as a (or "The") mechanism for reliable retransmission in their framework (or class structure driven application characterization).] --------------------------------------------------------- 3. Status reports from all interested parties Allison introduced this session and gave each person 10 minutes to report o ntheir work in progress: The work had to orient Is the work a framework (or towards) is the work applications requirement pulled is the work commercial or network requirement driven? 3.1 Carsten Bormann presented (slides) MTP SO fits Steve's SRM framework have several Applications (shared X, sccp, mmap, newscaster) - talked about repair techniques 3.2 Joe Macker talked about the nrl work on a wireless MOSPF network, with non realtime bulk data using Joe's IMM-derived software - DIS requirement were also mentions as were asymmetric networks 3.3 Jon Crowcroft talked about BMART (reliable one-to-many bulk transfer protocol with congestion control - driven by web cache mirror load requirement 3.4 Rak Yavatkar talked about the continuous media streaming applications and receiver choice (dynamic tree, playout monitoring and loss repair mechanism choice based on monitoring playout delay [have a modified vat which works with this now) 3.5 Ken Miller talked about MFTP and starburst approach - work on intranets with upwards of 50000 receivers..... Q: satellite - is a very "flat" multicast tree right? not generic big-I Internet? Ran Atkinson pointed out that @home have HFC (Hybrid Fibre-Coax) network which has very similar asymmetric bandwidth to satellite up and down links.... (HFC networks might have several (5-8) Mbps downstream to subscribers, but only about 1-3 Mbps upstream from subscribers. There might be contention for upstream channel access, depending on which cable modem vendor one uses.) 3.6 Sanjoy Paul talked about RMTP RMTP uses hierarchical domains and repair servers (can be static or dynamic) Applications include - bulk transfer and push and pull reliable multicast FTP, as well streaming (time constrained or "resilient:" rather than completely reliable...) There is a demonstrator - of a tree based whiteboard Sanjoy comment was RMTP can also be seen as a framework. 3.7 Brian Whetten from GlobalCast talked about their approach (NOT 1 size fits all!) He presented this taxonomy: RMP SRM ARM topology ring cloud tree scale goods better best net type lan wan both needs this now needs common api congestion control is "tough, maybe impossible" not (NOT) convinced we need integrated security 3.8 Vogels gave a talk on the conrnell views on fault tolerance (coming from ISIS HORUS Ensemble history) can build a LOT on top of virtual synchrony - probabilistic reliability they have applications in stock exchange and in Aviation 3.9 Ross Finlayson gave a talk on the multi kit framework for generalising the session directory - he has many directories and arbitrary attributes - http://www.lvn.com/multikit 3.10 Roger Kermode gave a talk on the scalable delivery of very large objects - use of late joins content based naming scheme (RLMish) and hierarchical data partition 3.11 Teruju Shiroshita gate a talk on the NTT RMTP (different ) see doc/slides for details designed to co-exist with tcp.... see slides at: http://www.ntinfo.ntt.jp/JOINT/jointH.html 3,12 Steve Seidensticker..*** gave an LSMA/HLA/DIS presentation...... 3.13 Mostafa Ammar talked about the single connection emulation approach (see document/slides..... http:///??? which is an extension of TCP for reliable multicast. --------------------------------------------------------- Sunday Agenda (rm -rf rmtp* ) Application Taxonomy Congestion Control FEC Deployment Goals and Charter and Workplan --------------------------------------------------------- 4. Mark Handley Gave an application Taxonomy which h stimulated a lot of discussion This started with 1/ the BIG I Internet requirement (we are not addressing Intranet or other network specific solutions) ii)assumption most things scale to nearly arbitrary numbers (100s, 100s of millions) of receivers/ 4.2 Categories of parameters of the system: number of sources start time (is there one real timeness consistency non factors congestion control ordering Both can be done orthogonally to the above categorisation..... Applications grouped into Bulk Transfer (VR database, Usenet news etc) Live data feeds (stock exchange etc) semi reliable (streamed server data - like RTSP from video/audio from web site) shared applications (wb, nte etc) Hybrid 4.3 Bulk Txfer 1-many relaxed real time synchronised start time file (object) level consistency 4.3 live data feeds 1 source no start time quasi real time may need synchronised consistency there was a brief discussion about use of integrated services to make these actually fly!) 4.3 semi reliable (name caused a problem and was changed to resilient!) 1 data source no start time stream consumed in near real time non consistency large commonality 4.4 shared apps distributed data set all sites fold most relaxed consistency many sources of changed 4.5 Mark then gave the NTE talk about consistency and concurrency and determinism and so on 4.5 there was a long talk about the value of one size fits all (Myjak said "yes) and categorisation factors - vogels asked if we had any more Crowcroft suggested "end time" Carsten pointed out another view (size of object over which consistency reliability or whether is needed, versus RTT, versus history... --------------------------------------------------------- 5. Congestion Jim Kurose introduced the session on Congestion and asked i) difference between Intranet and Internet ii) fairness definitions for mcast v. uni cast iii) scaling and hierarchy iv) goals [Crowcroft aside - many intranets now are large, unwieldy and badly engineered - e.g. Texaco's international Intranet has higher losses than much of the "open" Internet in Europa, even!] 5.1 Mbone Performance Mark Handley outlined some neat masurement results he had on congestion (see slides) He outlined the loss distributions (spatial and temporal) in the current mbone - statistics from shuttle video....based on RTCP log + mtrace are available at http://buttle.lcs.mit.edu/~mjh/mbone.ps We see lots of groups discernible with correlated loss, plus a group of a few with really bad (might as well get off the net) ... since probability is high that some receiver needs a packet again, argues for FEC (and for local repair.... see debate later about tradeoffs) He did a least square correlation fit for loss v. data rate - it is largely uncorrelated Actually even on quietest time, 128 kbps of shuttle video is a small portion of the overall (t3 or OC3 saturated) backbone traffic anyhow Matt Mathis noted that TCP would see something pretty similar........ maybe slightly better; sally noted that in other words, problems for any e2e scheme with no net help Vern Paxson had statistics that showed the Internet aggregate loss was roughly doubling each year (actually 1% in 91, 3 in 93 and 5% in 97) Steve McCanne pointed out that while this is measuring a small fraction, this is not an argument for designing a protocol to service in this ecosystem - we want to design a protocol that everyone used and avoids targetting into this state at all.a 4th slide looked at conditional loss probability (if i lose one, will i lose next, and next etc) [partly caused by "cisco disease"...(route updates cause cessation of packet forwarding service for a longish hiatus). 5.2 ISI Congestion Control Work katia (with dante de lucia) - (see slides) assume: IP mcast no assumptions on group (size/distribution etc) 1- many bulk Txfer had done something consistent with Mark's stats - using "representative election" algorithm,, then explicit feedback.... (acks and nacks.....) - seems to work pretty well (using ns simulator (http://www-nrg.ee.lbl.gov/ns/??) The exact congestion signal is still a subject for research )what metric....). See http://www.isi.edu/people/katia/ and in particular, see "Multicast Transport Mechanisms: A Survey and Taxonomy", Katia Obraczka, submitted to IEEE Communications Magazine. "Multicast Feedback Suppression Using Representatives", Dante De Lucia, Katia Obraczka, to appear at the IEEE Infocom97. "Multicast Feedback Suppression Using Representatives", Dante DeLucia, Katia Obraczka, Computer Science Department - University of Southern California - Technical Report No. 96-638, June 1996. status ....conserviatve see also' http://buttle.lcs.mit.edu/~mjh/mbone.ps 5.3 Crowcroft talked about FEC+layer+sender probe congestion scheme......use TCP mimic model.... have implementation (see HPCS paper and bmart paper...) and runs on rmfp..... sally asked q. about congestion control;l short term fair (with tcp) congestion. collapse rough consensus? long term how make use of available capacity best effort compete with tcp not only option'' maybe need category.... van talked about limits of TCP VJCC and FIFO drop tail, and now we need more gateway mech..... RED debate (ref - sally will write a note on this to point people at the red queue management debate....) raj and ran said - this looks neat crowcroft - yes - lets you me more aggressive when congestion signal has more noise (e.g. for aggregated signal from mcast elected reps or whatever...) Jim said: what are adaption time scales; ? van: problem is bad guys etc... [Crowcroft aside: it should be possible to build a system that uses layered, striped, and integrated FEC, as well as ARQ, with local repair, and extend the SRM local repair mecxhanisms to distribute "scope of congestion" signals to nearby - these can be used to drive either group selection in the BMART congestion scheme, or to select representatives to ask for "slow down" in the ISI congestion scheme - these can then result in senders (whether sending original data, parity bits, new layers with FEC repair data) to adjust rates - a suitable 2 layer technique might use representatives within each layer (group) and multiple groups to get the expontneial rate increase (and cope with heterogeneity) where suitable (selected by application/session?). ] --------------------------------------------------------- 6. FEC Joe Macker (slides? gave a presentation on use of FEC 1/. integrated FEC 2/ layered FEC 3/ local repair 4/ max loss report 5/ scale of group and high loss (>50% high loss... q: how does FEC fit with (dynamic) heterogeneity problem, (McCanne) A: maybe maybe not.... 6.2 Down Towsley reported his FEC work [paper) showing integrated work is a big win....slides are on the web at http://www.cs.ucl.ac.uk/staff/jon/arpa/rm-www/index.htm The downside is implementation cost (see Rizzo's code - see Vicisano and Rizzo paper to... http://www.arl.wustl.edu/arl/workshops/hpcs/??? --------------------------------------------------------- 7. Application matrix - revisited Allison, Mark and Ran fielded their application matrix (slide?) properties v., applications as per follows: num streams start-time realtime consistent ordered reliable bulk 1 sync no file-level no yes live 1 no maybe probably probably yes stream 1 no yes no no semi shared many many yes eventual no yes Seems that the "live-feed" category is poorly defined... "ordered" is the order of delivery at the application level. vogels asked about ordering and stability ...etc ran asked the group did we have consensus Allison proposed the group look at producing some documents on the bulk Txfer and shared data apps van said fine (as did others....) 7.5 security van made some remarks about open security - key point is that groupware!=telnet The publish /subscribe model is different from conversational model - can leverage maybe off namespace to establish right security model...? --------------------------------------------------------- 8. Charter specifiable congestion control scaling security deliverables: mid and near term.... Carsten interjected elastic time critical 1:n bulk data live data? n:m shared data control (DIS???) Allison will pursue writing up the charter online. --------------------------------------------------------- 9. next meet - at Munich IETF (which will be mboned) briefly, then at sigcomm (dates - sep 14?)(Nice......which can be mboned). required reading - axiomatic fairness...etc and other useful mcast refs: see http://www.acm.org/sigcomm//ccr/index.html for back issues includign SRM paper....for which, also see: http://ftp.ee.lbl.gov/floyd/srm.html For surveys of protocols, see http://hill.lut.ac.uk/DS-Archive/MTP.html http://www.tascnets.com/mist/doc/mcpCompare.html http://www.inria.fr/rodeo/personnel/mgross/WWW/multicast.html http://www.cs.ucl.ac.uk/staff/jon/sigbone.html See also ftp://cs.ucl.ac.uk/darpa/jsac-group.ps.Z