Applications of NewsTools

The NewsTools Project




Sourcing news for applications


There is an enourmous amount of news available on the web from a variety of sources, both mainstream (eg. www.ft.com, news.bbc.co.uk, www.cnn.com) and specialized. News indexing sites may offer ways to adequately locate news stories, and to aggregate and filter them using indexes and keyword technology (eg. www.newsnow.net, www.moreover.com, www.newsindex.com). However, with the frequency and depth of some of this news, there is a pressing need for further tools and services to help manage this information.

The way forward is to switch to structured text for representing news. This is text marked up in the form of XML - a document processing standard proposed by W3C (the World Wide Web Consortium) - the same people behind HTML. XML can be viewed as a simplified form of SGML for the web. It is intended for the creation of document markups for specific applications. This is done by creating a document type definition (DTD). Furthermore, the tags used are not restricted to describing the style or layout of documents, but support tagging of the type of content. Tagged text can also be nested within larger chunks of tagged text according to the DTD used in the application.

Some aggregation sites (such as Moreover) can deliver news headlines marked-up in XML, and some of the major news agencies, including Reuters and Dow Jones, have collaborated on an XML standard called NewsML (www.iptc.org). Though in both these cases the tags are limited to giving meta-level information such as the source of a report, publication date, index keywords, etc, rather than on the content of the report. Nevertheless, these are important steps towards marking up content.

Whilst content can be marked up in XML by hand, news reports can be automatically marked up in XML. News reports that are already in a tabulated form (eg. the mergers and acquistions reports in Zephus at www.zephus.com) can be rewritten in XML in a straightforward way. Furthermore some major news agencies store news reports in relational data format for efficient translation into various languages around the world. Each attribute in an entry in the database can then be just tagged according to the attribute type.

More complex is free text. For focussed domains, information extraction technology can be used. This technology is intended to "read" reports and pick out the bits of information that are needed. For example, in a news report on a takeover bid, it may pick out the name of the buyer, the target, the value of the bid, and the status of the bid such as whether it is agreed, aggressive, or being refered to an anti-trust agency. Key information extraction research groups include the Natural Language Processing Group at the University of Sheffield (www.dcs.shef.ac.uk/nlp). Commercialisation of information extraction technology is now starting with companies including Cymfony (www.cymfony.com), Inxight (www.inxight.com), GTE (www.gte.com), Whizbang (www.whizbang.com), and Volantia (www.volantia.com). The technology has useful reliability within focussed domains such as when tailored for particular business sectors or business activities. For a very short introduction to information technology, see the information extraction article that was published in Information World Review last year.

Once news reports are in the form of structured text, knowledge-based systems technology can be used for automated reasoning with the information in each news report. This may include tools for looking for inconsistencies in news reports, and for merging potentially conflicting news reports from multiple sources, for ramification analysis, and for constructing arguments based on the potentially conflicting information. The NewsTools Project assumes news is in the form of structured text, and therefore the research can be viewed as interfacing with information extraction technology.

For more information about the NewsTools Project, contact Tony Hunter (a.hunter@cs.ucl.ac.uk or +44 20 7679 7295), or look at the NewsTools Project home page.