FAST winning the search engine race

Originally published on EuropeMedia.net, 01/02/2002.

Search engine technology company Fast Search and Transfer (FAST) provides the technology used to run search services across Lycos' portals around the globe and also showcases it at its own search engine, www.AlltheWeb.com.

In the continuation of a series of interviews, Duncan Parry talks to Stephen Baker, Director of Business Development and Marketing for FAST's internet Business Unit, about the company's future plans.

The "deep web" of web pages built using technologies that normal HTML spiders can't index (e.g. databases, dynamically generated pages, etc.) has been something of a hot topic over recent months in the search community. Companies like Teoma (now part of Ask Jeeves), Quigo and Google and FAST are all developing technology to allow search engines to include more and more results from these web pages. What sort of web pages and content can FAST's search technology currently index, and what sort of pages do you intend to add to this?

FAST: FAST was the first engine to handle deep web content when we rolled out our FTP and MP3 Search products in 1997. These services enable us to index and search the contents of file servers, which contain a variety of document types, including PDF's, MP3's, JPEG's, executables, zip files, etc.

In 1999, we took the concept to HTTP servers and created the world's first integrated index of multimedia documents, including pictures, audio and video files. Today, and moving forward, our crawler is becoming more sophisticated and is able to seamlessly integrate numerous file types (including content served by app-servers) into the web search index. We also possess the ability to work with online e-tailors, for example, to receive XML data feeds of database content.

FAST's search engine, AlltheWeb, offers search options for web pages, news, images, MP3s and videos. What other content-specific search options do you intend to add? An option to only search public newsgroups, message boards and forums, perhaps?

FAST: We will continue to add verticals of information that users demand and ultimately improve the user experience. Scirus.com, the leading science search engine, is an example of a vertical search index that we built for Reed Elseveier.

This has turned into a very exciting portal for the science community. Indexing newsgroups and message boards is something that we do not see tremendous demand in, currently. We will continue to increase the size of our index with a variety of visible and invisible web content, with the goal of reaching 2BB objects in the near future. However, the index will contain objects that searchers are actually interested in. We will also continue to make improvements in the freshness area, as exemplified by our real-time news search.

If these improvements require FAST to build additional vertical categories, for example an index of live streaming radio stations, we will.

Do you intend to promote AlltheWeb.com as a mainstream search destination in its own right, competing with the likes of Google and Yahoo!, or is your focus primarily on providing search technology for other search engines and enterprise customers?

FAST: FAST is a provider of OEM search to the world's leading portals. AlltheWeb is purely FAST's technology showcase. It is where we roll out, test, and understand the ROI of new features before deploying them to our portal customers.

There is no intention of making AlltheWeb a world-class destination site, like Google or Yahoo However, it has quickly become a site which people with a deep understanding of search have been turning to as an "in-the-know" site for fresh and relevant results.

AlltheWeb offers language-specific searches; do you intend to offer nation-specific searches, i.e. domain-specific, at their own websites in future (e.g. AlltheWeb.co.uk, AllthWeb.fr, etc?)

FAST: The jury is still out on this. If adding country specific UI's enables us to better test new features that our portal customers demand, then yes, we will roll out country specific UI's. However, since we are not a destination site, we do not staff ourselves with country specific editors or UI developers, so I doubt we will do this in the short term.

There are already banner and button adverts on AlltheWeb – are these signs of moves to make this website a profitable, standalone entity beside the search technology services under the FAST brand?

FAST: In order to cover the costs of operating ATW, we have forged a partnership with Lycos to monetise AlltheWeb. This enables us to run a technology showcase at break-even levels, while demonstrating our commitment to our portal customers and partners.

Search results on AlltheWeb feature beta "FAST topics" are these based on the open directory, or the categorisation of web pages based on their content, as determined by your spiders.

FAST: We use the ODP as a training taxonomy. Those documents which can't be classified to an ODP directory, are dynamically catagorised based on linguistics analysis.

Does FAST plan to offer a fee-based URL inclusion program similar to those offered by AltaVista and Inktomi? I've seen some indications of a beta test of this on the FAST website scheduled for the summer of 2002.

FAST: FAST PartnerSite, our "pay for inclusion" program, will be launching in mid-February 2002. Our initial distribution partners will be Terra Lycos in the US and Lycos Europe in Europe. Additional portal distribution partners will be added soon.

What are FAST future plans for WAP and 3G? Will you be looking to offer search services for mobile internet devices ?

FAST: FAST launched the first and the largest Mobile Search in 1999. Our enterprise division has created a mobile search platform designed to be integrated with Nokia and Ericsson mobile portal platforms.

Although our mobile features are not something we are currently promoting, FAST is prepared to lead in this area when the market is ready for it.

Some analysts have speculated that localisation services – e.g. locating the nearest Pizza takeaway via a 3G mobile and the local postal code – will be a popular use of mobile internet search services. Is FAST looking to develop these sort of services?

FAST: FAST has developed geo-location software that achieves these goals. As these applications are demanded by the mobile phone user and as standards for providing geo-coded content are developed, FAST will be a lead player in the mobile industry.

As the number of broadband users grows, does FAST have plans to serve their unique search needs? For example, a search option for streaming media broadcasts or full-screen video?

FAST: Yes, we are already doing this with our Audio and Video Search. To date, we have not invested significantly in developing these products because they are largely broadband driven.

As broadband penetration increases however, we expect to see the demand for streaming media search to increase. We will definitely be a leader in this arena when demand requires it.