Bird-Doggin' the Internet

Metasearching & Exploring Parallel Worlds

By Dr. Steve O'Neil, Micro Mo Electronics, Inc.

I n past articles we have touched on individual search engines and directories, and how to make the best use of them. Unfortunately, even the best search engines do not catalog or list all the information on the WWW. A 1998 study on search engines by NEC Research Institute (cited on C/Net News.Com) estimated that even the most comprehensive search engines index no more than 1/3 of the Web's estimated 325 million pages. A widely quoted Associated Press report around the same time estimated that all search engines put together list less than 50% of Internet sites. An interesting aside in the NEC report was that when a given scientific search phrase or keyword was entered into multiple search engines, the number of unduplicated resources returned increased about 3 1/2 times. Combine this with the fact that each search engine has subject-specific strengths and weaknesses, and it becomes clear that searching in parallel using several search engines can yield superior results. This is the principle used by so-called "metasearch" engines.

Using metasearch resources gives you access to several search engines with one entry. The entry, however, will not utilize each engine's special capabilities. Boolean searches are usually a safe bet. You will typically need a forms-capable browser to make the most of these tools. Start with a few inquiries in your area of interest at one of the most famous of them all--MetaCrawler at http://www.go2net.com/search.html. Be prepared to wait a few minutes for your search results. My searches typically tap Alta Vista, Lycos, Infoseek, Excite, and Thunderstone (a new experimental search engine). Duplicates are eliminated, and results are collated with keyword(s) bolded. Among the new, Mamma at http://www.mamma.com gets high marks for fast processing and a good interface. With her(?) you can search Web sites, audio and graphics files, and newsgroups. Typical results may include returns from Excite, Infoseek, WebCrawler, and Alta Vista. Your returned links aren't checked for validity, so expect a few clunkers. An engine I use regularly is Dogpile at http://www. dogpile.com. Enter your terms here and they go to Yahoo, Lycos, A2Z, Excite, and a dozen other engines, as well as newsgroup, wire service, and ftp search tools. Returns are listed by source of information ranked by number of hits for each source. Search engines are queried sequentially. This is a source for filling in potential holes from other search tools. Fast and fairly extensive.

Other metasearch engines which can be useful for quick searching include Metafind at http://www.metafind.com/, and SavvySearch at http://www.savvysearch.com. Metafind searches several search engines in parallel, bringing returns from each. You can sort alphabetically, by URL, or by your keywords. Metafind includes a time-out option if you hate waiting. Savvysearch gives you half a dozen default search engines. If you don't like what you see there, choose the option to expand the scope of the search. This site includes some engines not commonly found on other sites such as Google, Galaxy, Direct Hit, and National Directory. ProFusion at http://www.profusion.com, and Highway 61 at http://highway61.com also give useful information. Highway 61 allows you to adjust your search returns to "lots" (35~75) or "bury me" (60~125). You can also pick your waiting time. Results were sparse, but the interface is flexible if you are a patient type. Profusion provides default or manual selection options based on your searching preferences.

Many of the sites, which I formerly used, have been merged, shut down, turned into commercial "portals," or provide little content and too many links to commercial products (generally unrelated to my search terms). That's why I no longer bother with metasearch engines such as OneSeek, Search.com, Inference Find, Cyber411, or Profind (the latter being down and out on my last several visits; it may be back up next month). If you want to link to a site that houses most of the better known search engines (not a metasearch tool in itself) go to the DGAF Search Page at http://www.dgaf.com/search/. Don't get lost!

Another type of multiple query tools, which are, strictly speaking, not metasearch engines, are worth looking at if you have search envy. These are the desktop "smart agents," special search engines, and "bots" which launch customized searches from your PC desktop using your IP connection. Included here are the likes of Copernic, WebSleuth, QueryN Metasearch, Inforian Quest, Mata Hari, and Hurricane. There are dozens of others, but these can be accessed for a free trial download (typically 30 days) at the ZDNet Software Library download page at http://www.zdnet.com/swlib/toolkits/find fast/tlk1098.html. Prices range from $15 to $40 although there is a free version of Copernic 98 (2.22 MB compressed). Other specialized engines can be found at the BotSpotŪ. Hook yourself up at http://www.botspot.com/ best/ and at http://www.botspot.com /search/s-search.htm.

Steve O'Neil can be reached on the Internet at Steveo@micromo.com.

URL Alert! We recognize the URLs printed in these articles may change by the time you try them out on the Internet. Most of the past Bird-Doggin' articles we've printed, with updated URLs and links, can be found at <http://www.micromo.com/related.html>.