Very informative.

Challenges of the Deep Web Explorers

Web spiders these days, it seems, are a dime a dozen. Not to minimize the tremendous value that Google and other search engines provide, but the technology that gathers up or “spiders” web pages is pretty straightforward. Spidering the surface web, consisting mostly of static content that doesn’t change frequently, is mostly a matter of throwing lots of network bandwidth, compute power, storage and time at a huge number of web sites. Merely throwing lots of resources at the deep web, the vast set of content that lives inside of databases and is typically accessed by filling out and submitting search forms, doesn’t work well. Different strategies and a new kind of “deep web explorer” are needed to mine the deep web.

Surface web spiders work from a large list, or catalog, of known and discovered web sites. They load each web site’s home page and note its links to other web pages. They then follow these new links and all subsequent links recursively. Successful web crawling relies on the fact that site owners want their content to be found and that most of a site's content can be accessed directly, or by following links from the home page. We can say that surface web content is organized by an association of links, or in HTML jargon, an association of tags

We should note that spidering is not without its hazards. Spiders have to be careful to not recrawl links that they’ve previously visited lest they get tangled up in their own webs! If spidering the surface web is not an impressive achievement then what makes Google’s technology so highly touted? In the case of Google and of other good search engines what’s impressive is not the ability to harvest lots of web pages (although Google currently searches over four billion pages) but what the engine does with the content once it finds it and indexes it. Because the surface web has no structure to it good search technology has to make relevant content easy to find. In other words, a good search engine will create the illusion of structure, presenting related and hopefully relevant web pages to a user.Google’s claim to fame is its popularity-based ranking.

It structures content by presenting first to the user web pages that are most referenced by other web pages. The deep web is a completely different beast. A web spider trying to harvest content from the deep web will quickly learn that there are none of those links to content and no association of links to follow. It will realize that most deep web collections don’t give away all of their content as readily as surface web collections do. It will quickly find itself faced with the need to speak a foreign language to extract documents from the collection. This need is definitely worth meeting since the quantity and quality of deep web content is so much greater than that of the surface web.

Deep web explorers approach content searching in one of two ways, they either harvest documents or they search collections on the fly. A deep web explorer may attempt to harvest content from a collection that doesn’t support harvesting but, for reasons cited below, the effort will likely not be very fruitful. Dipsie and BrightPlanet are harvesters. They build large local repositories of remote content. Deep Web Technologies and Intelliseek search remote collections in real time. Harvesting and real time search approaches each have their pluses and minuses. Harvesting is great if you have adequate infrastructure to make the content you’ve collected available to your users and if you have a sufficiently fat network pipe plus enough processing and storage resources to get, index and save the content you’ve obtained. Harvesting isn’t practical if the search interface doesn’t make it easy to retrieve lots of documents or if it’s not easy to determine how to search a particular collection.

If the collection doesn’t support a harvesting protocol then harvesting will not retrieve all documents. Additionally, not having the network bandwidth and other resources makes harvesting impractical. And, if a collection is constantly adding documents then either the collection is going to somehow identify new content or you’re going to waste lots of resource retrieving the documents already in your local repository just to get a few new documents.

OIA, the Open Archives Initiative, is an example of a harvesting protocol. OIA describes a client-server model useful for aggregating multiple collections into a single centralized collection. The server tells the client, among other things, what documents are new in its collection and the client updates its repository with them.

Deep Web Technologies’ (DWT) Distributed Explorit application implements the other approach, the real-time search approach, which also has its pluses and minuses. A tremendous plus is that most deep web collections lend themselves to real-time searching even if they don’t lend themselves to harvesting. This is because by not implementing a harvesting protocol the content owner doesn’t have to do anything to its documents to allow them to be searched; it doesn’t need to generate metadata or otherwise structure its content. An on-the-fly search client uses the simple HTTP protocol to fill out and submit a web-form that initiates a query against the content database. The client then processes (parses) the content returned and displays search results to the user. DWT’s Distributed Explorit does multiple simultaneous real-time searches against different collections then aggregates

An on-the-fly search client uses the simple HTTP protocol to fill out and submit a web-form that initiates a query against the content database. The client then processes (parses) the content returned and displays search results to the user. DWT’s Distributed Explorit does multiple simultaneous real-time searches against different collections then aggregates the results and displays them to the user. The minuses of the harvesting approach become pluses in real-time searching. That entire infrastructure you needed to retrieve, store, refresh and index remote content and to then provide access to it disappears.

Minuses of real-time searching are the ongoing demands placed on the remote collection, the reliance on the availability of the remote content, the vulnerability of depending on search forms that change or break, and the inability to rank documents in a homogenous and effective way. (Search engines are notorious for ranking poorly or not at all and even collections that do rank documents in a relevant way can’t deal with the fact that their well-ranked documents will likely be aggregated with documents from other poorly ranked documents.)

Now that we’ve tapped into the vast content of the deep web we quickly discover that we’re drowning in content and not all of it is so relevant. What’s a web explorer to do with so many documents? We’ll explore this question next time.

Beyond Information Clutter

Let’s face it, there’s way too much content in the Internet. While finding enough of it was once a big deal that’s not the case anymore. Huge armies of web crawlers now make their home on the Internet, continuously examining and cataloging documents from all corners of the Web, making them all accessible through their search engines. Today’s problem is sorting through all that content, finding what you want, and keeping the clutter down.

Google is my favorite search engine but it has some serious limitations. In 0.23 seconds Google can identify more documents than I can read in a thousand lifetimes! Fine, you say, Google ranks them for you so what you’re likely to be most interested in will show up in the first 10 hits, right? Not always. Google mostly searches the surface web. That’s the collection of documents that web crawlers can easily catalog. It doesn’t include the deep web, which is the collection of content (much, much larger than the surface web collection) that typically lives inside of databases and has the higher quality scientific and technical information. Web crawlers don’t know how to search deep web collections so they miss much of the content that serious researchers are seeking. So, the millions of documents that Google finds might not include the ones you’re looking for, either in its first page of hits or in the first thousand pages of hits.

A more serious problem that Google faces, and I’m not picking on Google – all web crawlers have limitations – is that it doesn’t necessarily rank documents the way you would rank documents. Google ranks a web page highly if it’s popular, i.e. if lots of web pages reference it. Popular documents are not necessarily the ones most relevant to you. Google should be praised, however, in that it does rank in a way that is very useful to many people in many situations. Other search engines rank very poorly and some don’t rank in any useful way at all. This is especially true for deep web search engines.

Let’s consider one more problem that keeps you from finding what you really want, especially if you’re looking for technical or scientific content. For this type of content you’ll likely use a deep web search engine, like the one that Deep Web Technologies developed for www.science.gov which searches a number of technical and scientific databases and aggregates results into a single results page. If we consider that each of the document sources that returns documents ranks them in its own way then we have the problem of how to rank the documents within the aggregate result set. In other words, if one source ranks documents more highly if they’ve been published recently and a second source ranks documents in alphabetical order by title and a third source ranks documents by frequency of search terms within them how does one rank the aggregate set of documents returned from the three sources?

This relevance ranking problem as it’s called is a messy one but a very important one to solve because without good relevance ranking the result set becomes filled with clutter, i.e. with documents you didn’t want. The solution has two parts to it. First, identify what makes a document relevant to a researcher and, second, analyze all documents against those criteria. The first part is fairly straightforward. Good current and envisioned search utilities can allow the user to select from a set of criteria. Examples are publication date of article, length of article, frequency of search terms within it, proximity of search terms to one another, and presence of search terms early in the document. Solving the second part of the problem, the rank and aggregation part, turns out to be difficult to do and requires lots of CPU, storage and network resources and it places a burden on the computer hosting the documents. It essentially requires retrieving the full text of the documents being compared in the ranking process. Without retrieving and analyzing entire documents we have no way of measuring the worth of the document against the user-specified criteria since the collection itself may rank documents poorly and most likely not according to our standards.

Additionally, because different document sources rank differently the only way to rank all documents against one set of criteria, regardless of which collections they came from, is to ignore the order in which the source returns (ranks) the documents and apply our own ranking approach.

This approach which we call “deep ranking” is not for the impatient but for he who values a thorough search. Retrieving and analyzing the full text of thousands, perhaps tens of thousands of documents in response to a single query is best done in a batch- oriented environment, as it doesn’t lend itself to real-time processing. The model is one of submitting a search and receiving an email when the processing is completed with links to the most relevant documents. There is some instant gratification here, however, as our approach will retrieve and rank a number of documents quickly using our “QuickRank” technology, currently in production at www.science.gov. The second benefit is the knowing that a large number of potentially relevant documents have been scoured and that only the best ones have been retained. Additionally, that small set has been optimally ranked and tailored to the needs of the researcher. It’s worth the wait.

Deep Web Technologies has researched different ways to perform relevance ranking and has created a novel approach that can effectively mine large numbers of text document from heterogeneous sources and document type and produces a single set of well-ranked documents. The approach utilizes different algorithms for processing different types of documents at varying degrees of thoroughness. This approach can yield great benefits to pharmaceuticals, legal firms, biotechnology companies and other enterprises needing to effectively separate the clutter from the content.

[ Abe Lederman is founder and president of Deep Web Technologies, LLC, (DWT), a Los Alamos, New Mexico company that develops custom deep web mining solutions. His brother Sol supports Abe in a variety of ways. Visit www.deepwebtech.com for more information including links to sites successfully deploying DWT's sophisticated search applications. ]