Deep Web, Anyone?

As I work with businesses to help them become more visible on the web using eMarketing and search engine optimization techniques, I'm often asked about the "invisible web" or "deep web" whenever it's highlighted by the media.

It's a loosely term that refers to web content often requiring password authorization to access. It's not available without registration and/or payment, so it's "hidden" from the big search engines like Google, Yahoo , and AOL .

Reid Goldsborough wrote in the Philly Inquirer about several web sites that help plumb the depths of the deep web. Here are several of his cited sites:

www.brightplanet.com - informative resource; be sure to check out their faq's.

www.ga.gov.au/oracle/nukexp_form.jsp - an example of specialized knowledge found in a research database

infomine.ucr.edu - search scholarly journals

search.yahoo.com/subscriptions - get a peek at what's in a dozen subscription-only services, such as Factiva, Consumer Reports, and the Wall Street Journal online

turbo10.com - aggregates search results across a configurable group of search engines; works best with Firefox.



In my experience, the deep web is interesting and useful when you are looking for detailed (or classified) information on a particular subject written by an expert. Examples would be industry or company reports for publically traded companies written by stock analysts, tracking down an individual's contact information, gathering scientific data that's privately published.

Metropolitan and national newspaper sites (ones that have the resources to create and maintain archives), often make all stories available for 30 days, and require special access for content that's older.

Yahoo is creating an aggregate site of subscriber-only content from the Wall Street Journal, Consumer Reports, and Lexis/Nexus. You'll be able to get search summaries of the results, but to read the full article, you'll need to subscribe to the source site and pay for the content.

Google has several projects under way that are exposing deep web content to the surface searches because it fits their business model. Google was also a pioneer in searching and indexing non-html content, such as Word docs and PDF files.

When you're building astronger business, you want to be aware of options and resources available to help you.