The Web That's Hidden From You
Finding information online isn't as easy as it seems. Search sites are only able to view a tiny portion of what's available on the Internet. The rest of the information remains hidden in what experts call the deep or invisible Web. Digital culture commentator David Kushner joins host Liane Hansen to tell us more.
LIANE HANSEN, host:
Finding information online isn't as easy as it seems. Search sites are only able to view a tiny portion of what's available on the Internet. The rest of the information remains hidden in what experts call the deep or invisible Web. Digital culture commentator David Kushner joins us from our New York bureau to tell us more. Welcome back, David.
DAVID KUSHNER: Thank you.
HANSEN: So, how much and what kind of information is contained in the deep Web?
KUSHNER: Well, how much is a lot. Michael Bergman, who's the computer scientist who first coined the term earlier this decade, says that we're actually only searching maybe one percent of the total information that's available online. The rest of the information is there. The challenge is just, how do we get to it?
HANSEN: Right. But why couldn't you use, you know, your normal search engines?
KUSHNER: Well, the reason is is that, you know, search engines, despite really being pretty incredible, are still kind of dumb. And if you think about a search engine being like a spider that goes off and crawls across the Web looking for information that you're requesting, when it's doing this, it's only able to go from link to link. So, if there's something that's not formatted like a typical Web page, it will miss it entirely.
HANSEN: So, what's being done to improve access to this type of information?
KUSHNER: A few things. I mean, one is just that search engines are getting smarter. I mean, of course, you know, Google has a team devoted to the deep Web issues. They're finding more and more databases that they're putting up and making available to search.
Also, people, I think are learning to not just sit back and just become a little more proactive. Number one is that people who have this data are actually letting search engines know that their data exists. And they can do that just by contacting them.
And, also, you know, there are ways that you can kind of trick popular search engines into being more precise and finding information that's in the deep Web. For example, researchers at Berkeley found that you can improve results just by adding the word database to your searches. So, you could, for example, write, you know, toxic chemicals database, which would get you deeper than you could go otherwise.
HANSEN: Are there any deep Web search engines that can be used now?
KUSHNER: Yes, there are. The National Science Foundation and University of Utah has a site called DeepPeep.org, which is really great. There's a mathematician who launched a site called Wolfram Alpha, which is W-O-L-F-R-A-M-alpha.com. And there's Info Mind, Direct Search and others. And the good news is that you can look these sites up on Google or any of these other popular search sites to find out the names.
HANSEN: David Kushner is a writer who covers digital culture. He joined us from our New York bureau. Thanks, David.
KUSHNER: Okay. Thanks.
Copyright © 2009 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.