THE DEEP WEB

CHAPTER ONE

1.0     INTRODUCTION

The volume of information on the web is already vast and is increasing at a very fast rate according to Deepweb.com [1]. The Deep Web is a vast repository of web pages, usually generated by database-driven websites, that are available to web users yet hidden from traditional search engines. The computer program that searches the Internet for newly accessible information to be added to the index examined by a standard search tool search engine [2] used by these search engines to crawl the web cannot reach most of the pages created on-the-fly in dynamic sites such as e-commerce, news and major content sites, Deep web.com [1].

According to a study by Bright Planet [3], the deep web is estimated to be up to 550 times larger than the ‘surface web’ accessible through traditional search engines and over 200,000 database-driven websites are affected (i.e. accessible through traditional search engines). Sherman & Price [4], estimates the amount of quality pages in the deep web to be 3 to 4 times more than those pages accessible through search engine like Google, About, Yahoo, etc. While the actual figures are debatable, it made it clear that the deep web is far bigger than the surface web, and is growing at a much faster pace, Deep web.com [1].

In a simplified description, the web consists of these two parts: the surface Web and the deep Web (invisible Web or hidden Web) but the deep Web came into public awareness only recently with the publication of the landmark book by Sherman & Price [4], “The invisible Web: Uncovering Information Sources Search Engines Can’t See”. Since then, many books, papers and websites have emerged to help further explore this vast landscape and these needs to be brought to your notice too.

• Statement of Problem

Most people access Web contents with Surface Search Engines and 99% of Web content is not accessible through Surface Search Engines.

A complete approach to conducting research on the Web incorporates using surface search engines and deep web databases. However, most users of the Internet are skilled in at least elementary use of search engines but the skill in accessing the deep web is limited to a much smaller population. It is desirable for most user of the Web to be enabled to access most of the Web content.  This work therefore seeks to address problems such as how Deep Web affects: search engines, websites, searchers and proffered solutions.

• Objective of the study

The broad objective of this study is meant to aid IT researchers in finding quality information in less time. The main objective of the project work can be stated more clearly as follows:

1. To describe the Deep Web and Surface Web
2. To compare deep web and surface web
3. To develop a piece of software to implement a Deep Web search technique

• Significance of the study

The study on deep web is necessary because, it brings to focus problems encountered by search engines, websites and searchers. More importantly, the study will provide information on the results of searches made using both surface search engines and deep web search tools. Finally, it presents deep web not only as a substitute for surface search engines, but as a complement to a complete search approach that is highly relevant to the academia and the general public.

 

• Literature review

What is Deep Web?

Wikipedia [5], defined the surface Web (also known as the visible Web or indexable Web) as that portion of the World Wide Web that is indexed by conventional search engines. Search engines construct a database of the Web by using programs called spiders or Web crawlers that begin with a list of known Web pages. For each page the spider knows of it retrieves the page and indexes it. Any hyperlinks to new pages are added to the list of pages to be crawled. Eventually all reachable pages are indexed, unless the spider runs out of time or disk space. The collection of reachable pages defines the surface Web.

For various reasons (e.g., the Robots Exclusion Standard, links generated by JavaScript and Flash, password-protection) some pages cannot be reached by the spider. These ‘invisible’ pages are referred to as the Deep Web.

Bergman [6], defined the deep Web (also known as: Deepnet, invisible Web or hidden Web) to mean World Wide Web content that is not part of the surface Web indexed by search engines. Dr. Jill Ellsworth coined the term “Invisible Web” in 1994 to refer to websites that are not registered with any search engine.

Sherman and Price [4], defined deep web as text pages, files, or other often high-quality authoritative information available via the World Wide Web that general-purpose search engines cannot, due to technical limitations, or will not, due to deliberate choice, add to their indices of Web pages. Sometimes referred to as invisible web” or “dark matter”

          Origin of Deep Web

In 1994, Dr. Jill H. Ellsworth a university professor who is also an Internet consultant for Fortune 500 companies was the first to coin the term “Invisible Web” [6]. In a January 1996 article, Ellsworth states: “It would be a site that’s possibly reasonably designed, but they didn’t bother to register it with any of the search engines. So, no one can find them! You’re hidden. He called that the invisible Web”.

The first commercial deep Web tool (although they referred to it as the “Invisible Web”) was AT1 (@1) from Personal Library Software (PLS), announced December 12^th, 1996 in partnership with large content providers. According to a December 12^th, 1996 press release, AT1 started with 5.7 terabytes of content which was estimated to be 30 times the size of the nascent World Wide Web.

Another early use of the term “invisible web” was by Bruce Mount (Director of Product Development) and Dr. Matthew B. Koll (CEO/Founder) of Personal Library Software (PLS) when describing AT1 (@1) to the public. PLS was acquired by AOL in 1998 and AT1 (@1) was abandoned [7], [8].

AT1 is an invisible web which allows users to find content “below, behind and inside the Net” therefore users can now identify high quality content amidst multiple terabytes of data on the AT1 Invisible Web; top publishers join as charter members.

• The Internet and the Visible Web

The primary focus of this project work is on the Web and more specifically, the parts of the Web that search engines can’t see (known as the invisible Web) but in order to fully understand the phenomenon called the Invisible Web, it is important to first understand the fundamental differences between the Internet and the Web.

Most people tend to use the words “Internet” and “Web” interchangeably, but they are not synonyms. The Internet is a networking protocol (set of rules) that allows computers of all types to connect to and communicate with other computers on the Internet. The Internet’s origin traced back to a project sponsored by the U.S. Defense Advanced Research Agency (DARPA) in 1969 as a means for researchers and defense contractors to share information. The World Wide Web (Web), on the other hand, is a software protocol that allows users to easily access files stored on the Internet computers. The Web was created in 1990 by Tim Berners-Lee, a computer programmer working for the European Organization for Nuclear Research (CERN). Prior to the Web, accessing files on the Internet was a challenging task, requiring specialized knowledge and skills. The Web made it easy to retrieve a wide variety of files, including text, images, audio, and video by the simple mechanism of clicking a hypertext link. Hypertext is a system that allows computerized objects (text, images, sounds, etc.) to be linked together, while a Hypertext link points to a specific object, or a specific place with a text; clicking the link opens the file associated with the object [4].

The Internet is a massive network of networks, a networking infrastructure. It connects millions of computers together globally, forming a network in which any computer can communicate with any other computer as long as they are both connected to the Internet. Information that travels over the Internet does so via a variety of languages known as protocols.

The World Wide Web, or simply Web, is a way of accessing information over the medium of the Internet. The Web uses the Hypertext Transfer Protocol (HTTP protocol), as one of the languages spoken over the Internet, to transmit data. The Web also utilizes browsers to access Web documents called Web pages that are linked to each other via hyperlinks. Web documents also contain graphics, sounds, text and video.

The Web is just one of the ways that information can be disseminated over the Internet. The Internet, not the Web, is also used for e-mail, relies on Simple Mail Transfer Protocol (SMTP, the standard protocol used in the Internet for mail transfer and forwarding), Usenet, news groups, instant messaging and File Transfer Protocol (FTP, a network protocol used to transfer data from one computer to another through a network, such as the Internet). So the Web is just a portion of the Internet, albeit a large portion, but the two terms are not synonymous and should not be confused.

1.4.2  How the Internet came to be

Up until the mid-1960s, most computers were stand-alone machines that did not connect to or communicate with other computers. In 1962 J.C.R. Licklider, a professor at The Massachusetts Institute of Technology (MIT, a private, coeducational research university located in Cambridge, Massachusetts.), wrote a paper envisioning a globally connected “Galactic Network” of computers [4]. The idea was far-out at the time, but it caught the attention of Larry Roberts, a project manager at the U.S. Defense Department’s Advanced Research Projects Agency (ARPA). In 1996 Roberts submitted a proposal to ARPA that would allow the agency’s numerous and different computers to be connected in a network similar to Licklider’s Galactic Network.

Robert’s proposal was accepted, and work began on the “ARPANET”, which would in time become what we know as today’s Internet. The first “node” on the ARPANET was installed at UCLA in 1969 and gradually, throughout the 1970s, universities and defense contractors working on ARPA projects began to connect to the ARPANET.

In 1973, the U.S. Defense Advanced Research Projects Agency (DARPA) initiated another research program to allow networked computers to communicate transparently across multiple linked networks. Whereas the ARPANET was just one network, the new project was designed to be a “network of networks”. According to Vint Cerf, widely regarded as one of the “fathers” of the Internet, the Internetting project and the system of networks which emerged from the research were known as the “Internet” [9].

 

https://www.modishproject.com/get-the-complete-research-project-topics-and-materials/