|
Introduction
This article explains how a Search Engine can search a Website for words and phrases that are exact matches or likely matches to all of the words or some of the words a user is searching for. It reviews how a Text Search Engine is incorporated into a Website to search Web pages in the form of Weblogs, Forums and Picture Galleries. The Website in review is the popular CommunityServer used by Fortune 100 companies, small start-up businesses and schools. The Text Search Engine powers the popular ASP.NET Forums website with almost 1 million posts, MSDN Blogs and Xbox Forums. The Search Engine was created with C#, ASP.NET and SQL Server.
This Search Engine does not incorporate web crawling used by Google, MSN, Yahoo and other Search Engines for spidering content hosted on alien websites and databases. Community Server’s ASP.NET Search Engine assumes that the content in the Web pages are already available on a database, hence, it does not spider the Web page to access it like some other Search Engines. While the tasks common to Search Engine include
· Spidering a Website
· Indexing static or dynamics text in Web pages
· Searching the indexed text
Community Server’s ASP.NET Search Engine only indexes text and searches indexed text.
Why Learn How A Real World ASP.NET Search Engine Works?
Learning how a real world ASP.NET Search Engine works is important because
· Searches by users on individual Websites form a sizeable part of the overall number of searches performed by users all over the world
· Text searching has become an important feature of most websites. Many Sites thrive on the traffic created by the big three Search Engines (Google, MSN and Yahoo). On arriving at a website, an online user expects a local Search Engine to ferry the user to the set of pages that contain the most exact matches to the user’s topics of interest.
· Local Searches on a Website have to be very quick. The days of long running search queries are over. If a Website cannot quickly and reliably retrieve a match for the user’s Search, the user typically hops to a competitor.
· Giving the reasons above, you will think that there is a plethora of readily usable real world information on ‘How To Create And Add Your Own Search Engine’, but this is not true. You will find some good information on ‘how to search text on Web pages saved as files on a hard drive’, but not a lot on how to search text on web pages saved as text on a database.
· A lot of Web pages are generated from a database both for dynamic pages and the so-called static pages. SQL Server has emerged as a common and scalable database platform. This article clearly exposes how you can integrate a database platform like SQL Server or any other relational database with ASP.NET, another popular web development framework.
· There will be instances when you have to write your own Search Engine for mining text in formats and media which are not readily accessible to other Search Engines. Furthermore, you may need your own custom Searching solution for several different reasons. This article offers you a template for a customizable, real world Search Engine. Understanding the architecture of this Search Engine could be valuable to you in designing a Search Engine for your own custom searching needs.
In
the first part I
explained how Community Server’s ASP.NET Search Engine first indexes text or the content of a Web page and saves the indexed text to a database. This article is Part 2, and refers to searching text which has been indexed and saved in the database as covered in Part 1.
After the Text contained in a Weblog, Forums, Gallery or Web page is indexed, it is ready to be searched for matches to a user's query. Text is searched when the GetSearchResults method of the Search class is called with the user’s search query. The ForumsSearch, GallerySearch and WeblogSearch classes derive from the Search class and are used in searching text in Weblogs, Forums and Picture Galleries specifically.
The GetSearchResults method is overridden by classes deriving from the Search class including the ForumsSearch and WeblogSearch class to execute a local search for possible matches to a user’s search query. This means that in the ForumsSearch class, for example, an instance of the ForumsDataProvider is created and the GetSearchResults method executed:
protected override
SearchResultSet GetSearchResults(SearchQuery query, SearchTerms terms){
ForumDataProvider fdp = ForumDataProvider.Instance();
SearchResultSet results = fdp.GetSearchResults(query, terms);}
The ForumsSearch GetSearchResults method delegates the actual work of searching the database for possible matches to a Search to the ForumsSqlDataProvider.GetSearchResults method, which creates a dynamic SQL statement in
string
searchSQL = SqlGenerator.SearchText(query,terms,GetSettingsID(),ApplicationType.Forum);
and searches the indexed posts for matches to the search terms with a call to the stored procedure cs_forums_Search.
Note that the size of the dynamic SQL statement which includes the search terms is limited to a variable length Unicode string which is less than 4000 characters. The dynamic sql search statement is a sql statement for searching against the Search Barrel table cs_SearchBarrel.
While the GetSearchResults() method is responsible for matching a set of words otherwise known as search terms against the cs_Posts and cs_SearchBarrel tables, the GetSearchTerms() method is responsible for creating a set of words or search terms which can be matched in the cs_Posts and cs_SearchBarrel tables.
The GetSearchTerms of the Search class receives a user's search as a SearchQuery object, strips HTML tags from the Search Query, tokenizes the clean Search Terms and breaks up the tokenized into "And" and "Or" search terms and returns the final result set to the GetSearchResults method of the Search class. It converts words created by a user into a form suitable for matching against the cs_SearchBarrel and cs_Posts table.
|