Introduction

This article explains how a Search Engine can search a Website for words and phrases that are exact matches or likely matches to all of the words or some of the words a user is searching for. It explains how a Text Search Engine is incorporated into a Website to search Web pages in the form of Weblogs, Forums and Picture Galleries. This Text Search Engine known as the Community Server Search Engine powers Text Searches on a number of high traffic Websites based on the CommunityServer Website including the popular ASP.NET Forums website with almost 1 million posts, MSDN Blogs and Xbox Forums. The Search Engine was created with C#, ASP.NET and SQL Server. CommunityServer is also used by Fortune 100 companies, small start-up businesses and schools.

This Search Engine does not incorporate Web crawling as used by Google, MSN, Yahoo and other Search Engines for spidering content hosted on alien websites and databases. Community Server’s ASP.NET Search Engine assumes that the content in the Web pages are already available on a database, hence, it does not spider a Web page to access it. Note : While the tasks common to Search Engines include

Community Server’s ASP.NET Search Engine only indexes text and searches indexed text saved on a database. It does not spider and index text on a Web page.

Why A Real World ASP.NET Search Engine Article?

Learning how a real world ASP.NET Search Engine works is important because

There will be instances when you have to write your own Search Engine for mining text in formats and media which are not readily accessible to other Search Engines. Furthermore, you may need your own custom Searching solution for several different reasons. This article offers you a template for a creating a flexible real world Search Engine. Understanding the architecture of this Search Engine could be valuable to you in designing a Search Engine for your Website’s custom searching needs.

Text Searching and Text Indexing

Text searching in CommunityServer's ASP.NET Search Engine is done mainly by the Search Class .

The Search class consists of two major functions public override void IndexPosts(int setSize, int settingsID) which indexes the text on a Web page and protected override SearchResultSet GetSearchResults(SearchQuery query, SearchTerms terms) which searches text which has been indexed and saved to the database.

Both functions used in text indexing and searching are members of the Search class and classes deriving from the Search class including the WeblogSearch class, ForumsSearch and GallerySearch class.

The WeblogSearch class implements these methods from the base Search class:

The IndexPosts method indexes the text of a Weblog in a website (each website has an associated SettingsId) in the database for search queries created by the GetSearchResultts method.

To make a search a Weblog, the GetSearchResults method executes the two statements below:

Text Indexing

To index the text of a Weblog, the IndexPosts method of the WeblogSearch class creates an instance of the WeblogDataProvider class and calls the SearchReindex method to get a list of Weblog posts or text which will be indexed:

The PostSet object is a collection of Weblog post objects . By iterating through the Weblog posts in the PostSet collection, each post can be indexed for searching. The PostSet collection could be a collection of Forum posts or Web pages where each post in the collection is a single Web page. If a collection of Forum posts needs to be indexed for example, an instance of the ForumDataProvider class is created and the SearchReindexPosts method executed to return a collection of posts:

An index-able Post object could consist of different types of text. For example, if a post object represents a Web page, then the Web page can be defined as a Post object consisting of Title, Keywords, Description and Author Meta Tags, a Subject and Body. Likewise, a Forum post consists of a Username (post author), Subject and Body. To index text, a call is made to the Index method of the Post object:

//Index the Post Author

//Index the Post Subject

//Index the Post Body

words = Index(post.Body, words, WordLocation.Body, settingsID);

if the Post object represented a Web page and the meta tags needed to be considered by this Search Engine, the indexing of the meta tags could be made by calls similar to:

//Index the Keywords Meta Tag

words = Index(post.Keywords, words, WordLocation.Keywords, settingsID);

//Index the Description Meta Tag

words = Index(post.Description, words, WordLocation.Description, settingsID);

After the specific parts of a Post are indexed, a call is made to the famous SearchBarrel to save the indexed text in the database:

InsertIntoSearchBarrel(words, post, settingsID, totalBodyWords) where the totalBodyWords parameter is created by counting the total number of words in the Post body with the method

totalBodyWords = CleanSearchTerms(post.Body).Length;

Now, to the InsertIntoSearchBarrel method, public void InsertIntoSearchBarrel(Hashtable words, Post post, int settingsID, int totalBodyWords) which inserts all the indexed words in a post in the Community Server cs_SearchBarrel table.

InsertIntoSearchBarrel iterates through all the keys in the Hashtable object words, retrieves the word or indexed text for each key and calculates the weight of the word using the CalculateWordRank method,

Word w = words[key] as Word;

w.Weight = CalculateWordRank(w, post, totalBodyWords);}

and then saves the Hashtable of indexed words in the database:

CommonDataProvider dp = CommonDataProvider.Instance();

dp.InsertIntoSearchBarrel(words, post, settingsID);

Finally, the InsertIntoSearchBarrel method of the SqlCommonDataProvider class is executed to save the indexed text in the database using the stored procedure cs_Search_Add

Since the SqlCommonDataProvider.InsertIntoSearchBarrel method receives a Hashtable containing indexed words, the entire Hashtable of indexed words or posts is saved by repeatedly calling the cs_Search_Add stored procedure passing in an indexed word each time

word = (Word) words[wordHash];