[Paper]: Revisiting Crawlers’ Role in a Search Engine

Following are the slides of my paper presentation at ICISA, 2010 in Seoul, Korea.

Analyzing Web Crawler as Feed Forward Engine for Efficient Solution to Search Problem in the Minimum Amount of Time through a Distributed Framework from M Atif Qureshi

This paper considers tradeoffs in web crawler design especially from the perspective of events versus threads[1,2]. The paper also makes some recommendations for better OS support for web crawling. It points out that the two principal problems with web crawling are:

• Choosing the right pages to crawl
  
• Basic architecture for performing the crawl

The focus of the work lies on the second problem with our proposition that events are the ideal way for implementation of web crawlers as events give better throughput while crawling the web. Furthermore we argue that the growing usage of search engines needs a careful redesign of the constituents of the search engine and that too from the perspective of systems software with the conclusion that the exokernel[3] is the right answer in removing some of the limitations of search engines today. We recommend having a future operating system dedicated to search engines.

If any of you is interested in more details I recommend him to contact me through email at atifms@kaist.ac.kr or matifq@yahoo.com. Moreover you can also request for a copy of the paper by personal email.

References

[1] von Behren, R., Condit, J., and Brewer, E. Why Events are a Bad Idea (for High-concurrency Servers). In 10th Workshop on Hot Topics for Operating Systems (HotOS IX), Lihue, Hawaii, May 2003.

[2] Ousterhout, J. Why threads are a bad idea (for most purposes). In Invited talk presented at 1996 USENIX Annual Technical Conference, San Diego, CA, October 1996.

[3] Engler, D. R., Kaashoek, M. F., and O'Toole, J. 1995. Exokernel: an operating system architecture for application-level resource management. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (Copper Mountain, Colorado, United States, December 03 - 06, 1995). M. B. Jones, Ed. SOSP '95. ACM, New York, NY, 251-266.

Blog Research: 19 GB data processing with 2 GB active RAM

Few days I was performing blogosphere analysis using my crawler "VisionerBot" that I recently presented at ICISA, Seoul, 2010. I had quite a tough time and not to forget a number of sleepless nights while I was on this task as the amount of data was extremely huge :), my task was to process blogs for interest of users over the blogspot domain. In this post I present the problem I was facing and the technique I used to overcome it:

Input: 69 Blogs

Objective:

1) Find 5,000 Blogs

2) Process at most 2,000 Blog posts per blog

Achievement

Total Blogs found: 5,067

Total Alive Blogs: 4,552

Total Number of Posts: 1,704,587

Problems:

Size of data that needs to be processed: 19 GB+

Size of available active RAM: 2 GB

Now what to do from here... When I started working on this, I never expected to find 1,704,587 posts with data size of 19 GB, I worked almost days and nights to get this data fit inside my desktop machine. If I used database for this experiment then it will cost me months to download this where as I had a deadline to complete this task within 10 days. There I gave birth to a new algorithm that I call as "Rack Algorithm" which downloads data in RAM until RAM gets filled and then it flushes the data on disk and cleans up RAM for remaining download process and this exercise continues until data is downloaded completely. After download comes the process of finding meaningful data out of that 19 GB and calling it to RAM to start processing and there I mananged to shrink its size size enough to manage it inside RAM. In this process I used (Key,Value) pairs along with lists.

Finally a sigh of relief, I have accomplished what seemed nearly impossilbe within 10 days, I am happy I managed to find opinion clusters of 1,704,586 posts by my coming algorithm that I call TDR (Topic Discussion Rank).

Welcome Post

So what should I write here? I think its time to write something random; something that gives me freedom to write whatever I do in small world of daily science.

Consider if computers were to be human, then I will have three more close friends in my life. Desktop machine, laptop and notebook. Why??? the reason is I spend most of my time with computer :)

I plan to share what ever I think , no matter how scattered or how confined.

I also intend to write things related to Linux; yeah! I am PC powered happily by Linux. So those of who are looking to try Linux feel free to come in contact with me. My prescription is Windows is a cancer for researchers and the cure is Linux. Even if you don't know how to operate Linux, then don't be afraid Linux is now more friendly than ever before, try Ubuntu.