User-Defined Query Term Weighting in Lucene

I am sharing a simple code with explanation on how Lucene (pyLucene to be specific) can be used for Query Expansion.

What I will not discuss here is how to devise a strategy for finding new terms for Query Expansion (a person can implement this on his/her own). But what I will explain here is, how one can assign different weights to query terms for retrieval task.

Consider four documents having following content
D1 -> 'pagerank pagerank algorithm'
D2 -> 'pagerank algorithm algorithm',
D3 -> 'pagerank',
D4 -> 'algorithm'

It implies our vocabulary of corpus is just 'pagerank' and 'algorithm', while corpus frequency of each term is 4 and document frequency is 3. Hence now idf and cf does not influence the scoring technique.

In the attached source code you can see that we have boosted term 'pagerank' by 10 times compared to term 'algorithm'. The query is 'pagerank algorithm'.

Upon retrieving the document D3 has 10 times higher score than D4 and likewise D1 has higher score than D2 (but not 10 times since the document's total terms are 3 which influences the scoring unlike in previously discussed case). Please run the source code and observe the results.

Source code: http://codeviewer.org/view/code:35db

Version: pylucene-3.6

EuroHCIR2013 Work Towards a New Search Interface namely Perspective-Aware Search

Recently an updated version was demoed  in SIGIR 2014: http://dl.acm.org/citation.cfm?id=2611184

There are occasions when search results do not satisfy the information need and give a completely undesirable set of results than what the user is looking for. A possible reason for this lies inside the returned documents which contain some perspectives while giving coverage to the topic and this perspective may be observed as bias by the user.

Lets take the following example scenarios:

• Consider a case where a user wishes to find information about a certain event (say, a bomb attack in a certain region). The search results returned, contain a majority of news reports blaming Islam (its implicit writing style) relating it with terrorism in most of the cases. This prompts the user to explicitly observe how much Islam is related with terrorism in the returned set of search results.
• Consider another case where a user wishes to find information about roles and rights of women in Islam but the search engine returns articles that contain a tendency of highlighting oppression against women instead of women rights and roles. In this case the user observes a correlation between women and oppression instead of factual position on rights.

In the above cases, the user's information need may lead him towards an explicit investigation of the underlying document collection and he/she may be interested in observing the amount of perspective tendencies in various search results (e.g., news reports). Current search engines do not facilitate this need by highlighting perspectives while displaying the search results. Hence, we propose the concept of "perspective-aware search." The proposed search interface enables the user to explicitly analyze search results with a touch of perspective awareness.

The following presentation contains some screen-shots of the proposed search interface; I will be giving a demo of this system at EuroHCIR Workshop that is co-located with SIGIR2013.

The system is built on top of the WikiMadeEasy API which is an API for mining Wikipedia data and is the output of work I am doing towards my PhD thesis. Feel free to contact me for more details of the API. The full paper describing the system can be found here.

Python: Reading large bz2 file with bz2.BZ2File()

There might arise a problem of partial (incomplete) reading of a file while reading a bz2 file in python.

The tip to overcome such a problem is very simple, uncompress the bz2 file using extraction utility (Ubuntu has the graphical utility by default). Once extracted, zip it back as bz2 and now try reading it again, this time you may have solved the problem.

Reason for the problem: the side that produced the bz2 file may have produced the bz2 file from multiple files which is not well recognized by bz2.BZ2File() functionality in python.

WWW2012: Solving the Media Crisis

Last week marked a significant period for the Internet as Lyon in France hosted the famous scientific conference namely World Wide Web (WWW) and was declared the World's Web capital for that week. Few people know that this is the same venue where Google was born as this is where Google co-founders Larry Page and Sergey Brin first presented their PageRank algorithm. This year's WWW conference was highly political in nature mainly due to the changing nature of today's Web and its significant role in major events all over the world. Many sessions focused on how the voices over new media (i.e. social media) is affecting the traditional media both in terms of new paradigms of news dissemination and the credibility of that news. In particular one panel named "Social Media Applications in News and Entertainment" comprised of journalists and media persons from famous media outlets of Europe (BBC, Deutsche Welle and AFP- Agence France-Press). Twitter was the top platform of choice for all the journalists on account of its speed and ease-of-use. However, all the journalists in the panel pointed to one thing namely the credibility of news on social media and methods for its verification. They pointed to some examples of false happenings disseminated through social media platforms. Denis Teyssou of AFP told the audience about how a fake picture of Osama bin Laden's death revolved around social media in 2007; AFP then used a computational graphics software to detect that the picture was fake and the news was false. A similar problem arose during the 2010 Haiti earthquake when a picture of 2008 Sichuan earthquake was included in a slideshow of 54 images from Haiti by Daily News. The photo was traced out to a random social media  activist and Daily News was heavily criticized by social media and traditional media circles alike. The point in focus is the huge challenge traditional media outlets face due to faster means of news dissemination: the problem is huge, the criticism from various circles is immense and the time is crucial. Even the recent Bhoja Air tragedy and corresponding burst of criticism on the traditional media by various social media activists points out the need for a solution to this traditional vs. social media crisis. We all speak of problems and shortcomings in various aspects of the media but when it comes to solutions, we are somewhat clueless and this for the better has to change.

Osama bin Laden fake death picture sent to AFP by a random social media activist

Siuchan Eartquake picture that was labeled to be Haiti Earthquake picture
by random social media activist

Being a Computer Scientist and having an analytical eye for things makes one approach problems differently: this was the key-point highlighted by all journalists during WWW2012 and Computer Science is the field where the journalists are turing for solutions. Most of them admitted that the advent of citizen journalism has left many in the media industry baffled and some even outraged as journalists by nature are arrogant people not having a habit of public engagement. Social media has considerably changed this leading to more and more public-media engagement but crucial problems still remain; it is the solution to these problems that will bridge the trust gap between the public and the media.


As part of a panel on Social Computing and Social Machines: A Research Agenda in Web Science track of WWW2012 I could relate to many of the concerns raised by the journalists and interaction with various panelists further strengthened my hypothesis on marriage between Computer Science and journalism. A similar belief about the way forward is held by sociologists and psychologists with data journalism seeming to be the way forward for all of us. As is the case with most analytic fields not many in the Pakistani media industry are familiar with data journalism and lack far behind than in this area unlike the well-established names such as Guardian, Huffington Post, and BBC. It could very well be that the solution to the media crisis lies right before us and we in our arrogance or let's put it softly ignorance are not turning towards it. Both the traditional media and social media circles have their own sets of problems and the way I see it collaboration can help solve the problems of both of them, and Computer Science is a good means of attaining that goal. Problems such as this is what has given birth to the new field of Web Science and this emerging field of Web Science aims to bring together scientists from various disciplines as the Web Science diagram shows.

Being a programmer from quite some time now and having been into the Computer Science research arena for about 3-4 years I along with my colleague and wife Arjumand Younus have set up a small Web Science research group at Institute of Business Administration in Karachi, Pakistan and this year our biggest achievement of so far came in the form of two research works by WebST IBA at WWW2012. Both of our works focus towards the media crisis. The first one titled "Investigating Bias in Traditional Media through Social Media" quantitatively studies bias in traditional media platforms through the use of social media text-mining and utilizes various text-mining similarity measures to identify differences in how both platforms report a news event. The second one titled "Traces of Social Media Activism from Malaysia and Pakistan" studies regional differences in activity of social media activists from two regions namely Malaysia and Pakistan and reports on some significant findings. We came to the conclusion that social media activists in Pakistan tend to do a large amount of mentions and retweets thereby endorsing each other's statements whereas Malaysian activists tend to post status updates more frequently. The research includes many other findings, feel free to contact us at the Web Science lab of IBA for further insights or if you are interested in joining any of these projects.

My visit to Russia, RuSSIR/EDBT 2011

Past week of my life was a very exciting experience (14 Aug.- 20 Aug. 2011), I along with my wife and colleague Arjumand went to St. Petersburg (Russia) for RuSSIR/EDBT summer school and International Conference centered around Information Retrieval (Computer Science). This was my first time ever to visit Russia and I was little confused about how it would be (as very little is known about Russia in Pakistan). Reaching there after long stretched hours of flight consumed me a lot in terms of energy. However, as soon I arrived I found my surroundings very normal and helpful as it was in South Korea, China and Malaysia (my previous experiences); everything looked usual and cooperative. Team of RuSSIR/EDBT arranged a nice hostel for the two of us which was in fact a studio apartment and better than where I had ever stayed previously during my short trips. Russian people could barely talk in English but were polite and helpful (thumbs up!).

On 15 Aug. we went to the school venue for our educational commitment through a bus followed by metro (simply subway or rail) and then a short walk to the campus of St. Petersburg State university. We faced little confusion in reaching our destination but managed to reach through words of passing by people. In the gathering, there were fine lectures for full five days as part of summer school and nice papers and posters during the conference. Our daily routine from 15-19 Aug. had a duration from 9:00 AM till 9:00 PM and every second of it paid off in terms of learning. In breaks we had chances to socialize with people hence forming future possible collaborative/cooperative connections. Suggestions and fine word of questions led us to have a feedback which could not be possible to generate within a week (usually). During the gathering, we witnessed people from social sciences taking keen interest in aspects of social networks which shows how aware they were about fields getting merged. A guy from industry, Yandex (similar to Google in Russia) came to me after reading my research paper for interacting and this shows how practical and serious things were there. Similar meaningful was my interaction with guy from mail.ru (another giant in Russian industry). All in all it was a very nice experience and I feel like typing in something for it as a token of appreciation.

I close my post by saying thanks to the team of RuSSIR/EDBT for organzing such an event and I also want to thank to the participants who put further five stars to the gathering.

As a note for Pakistani people who know a very little about Russia: it is a nice place with nice people and very open place for knowledge cooperation and collaborations, do consider it as a meaningful option for academics and industry. Further, feel free to write back to me.

RuSSIR/EDBT 2011 link: http://romip.ru/edbt-russir2011/

WWW2011: Fellowship Achieved, Research Paper Approved But Indian VISA Denied

I had planned to attend WWW 2011 this year which begins today. Every researcher working on the Web and its related technologies knows that WWW is the greatest academic conference for Web researches with an acceptance rate as low as below 20% . It is the same conference where Google founders Larry Page and Sergey Brin presented their famous PageRank algorithm. The venue for this year's WWW Hyderabad, India was particularly exciting for me as I always had a wish to visit my neighboring country India and this year I felt it's a natural chance for me.

WWW 2011 is being organized by IIIT Bangalore and they also announced few fellowships for attending the conference. I applied for the NIXI fellowship which was a fellowship program aimed at encouraging faculty and students from under-represented parts of the world, e.g. developing countries and poor regions of developed countries, etc. to have representation in this year's WWW as the theme in particular was "Web for All". Hence, in line with the theme preference for NIXI fellowship was given to those who had not attended a WWW conference in the past, who were from a region not already having considerable presence at WWW, and who had a demonstrated interest in web and its technologies. Based on my publications related to the Web and my credentials in this field, I got the NIXI fellowship. I along with my wife were the only Pakistanis attending WWW 2011, and this to the best of my knowledge is the first ever representation of Pakistanis in a WWW conference.

Then began the process of applying for Indian VISA which turned out to be the bottleneck. I was short on time and Indian VISA application center in Seoul informed me that the process for Pakistani passport holders takes a month. I then approached the Indian Embassy in Seoul and I found them to extremely co-operative, they called me to their office in Seoul and were ready to give me special consideration on account of my research profile. They said they could give me VISA in one day if I showed them clearance from Ministry of Home Affairs, India which happens to be the main requirement for getting conference VISA for India. And then began a huge round of email exchanges with the WWW Secretratriat who happened to be from IIIT Bangalore and was highly, highly co-operative throughout. Despite their tremendous efforts the Ministry of Home Affairs (MHA), India refused to give me clearance without stating any reason for the denied clearance.

IIIT Bangalore and their staff did all what they could (they were simply great people to interact with). However MHA stood against values of science and they were unable to understand importance of scientific merit. Indian leading school would learn a lot from this, MHA is shutting off doors of science (without even bothering for any sort of explanation). I was invited there with delegate status, and even then MHA ignored seemingly because of carrying Pakistani Passport with status on merit i.e., got a research paper approved and got the fellowship. If such profiles are not invited then who else could they invite naturally and intellectually. I believe there is a lesson to learn on both sides (specially for intellectuals who could turn things by using intellect). MHA has no idea, they are killing science and I am not the only one who is denied, one German scientist was asked for Birth certificate (God knows what on earth they are thinking) and a Russian scientists was denied VISA due to her passport expiry period within 3 months (however, she is renowned name in the field and co-founder of famous www.tweetedtimes.com).

So today, WWW 2011 begins in Hyderabad, India. I had thought of writing a blog post covering the various sessions of the conference but it was not to be. But despite all that I wish all the best from my side to WWW 2011 organizers and attendees; would love to see the tweets coming from there: #www2011. I have already read some of the WWW 2011 papers and seems to be a pretty exciting conference this year.

The Yellow Banded Graduates

These were my feelings on the day of KAIST Graduation Ceremony 2011 i.e, 11th February, 2011 . I have managed the time to post this on my blog today. Here are my feelings for that day:

Finally, the long awaited day has arrived, the tradition of passing on the light at KAIST. The difference this time, me and my wife get two degrees of MS by Research as only Pakistanis in Computer Science in world's 21st (this year it's 24th) ranked university under Technology (simply, Asian MIT). With a lot of hopes and a mix of emotions, the yellow banded graduates head to the ceremony. This is a product of hard work of everyone associated with us (in terms of academics, advices, emotions, etc.,) and importantly a responsibility to pay back to the seed land, Pakistan. Never felt but the truth is, every single degree counts for Pakistan (part of Muslim world). Hopes went on to height as we went on to recieve something which looked impossible if life goes into flash back of more than two and a half years with no money and no sense of how to pursue higher education in ever challenging field of Computer Science. I can't mention everyone who stood at our backs in the tough times, but one person I can never forget to mention is my mother as she is the lady who turned me into what I am and led me after passing of my father when I had just passed 7 grades of basic schooling. My mother made me look to the world through her eyes and turned me into what made me a yellow banded KAISTIAN, she completed her PhD after the passing of my father and taught me by her living example before me to target higher than the known limits of the world, a lady who not only took great care of her children but also attained biggest pride in education (i.e., PhD) after passing of her life partner and my father. This by example, shaped me well and I am taking not one degree but two, one of mine and other of my wife.

Having said some of my background now comes some snaps that now become part of golden memories :)