Nation and its Progress: Values and Material Strength

According to my personal understanding values and material strength define a nation's progress. Values can be easily understood as ethical lines such as liberalism, secularism, faith, etc. Material strength implies understanding/application of science and technology in order to benefit from it. I will give an analogy to explain my understanding: value or ethical line is like a spirit and material strength is the body in which the spirit lives, and together these two make up a living body. Absence of any of the two would lead to death, i.e., a body cannot survive without spirit in it and likewise spirit without a body is meaningless. Therefore, good values with no material strength would leave a nation nowhere but at the mercy of other nations who have better material strength. Likewise, a nation with no sense of values can lead a barbaric lifestyle and hence, may enter into committing societal suicide with its own technology such as inflicting wars inside and outside for selfish gains. These unjust wars inflicted outside the nation will make the (waging war) nation insecure from within morally because the returning ruthless soldiers will themselves make their own land insecure with their ruthless behaviours. Similarly without understanding of values the rich will exploit the poor and for this the rich will use best-known technology, laws and rules (twisting them in their favour) to exploit the poor, and hence, eventually the nation will observe protests and friction within its own citizens causing further deterioration.

Being a "Muslim" and from "Pakistan", I will now limit the scope of my writing exclusively towards these two entities. however, I am confident that people belonging to similar (specially from Muslim World) or dissimilar entities can find some general points as a take-away. So, being a Muslim, I will simply state Islam as being the ethical line for a Muslim (I assume that the person is practising the faith), and being a Pakistani I know we have not reached scientific enlightenment in different spheres of life by a long shot. So what goes around in Pakistan is like this, some of us observe that the problem lies in not practising good values, therefore those of us choose to learn and practice values and this is where our youth takes inspiration in learning Islam. On the other hand others consider that improvement of scientific progress is the way forward, and therefore, these people resume to take inspiration from science, a few of them actually learn it or perfect it but most of them only get satisfied by finding heroes in science (of-course scientific enlightenment is not easy for everyone). I personally believe that a win-win here is to have people who addresses both issues simultaneously. However, they may choose to gain expertise in one but they should have understanding of both. A lot of us have heard the phrase "education brings progress", I agree but I feel sometimes people who say this sentence do not actually comprehend the deep meaning behind this phrase because education is not only about science (which some people actually imply by the sentence) but it is also about learning good values and practising them. It would be last thing for a progressive society to find itself with good technology but with barbaric behaviours towards each other (I don't see harmony here). Some examples where we are deficient of good values are the following:

1- We comment on others' mistakes but when someone points our own error that becomes the last day of friendship with an implicit expression of "how dare you?"
2- We are friends with each other but say bad things behind their backs.
3- We follow a political party and the wrong of that political party becomes justified because other political parties are also doing wrongs.
4- We break our promises so easily; I will be there at 8:00 PM but until 9:00 PM there is no sign of me being there.
5- In weddings showing off money is more important than actually collecting the prayers of guests for the new couple.



I will conclude my article to show how much we much we actually respect ethical lines (ours is Islam) and scientific progress as a nation.

"If a child is not sharp or intelligent send him/her to Madersa (Islamic school) and if a person is not fit for a challenging job then let him/her do a PhD degree". 

We need our top brass in Islamic schools and likewise, we need our top brass in PhD programs, but so far our collective efforts as a nation is less than what is sufficient.

[SIGIR2014 Demo] A System-Oriented View Towards Bias in Search Process: Visualizing Perspectives in News

The notion of "bias" in search results has been investigated by the information retrieval research community. However, so far all investigations seem to take a user-oriented view of "bias" when considering the search process i.e., the user's tendency to click on results that are highly positioned within a search engine's ranked list, or the user's tendency to click on results that have more query terms in the search result title or summary. In the proposed demonstration accepted at SIGIR 2014, we take a system-oriented approach towards "bias" within the search process and offer a new interface for users to investigate the "perspective biases" in documents returned by a search engine.

To clearly illustrate what we mean by "perspective bias", we have focused on the news domain where the inherent bias lies for the most part within the news collection itself such as news web sites having a "leftist" or "rightist" agenda. Consider a case in which a user wishes to find information about a certain event (say, a bomb blast in a certain region). The search results returned may be polarized instead of focusing on factual aspects i.e., relating to a certain race, ethnicity, or political movement which caused violence. This can prompt a user to explicitly evaluate a move from objective factual reporting to subjective reporting within the top results and this is where perspective-aware search comes to the rescue as shown in the figure below. Here, the user is asked to input a normal search query and a perspective allowing the user to highlight the presence of a perspective in the search results.

For the purpose of demonstration, the system returns the top 10 news stories for the query from Bing, Yahoo and Google and then calculates a perspective score for each result while at the same time using graph visualizations to illustrate the perspective scores for each news source and each search engine.

Below is a video demonstration of the "perspective-aware search system". I will be attending SIGIR 2014 in Gold Coast, Australia and for that I owe a special thanks to SIGIR Travel Grants Committee who has funded my travel to SIGIR 2014. See you Information Retrieval folks in Australia where I will be available to explain more aspects of this novel search interface.

User-Defined Query Term Weighting in Lucene

I am sharing a simple code with explanation on how Lucene (pyLucene to be specific) can be used for Query Expansion.

What I will not discuss here is how to devise a strategy for finding new terms for Query Expansion (a person can implement this on his/her own). But what I will explain here is, how one can assign different weights to query terms for retrieval task.

Consider four documents having following content
D1 -> 'pagerank pagerank algorithm'
D2 -> 'pagerank algorithm algorithm',
D3 -> 'pagerank',
D4 -> 'algorithm'

It implies our vocabulary of corpus is just 'pagerank' and 'algorithm', while corpus frequency of each term is 4 and document frequency is 3. Hence now idf and cf does not influence the scoring technique.

In the attached source code you can see that we have boosted term 'pagerank' by 10 times compared to term 'algorithm'. The query is 'pagerank algorithm'.

Upon retrieving the document D3 has 10 times higher score than D4 and likewise D1 has higher score than D2 (but not 10 times since the document's total terms are 3 which influences the scoring unlike in previously discussed case). Please run the source code and observe the results.

Source code: http://codeviewer.org/view/code:35db

Version: pylucene-3.6

EuroHCIR2013 Work Towards a New Search Interface namely Perspective-Aware Search

Recently an updated version was demoed  in SIGIR 2014: http://dl.acm.org/citation.cfm?id=2611184

There are occasions when search results do not satisfy the information need and give a completely undesirable set of results than what the user is looking for. A possible reason for this lies inside the returned documents which contain some perspectives while giving coverage to the topic and this perspective may be observed as bias by the user.

Lets take the following example scenarios:

• Consider a case where a user wishes to find information about a certain event (say, a bomb attack in a certain region). The search results returned, contain a majority of news reports blaming Islam (its implicit writing style) relating it with terrorism in most of the cases. This prompts the user to explicitly observe how much Islam is related with terrorism in the returned set of search results.
• Consider another case where a user wishes to find information about roles and rights of women in Islam but the search engine returns articles that contain a tendency of highlighting oppression against women instead of women rights and roles. In this case the user observes a correlation between women and oppression instead of factual position on rights.

In the above cases, the user's information need may lead him towards an explicit investigation of the underlying document collection and he/she may be interested in observing the amount of perspective tendencies in various search results (e.g., news reports). Current search engines do not facilitate this need by highlighting perspectives while displaying the search results. Hence, we propose the concept of "perspective-aware search." The proposed search interface enables the user to explicitly analyze search results with a touch of perspective awareness.

The following presentation contains some screen-shots of the proposed search interface; I will be giving a demo of this system at EuroHCIR Workshop that is co-located with SIGIR2013.

The system is built on top of the WikiMadeEasy API which is an API for mining Wikipedia data and is the output of work I am doing towards my PhD thesis. Feel free to contact me for more details of the API. The full paper describing the system can be found here.

Python: Reading large bz2 file with bz2.BZ2File()

There might arise a problem of partial (incomplete) reading of a file while reading a bz2 file in python.

The tip to overcome such a problem is very simple, uncompress the bz2 file using extraction utility (Ubuntu has the graphical utility by default). Once extracted, zip it back as bz2 and now try reading it again, this time you may have solved the problem.

Reason for the problem: the side that produced the bz2 file may have produced the bz2 file from multiple files which is not well recognized by bz2.BZ2File() functionality in python.

WWW2012: Solving the Media Crisis

Last week marked a significant period for the Internet as Lyon in France hosted the famous scientific conference namely World Wide Web (WWW) and was declared the World's Web capital for that week. Few people know that this is the same venue where Google was born as this is where Google co-founders Larry Page and Sergey Brin first presented their PageRank algorithm. This year's WWW conference was highly political in nature mainly due to the changing nature of today's Web and its significant role in major events all over the world. Many sessions focused on how the voices over new media (i.e. social media) is affecting the traditional media both in terms of new paradigms of news dissemination and the credibility of that news. In particular one panel named "Social Media Applications in News and Entertainment" comprised of journalists and media persons from famous media outlets of Europe (BBC, Deutsche Welle and AFP- Agence France-Press). Twitter was the top platform of choice for all the journalists on account of its speed and ease-of-use. However, all the journalists in the panel pointed to one thing namely the credibility of news on social media and methods for its verification. They pointed to some examples of false happenings disseminated through social media platforms. Denis Teyssou of AFP told the audience about how a fake picture of Osama bin Laden's death revolved around social media in 2007; AFP then used a computational graphics software to detect that the picture was fake and the news was false. A similar problem arose during the 2010 Haiti earthquake when a picture of 2008 Sichuan earthquake was included in a slideshow of 54 images from Haiti by Daily News. The photo was traced out to a random social media  activist and Daily News was heavily criticized by social media and traditional media circles alike. The point in focus is the huge challenge traditional media outlets face due to faster means of news dissemination: the problem is huge, the criticism from various circles is immense and the time is crucial. Even the recent Bhoja Air tragedy and corresponding burst of criticism on the traditional media by various social media activists points out the need for a solution to this traditional vs. social media crisis. We all speak of problems and shortcomings in various aspects of the media but when it comes to solutions, we are somewhat clueless and this for the better has to change.

Osama bin Laden fake death picture sent to AFP by a random social media activist

Siuchan Eartquake picture that was labeled to be Haiti Earthquake picture
by random social media activist

Being a Computer Scientist and having an analytical eye for things makes one approach problems differently: this was the key-point highlighted by all journalists during WWW2012 and Computer Science is the field where the journalists are turing for solutions. Most of them admitted that the advent of citizen journalism has left many in the media industry baffled and some even outraged as journalists by nature are arrogant people not having a habit of public engagement. Social media has considerably changed this leading to more and more public-media engagement but crucial problems still remain; it is the solution to these problems that will bridge the trust gap between the public and the media.


As part of a panel on Social Computing and Social Machines: A Research Agenda in Web Science track of WWW2012 I could relate to many of the concerns raised by the journalists and interaction with various panelists further strengthened my hypothesis on marriage between Computer Science and journalism. A similar belief about the way forward is held by sociologists and psychologists with data journalism seeming to be the way forward for all of us. As is the case with most analytic fields not many in the Pakistani media industry are familiar with data journalism and lack far behind than in this area unlike the well-established names such as Guardian, Huffington Post, and BBC. It could very well be that the solution to the media crisis lies right before us and we in our arrogance or let's put it softly ignorance are not turning towards it. Both the traditional media and social media circles have their own sets of problems and the way I see it collaboration can help solve the problems of both of them, and Computer Science is a good means of attaining that goal. Problems such as this is what has given birth to the new field of Web Science and this emerging field of Web Science aims to bring together scientists from various disciplines as the Web Science diagram shows.

Being a programmer from quite some time now and having been into the Computer Science research arena for about 3-4 years I along with my colleague and wife Arjumand Younus have set up a small Web Science research group at Institute of Business Administration in Karachi, Pakistan and this year our biggest achievement of so far came in the form of two research works by WebST IBA at WWW2012. Both of our works focus towards the media crisis. The first one titled "Investigating Bias in Traditional Media through Social Media" quantitatively studies bias in traditional media platforms through the use of social media text-mining and utilizes various text-mining similarity measures to identify differences in how both platforms report a news event. The second one titled "Traces of Social Media Activism from Malaysia and Pakistan" studies regional differences in activity of social media activists from two regions namely Malaysia and Pakistan and reports on some significant findings. We came to the conclusion that social media activists in Pakistan tend to do a large amount of mentions and retweets thereby endorsing each other's statements whereas Malaysian activists tend to post status updates more frequently. The research includes many other findings, feel free to contact us at the Web Science lab of IBA for further insights or if you are interested in joining any of these projects.

My visit to Russia, RuSSIR/EDBT 2011

Past week of my life was a very exciting experience (14 Aug.- 20 Aug. 2011), I along with my wife and colleague Arjumand went to St. Petersburg (Russia) for RuSSIR/EDBT summer school and International Conference centered around Information Retrieval (Computer Science). This was my first time ever to visit Russia and I was little confused about how it would be (as very little is known about Russia in Pakistan). Reaching there after long stretched hours of flight consumed me a lot in terms of energy. However, as soon I arrived I found my surroundings very normal and helpful as it was in South Korea, China and Malaysia (my previous experiences); everything looked usual and cooperative. Team of RuSSIR/EDBT arranged a nice hostel for the two of us which was in fact a studio apartment and better than where I had ever stayed previously during my short trips. Russian people could barely talk in English but were polite and helpful (thumbs up!).

On 15 Aug. we went to the school venue for our educational commitment through a bus followed by metro (simply subway or rail) and then a short walk to the campus of St. Petersburg State university. We faced little confusion in reaching our destination but managed to reach through words of passing by people. In the gathering, there were fine lectures for full five days as part of summer school and nice papers and posters during the conference. Our daily routine from 15-19 Aug. had a duration from 9:00 AM till 9:00 PM and every second of it paid off in terms of learning. In breaks we had chances to socialize with people hence forming future possible collaborative/cooperative connections. Suggestions and fine word of questions led us to have a feedback which could not be possible to generate within a week (usually). During the gathering, we witnessed people from social sciences taking keen interest in aspects of social networks which shows how aware they were about fields getting merged. A guy from industry, Yandex (similar to Google in Russia) came to me after reading my research paper for interacting and this shows how practical and serious things were there. Similar meaningful was my interaction with guy from mail.ru (another giant in Russian industry). All in all it was a very nice experience and I feel like typing in something for it as a token of appreciation.

I close my post by saying thanks to the team of RuSSIR/EDBT for organzing such an event and I also want to thank to the participants who put further five stars to the gathering.

As a note for Pakistani people who know a very little about Russia: it is a nice place with nice people and very open place for knowledge cooperation and collaborations, do consider it as a meaningful option for academics and industry. Further, feel free to write back to me.

RuSSIR/EDBT 2011 link: http://romip.ru/edbt-russir2011/