ABSTRACT
The purpose of this research is to investigate whether expert information can improve the ranking function of academic search engines. We chose CiteSeerX as the testbed which is a well-known academic paper search engine where the ranking function takes the number of citations and the similarity between the query and the paper. Therefore, more cited papers easily get higher positions in the ranking list. Thus, adding more features to the ranking function may reduce effects on one or a few factors. Intuitively, if an author of a paper is an expert in the area, the paper should be more credible and also searchers should be more interested in. This research found that a document of which an author who is an expert has higher probability to be clicked than a document of which the author who is not an expert. Furthermore, we included expert information as another feature of CiteSeerX’s ranking function. A supervised ranking approach that considers expert information was used. The evaluation shows that the two learned ranking functions perform better than the existing CiteSeerX’s one.
Chapter 1
INTRODUCTION AND RELATED WORK
INTRODUCTION
CiteSeerX is a well-known digital library search engine [1] where its ranking function takes the number of citations and the similarity between the query and the paper into consideration. However, for some queries, the top most clicked documents which are possibly more relevant to the queries [2] are not shown on the top positions in the search results.
Table 1-1 A ranking list of query “cloud computing” ordered by CiteSeerX and the number of clicks.
rank | Ordered by CiteSeerX | Ordered by number of click | Number of click |
1 | 10.1.1.149.7163 | 10.1.1.158.2549 | 1018 |
2 | 10.1.1.462.4311 | 10.1.1.148.1726 | 760 |
3 | 10.1.1.158.2549 | 10.1.1.149.2162 | 400 |
4 | 10.1.1.413.4719 | 10.1.1.178.1323 | 291 |
5 | 10.1.1.453.6574 | 10.1.1.173.686 | 231 |
6 | 10.1.1.352.6525 | 10.1.1.149.7163 | 125 |
Table 1-1 shows that a ranking list of the query ‘cloud computing’ ordered by
CiteSeerX and by the number of clicks, which is obtained from the CiteSeerX’s search logs from 2009 to 2013, are different in order. The document gets the most number of clicks is 10.1.1.158.2549 which is at the rank 3 in the CiteSeerX’s ranking list. Also, the top ranked document 10.1.1.149.7163 in the CiteSeerX’s ranking list only has the sixth most number of clicks among clicked documents associated with the query. Based on the assumption that a document gets more clicks should be more relevant, the CiteSeerX’s ranking has spaces to be improved.
We analyzed search results from CiteSeerX and investigated the correlation between the features extracted from a paper, such as title, author, venue, and so on, and the number of clicks. We narrowed down the scope of features and focus on author information since intuitively if an author of a paper is an expert in the area, the paper should be considered as more credential which might mean that it would get more clicks. Therefore, we aim to answer the following two research questions in the research:
Research Question 1:
How does expert information affect academic search engines?
Research Question 2:
Can expert information be used to improve the ranking function?
RELATED WORK
The purpose of our work is to improve the ranking function of an academic search engine which combines knowledge of multiple domains in information retrieval including information integration, digital library, learning to rank and expert finding. In our work, in order to add more features into the ranking function, results from different resources are selected and integrated in a reasonable way. Since the research target is a digital library search engine, understanding other digital libraries and their ranking functions, which state the general concepts of what are features are used in terms of ranking functions in digital libraries, provides clues for improvement. Furthermore, in order to include expert information, one of expert finding algorithms among expert finding literatures is needed. From the search logs, we extracted user’s search activities and use them as training dataset to learn a ranking function with higher retrieval efficiency.
INFORMATION INTEGRATION
Information integration is to merge information from heterogeneous resources and present the integrated result in different perspectives. [3] is one of examples of information integration being used in digital libraries. The proposed academic search engine searches across journal publisher collections and merges returned result into a single search result page. Although the search results are merged from multiple resources, the contents of the search results are similar, that is, all are academic papers.
Our research aims to collect diverse information from different resources and integrate them to improve the ranking function.
Furthermore, another way to integrate information is to intersect results from search resources and return the intersection which is upon an assumption that each search resource has the same representation of collections. [4] uses the intersection as one of the features of learning to merge search results. However, in our case, intersection between four selected verticals is really small; hence, the methods are not applicable to the problem.
The search result blending methods proposed by [5] are built on an assumption that the relative order of documents coming from the same sources is not allowed to change since they believe that each specialized search engine generates the best ranking about their own documents. Thus, we assume that each search result from different verticals can help to refine the ranking function since all of them are in different nature and correlated in some ways.
RANKING FUNCTIONS OF DIGITAL LIBRARIES
Academic search engines provide open and easily accessible information retrieval platforms for academic publications. The data a popular academic search engine collects is broadly representative of the impact of journals, conferences, and publications [6] . The number of citation is one of the important indicators to be a successful publication and in [7] , the research discovered that there are positive correlations between citation counts and the position of documents mostly. However, for some quires, no correlation existed which means some more factors affect ranking.
The age of the publication could be another feature in ranking functions. For Google Scholar, although [8] shows that no significant correlation between an paper’s age and the ranking, older papers are ranked in top positions more often than recent ones. The possible reason is that Google scholar highly depends on citation counts for ranking. There are some more options for ranking academic documents such as h-index and g-index. [9] used a combination of g-index and h-index, known as hg-index, to rank marketing journals which have highly correlation in terms of ranking with Journal Impact Factor. Furthermore, [10] aims to provide guidelines on optimizing scholarly literature for academic search engines as well as discuss the concept of academic search engine optimization.
LEARNING TO RANK
Learning to rank approaches are used in rankers including abounding and diverse sets of features [4, 5, 11] and use a judged training set of query-document pair and apply machine learning techniques to learn complicated combination of features. Although the approaches are not readily applicable to use here since different search verticals have different feature spaces which must be merged somehow. In [5, 12] , for every source, a copy of features for each source allows the learning to rank model to learn a relationship between relevance and features for each vertical. [13] proposed SVM-based method to learn retrieval functions and illustrated the features and the results of weighting the features which would be a good baseline for reference of learning to rank.
[12, 14] focus on aggregated searches and learn to merge results in the search engine’s final result page with block-based ranking which means search results of the same searching resources must be grouped together while presenting. The papers address the difficulty of the representation of different features across search resources as well as propose approaches to allow learning algorithm to learn across features.
There are three categories of learning to rank, pointwise, pairwise, and listwise according to [11] . [2, 13] proposed pairwise learning to rank algorithms and both of them used clickthrough data to optimize with different strategies where [2] involved human judgement to develop the learning. Interestingly, clickthrough data perform really well for learning, however, involvement of human judgement works less reliable and informative since large size of clickthrough data represent decisions of large amount of users.
EXPERT FINDING
Expert finding is a common task and popular research area in digital libraries and in different perspectives as well. Basically, expert finding algorithm takes a query as an input and returns a list of experts of the area. A Citation-Author-Topic (CAT) model is proposed by [15] which models the linkage among cited authors, words, and paper authors together. [16] proposes large-scale expert finding algorithms in a specific field with the data supplementation of DBLP biblography and Google Scholar. [17, 18] use CiteSeerX document collection as the corpus to build the models where [17] proposes a graph-based algorithm which accommodates multiple features extracted from documents and links to other documents and [18] generates keyphrases which are used to gather authors’ expertise.
IMPROVING DIGITAL LIBRARY RANKING WITH EXPERT INFORMATION