This blog is the communication channel between the Odysci Team and you – our users! We will use this space to let you know about what is happening and new features we develop. We value your opinion – send us your feedback via comments below or email to feedback@odysci.com.

Search Habits of Researchers

Introduction

Prior to the launch of the Odysci Academic Search site, we commissioned a marketing firm to conduct a poll among researchers to find out where, how and what they searched for. This report describes some interesting results we collected.

We believe these results can be useful to the scientific community and help improve academic search engines. Working together with researchers we can tune the search engine to better serve the needs of the researchers in finding  the most relevant information for their work in the least amount of time.  During the development of the Odysci Academic Search Engine we paid close attention to the opinions expressed in this survey. We continue to work with researchers to improve everything on the site, including navigation mechanisms, ranking algorithms, peer collaboration and additional content. If you want to participate in this quest, send us your views, tell us what you need and how you use an academic search engine. You can send your views by posting comments on the blog (blog.odysci.com) or by email to feedback@odysci.com.

First a word on the methodology used. This was an informal poll with no pretense of being statistically relevant. In the span of 3 months in mid 2010, we sent email invitations to about 20,000 researchers (i.e., people who had published at least one paper in conferences and journals in the computer science and eletrical engineering fields). The invitations contained a link to a polling site with the survey. We did not identify the company, we simply stated that the survey was being conducted by a marketing firm on behalf of an academic search site.

We collected 400 responses, i.e., people who clicked on the link and completed the survey. This comes to 2% of respondents, which (as the marketing firm told us) is pretty good! The geographic distribution of the respondents is shown below.

We asked several questions related to how researchers use academic search sites, what they look for, and what they think of the results they get. Most questions were multi-valued and multiple choice (e.g., the respondent could say that he/she uses site-A 80% and site-B 20% of the time). The values plotted in the following sections were, in most cases, the Expected value of the possible choices, that is, the weighted average of all possible values that each choice can take, normalized with respect to the highest value.

Survey Results

The following sections describe the questions we asked the researchers and the aggregate values of their answers.

Question 1: What are your main reasons for searching for scientific articles on the web? (please mark the relative percentage on all choices)

The purpose of this question was to assess how specific the searchers are and what the intent of the researcher is when searching. The answers are plotted below.

The percentages represent the expected value (sum of the number of responses X percentage product), normalized with respect to the highest expected value. The most popular answer was “I look for articles in a given topic in order to find the Best papers in the topic” – this was plotted as 100% and used as normalization factor of the other choices. It does not mean that this option was selected by all respondents, but it means it was the option with the highest expected value.

This graph shows a strong bias toward the first 3 choices. That is, researchers tend to search for specific topics, articles they know of (to find the pdf), and recent articles. They showed less interest in searching for specific authors or for specific conferences.

Question 2: How long do you spend searching for scientific articles on the web? (Please report average per week, including all academic search sites you currently use)

How long do you spend searching for scientific articles on the web? (Please report average per week, including all academic search sites you currently use)

The answers are plotted below. Almost 50% of respondents use between 15min and 1 hour searching, whereas 31% reported spending more than 1 hour per week searching for papers on the web.

Question 3: The time you spend searching for articles on the web is…

The purpose of this question was to gauge the researchers’ perception regarding the time they spend searching. Clearly this correlated with how much time they use and whether they find the articles they are looking for. The answers are given below:

About 21% of respondents did not mind the time spent, whereas 59% indicated that the time was significant but well spent. About 18% said they spend too much time for what they get.

By cross-analysing these results with the previous question, and we can trace the researchers’ perception on time spent in relation to the actual time they spent, as the graph below illustrates.

This graph says, for example, that amongst all those who spend between 15 min and 1 hour per week searching, 61.6% consider it a measurable time, but well spent; 20.5% consider it short and 17.8% consider it too much time.

Question 4: Which sites do you use when searching for scientific articles? (please mark relative percentages)

The obvious purpose of this question was to find out which academic search sites were most used by researchers. The responses are plotted below.

It is probably not surprising that Google Scholar was the most popular choice. The 100% value associated with Google Scholar in the graph below does not mean that 100% of the respondents use Google Scholar; but it means that it was the choice most picked by our respondents. The other percentages are all with respect to Scholar’s percentage.

In the actual survey we named 7 other sites for respondents to pick from, some were publisher sites, others were company sites. In this report we chose not to divulge the names of those sites explicitly. The site in 2nd place (anonimously called Site-D n the graph below) had 61% of the preference when compared to Google Scholar, Site-E had 53% of the preference and so on. Odysci had not been launched when we conducted this survey.

We also asked researchers if they used any other search site (when all else failed to give them good results), and the answer here was also expected. In addition to the sites above, people use plain Google when searching for articles. This shows there is definitely need for a good academic search engine, a void that Odysci plans to occupy.

Question 5: Choose the 3 main reasons, in order of importance, for why you prefer one academic search site over another. (Select each alternative: 1=somewhat important; 2=important; 3=very important; only once).

The purpose of this question was to understand the relative reasons for using one search engine instead of another. Results are plotted below.

The most important reasons for their choice of search engine was clearly the quality of search results, represented by the choice “I like to find the best articles on the first page of search results”. In 2nd place came the choice “I need to search articles from known publishers”. The amount of coverage of the engine (represented by the choice “I like to find articles from all over the world”) came in 3rd place, but somewhat less important than the first two.

Question 6: Mark the options that you agree with, regarding the sentence: “The most annoying things in the site for search of scientific papers that I use are:”  (Please mark at most 3 choices)

The purpose of this question was to understant what researchers don’t like about their search sites and contrast it with the previous question in which they stated what they liked the most. Results are plotted below.

The most annoying factor reported was the failure by the search engine in bringing relevant results, as represented by choice “It usually brigs a lot of stuff that has nothing to do with my search” in 1st, and by choice “I have to repeat the search in different ways to get what I want” in 2nd place. This is consistent with the previous question where respondents considered the most important reason for using a search engine to “be able to find the best articles in the first page of the results” (and they find it most annoying when that does not happen). These results show that users are not fully satisfied with their current academic search engine.

Once again, Odysci had not been lauched when this question was asked.

Question 7: Mark as many options as you see fit regarding the following sentence: “I would start using a new search engine for scientific articles if…”:

This questions aimed at understanding what characteristics of an academic search engine could be improved upon, in the eyes of the researchers using it. The results are plotted below.

The most popular answer was “…it saved me time, by yielding better search results”, corroborating previous answers which pointed to the importance of returning relevante results without wasting time having to repeat the search in different ways.

Question 8: When you search for scientific papers, how would you like to receive the following content in addition?

This question attempted to gauge the researchers interest for additional content related to his/her search. Results are plotted below. Within each type of content, the graph shows the percentages of respondents choose a given answer.

Wrap Up

This report described the results of a polling survey conducted by a marketing research firm on the behalf of a new academic search site – Odysci – www.odysci.com. Our goal was to gather information from researchers on what they need from an academic search engine. This information has been and continues to be used in the development of the Odysci search site.

We invite all researchers to help us improve everything on the site, including navigation mechanisms, ranking algorithms, peer collaboration and additional content. If there is something you would like to see in the Odysci search engine, like a special way to navigate through the papers, or a different way to list them, or some new content you would like to have, let us know! You can send your views by posting comments below or by email to feedback@odysci.com.

Categories: Academic Search, Search Habits. Tags: , .

Add comments... »

Post a comment.