Performance Evaluation of Google Search Engine in Turkish Natural Language Searches

Abstract

With this research, it is aimed to evaluate the performance of the Google search engine on the searches made with Turkish natural sentences. In this direction, Google was asked 20 questions of different types. The first 10 questions are comprised factoid questions with specific and concise answers. On the other hand, the other 10 questions are comprised of list questions with long answers or complex questions with no specific answer. Within the scope of the research, first 10 answers brought up by the Google search engine were evaluated. After the documents retrieved during the study were classified as relevant and irrelevant, precision and general normalized recall measure of the results were calculated. Whether the precision and general normalized recall measure differ depending on the question types, the difference between the first 5 results and the first 10 results, and the relevance level of the snippets and the videos brought on the result page were examined. Mann-Whitney U, Spearman and Chi-square tests were used in the analysis of the data. As a result of the research, the relevance level of the results brought by the Google search engine in regards to Turkish natural language searches were found to be average. It was revealed that the relevance levels did not differ according to the question type, but the relevance ranking did differ. Compared to other questions, relevant documents ranked higher for factoid questions. In addition, Google has been successful in showing relevant documents at the top. It turned out that mostly in the first five results relevant results can be reached. A snippet was brought for almost half of the questions and the majority of these were determined to be relevant with the query. Snippets are available not only in factoid questions but also in list questions. Videos brought up in the result pages were revealed to be mostly irrelevant to the query. Videos for factoid questions are less relevant than other questions. It is thought that the results of the research will benefit the work carried out to improve the Google search engine. It is recommended to increase the performance tests made with real user queries.