Abstract

With this research, it is aimed to evaluate the performance of the Google search engine on the searches made with Turkish natural sentences. In this direction, Google was asked 20 questions of different types. The first 10 questions are comprised factoid questions with specific and concise answers. On the other hand, the other 10 questions are comprised of list questions with long answers or complex questions with no specific answer. Within the scope of the research, first 10 answers brought up by the Google search engine were evaluated. After the documents retrieved during the study were classified as relevant and irrelevant, precision and general normalized recall measure of the results were calculated. Whether the precision and general normalized recall measure differ depending on the question types, the difference between the first 5 results and the first 10 results, and the relevance level of the snippets and the videos brought on the result page were examined. Mann-Whitney U, Spearman and Chi-square tests were used in the analysis of the data. As a result of the research, the relevance level of the results brought by the Google search engine in regards to Turkish natural language searches were found to be average. It was revealed that the relevance levels did not differ according to the question type, but the relevance ranking did differ. Compared to other questions, relevant documents ranked higher for factoid questions. In addition, Google has been successful in showing relevant documents at the top. It turned out that mostly in the first five results relevant results can be reached. A snippet was brought for almost half of the questions and the majority of these were determined to be relevant with the query. Snippets are available not only in factoid questions but also in list questions. Videos brought up in the result pages were revealed to be mostly irrelevant to the query. Videos for factoid questions are less relevant than other questions. It is thought that the results of the research will benefit the work carried out to improve the Google search engine. It is recommended to increase the performance tests made with real user queries.

Keywords: Information retrieval, semantic search, Google search engine, precision, general normalized recall measure

References

  1. Akbulut, M., Tonta, Y. ve White, H.D. (2020). Related records retrieval and pennant retrieval: An exploratory case study. Scientometrics, 122, 957–987. doi: 10.1007/s11192-019-03303-9
  2. Alexa. (2020). Alexa top sites. https://www.alexa.com/topsites adresinden erişildi.
  3. Amasyalı, M. F. ve Diri, B. (2005). Bir soru cevaplama sistemi: Baybilmiş. Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi, 1(1), 37-51. https://dergipark.org.tr/tr/pub/tbbmd/issue/22240/238746 adresinden erişildi.
  4. Azeez, S. S. (2017). An evaluation of three search engines (Google, Yahoo, Bing) based on Arabic user perception (Yayımlanmamış Yüksek Lisans Tezi). Çankaya Üniversitesi, Ankara. http://earsiv.cankaya.edu.tr:8080/xmlui/bitstream/handle/20.500.12416/1950/Azeez%2c%20Suad%20Shatti.pdf?sequence=1&isAllowed=y adresinden erişildi.
  5. Bar-Ilan, J. ve Gutman, T. (2005). How do search engines respond to some non-English queries? Journal of Information Science, 31(1), 13-28. doi: 10.1177/0165551505049255
  6. Bilal, D. (2012). Ranking, relevance judgment, and precision of information retrieval on children’s queries: Evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids. Journal of the American Society for Information Science and Technology, 63(9), 1879-1896. doi: 10.1002/asi.22675
  7. Bitirim, Y. ve Görür, A. K. (2017). A comparative evaluation of popular search engines on finding Turkish documents for a specific time period. Tehnički vjesnik, 24(2), 565-569. doi: 10.17559/TV-20140512181430
  8. Bollmann, P. (1983). The normalized recall and related measures. ACM SIGIR Forum, 17(4), 122-128. doi: 10.1145/1013230.511811
  9. Brin, S. ve Page, L. (1998). The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems, 30(1-7), 107-117. doi: 10.1016/S0169-7552(98)00110-X
  10. Ceylan, N. Ö. (2019). Parametrik ve parametrik olmayan testler için güç karşılaştırmaları (Yüksek Lisans Tezi). Selçuk Üniversitesi, Konya.
  11. Chaoticity. (2010). Google as a question answering system. https://chaoticity.com/google-as-a-question-answering-system/ adresinden erişildi.
  12. Demirci, R. G., Kişmir, V. ve Bitirim, Y. (2007). An evaluation of popular search engines on finding Turkish documents. Second International Conference on Internet and Web Applications and Services (ICIW’07) içinde (s. 61). Morne. doi: 10.1109/ICIW.2007.15.
  13. Dwivedi, S. K. ve Singh, V. (2013). Research and reviews in question answering system. Procedia Technology, 10, 417-424. doi: 10.1016/j.protcy.2013.12.378
  14. Gul, S., Ali, S. ve Hussain, A. (2020). Retrieval performance of Google, Yahoo and Bing for navigational queries in the field of “life science and biomedicine”. Data Technologies and Applications, 54(2), 133-150. doi: 10.1108/DTA-05-2019-0083
  15. Kesen, S., Şenol, C. ve Yanar, Z. (2008). Google Scholar ve Scirus arama motorlarında Türkçe anahtar sözcüklerle yapılan aramalar üzerine bir değerlendirme. Bilgi Dünyası, 9(1), 140-157. http://bd.org.tr/index.php/bd/article/view/175 adresinden erişildi.
  16. Khan, J. A., Sangroha, D., Ahmad, M. ve Rahman, M. T. (2014). A performance evaluation of semantic based search engines and keyword based search engines. 2014 International Conference on Medical Imaging, m-Health and Emerging Communication Systems (MedCom) içinde (ss. 168-173). Greater Noida. doi: 10.1109/MedCom.2014.7005997
  17. Kumar, B. T. S. ve Prakash, J. N. (2009). Precision and relative recall of search engines: A comparative study of Google and Yahoo. Singapore Journal of Library & Information Management, 38, 124-137.
  18. Mala, V. ve Lobiyal, D. K. (2016). Semantic and keyword based web techniques in information retrieval. 2016 International Conference on Computing, Communication and Automation (ICCCA) içinde (ss. 23-26). Noida. doi: 10.1109/CCAA.2016.7813724
  19. Singhal, A. (2012). Introducing the knowledge graph: Things, not strings. https://blog.google/products/search/introducing-knowledge-graph-things-not/ adresinden erişildi.
  20. Soricut, R. ve Brill, E. (2004). Automatic question answering: Beyond the factoid. Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004 içinde (ss. 57–64). Boston. https://www.aclweb.org/anthology/N04-1008 adresinden erişildi.
  21. Soydal, İ. (2000). Web arama motorlarında performans değerlendirme (Yayımlanmamış Yüksek Lisans Tezi). Hacettepe Üniversitesi, Ankara. http://bby.hacettepe.edu.tr/yayinlar/70.pdf adresinden erişildi.
  22. Sullivan, D. (2016). All about the Google RankBrain algorithm. https://searchengineland.com/faq-all-about-the-new-google-rankbrain-algorithm-234440 adresinden erişildi.
  23. Tonta, Y. (1995). Bilgi erişim sistemleri. Türk Kütüphaneciliği, 9(3), 302-314. http://www.tk.org.tr/index.php/TK/article/view/1010/1011 adresinden erişildi.
  24. Tonta, Y., Bitirim, Y. ve Sever, H. (2002). Türkçe arama motorlarında performans değerlendirme. Ankara: Total Bilişim. http://yunus.hacettepe.edu.tr/~tonta/yayinlar/tonta-bitirim-sever-arama-motorlari.pdf adresinden erişildi.
  25. Tümer, D., Shah, M. A. ve Bitirim, Y. (2009). An empirical evaluation on semantic search performance of keyword-based and semantic search engines: Google, Yahoo, Msn and Hakia. 2009 Fourth International Conference on Internet Monitoring and Protection içinde (ss. 51-55). Venice. doi: 10.1109/ICIMP.2009.16
  26. Usmani, T. A., Pant, D. ve Bhatt, A. K. (2012). A comparative study of Google and Bing search engines in context of precision and relative recall parameter. International Journal on Computer Science and Engineering, 4(1), 21-34.
  27. Yozkat, S. A. (2017). Arama motoru sonuçları güvenilirliğinin analizi: Google örneği. Yeni Medya Elektronik Dergisi, 1(1), 72-81. http://static.dergipark.org.tr/article-download/b3c9/be9f/55f6/imp-JA67PK24SM-0.pdf? adresinden erişildi.
  28. Zadeh, L. A. (2006). From search engines to question answering systems: The problems of world knowledge, relevance, deduction and precisiation. E. Sanchez (Ed.). Capturing Intelligence içinde (ss. 163-210). New York: Elsevier. doi: 10.1016/S1574-9576(06)80011-0
  29. Zeden, R., Ayalp, E., Bitirim, Y. ve Sever, H. (2006). Duyarlılık ve normalize sıralama: Gövdelemenin Türkçe arama motorları üzerindeki etkisi. The Fifteenth Turkish Symposium on Artificial Intelligence and Neural Networks (TAINN) içinde (ss. 215-222).
  30. Zhao, Y., Zhang, J., Xia, X. ve Le, T. (2019). Evaluation of Google question-answering quality. Library Hi Tech, 37(2), 312-328. doi: 10.1108/LHT-10-2017-0218

License

Copyright (c) 2020 The author(s). This is an open access article distributed under the Creative Commons Attribution License (CC BY), which permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited.

How to Cite

Dişli, M. (2020). Performance Evaluation of Google Search Engine in Turkish Natural Language Searches. Information World, 21(2), 247-267. https://doi.org/10.15612/BD.2020.554