A Comparative Analysis of TF-IDF Rating over Query-based Response Relevance
Abstract
With the development of Artificial Intelligence (AI) chatbot, there has been increasing number of conversations made between human and AI. This study aims to evaluate the efficacy of the Term Frequency-Inverse Document Frequency (TF-IDF) model in scoring the
relevance of AI-generated responses to specific user queries. Using the DeepSeek language model, a dataset regarding two mental-health
queries were generated. I implemented both a standard TF-IDF algorithm and another version incorporating synonym detection via WordNet
to address the model's limitation with only exact keyword matching. Performance was evaluated across multiple corpus sizes using metrics including average TF-IDF score, top 10% average, and score variance. Results indicate the synonym-aware TF-IDF can consistently identify the
actual relevance between query and responses. The marked improvement is particular in the top 10% of scores, suggesting the improvement
that can be made from a more context-aware TF-IDF model.
relevance of AI-generated responses to specific user queries. Using the DeepSeek language model, a dataset regarding two mental-health
queries were generated. I implemented both a standard TF-IDF algorithm and another version incorporating synonym detection via WordNet
to address the model's limitation with only exact keyword matching. Performance was evaluated across multiple corpus sizes using metrics including average TF-IDF score, top 10% average, and score variance. Results indicate the synonym-aware TF-IDF can consistently identify the
actual relevance between query and responses. The marked improvement is particular in the top 10% of scores, suggesting the improvement
that can be made from a more context-aware TF-IDF model.
Keywords
Term Frequency-Inverse Document Frequency (TF-IDF); Artificial Intelligence (AI); User queries
Full Text:
PDFReferences
[1] Dai, S., Li, K., Luo, Z., Zhao, P., Hong, B., Zhu, A., & Liu, J. (2024). AI-based NLP section discusses the application and effect of bagof-words models and TF-IDF in NLP tasks. Journal of Artificial Intelligence General Science (JAIGS) ISSN:3006-4023, 5(1), 1321.
[2] Das, M., K., S., & Alphonse, P. J. A. (2023). A Comparative Study on TF-IDF feature Weighting Method and its Analysis using Unstructured Dataset.
[3] Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure.
[4] Wang, Y. (2024). Research on the TFIDF algorithm combined with semantics for automatic extraction of keywords from network news
texts. Journal of Intelligent Systems, 33(1), 20230300.
DOI: http://dx.doi.org/10.70711/aitr.v3i3.8035
Refbacks
- There are currently no refbacks.
