The week's pick
Random Articles
Reseach Article
Assessing LLMs as Cognitive Interpreters of Student Prompts: A Typological Framework
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 187 - Number 27 |
| Year of Publication: 2025 |
| Authors: Tadeu da Ponte, Matevz Vremec, Matej Mertik |
10.5120/ijca2025925477
|
Tadeu da Ponte, Matevz Vremec, Matej Mertik . Assessing LLMs as Cognitive Interpreters of Student Prompts: A Typological Framework. International Journal of Computer Applications. 187, 27 ( Aug 2025), 1-11. DOI=10.5120/ijca2025925477
Abstract
This paper introduces a typology of student cognitive actions in interactions with large language model (LLM)-based tutors. Drawing on the CoMTA dataset of 188 anonymized math tutoring dialogues from Khan Academy, student-generated questions were analyzed as evidence of reasoning processes. The methodology combines a natural language processing (NLP) pipeline for semantic clustering with a dual-stage human classification of communicative intent and cognitive action. The resulting typology is synthesized into a partially ordered taxonomy that captures the complexity and multidimensionality of student thinking in AI-mediated learning contexts. Two research questions guide this investigation: (1) Can a typology be derived directly from unsupervised NLP clustering methods? and (2) To what extent can LLMs replicate expert-driven classification schemes? Findings from RQ1 reveal that semantic clustering via PCA and KMeans offers only limited alignment with pedagogically meaningful distinctions. In contrast, results from RQ2 show that several LLMs–particularly Deepseek, Grok, and Gemini–can reliably extend the typology to unseen data, demonstrating high accuracy in classification. These results suggest that scalable, cognitively informed AI tutoring may be supported by combining expert frameworks with strategically configured LLM architectures.
References
- Perplexity AI. Perplexity assistant semantic classifier, 2025. Zero-/few-shot classification based on semantic alignment with labeled examples.
- Anthropic. Claude 3.7 sonnet classification process, 2025. Few-shot reasoning using semantic patterns from labeled student questions.
- Simon P. Bates, Ross K. Galloway, Jonathan Riise, and Danny Homer. Assessing the quality of a student-generated question repository. Physical Review Special Topics - Physics Education Research, 10(2):020105, 2014.
- Benjamin S. Bloom, Max D. Engelhart, Edward J. Furst, Walker H. Hill, and David R. Krathwohl. Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook I: Cognitive Domain. David McKay Company, New York, 1956.
- Ng D., LuoWanying, Chan H., and Chu S. Using digital story writing as a pedagogy to develop AI literacy among primary students. Computers and Education: Artificial Intelligence, 2022.
- Ng D., Wu Wenjie, Leung J., Chiu T., and K. Chu S. Design and validation of the AI literacy questionnaire: The affective, behavioural, cognitive and ethical approach. British Journal of Educational Technology, 2023.
- Jennifer Daries and colleagues. Ai literacy framework: Review draft. Unpublished draft, 2024.
- Google DeepMind. Gemini 25pro question classification process, 2025. Few-shot classification based on phrasing and inferred student intent.
- Michael Hansen, Claire Scoular, and Patrick Griffin. Evidence for assessment of 21st century competencies: A review of 19 instruments. Assessment in Education: Principles, Policy & Practice, 30(1):43–68, 2023.
- H. Y. Yim I. and Su Jiahong. Artificial intelligence (AI) learning tools in K-12 education: A scoping review. Journal of Computers in Education, 2024.
- Ian T. Jolliffe. Principal Component Analysis. Springer, 2nd edition, 2002.
- Stacy M. Kula and Ting Chang. Student-generated questions: Encouraging academic engagement, critical thinking, and student voice. College Teaching, 65(3):126–135, 2017.
- Xin Li and Dan Roth. Learning question classifiers. In Proceedings of the 19th International Conference on Computational Linguistics, pages 1–7, 2002.
- David Long and Brian Magerko. What is ai literacy? competencies and design considerations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), pages 1–16, New York, NY, USA, 2020. ACM.
- Casal-Otero Lorena, Catal´a Alejandro, Fern´andez-Morante Carmen, Taboada M., Cebreiro Beatriz, and Barro S. Ai literacy in K-12: a systematic literature review. International Journal of STEM Education, 2023.
- J. MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, pages 281–297. University of California Press, 1967.
- Pepper Miller and Kristen Dicerbo. Llm based math tutoring: Challenges and dataset. Technical report, Khan Academy, 2024. Available at: https://github.com/Khan/tutoringaccuracy- dataset/blob/main/LLM Based Math Tutoring.pdf.
- Davy Tsz Kit N.G., Luo Wanru, Chan H., and Chu S. An examination on primary students’ development in AI literacy through digital story writing. Computers and Education: Artificial Intelligence, 2022.
- OECD. Introducing the OECD AI Capability Indicators. OECD Publishing, Paris, 2025.
- OpenAI. Chatgpt-4o nlp classification meta-process, 2025. Performed classification of student questions using TF-IDF + kNN pipeline in Python.
- OpenAI. Chatgpt-o3 classification meta-process, 2025. Executed TF-IDF + logistic regression on labeled student questions using scikit-learn.
- OpenAI. Chatgpt-o4-mini classification meta-process, 2025. TF-IDF + logistic regression classification from 42 labeled examples.
- OpenAI. Chatgpt-o4-minihigh classification meta-process, 2025. Same approach as o4-mini, applied to the same task using TF-IDF + logistic regression.
- Stacey Pylman. Student-generated questions as formative assessment: A review of the literature. Educational Practice and Theory, 42(2):5–25, 2020.
- Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
- Deepseek Team. Gpt-4 based semantic classification, 2025. Used semantic matching to assign question categories without retraining.
- Manus AI Team. Manus ensemble classification engine, 2025. Used Naive Bayes, SVM, and Random Forest in soft-voting mode with TF-IDF inputs.
- xAI. Grok 3 student query classification, 2025. Manual category assignment with semantic and intent-based rule application.
- Zhou Xiaofei, Van Brummelen Jessica, and Lin Phoebe. Designing AI Learning Experiences for K-12: Emerging Works, Future Opportunities and a Design Framework. arXiv.org, 2020.
- Fu-Yun Yu. Student question-generation: The learning processes involved and their relationships with students’ perceptions of a learning environment. Instructional Science, 39(3):325–346, 2011.
Index Terms
Keywords