Artificial Grammar Learning: Rule Abstraction and Production in Adults and LLMs

Abstract

Language processing is a complex cognitive process that is yet to be understood in its entirety. Alongside statistical learning mechanisms, hierarchical structure processing may be a core cognitive mechanism involved in language processing. The hierarchical structure of language allows for the organisation of phonemes into syllables, syllables into words, words into phrases, and phrases into sentences, forming recursive hierarchies that enable complex connections [1]. Language processing also underpins much of today’s development in artificial intelligence. With recent advancements in large language models (LLMs), many comparisons have been drawn between humans and LLMs; yet the underlying mechanisms in both have not been fully disentangled.

In psycholinguistics, linguistic structure processing is often examined through artificial grammar learning (AGL) experiments. In an AGL experiment, the participants are presented with a made-up grammar, whereupon they are asked to determine the grammaticality of novel stimuli. Eye-tracking studies have demonstrated that infants as young as seven months old can discern that the stimulus ga-ti-ga follows an ABA clause pattern [2], and that the sentence du-ba-ba, lo-mo-mo, za-vu-za. follows the hierarchical pattern AAB at sentence level, and the patterns ABB, ABB, ABA at clause level, respectively [1]. A number of AGL experiments attest to such processing capabilities in humans [2]. AGL experiments with LLMs have shown that they possess structure abstraction abilities to a limited degree [3]. However, since LLMs are generative models and their abilities were evaluated based on their output, and AGL experiments in humans have thus far only focused on language comprehension, an experimental paradigm in which humans would be asked to not only recognize, but also produce linguistic patterns, is missing.

Our research explores hierarchical abstraction abilities in adults by focusing on language production. In a planned behavioral AGL experiment, adults (n ≈ 60) will be exposed to linguistic patterns of varying complexity and asked to generate novel patterns adhering to the same rule. In the analysis, the mean accuracy of the generated responses will be calculated and compared across conditions and groups, as well as to the generation success rates of LLMs. Preliminary pilot study data has shown that humans outperform LLMs, which might indicate a difference in hierarchical abstraction mechanisms between humans and LLMs. By examining the mechanisms underlying language production, we aim to address a gap in AGL research, contribute to a better understanding of language processing in both humans and artificial systems, and inform research on the cognitive plausibility of LLMs.

References

[1] Á. M. Kovács, and A. D. Endress. “Hierarchical Processing in Seven-Month-Old Infants.” Infancy, vol. 19, no. 4, pp. 409–425, 2014. doi: 10.1111/infa.12052.

[2] H. Rabagliati, B. Ferguson, and C. Lew-Williams. “The profile of abstract rule learning in infancy: Meta-analytic and experimental evidence.” Developmental Science, vol. 22, no. 1, e12704, 2019. doi: 10.1111/desc.12704. 

[3] L. Thoma, I. Weyers, E. Çano, J. L. Mueller, and B. Roth, “Generative large language models are capable of processing artificial grammar inputs at abstract levels – but only non-hierarchically”, forthcoming, 2025.