Evaluation of LLMs Effectiveness in Generating Slovak Texts for Children’s Picture Books

Abstract

Picture books play a significant role in fostering early cognitive and emotional development in children. They are a unique literary genre of a short format that combines visual images with little text. The mutual relationship and complementarity of pictures and text are important for the overall understanding of the story. The process of creating a good picture book is usually laborious and involves the collaboration of many people. The integration of AI might facilitate the process and bring some new dimensions to storytelling. AI tools have already started to be used for picture books, mainly in the English-speaking world [1].

In our work, we aimed to explore the effectiveness of LLMs in generating texts for children’s picture books, in Slovak. We mapped the Slovak population's attitude towards using AI for children’s literature. Inspired by a similar Chinese study [2], we generated texts by three LLMs (ChatGPT-4o, Gemini 2.5 pro, Copilot) based on an illustration from a human-authored picture book. An extract from a Slovak picture book [3] was used for the study. We provided participants with a questionnaire in which they had to evaluate on a 10-point Likert scale the relevance, fluency, vocabulary, creativity and overall impression of the original text and three generated ones to the provided illustration. The evaluation process was blind, participants were unaware of the source of the texts. Participants were also asked to express their opinion on the use of AI in children's literature. Our main hypothesis was that LLMs are still insufficient in generating good texts for picture books in small languages, such as Slovak. We also supposed that Slovaks would be reserved in their attitude towards the use of AI.

We collected a sample of 63 participants, mostly students or people under 30 years (45 participants). The results showed that in most cases, participants preferred text generated by Gemini 2.5 pro, suggesting that this LLM is most suitable for generating Slovak literary texts, and its performance surpasses that of a human. Nevertheless, the results slightly differed across the conditions, namely for participants rarely or never using AI, older participants or parents, who preferred some aspects of the original text (e.g. relevance, originality or fluency). The results also showed that while Slovaks are open towards the use of AI in children's literature, they prefer the process to be controlled by human specialists, indicating some degree of reservedness. The study has some limitations, especially concerning the diversity and size of the participants’ sample, which could be improved in future research.

References

[1] D. Kolednjak, “A comparison of human-authored and AI-generated picturebooks in read-alongs with young learners,” M.S. Thesis, Fac. of Teacher Education, Univ. of Zagreb, Zagreb, 2024. [Online]. Available: https://zir.nsk.hr/islandora/object/ufzg%3A4865/datastream/PDF/view

[2] D. Zhou, F. Liu, W. Liu, and J. Du, “Can MLLMs replace humans in writing Chinese children’s fairy tales based on pictures?,” Preprint, ver. 1, Research Square, Aug. 18, 2024. [Online]. doi:/10.21203/rs.3.rs-4766954/v1

[3] M. Juhász and K. Ilkovičová, Glória vo veľkom svete. Bratislava, Slovakia: Slovart, 2022.