Compositionality in Meta-Learning Neural Network

Abstract

The capacity of human language and thought to express complex ideas is a consequence of compositionality. Compositionality is the idea that the meaning of complex expressions or structures can be derived from the meanings of their constituent parts and the rules used to combine them. It suggests that the meaning of the phrase "black cat" can be understood by combining the meanings of its constituent parts ("black" and "cat") and the rules used to combine them. 

This project examines the compositional abilities of a meta-learning neural network model. While considering the open nature of compositionality and lack of consensus on exact interpretation, we proceeded to test the model on the following three aspects that are motivated by theoretical literature. These aspects include systematicity, productivity, and localism [1]. The systematicity test evaluates a model's ability to combine constituents not seen together during training, focusing on plausible combinations. Productivity is assessed through the unboundedness of understanding longer sentences than those in training data. Localism is tested whether the neural network processes sentences as a whole or by understanding and combining individual parts separately. 

To assess these aspects, we adopt a meta-learning compositionality (MLC) approach for the few-shot learning task [2]. The few-shot learning task involves mapping input words to output symbols using a combination of primitive mappings and functional terms. Primitive words directly map to specific output symbols (e.g., 'dax' to RED, 'wif' to GREEN, 'lug' to BLUE). Functional terms, such as 'fep', 'blicket', and 'kiki', operate on primitives or strings of primitives to produce outputs. For example, 'fep' repeats the output of the preceding primitive multiple times, 'blicket' alternates outputs from preceding and following primitives, and 'kiki' concatenates and reverses outputs from preceding and following strings. Function composition is also tested, where outputs from one function serve as inputs for another function. 

For our project, we employ MLC implementation trained on algebraic data [3]. This implementation uses a standard seq2seq transformer architecture, comprising two interconnected neural networks: an encoder transformer for query input and example processing, and a decoder transformer for output sequence generation. In the encoder network, a concatenated source string is processed, incorporating both the query input sequence and a set of study examples in the form of input/output sequence pairs. In the decoder network, messages are received from the encoder, and the output sequence is generated. 

The results of our experiments demonstrate the model’s capability to handle compositional tasks effectively. The project provides insights into the potential and limitations of the MLC approach in understanding and generating complex expressions. 

References

[1] D. Hupkes, V. M. Dankers, and E. Bruni, Compositionality Decomposed: How do Neural Networks Generalise?. Journal of Artificial Intelligence Research. 67. pp. 757-795, 2020.

[2] B. M. Lake, and M. Baroni, Human-like systematic generalization through a meta-learning neural network. Nature 623, pp. 115–121, 2023.

[3]B. Lake, “brendenlake/MLC: Meta-Learning for Compositionality”. Zenodo, aug. 22, 2023. doi: 10.5281/zenodo.8274610.