ChatGPT as an informant
While previous machine learning protocols have failed to achieve even observational adequacy in acquiring natural
language, generative large language models (LLMs) now produce large amounts of free text with few grammatical errors. This is
surprising in view of what is known as “the logical problem of language acquisition”. Given the likely absence of negative
evidence in the training process, how would the LLM acquire the information that certain strings are to be avoided as ill-formed?
We attempt to employ Dutch-speaking ChatGPT as a linguistic informant by capitalizing on the documented “few shot learning”
ability of LLMs. We then investigate whether ChatGPT has acquired familiar island constraints, in particular the CNPC, and compare
its performance to that of native speakers. Although descriptive and explanatory adequacy may remain out of reach, initial results
indicate that ChatGPT performs well over chance in detecting island violations.
Article outline
- 1.Introduction
- 2.Test methodology: ChatGPT as an informant
- 3.Experiment 1: ChatGPT
- 3.1Language models
- 3.2Materials
- Set 1: Base line examples
- Set 2: English island violations
- Set 3: Dutch island violations
- 3.3Procedure
- 3.4Predictions
- 3.5Analysis, results and discussion
- 4.Experiment 2: Humans versus few-shot GPT 4
- 4.1Participants
- 4.2Materials
- 4.3Procedure
- 4.4Analysis and results
- 4.4.1Humans
- 4.4.2Humans versus GPT 4 with a few-shot prompt
- 5.Conclusions
- Supplementary materials and data archive
- Acknowledgements
- Notes
-
References