C-LARA

We just received notification that the paper we submitted to the 2023 edition of the Australasian Language Technology Association workshop has been accepted. Here are the details:

• Title: “Using C-LARA to evaluate ChatGPT-4’s multilingual processing”.
• Authors (alphabetical order): ChatGPT C-LARA-Instance, Belinda Chiera, Cathy Chua, Chadi Raheb, Manny Rayner, Annika Simonsen, Zhengkang Xiang, Rina Zviel-Girshin.
• Abstract:
  We present a cross-linguistic study in which the open source C-LARA platform was used to evaluate ChatGPT-4’s ability to perform several key tasks relevant to Computer Assisted Language Learning. For each of the languages English, Farsi, Faroese, Mandarin and Russian, we instructed ChatGPT-4, through C-LARA, to write six different texts, using prompts chosen to obtain texts of widely differing character. We then further instructed ChatGPT-4 to annotate each text with segmentation markup, glosses and lemma/part-of-speech information; native speakers hand-corrected the texts and annotations to obtain error rates on the different component tasks. The C-LARA platform makes it easy to combine the results into a single multimodal document, further facilitating checking of their correctness. ChatGPT-4’s performance varied widely across languages and processing tasks, but performance on different text genres was roughly comparable. In some cases, most notably glossing of English text, we found that ChatGPT-4 was consistently able to revise its annotations to improve them.

The workshop is Nov 29 to Dec 1.