Sign In

Ekaterina Kochmar

Assistant Professor at MBZUAI | Co-Founder & CSO at Korbit AI

Ekaterina Kochmar is an Assistant Professor in the Natural Language Processing Department at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI). She specializes in artificial intelligence and natural language processing, with a focus on intelligent tutoring systems. Her research interests include author profiling, computational semantics, readability assessment, language complexity, text simplification, summarization, and error detection and correction. A significant aspect of her work involves applying machine learning techniques to educational contexts, particularly in second language learning and assessment.13

Before her current position at MBZUAI, Kochmar was a Lecturer at the University of Bath from 2021 to 2023 and a postdoctoral researcher at the ALTA Institute at the University of Cambridge, where she developed educational applications for second language learners. She is also the co-founder and Chief Scientific Officer of Korbit AI, which aims to create an AI-powered dialogue-based tutoring system for personalized education in STEM subjects.123

Kochmar holds a PhD in Natural Language Processing and has authored over 30 peer-reviewed research papers. She is actively involved in the academic community, serving as the President of the ACL Special Interest Group on Educational Applications (SIGEDU) and the International Alliance to Advance Learning in the Digital Era (IAALDE).13

Highlights

Mar 30 · linkedin

Large language models (LLMs) are now commonly used for NLP tasks that involve text generation. These include text simplification, machine translation, grammatical error detection and correction, summarization, and so on. Traditionally, the outputs in such tasks have been evaluated against references – texts representing target outputs that are typically collected from humans. Such references are, by default, limited in number. However, state-of-the-art NLP models, including LLMs, are capable of producing much more diverse outputs not always covered by the available references, so evaluation becomes a real challenge.

In this work, we propose a new reference-free evaluation metric for text simplification, which we call REFeREE. It uses a three-stage curriculum, mimicking human quality estimations, and can be applied to any quality standard even if just a small number of human annotations are available. We show that our metric performs competitively while requiring no reference simplifications at inference time.

You can find out more about our research in the paper – https://lnkd.in/dtxPdMey. This is joint work with Yichen Huang, which will be presented in May at LREC-COLING 2024 (https://lnkd.in/dAjdsbVn).

#nlp #evaluation

Mar 20 · linkedin

Large language models (LLMs) demonstrate huge potential across a range of tasks and are constantly reported to exhibit emergent abilities. Yet an equally interesting question to that of what they do well is when and, importantly, why they fail.

One of the tasks that even state-of-the-art powerful LLMs still struggle with is solving math problems. In this work, we have tested abilities of a number of open-source LLMs in solving word math problems from the popular GSM8K dataset. This dataset consists of math exercises at the grade school math level, but the tasks themselves are linguistically diverse. In other words, a bright middle school student can be expected to solve every problem in this set, but we find that most LLMs we tested cannot do that. Moreover, we discover that they find various factors (e.g., the phrasing of the problem or the amount of world knowledge needed) challenging and identify which of these factors confuse each of the models.

You can find out more about our research in the paper – https://lnkd.in/dkCgQ7Bh. This is joint work with KV Aditya Srivatsa, which will be presented in June at NAACL 2024 (https://2024.naacl.org/).

#llms #emergent #ai

Related Questions

What are some of the key projects Ekaterina Kochmar has worked on at MBZUAI?
How does Korbit AI's tutoring system differ from traditional educational methods?
What inspired Ekaterina Kochmar to focus on AI applications in education?
Can you provide more details about the Read & Improve tool developed by Kochmar?
What are the main goals of the International Alliance to Advance Learning in the Digital Era?
Ekaterina Kochmar
Ekaterina Kochmar, photo 1
Ekaterina Kochmar, photo 2
Add to my network

Location

Greater Cambridge Area