CheckThat!Lab 2024, a competition to evaluate access to multilingual information

This was a binary classification task in which systems were challenged to distinguish whether a sentence from a news article expresses the subjective view of the author behind it or presented an objective view on the covered topic instead. Égré's team, involving Benjamin Icard (LIP6/IJN) and partners from IRISA Rennes, Mondeca, and Airbus Defence and Space, ranked 1st in English, out of 15 competing teams. "In other languages, Égré says, the results were more mixed, likely due to the translation step, but English was the main target of our hybrid approach, combining transformer models with expert rules from the system VAGO we created, so we are very happy, and see better where to go next”.

The approach basically integrates a RoBERTa-base model fine-tuned for the task with scores of subjectivity and objectivity produced by the expert system and lexical database VAGO, developed at IJN by Égré and Icard in collaboration with Mondeca since 2020, and connected to deep learning methods with their associates from IRISA and Airbus. The results and details of the method will be presented at the CLEF conference in Grenoble in September 2024.

The CLEF initiative

The CLEF initiative (Conference and Labs of the Evaluation Forum) promotes research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. CLEF 2024 is the 15th CLEF conference continuing the popular CLEF campaigns which have run since 2000 contributing to the systematic evaluation of information access systems, primarily through experimentation on shared tasks.

About HYBRINFOX

The HYBRINFOX project aims to contribute to the fight against online misinformation by studying and developing possible synergies between symbolic AI and deep learning approaches for the detection of fake news (aka. infox). The main lever is the identification of vague information, likely to introduce or promote bias (subjectivity, evaluativity). The projet is led by Paul Egré.
 

Publications:

• Benjamin Icard, Vincent Claveau, Ghislain Atemezing, Paul Égré (2023). Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO. IEEE International Conference on Web Intelligence and Intelligent Agent Technology
   
• Géraud Faye, Benjamin Icard, Morgane Casanova, Julien Chanson, François Maine, François Bancilhon, Guillaume Gadek, Guillaume Gravier, Paul Égré (2024). Exposing propaganda: an analysis of stylistic cues comparing human annotations and machine classification. EACL 2024

Copyright: image de pikisuperstar sur Freepik