SCIENTIFIC PUBLICATIONS
Explore research papers and studies that contribute to CERTAIN’s goal of making AI systems across Europe more trustworthy, ethical, and sustainable.
Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification
Ünsal Öztürk, Hatef Otroshi Shahreza, Sébastien Marcel
Multimodal Large Language Models (MLLMs) have recently been explored as face verification systems that determine whether two face images are of the same person. Un-like dedicated face recognition systems, MLLMs approach this task through visual prompting and rely on general visual and reasoning abilities. However, the demographic fairness of these models remains largely unexplored. In this paper, we present a benchmarking study that evaluates nine open-source MLLMs from six model families, ranging from 2B to 8B parameters, on the IJB-C and RFW face verification protocols across four ethnicity groups and two gender groups. We measure verification accuracy with the
Equal Error Rate and True Match Rate at multiple operating points per demographic group, and we quantify demographic disparity with four FMR-based fairness metrics. Our results show that FaceLLM-8B, the only face- specialised model in our study, substantially outperforms general-purpose MLLMs on both benchmarks. The bias patterns we observe differ from those commonly reported for traditional face recognition, with different groups being most affected depending on the benchmark and the model.
We also note that the most accurate models are not necessarily the fairest and that models with poor overall accuracy can appear fair simply because they produce uniformly high error rates across all demographic groups. Project page:
https://www.idiap.ch/paper/mllm-fairness.
Technology-mediated screening interviews for youth mental health: Content validation, randomized controlled trial, and expert evaluation
Izidor Mlakar, Valentino Šafran, Nejc Plohl, Hojka Gregorič Kumperščak, Sara Močnik, Žan Smogavc, Urška Smrke
Early identification of mental health disorders in children and adolescents is essential for optimal outcomes, yet access to specialist psychiatric assessment remains limited by severe workforce shortages and geographic barriers. This study examines content validity, feasibility and acceptability from youths’ perspective, and clinical utility from experts’ perspective of technology-mediated screening interview for youth mental health assessment through three sequential investigations (ISRCTN Registry, ISRCTN68006163, registered on 16/04/2025). First, content validity was established through expert evaluation, demonstrating that all screening questions met established psychometric criteria for relevance and clarity. Next, a randomized controlled trial evaluated feasibility and acceptability (satisfaction and willingness to repeat with the same modality) across psychiatrist-led, chatbot, and robot modalities among 106 young patients aged 10–19 receiving mental health treatment. Finally, expert clinical evaluation of recorded interviews assessed clinical utility across modalities. Psychiatrist-led interviews achieved higher satisfaction scores, particularly for communication quality, while willingness to repeat screening did not differ significantly across modalities. Personality characteristics, specifically conscientiousness and open-mindedness, predicted satisfaction with communication across all conditions. Finally, expert clinical evaluation of recorded interviews revealed comparable ratings across modalities for most dimensions, with only alignment with other assessment methods favoring psychiatrist-led approaches over robot-delivered interviews. The results show technology-mediated screening, using the proposed interview, demonstrates acceptable content validity, user feasibility from youth perspective, and clinical utility from expert perspectives. These findings support implementation of accessible, self-administered screening tools as complementary first-step assessments that maintain clinical relevance while expanding access to mental health evaluation for youth populations in underserved settings where specialist availability is limited.
Towards a Framework for Supporting the Ethical and Regulatory Certification of AI Systems
Fabian Kovac, Sebastian Neumaier, Timea Pahi, Torsten Priebe, Rafael Rodrigues, Dimitrios Christodoulou, Maxime Cordy, Sylvain Kubler, Ali Kordia, Georgios Pitsiladis, John Soldatos, Petros Zervoudakis
Artificial Intelligence has rapidly become a cornerstone technology, significantly influencing Europe’s societal and economic landscapes. However, the proliferation of AI also raises critical ethical, legal, and regulatory challenges. The CERTAIN (Certification for Ethical and Regulatory Transparency in Artificial Intelligence) project addresses these issues by developing a comprehensive framework that integrates regulatory compliance, ethical standards, and transparency into AI systems. In this position paper, we outline the methodological steps for building the core components of this framework. Specifically, we present: (i) semantic Machine Learning Operations (MLOps) for structured AI lifecycle management, (ii) ontology-driven data lineage tracking to ensure traceability and accountability, and (iii) regulatory operations (RegOps) workflows to operationalize compliance requirements. By implementing and validating its solutions across diverse pilots, CERTAIN aims to advance regulatory compliance and to promote responsible AI innovation aligned with European standards.