Certifiable Auto-supervised Large Models (IRIT, ANITI)

Chaire aniti 2
The CALM project has a high degree of continuity with the great success of the 3IA Aniti and DEEL program: studying 1-Lipschitz neural networks, a class of robust by-design neural networks. The team, built in this first round, was composed of Mathematicians, Computer Science researchers, Data-scientists, and Industry experts, and has published several papers in major conferences and journals. They have laid the groundwork for classification with 1-Lipschitz neural networks, both on theory, on the definition of optimal loss linked to optimal transport, and proof of robustness, certifiability and explainability. Additionally a full library has been developed, called DEEL-LIP, to learn these kind of neural networks as classical Tensorflow models.
In this project, we propose to further investigate the 1-Lipschitz neural networks in the scope of self-supervised learning: the objective is to be able to learn large models with unannotated data in several domains (medical/satellite images, time series, natural language processing) while maintaining the guarantees in terms of robustness, certifiability and explainability. Self-supervised learning is a high trend for classical networks with applications in few-shot learning, semi-supervised learning or as backbones. But, as far as we know, there is no contribution in the literature on self-supervised 1-Lipschitz neural networks.
We propose to tackle the unexplored domain of self-supervised 1-Lipschitz large models with three research axis: In the first axis, we will explore methods for self-supervised learning using optimal transport loss, to learn from unannotated data while still promoting the robustness of the neural network. We will also investigate more recent and deeper 1-Lipschitz architectures, such as transformer, to enhance the learning capabilities of these networks on very large datasets and their generalization. To end with, we will work to establish the theory and certifiable guarantees for these self-supervised learnt 1-Lipschitz Neural Networks. For industrial safety-critical applications, we will develop a set of pre-trained 1-Lipschitz Networks for various domains, including satellite images, time series, language processing, and medical imaging, where data quantity and annotation are crucial.

• Large 1-Lipschitz network architecture for visio and natural language processing (NLP)
• Self-supervised training for 1-Lipschitz network with large unlabelled dataset based on optimal transport approaches
• Distillation, finetuning and few shots approaches with transferable robustness, explainability and fairness properties for 1-Lipschitz network