\n
\n
\n\n\n 
\n
Abstract:** *In this work, we introduce FaceXFormer, an end-to-end unified \ntransformer model capable of performing nine facial analysis tasks including face parsing, landmark detection, \nhead pose estimation, attribute prediction, and estimation of age, gender, race, expression, and face visibility \nwithin a single framework. Conventional methods in face analysis have often relied on task-specific designs \nand pre-processing techniques, which limit their scalability and integration into a unified architecture. \nUnlike these conventional methods, FaceXFormer leverages a transformer-based encoder-decoder architecture \nwhere each task is treated as a learnable token, enabling the seamless integration and simultaneous processing of \nmultiple tasks within a single framework. Moreover, we propose a novel parameter-efficient decoder, FaceX, which \njointly processes face and task tokens, thereby learning generalized and robust face representations across \ndifferent tasks. We jointly trained FaceXFormer on nine face perception datasets and conducted experiments \nagainst specialized and multi-task models in both intra-dataset and cross-dataset evaluations across multiple benchmarks, showcasing state-of-the-art or competitive performance. Further, we performed a comprehensive analysis of different \nbackbones for unified face task processing and evaluated our model in-the-wild, demonstrating its robustness and generalizability. To the best of our knowledge, this is the first work to propose a single model capable of \nhandling nine facial analysis tasks while maintaining real-time performance at 33.21 FPS.*
\n\n# :rocket: News\n- [03/19/2024] 🔥 We release FaceXFormer.\n\n## Installation\n```bash\nconda env create --file environment_facex.yml\nconda activate facexformer\n\n# Install requirements\npip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 torchaudio==2.0.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117\npip install -r requirements.txt\n```\n## Download Models\nThe models can be downloaded manually from [HuggingFace](https://huggingface.co/kartiknarayan/facexformer) or using python:\n```python\nfrom huggingface_hub import hf_hub_download\n\nhf_hub_download(repo_id=\"kartiknarayan/facexformer\", filename=\"ckpts/model.pt\", local_dir=\"./\")\n```\nThe directory structure should finally be:\n\n```\n . ── facexformer ──┌── ckpts/model.pt\n ├── network\n └── inference.py \n```\n## Usage\n\nDownload trained model from [HuggingFace](https://huggingface.co/kartiknarayan/facexformer) and ensure the directory structure is correct.\n 
\n
![\"\"](\"./static/images/intro.png\")
\n ![\"\"](\"./static/images/intro_table.png\")
\n Comparison with representative methods under different task settings.\n FaceXformer\n can perform various facial analysis tasks in single model. FP - Face Parsing, LD - Land-\n marks Detection, HPE - Head Pose Estimation, Attr - Attributes\n Recognition, Age - Age, Gen - Gender, Race - Race Estimation,\n Exp - Facial Expression Recognition, and Vis - Face Visibility\n
\n![\"\"](\"./static/images/main_archi.png\")
\n \n Overview of FaceXformer framework. It employs an encoder-decoder\n architecture, extracting multi-scale features from the input face image I,\n and\n fusing them into a unified representation F via MLP-Fusion. Task tokens\n T\n are processed alongside face representation F in the decoder, resulting in\n refined\n task-specific tokens \n T\n
![\"\"](\"./static/images/parsing.png\")
\n \n Comparison with specialized models and existing multi-task networks on Face Parsing.\n
\n![\"\"](\"./static/images/hpe_lnd_attr.png\")
\n \n Comparison with specialized models and existing multi-task networks on Headpose\n Estimation,\n Landmarks Detection and Attributes Prediction.\n
\n![\"\"](\"./static/images/results_others.png\")
\n \n Comparison with specialized models and existing multi-task networks on Facial\n Expression\n Recognition, Face Visibilty, and Age Estimation.\n
\n![\"\"](\"./static/images/qualitative.png\")
\n \n Qualitative results of FaceXFormer\n
\n@article{narayan2024facexformer,\n title={FaceXFormer: A Unified Transformer for Facial Analysis},\n author={Narayan, Kartik and VS, Vibashan and Chellappa, Rama and Patel, Vishal M},\n journal={arXiv preprint arXiv:2403.12960},\n year={2024}\n}\n