Ensemble Model for 3D Face Reconstruction using Dual UNET along with PNCC and Depth Filtering Fine-Tuning for Forensic Investigation
Keywords:
3D Facial Reconstruction, Plane Normalized Coordinate Cross-Correlation, Depth Filtering Fine-Tuning, Dual UNet, Face ShapesAbstract
In the realm of facial recognition, accurate reconstruction of three-dimensional (3D) facial structures plays a vital role in various applications such as verification, biometrics, and forensic investigation. Existing models depend on the availability of labeled data for effectively reconstructing facial images. However, it is challenging to obtain labeled datasets which provide a diverse set of facial images with 3D face geometry. As a result, most of the research works develop synthetic data using morphable facial images. This research paper presents a novel approach to enhance 3D facial reconstruction by implementing Dual UNet architecture with Projected Normalized Coordinate Code (PNCC) and Depth Filtering Fine-Tuning (DFFT). The proposed methodology leverages the semantic segmentation and feature extraction abilities of Dual UNet framework in order to obtain color and depth information. This information is used to understand facial geometry and texture which significantly helps in the reconstruction of facial images. The PNCC used in this research enhances the capacity of the Dual UNet model to represent the nonlinear relationships within facial features. In addition, the PNCC with DFFT helps in modelling complex facial expressions, and thereby improves the reliability of 3D facial reconstruction. Experimental results demonstrate that the reconstruction of 3D face shapes with geometry details from only a single input image can efficiently be performed using the proposed approach.
Downloads
References
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016a). Facevr: Real-time facial reenactment and eye gaze control in virtual reality. arXiv preprint arXiv:1610.03151.
Chen, S. Y., Gao, L., Lai, Y. K., Rosin, P. L., & Xia, S. (2018, March). Real-time 3d face reconstruction and gaze tracking for virtual reality. In 2018 IEEE Conference on virtual reality and 3d user interfaces (VR) (pp. 525-526). IEEE.
You, Y., Song, L., & Yang, Y. (2021). High-Quality Facial Expression Animation Synthesis System Based on Virtual Reality. In VR/AR and 3D Displays: First International Conference, ICVRD 2020, Hangzhou, China, December 20, 2020, Revised Selected Papers 1 (pp. 21-32). Springer Singapore.
Wang, L., & Zhang, R. (2023). Framework for facial recognition and reconstruction for enhanced security and surveillance monitoring using 3D computer vision. Journal of Electronic Imaging, 32(4), 042108-042108.
Zollhöfer, M., Thies, J., Garrido, P., Bradley, D., Beeler, T., Pérez, P., ... & Theobalt, C. (2018, May). State of the art on monocular 3D face reconstruction, tracking, and applications. In Computer graphics forum (Vol. 37, No. 2, pp. 523-550).
Tarassoli, S. P., Shield, M. E., Allen, R. S., Jessop, Z. M., Dobbs, T. D., & Whitaker, I. S. (2020). Facial reconstruction: a systematic review of current image acquisition and processing techniques. Frontiers in Surgery, 7, 537616.
Ichim, A. E., Bouaziz, S., & Pauly, M. (2015). Dynamic 3D avatar creation from hand-held video input. ACM Transactions on Graphics (ToG), 34(4), 1-14.
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016b). Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2387-2395).
Tran, L., Liu, F., & Liu, X. (2019). Towards high-fidelity nonlinear 3D face morphable model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1126-1135).
Tewari, A., Seidel, H. P., Elgharib, M., & Theobalt, C. (2021). Learning complete 3d morphable face models from images and videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3361-3371).
Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., & Tong, X. (2019). Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 0-0).
Jiang, L., Zhang, J., Deng, B., Li, H., & Liu, L. (2018). 3D face reconstruction with geometry details from a single image. IEEE Transactions on Image Processing, 27(10), 4756-4770.
Tewari, A., Zollhoefer, M., Bernard, F., Garrido, P., Kim, H., Perez, P., & Theobalt, C. (2018). High-fidelity monocular face reconstruction based on an unsupervised model-based face autoencoder. IEEE transactions on pattern analysis and machine intelligence, 42(2), 357-370.
Booth, J., Roussos, A., Ponniah, A., Dunaway, D., & Zafeiriou, S. (2018). Large scale 3D morphable models. International Journal of Computer Vision, 126(2), 233-254.
Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2014). Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3), 413-425.
Abada, L., & Aouat, S. (2016). Facial shape-from-shading using features detection method. International Journal of Advanced Intelligence Paradigms, 8(1), 3-19.
Richardson, E., Sela, M., Or-El, R., & Kimmel, R. (2017). Learning detailed face reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1259-1268).
Rotger Moll, G., Moreno-Noguer, F., Lumbreras, F., & Agudo Martínez, A. (2019). Detailed 3D face reconstruction from a single RGB image. Journal of WSCG (Plzen, Print), 27(2), 103-112.
Zhao, D., & Qi, Y. (2022, September). 3D Face Reconstruction with Geometry Details from a Single Color Image Under Occluded Scenes. In International Conference on Artificial Neural Networks (pp. 332-344). Cham: Springer Nature Switzerland.
Yang, L., Wu, J., Huo, J., Lai, Y. K., & Gao, Y. (2021). Learning 3D face reconstruction from a single sketch. Graphical Models, 115, 101102.
Tu, X., Zhao, J., Xie, M., Jiang, Z., Balamurugan, A., Luo, Y., ... & Feng, J. (2020). 3D face reconstruction from a single image assisted by 2D face images in the wild. IEEE Transactions on Multimedia, 23, 1160-1172.
Dou, P., Shah, S. K., & Kakadiaris, I. A. (2017). End-to-end 3D face reconstruction with deep neural networks. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5908-5917).
Sharma, S., & Kumar, V. (2020). Voxel-based 3D face reconstruction and its application to face recognition using sequential deep learning. Multimedia tools and applications, 79, 17303-17330.
Chen, F. F., Guan, B., Kim, S., & Choi, J. (2023, August). 3-D Face Reconstruction Method Using Deep Learning Based Simulated Annealing. In International Conference on Intelligent and Fuzzy Systems (pp. 215-221). Cham: Springer Nature Switzerland.
Sharma, S., & Kumar, V. (2022). 3D face reconstruction in deep learning era: A survey. Archives of Computational Methods in Engineering, 29(5), 3475-3507.
Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In ECCV, pages 534–551,2018.
Xiaoguang Tu, Jian Zhao, Mei Xie, Zihang Jiang, Akshaya Balamurugan, Yao Luo, Yang Zhao, Lingxiao He, Zheng Ma, and Jiashi Feng. 3d face reconstruction from a single image assisted by 2d face images in the wild. IEEE Trans_x0002_actions on Multimedia (TMM), 2020.
Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei,and Stan Z Li. Towards fast, accurate and stable 3d dense face alignment. In ECCV, 2020.
Wu, Cho-Ying, Qiangeng Xu, and Ulrich Neumann. "Synergy between 3dmm and 3d landmarks for accurate 3d facial geometry." In 2021 International Conference on 3D Vision (3DV), pp. 453-463. IEEE, 2021.
Martyniuk, T., Kupyn, O., Kurlyak, Y., Krashenyi, I., Matas, J. and Sharmanska, V., 2022. Dad-3dheads: A large-scale dense, accurate and diverse dataset for 3d head alignment from a single image. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition (pp. 20942-20952).
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7763–7772, 2019.
Vitor Albiero, Xingyu Chen, Xi Yin, Guan Pang, and Tal Hassner. img2pose: Face alignment and detection via 6dof, face pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7617–7627, 2021.
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J. Crandall. Hope-net: A graph-based model for hand-object pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755–1758, 2009.
Feng-Ju Chang, Anh Tuan Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, and Gerard Medioni. Faceposenet: Making a case for landmark-free face alignment. In International Conference on Computer Vision (ICCV) Workshops, Oct 2017.
Jianzhu Guo, Xiangyu Zhu, and Zhen Lei. 3ddfa. https: //github.com/cleardusk/3DDFA, 2018.
Xiangyu Zhu, Xiaoming Liu, Zhen Lei, and Stan Z Li. Face alignment in full pose range: A 3d total solution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1):78–92, 2017.
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li. Face alignment across large poses: A 3d solution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 146–155, 2016.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
