Face Swapping Using Convolutional Neural Network and Tiny Facet Primitive

HUANG Ruobing; JIA Yonghong

doi:10.13203/j.whugis20180500

Volume 46 Issue 3

Mar. 2021

Turn off MathJax

Article Contents

Abstract

References

Geomatics and Information Science of Wuhan University > 2021 > 46(3): 335-340. > DOI: 10.13203/j.whugis20180500

HUANG Ruobing, JIA Yonghong. Face Swapping Using Convolutional Neural Network and Tiny Facet Primitive[J]. Geomatics and Information Science of Wuhan University, 2021, 46(3): 335-340. DOI: 10.13203/j.whugis20180500

Citation:

PDF (29963 KB)

Face Swapping Using Convolutional Neural Network and Tiny Facet Primitive

HUANG Ruobing^1,,
JIA Yonghong^1, ,

1.
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

More Information

Author Bio:
HUANG Ruobing, master, specializes in computer vison. E-mail: hrbing23456@163.com
Corresponding author:
JIA Yonghong, PhD, professor. E-mail: yhjia2000@sina.com
Received Date: December 26, 2019
Published Date: March 04, 2021

Graphical Abstract

Abstract

Abstract

Objectives Face swapping technology has important application value in entertainment, virtual reality, film and so on. However, existing methods are limited by face pose consistency and can not overcome the influence of occlusion.
Methods We proposes a method of face swapping using convolutional neural network and tiny facet primitive. Firstly, detect the face using the cascade convolutional neural network and segment the face to determine the replacement region using fully convolutional network.Then, the Wallis transform is applied to adjust the skin color of the source image to make it consistent with the skin color of the face in the target image.After that, using facial key points detection method based on an ensemble of regression trees and Delaunay triangulation to construct the face triangulation network, then replacing faces based on tiny facet primitive. Finally, applying Poisson fusion to eliminate splicing traces between different images.
Results We evaluate the performance of the proposed method compared with existing method through qualitative and quantitative experiments. Experimental results show that face segment can well solve the problem of occlusion such as cap, glasses, and hair.Moreover, when source image and target image have different face poses, replacing face area using tiny facet primitive separately performs better than using the whole face area.
Conclusions Our method can well solve the problem of face pose consistency limitation and occlusion, which has certain practical application value.
- convolutional neural network,
- tiny facet primitive,
- Delaunay triangulation,
- face swapping,
- face segmentation

FullText(HTML)

References (13)

References

[1]	林源, 桂良琰, 王生进, 等. 基于真实感三维头重建的人脸替换[J]. 清华大学学报(自然科学版), 2012, 52(5): 602-606 https://www.cnki.com.cn/Article/CJFDTOTAL-QHXB201205006.htm Lin Yuan, Gui Liangyan, Wang Shengjin, et al. Face Swapping Based on 3-D Photo Realistic Head Reconstruction[J]. Journal of Tsinghua University (Science and Technology), 2012, 52(5): 602-606 https://www.cnki.com.cn/Article/CJFDTOTAL-QHXB201205006.htm
[2]	Thies J, Zollhofer M, Stamminger M, et al. Face2Face: Real-Time Face Capture and Re-enactment of RGB Videos[C].IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016
[3]	谈国新, 孙传明. 一种真实感三维人脸交互式生成方法[J]. 武汉大学学报·信息科学版, 2014, 39(8): 992-997 doi: 10.13203/j.whugis20130152 Tan Guoxin, Sun Chuanming. An Interactive Approach to Generate Realistic 3D Face[J]. Geomatics and Information Science of Wuhan University, 2014, 39(8): 992-997 doi: 10.13203/j.whugis20130152
[4]	Bitouk D, Kumar N, Dhillon S, et al. Face Swapping: Automatically Replacing Faces in Photographs[J]. ACM Transactions on Graphics, 2008, 27(3), DOI: 10.1145/1399504.1360638
[5]	Guo D, Sim T. Digital Face Makeup by Example[C]. IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009
[6]	Mahajan S, Chen L J, Tsai T C. SwapItUp: A Face Swap Application for Privacy Protection[C]. IEEE International Conference on Advanced Information Networking and Applications, Taipei, China, 2017
[7]	孙康, 李千目, 李德强. 基于级联卷积神经网络的人脸检测算法[J]. 南京理工大学学报, 2018, 42(1): 40-47 https://www.cnki.com.cn/Article/CJFDTOTAL-NJLG201801006.htm Sun Kang, Li Qianmu, Li Deqiang. Face Detection Algorithm Based on Cascaded Convolutional Neural Network[J]. Journal of Nanjing University of Science and Technology, 2018, 42(1): 40-47 https://www.cnki.com.cn/Article/CJFDTOTAL-NJLG201801006.htm
[8]	Li H, Lin Z, Shen X, et al. A Convolutional Neural Network Cascade for Face Detection[C]. IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015
[9]	Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation[C]. IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015
[10]	Reinhard E, Adhikhmin M, Gooch B, et al. Color Transfer Between Images[J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41
[11]	Kazemi V, Sullivan J. One Millisecond Face Alignment with an Ensemble of Regression Trees[C]. IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014
[12]	杨学习, 石岩, 邓敏, 等. 一种基于多层次专题属性约束的空间异常探测方法[J]. 武汉大学学报·信息科学版, 2016, 41(6): 810-817 doi: 10.13203/j.whugis20140425 Yang Xuexi, Shi Yan, Deng Min, et al. A New Method of Spatial Outlier Detection by Considering Multi-level Thematic Attribute Constraints[J]. Geomatics and Information Science of Wuhan University, 2016, 41(6): 810-817 doi: 10.13203/j.whugis20140425
[13]	Yi D, Lei Z, Liao S C, et al. Learning Face Representation from Scratch[EB/OL]. [2014-11-28]. https://arxiv.org/abs/1411.7923

[1]	LIAO Haibin, WANG Dianhua, CHEN Youbin. Face Age Estimation Based on Multi⁃layer Spare Representation[J]. Geomatics and Information Science of Wuhan University, 2021, 46(8): 1233-1240. DOI: 10.13203/j.whugis20190126
[2]	QIU Yiming, LIAO Haibin, CHEN Qinghu. Occluded Face Pose Recognition Based on Dictionary Learning with Discrimination Performance[J]. Geomatics and Information Science of Wuhan University, 2018, 43(2): 275-281, 288. DOI: 10.13203/j.whugis20150298
[3]	Fang Tianhong, Chen Qinghu, Liao Haibin, Qiu Yiming. Face Feature Weighted by Fusing Texture and Shape[J]. Geomatics and Information Science of Wuhan University, 2015, 40(3): 321-326+340.
[4]	Li Xi, Zheng Hong, Liu Cao. A Method of Face Recognition Based on HSI-PCNN[J]. Geomatics and Information Science of Wuhan University, 2014, 39(12): 1499-1503.
[5]	YAO Haitao, ZHU Fuxi, CHEN Haiqiang. Face Tracking Based on Adaptive PSO Particle Filter[J]. Geomatics and Information Science of Wuhan University, 2012, 37(4): 492-495.
[6]	YUAN Li, CHEN Qinghu, LIAO Haibin, DUAN Liuyun. 3D Face Reconstruction for Single Visual Images[J]. Geomatics and Information Science of Wuhan University, 2012, 37(4): 487-491.
[7]	LIAO Haibin, CHEN Qinghu, WANG Hongyong. Robust Face Recognition by Fusion Local Deformable Model[J]. Geomatics and Information Science of Wuhan University, 2011, 36(7): 877-881.
[8]	LIAO Haibin, CHEN Qinghu, YAN Yuchen. 3D Face Reconstruction of Morphable Model and Its Improvement[J]. Geomatics and Information Science of Wuhan University, 2011, 36(2): 176-180.
[9]	TANG Liang, XIONG Rong, LIU Yong. Discriminant Common Vectors in Principal Component Analysis Transformed Space for Face Recognition[J]. Geomatics and Information Science of Wuhan University, 2009, 34(4): 404-408.
[10]	LU Jian, WANG Hui. Comparison Between SVD and DCT Feature Extraction Methods in Face Recognition[J]. Geomatics and Information Science of Wuhan University, 2005, 30(2): 118-121.

Cited By

Cited by

Periodical cited type(3)

1.	胡永健，佘惠敏，刘琲贝，陈香全，刘光尧. 利用人脸3DMM重构信息检测深度伪脸视频. 武汉大学学报(信息科学版). 2024(02): 190-196 .
2.	李晓霖，李刚，张恩琪，顾广华. 行列式点过程采样的文本生成图像方法. 武汉大学学报(信息科学版). 2024(02): 246-255 .
3.	魏玮，张鑫，朱叶. 基于双重注意力和光流估计的人脸替换方法. 计算机工程与应用. 2023(07): 143-151 .

Other cited types(6)

Get Citation

PDF

XML

Article views (6569) PDF downloads (175) Cited by(9)

Face Swapping Using Convolutional Neural Network and Tiny Facet Primitive

Abstract

References

Related Articles

Cited by

Periodical cited type(3)

Other cited types(6)

Catalog

Related

Face Swapping Using Convolutional Neural Network and Tiny Facet Primitive

Abstract

References

Related Articles

Cited by

Periodical cited type(3)

Other cited types(6)

Catalog

Related

Export File

Citation

Format

Content