We propose a novel 3D generative model to translate a real-world face image into its corresponding 3D avatar with only a single style example provided. Our model is 3D-aware in sense and also able to do attribute editing, such as smile, age, etc directly in the 3D domain.
Dynamic Texture Transfer using PatchMatch and Transformers
Guo Pu, Shiyao Xu, Xixin Cao, Zhouhui Lian [PDF] finally available on arxiv... but this is my first project;-)
We propose an automatically method to transfer the dynamic texture of a given video to a still image.
How to automatically transfer the dynamic texture of a given video to the target still image is a challenging and ongoing problem.
In this paper, we propose to handle this task via a simple yet effective model that utilizes both PatchMatch and Transformers.
The key idea is to decompose the task of dynamic texture transfer into two stages, where the start frame of the target video with the desired dynamic texture is synthesized in the first stage via a distance map guided texture transfer module based on the PatchMatch algorithm.
Then, in the second stage, the synthesized image is decomposed into structure-agnostic patches, according to which their corresponding subsequent patches can be predicted by exploiting the powerful capability of Transformers equipped with VQ-VAE for processing long discrete sequences.
After getting all those patches, we apply a Gaussian weighted average merging strat- egy to smoothly assemble them into each frame of the target stylized video. Experimental results demonstrate the effectiveness and superiority of the proposed method in dynamic texture transfer compared to the state of the art.
2023.07 - present: Research Scientist at Cybever Inc., Mountain View (remotely).
2023.06 - 2023.10: Research Assistant at Prof. Yebin Liu's group, Tsinghua University.
2021.08 - 2023.07: Research Intern in DAMO Academy, Alibaba Group. Mentored by Lingzhi Li, Supervised by Dr. Li Shen.
2021.07 - 2021.08: Machine Learning Intern at Apple Inc., Beijing, China.
2020.09 - 2023.06: M.Sc. in Wangxuan Institute of Computer Techonology(WICT) at Peking University, China.
Supervised by Prof. Zhouhui Lian.
Worked on 3D-aware Generation, Style Transfer, Neural Rendering.
Thesis: 3D-aware Style Transfer based on Neural Radiance Field.
2016.09 - 2020.06: B.Eng. in School of Software at Dalian University of Technology, China.
Supervised by Prof. Kun Lu and Prof. Shimin Shan.
Major in Big Data and Machine Learning.