Are there parameters for making the source facial details stronger?

[ExponentialML]

First of all, great work. Are there parameters that can keep more of the details from the source image, or is this something that needs to be trained on? For example, sometimes there might be key details missing from the eyes, or maybe there are other features(piercings, tattoos, moles, and so on for example) that I may want to keep. Thanks!

[NNNNAI]

Thank you for carefully studying our project engineering. We do not have parameters to adjust the degree of facial feature retention, because this project is our open source code for ACMMM2020. From a research perspective, our goal at the time was that features such as tattoos and piercings should be removed, because this is not The nature of the character. Of course we will consider your suggestions, and maybe we will add an interface to control the degree of feature retention in the version 2 that we will release in the future. This should be cool, many thanks~

[NNNNAI]

If you have any other suggestions, please feel free to ask questions, and we will consider your comments and make improvements.
:)

[ExponentialML]

Thanks for your reply! I'm honestly impressed that you can throw a full video into the script without any pre-production editing, so great work. I have some suggestions that should be easy to implement.

1. The bounding box is very visible after inference. Filling the face based on facial landmarks, then using that as a blurred matte should solve that issue.

2. An option to perform super resolution on the face during or after inference. An example would be something like Face Sparnet.

3. Support for the side of faces, or more extreme angles. I'm sure this is due to how the model was trained and not the actual project itself.

All of these things I can do myself, but I have to use other projects to implement them. It would be nice to have them native to what the script uses like ONNX and insightface.

[NNNNAI]

Cool, I will try to add these modules to see if there is any improvement.

[ExponentialML]

Cool, I will try to add these modules to see if there is any improvement.

Great. Another idea for a problem I've just realized. If the face is tilted too far, there are a lot of jitters on the swapped face. For example, if the head goes from a 0 degree angle to a 90 degree one, the facial alignment struggles a bit. To fix this, you can set a parameter by getting the rotation matrix from the top of the head to the bottom. It would be something like this in steps.

1. Detect if head is tilted to far.
2. If the head tilted to far, rotate then entire target video so that it is aligned, and the head's the chin at the bottom, and the top of the head at the top.
3. Re-detect the face after the rotation, and swap.
4. Rotate the entire target video back to its regular rotation.
5. Continue inference and repeat if needed.

That should solve that issue. If this was Dlib I could probably implement this, but I'm not familiar with what you're using :).

[NNNNAI]

I see, it sounds very reasonable, let me spend some time to try this idea these two days.

[woctezuma]

2. An option to perform super resolution on the face during or after inference.
   An example would be something like Face Sparnet.

For info, I use an optional post-processing with GPEN [1] in one of my SimSwap-Colab notebooks: gpen/SimSwap_images.ipynb

If you want to try GPEN on Colab, independently from SimSwap, I have a small Github Gist for that purpose: https://gist.github.com/woctezuma/ecfc9849fac9f8d99edd7c1d6d03758a

Based on my experiments with super-resolution, GPEN looks the best among open-source super-resolution algorithms for faces. When it works, it is great. However, one has to be careful, because it can introduce artifacts, especially if the input photos had already been processed with a super-resolution algorithm. So it really has to be optional, to avoid inadvertently stacking super-resolution processing steps, which would degrade the final result.

[1] Yang, Tao, et al. GAN Prior Embedded Network for Blind Face Restoration in the Wild. arXiv preprint arXiv:2105.06070. 2021.
(code and paper)

[ExponentialML]

@woctezuma Wow, that works really well. Thanks for the tip!

@NNNNAI I actually tried implementing the video rotation idea, but that didn't solve the issue. The performance comes down to the function here. The alignment algorithm seems to make the face crop shake a lot, resulting in poor alignment on some more extreme angles.

[kkhaial]

Would also be good to have the mask saved as a separate output to bring the footage into after effects to edit

[AlonDan]

After playing with GPEN and DFNet (GPEN gives much better quality results in my opinion based on some tests).

Like the suggested above, but I would like to focus on local machine (Windows 10 + Anaconda)
It will be really nice and useful to have optional extra command to add: GPEN as post:

1️⃣ - GPEN should run on every frame that SimSwap generated.
2️⃣ - Then SimSwap merge-connect everything together hopefully with lossless quality from the source video.

I couldn't install GPEN locally so I could only test it via google colab, but if this could be merged to SimSwap as optional command so the user won't have to use GPEN (like the current version) but using a command to run GPEN as post with it's basic properties (scaling, generate ONLY the final GPEN without the other files to speed up things and focus SimSwap's goal) will be more than nice!

[instant-high]

@ftaker887 you say that you can solve the problem with the bounding box after inference.
Would you share your solution/ source code for that?

[NNNNAI]

You can try to use this face parsing repo to get the mask, and blend the original image and swapping image according to the mask. The results by using mask or by bounding box are shown below. Many thansk~.

[AlonDan]

Impressive! the smooth masking bounding box looking really nice!
@NNNNAI Any chance this feature will be added to SimSwap as extra command?

--MaskSmooth
or if needed extra parameters or values?

[NNNNAI]

I will release this feature may be within a week, cause there are works I got to busy with recently. Many thanks~.