Color and material properties not getting captured for make up

[fused-byte]

First of all thanks for putting your work out in open and such simplified effective documentation.

I am trying to use the codebase for transferring lipstick shade from reference image (containing full face) to source image (containing full face). However, the color and material characteristics of the reference image are not reflecting in the generated image using the source.

I played around a bit with reference image by using the cropped part of the lips, blurring out remaining part of the image, blacking out remaining part of the image

I am running the code with default setting. I am new to diffusion field and therefore, not fully aware of the control knobs that can be leveraged for getting right output any help would be appreciated. Sharing example of images.

Reference image:

Generated image (you can see that it is purplish & not pink in color and instead of glossy it is matty finish):

Thanks in advance

[ttchengab]

Hi @fused-byte . Thanks for taking interest in our work. The current ZeST work is a zero-shot approach, and thus sometimes it causes some albedo shifts. For your specific case of lipstick colour where the region is very specific from the exemplar, you may require some training by fine-tuning it.

You may also try to change the initialisation seed to see if you can remove the albedo shift which at times are rather random.

[fused-byte]

Thanks for the response @ttchengab
A bit of clarification on the exemplar I am using; I am not passing the whole image as exemplar, I have mask of lip area and I am extracting a transparent image from that mask and original image. Attaching the examples of exemplars for reference.

And for 3rd and 4th images in second row the generated images are extremely poor. Those exemplar are cut-out of the lip area. My assumption is because of cut-out the resolution of reference (exemplar) became very poor. Is there a resolution of exemplar images for which the generated images are good?

[ttchengab]

The resolution is suggested to be around 1024x1024, which the original SDXL and IP-Adapter is trained for.