Facial image processing method and facial image processing apparatus

Claim 2

A facial image processing method for synthesizing a facial image from first and second source facial images and a background facial image, each of the three facial images comprising associated shape data and associated image data, the associated shape data representing an associated shape model and including a plurality of points with associated coordinates, the associated image data representing a color of each pixel in the associated shape model, the method comprising the steps of: (a) grossly matching the first source facial image with the background facial image by converting coordinates of particular points in preselected feature areas in the shape model of the first source facial image to coordinates of corresponding points in the shape model of the background image so as to produce a first matched shape model; (b) grossly matching the second source facial image with the background facial image by converting coordinates of particular points in preselected feature areas in the shape model of the second source facial image to coordinates of corresponding points in the shape model of the background facial image so as to produce a second matched shape model; (c) interpolating the coordinates of the corresponding points in the matched shape models of the first and second source facial images and the shape model of the background facial image at a shape interpolation ratio so as to produce an interpolated shape model with associated shape interpolated data; and (d) interpolating a color of each pixel in the interpolated shape model at a color interpolation ratio, based on the shape data and the image data of the three images and the shape interpolated data so as to produce a synthesized image.

Claim 3

The image processing method of claim 1, wherein said step (c) comprises the step of changing the shape interpolation ratio according to a feature area being interpolated.

Claim 4

The image processing method according to claim 1, wherein the background facial image has luminance values associated with respective areas of said background facial image, and said step (d) comprises the step of setting the color interpolation ratio based on the luminance value of the corresponding area in the background facial image.

Claim 5

The image processing method of claim 3, further comprising the steps of: setting a color interpolation ratio for areas of the background facial image with a median luminance value of a skin color in the background image to a lowest value relative to a color interpolation ratio of the first and second facial source images, setting a color interpolation ratio for areas of the background facial image with a luminance value that is out of a predetermined range of the median luminance value of the skin color in the background facial image to a highest value relative to the color interpolation ratio of the first and second source facial images, and setting a color interpolation ratio for areas of the background facial image with a luminance value that is within the predetermined range of the median luminance value in accordance with its luminance value.

Claim 6

The image processing method of claim 1, wherein the first and second source facial images have luminance values associated with respective areas of said first and second source facial images, and said step (d) comprises the step of setting the color interpolation ratio based on the luminance values of the corresponding area in the first and second source facial images.

Claim 7

The image processing method of claim 5, further comprising the steps of: setting a color interpolation ratio for areas of the first and second source facial images with a median luminance value of a skin color in the first and second source facial images to a highest value relative to a color interpolation ratio of the background facial image, setting a color interpolation ratio for areas of the first and second source facial images with a luminance value that is out of a predetermined range of the median luminance value of the skin color in the first and second source facial images to a lowest value relative to the color interpolation ratio of the background image, and setting a color interpolation ratio for areas of the first and second source facial images with a luminance value within the predetermined range of the median luminance value in accordance with its luminance value.

Claim 8

The image processing method of claim 1, wherein said step (d) comprises the step of varying the color interpolation ratio according to a facial feature area being interpolated.

Claim 9

The image processing method of claim 1, further comprising the step of changing a facial expression of the synthesized image by moving a particular point in the synthesized image a predetermined distance, said step of changing a facial expression comprising the steps of: obtaining first, second and third vectors, the first vector being directed from a predetermined point in the synthesized image toward a contour of the synthesized image in a first direction, the second vector being directed from the predetermined point in the synthesized image toward the contour of the synthesized image in a second direction, and the third vector being orthogonal to the first and second vectors and having a magnitude that is a geometric mean of the first and second vectors; and calculating, based on preset constants and the first, second and third vectors, a displacement vector expressing a direction and amount of movement of the particular point in the synthesized image to be moved for changing the facial expression.

Claim 10

The image processing method of claim 1, further comprising the step of changing a facial expression of a preselected one of the first and second source images by moving a particular point in the preselected image a predetermined distance, said step of changing a facial expression comprising the steps of: obtaining first, second and third vectors, the first vector being directed from a predetermined point in the preselected image toward a contour of the preselected image in a first direction, the second vector being directed from the predetermined point in the preselected image toward the contour of the preselected image in a second direction, and the third vector being orthogonal to the first and second vectors and having a magnitude that is a geometric mean of the first and second vectors; and calculating, based on preset constants and the first, second and third vectors, a displacement vector expressing a direction and amount of movement of the particular point in the synthesized image to be moved for changing the facial expression.

Claim 11

A image processing apparatus for synthesizing a facial image from first and second source facial images and a background facial image, each of the three facial images comprising associated shape data and associated image data, the associated shape data representing an associated shape model and including a plurality of points with associated coordinates, the associated image data representing a color of each pixel in the associated shape model, the apparatus comprising: a first shape matching processor for grossly matching the first source facial image with the background facial image by converting coordinates of particular points in preselected feature areas in the shape model of the first source facial image to coordinates of corresponding points in the shape model of the background facial image so as to produce a first matched shape model; a second shape matching processor for grossly matching the second source facial image with the background facial image by converting coordinates of particular points in preselected feature areas in the shape model of the second source facial image to coordinates of corresponding points in the shape model of the background facial image so as to produce a second matched shape model; a shape interpolation processor which is coupled to said first and second shape matching processors for interpolating the coordinates of the corresponding points in the matched shape models of the first and second source facial images and the shape model of the background facial image at a shape interpolation ratio so as to produce an interpolated shape model with associated shape interpolated data; and a color interpolation processor which is coupled to said shape interpolation processor for interpolating a color of each pixel in the interpolated shape model at a color interpolation ratio, based on the shape data and the image data of the three facial images and the shape interpolated data so as to produce a synthesized image.

Claim 12

A image processing apparatus for synthesizing a facial image from first and second source facial images and a background facial image, each of the three facial images comprising associated shape data and associated image data, the associated shape data representing an associated shape model and including a plurality of points with associated coordinates, the associated image data representing a color of each pixel in the associated shape model, the apparatus comprising: a first shape matching processor for grossly matching the first source facial image with the background facial image by converting coordinates of particular points in preselected feature areas in the shape model of the first source facial image to coordinates of corresponding points in the shape model of the background facial image so as to produce a first matched shape model; a second shape matching processor for grossly matching the second source facial image with the background facial image by converting coordinates of particular points in preselected feature areas in the shape model of the second source facial image to coordinates of corresponding points in the shape model of the background facial image so as to produce a second matched shape model; a shape interpolation processor which is coupled to said first and second shape matching processors for interpolating the coordinates of the corresponding points in the matched shape models of the first and second source facial images and the shape model of the background facial image at a shape interpolation ratio so as to produce an interpolated shape model with associated shape interpolated data; and a color interpolation processor which is coupled to said shape interpolation processor for interpolating a color of each pixel in the interpolated shape model at a color interpolation ratio, based on the shape data and the image data of the three facial images and the shape interpolated data so as to produce a synthesized image.

Claim 13

The image processing apparatus of claim 10, wherein said shape interpolation processor comprises means for changing the shape interpolation ratio according to a feature area being interpolated.

Claim 14

The image processing apparatus of claim 10, wherein the background facial image has luminance values associated with respective areas of said background facial image, and said color interpolation processor comprises means for setting the color interpolation ratio based on the luminance values.

Claim 15

The image processing apparatus of claim 12, wherein the means for setting the color interpolation ratio sets a color interpolation ratio for areas of the background facial image with a median luminance value of a skin color in the background facial image to a lowest value relative to a color interpolation ratio of the first and second source facial images, sets a color interpolation ratio for areas of the background facial image with a luminance value that is out of a predetermined range of the median luminance value of the skin color in the background facial image to a highest value relative to the color interpolation ratio of the first and second source facial images, and sets a color interpolation ratio for areas of the background facial image with a luminance value that is within the predetermined range of the median luminance value in accordance with its luminance value.

Claim 16

The image processing apparatus of claim 10, wherein said color interpolation processor comprises: means for determining a luminance value of each area of the first and second source facial images, and means for setting the color interpolation ratio of a particular area of the first and second source facial images according to its luminance value.

Claim 17

The image processing apparatus of claim 14, wherein the means for setting the color interpolation ratio sets a color interpolation ratio for areas of the first and second source facial images with a median luminance value of a skin color in the first and second source facial images to a highest value relative to a color interpolation ratio of the background facial image, sets a color interpolation ratio for areas of the first and second source facial images with a luminance value that is out of a predetermined range of the median luminance value of the skin color in the first and second source facial images to a lowest value relative to the color interpolation ratio of the background facial image, and sets a color interpolation ratio for areas of the first and second source facial images with a luminance value within the predetermined range of the median luminance value in accordance with its luminance value.

Claim 18

The image processing apparatus of claim 10, wherein said color interpolation processor comprises means for varying the color interpolation ratio according to a feature area being interpolated.

Claim 19

The image processing apparatus of claim 10, further comprising an expression-changing processor for changing a facial expression of the synthesized image by moving a particular point in the synthesized image a predetermined distance, said expression-changing processor comprising: means for obtaining first, second and third vectors, the first vector being directed from a predetermined point in the synthesized image toward a contour of the synthesized image in a first direction, the second vector being directed from the predetermined point in the synthesized image toward the contour of the synthesized image in a second direction, and the third vector being orthogonal to the first and second vectors and having a magnitude that is a geometric mean of the first and second vectors; and means, based on preset constants and the first, second and third vectors, for calculating a displacement vector expressing a direction and amount of movement of the particular point in the synthesized image to be moved for changing the facial expression.

Claim 20

The image processing apparatus of claim 10, further comprising an expression-changing processor for changing a facial expression of a preselected one of the first and second source facial images by moving a particular point in the preselected image a predetermined distance, said expression-changing processor comprising: means for obtaining first, second and third vectors, the first vector being directed from a predetermined point in the preselected image toward a contour of the preselected image in a first direction, the second vector being directed from the predetermined point in the preselected image toward the contour of the preselected image in a second direction, and the third vector being orthogonal to the first and second vectors and having a magnitude that is a geometric mean of the first and second vectors; and means, based on preset constants and the first, second and third vectors, for calculating a displacement vector expressing a direction and amount of movement of the particular point in the synthesized image to be moved for changing the facial expression.