JPH11167626A - Image processing method, recording medium and image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method which forms a favorite face picture by easily and naturally moving visualized face parts. SOLUTION: A user determines which should be moved among eyes, an eyebrow and a mouth based on a desired facial expression (smile, anger, surprise and sorrow) and its movable range. That is, the user selects one of such expressions as 'smile', 'anger', 'surprise' and 'sorrow' by using scroll bars B7 to B10, watching a preview image Pv and further, freely inputs such feelings in 0 to 100% range as a parameter. Only by inputting the parameter of feeling, it is possible to easily form a desired facial expression and also to regulate and limitation of its movable range. Thus, facial expressions that are formed by it become very natural by preventing an eye, etc., from being moved to an unnatural position.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for displaying a person photographed by a digital camera on a display and changing the facial expression of the person.

[0002]

2. Description of the Related Art A person is photographed by a digital camera and displayed on a display. By the way, since the expression of a person is changing every moment, it is difficult to capture the best photo opportunity, and the best expression may not be captured. There are also cases where it is desired to record, for example, a smile or the like of a person who cannot change his / her own expression, such as a baby. In such a case, it is convenient if the facial expression of the person who has photographed can be changed.

[0003]

According to the prior art, for example, as disclosed in Japanese Patent Application Laid-Open No. 62-98472, the face of a photographed person is approximated three-dimensionally using a number of elements. In some cases, a distortion is calculated for each element to form an expression. However, according to such a conventional technique, although the formed facial expression is very natural, the amount of calculation of distortion performed for each of a large number of elements is enormous, which is inconvenient to handle.

On the other hand, software "KAI's Power Goo" marketed by Meta Tool is available.
According to (Registered Trademark), the expression can be freely changed by adding different processing to each part of the face. However, since a plurality of parts such as eyes, mouths, eyebrows, and the like are changed by separate processing, the operation is complicated and the handling is inconvenient.

According to another prior art, for example, as disclosed in Japanese Patent Application Laid-Open No. Hei 6-203123, when a weight is input, the face appears to be fat and the expression changes accordingly. is there. However, according to this technique, there is a problem that the facial expression changes uniformly, and the facial expression is not always the one desired by the user. Furthermore, in any of the prior arts, it is difficult to grasp the image after the change, so that it is necessary to form the best expression by trial and error, and this is also inconvenient to handle.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing method for forming a desired face state by easily and naturally moving a visualized face portion when a captured face is visualized. To provide.

[0007]

In order to achieve the above object, an image processing method according to the present invention comprises the steps of specifying each part of a face image based on image information including the face image, and changing the facial expression. Setting parameters for performing
In accordance with the set parameters, a plurality of parts of the face image to be changed are determined, and the content of processing for each of the determined parts is determined. Performing a process in accordance with the content of the process performed.

Further, in the image processing method according to the present invention, an image based on image information including a face image is displayed, and each part of the face image is specified by designating a part of the displayed face image. Setting a parameter for changing a facial expression; determining a part to be processed and processing contents from among the specified parts based on the set parameter; Changing image information corresponding to each part based on the determined face part and processing content.

An image processing apparatus according to the present invention includes: means for specifying each part of a face image based on image information including the face image; means for setting a parameter for changing a facial expression;
In accordance with the set parameters, a plurality of parts of the face image to be changed are determined, and a processing content for each of the determined parts is determined. Means for performing processing in accordance with the processing content.

Further, according to the image processing method of the present invention, an image based on image information including a face image is displayed, and each part of the face image is specified by designating a part of the displayed face image. Means for setting a parameter for changing a facial expression; means for determining a part to be processed and processing contents from among the specified parts based on the set parameters; Means for changing image information corresponding to each part based on the determined face part and processing content.

[0011]

According to the image processing method and the image processing apparatus of the present invention, a plurality of parts of the face image to be changed are determined according to the set parameters, and the processing content for each determined part is determined. Or, based on the set parameters, from among the specified parts, the part to be processed and the processing content are determined, so that, for example, the user can easily select a desired part. The face condition can be formed, and the formed face condition becomes very natural.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment according to the present invention will be described with reference to the drawings. FIG. 1 is a flowchart illustrating an image processing method for changing a facial expression displayed on a display of a computer. FIG. 2 is a perspective view showing a computer from which a program for executing the image processing method is read.

In FIG. 2, image information of a person photographed by a digital camera (not shown) is stored in a memory built in the computer 1, and a face F of the person is displayed on a display means based on the image information. A display 1a
Is displayed in. A first dialog box D1 including six scroll bars B1 to B6 is displayed around the displayed face F, and allows a user operation to be described later. That is, the buttons of the scroll bars B1 to B6 can be moved up and down or left and right by the mouse 1b. The floppy disk FD serving as a recording medium stores a program for executing an image processing method described later.
The computer 1 reads the program of
Image processing of changing the expression can be performed. Note that such a program may be stored on a CD, a ROM, or the like. Hereinafter, this image processing will be described.

Steps S101 to S103 in FIG.
It is an example of a method of determining each part of a face image. In step S101, the user first selects a face area. More specifically, in FIG. 2, a rectangular area Af surrounding a face F whose expression is to be converted is specified. This designation can be made, for example, by the user dragging the mouse 1b.

First, the computer 1 displays a first dialog box D1 (FIG. 2). Since the positions and areas of the plurality of rectangular areas A1 to A3 can be changed on the display 1a, the user moves these rectangular areas A1 to A3, and the eyes E and the mouth M which are parts of the face F are moved. , The eye E and the mouth M are selected (step S102). In addition, the user can use the scroll bar B1 in FIG.
Can be moved left and right, the rectangular areas A1 and A2 can be moved up and down by the scroll bar B2, and the area of the rectangular areas A1 and A2 can be changed by the scroll bar B3. The positions and areas of the rectangular regions A1 and A2 may be independently changed.

On the other hand, the user can move the rectangular area A3 surrounding the mouth M right and left by the scroll bar B4,
The rectangular area A3 can be moved up and down by the scroll bar B5, and the area of the rectangular area A3 can be changed by the scroll bar B6.

Further, in step S103, the computer 1 detects the eyes E and the mouth M. The manner in which the eyes E and the mouth M are detected in the selected area will be described below. FIG.
Is a flowchart showing a subroutine for detecting an eye E and a mouth M.

First, the computer 1 detects an edge pixel in the area selected in step S102 (FIG. 1) (step S103a). Specifically, first, a linear sum X _i , _j of the RGB values of the target pixel is obtained. here,
i and j are vertical and horizontal positions in the image. This linear sum is the eye,
Each mouth has a unique value, and in the case of eye detection,
For example, X _i , _j = 0.3 * R _i , _j + 0.59 * G _i , _j +
It is calculated with 0.11 * B _i , _j , and in the case of mouth detection, for example, X _i , _j = 0.21 * R _i , _j− 0.52 * G _i , _j +
Calculated as 0.31 * B _i , _j .

The computer 1 obtains a difference between a certain pixel and its surrounding four pixels, and obtains edge amounts E _i , _j . E
_i and _j are represented by the following equations. E _i , _j = | X _{i + 1} , _j −X _i , _j | + | X _i−1 , _j −X _i , _{j
| + | X i, j +} 1 -X i, j | + | X i, j-1 -X i, j |

Further, the computer 1 obtains the average Ave_edge of the edge amount in the target area. If E _i , _j is larger than Ave_edge in the target area, it is determined that the pixel is an edge pixel.

Next, in step S103b, the computer 1 labels the detected edge pixels in the vicinity of the top, bottom, left, and right of the detected edge pixel. More specifically, if there are edge pixels near the periphery 8 of the edge pixel, the same label is determined, and Ai (i = label) is obtained for the area of each label.

The computer 1 further performs step S103
In c, the area to which the label having the largest area is allocated in each area is determined as an eye and mouth area. If the positions of the eyes or the mouth are determined, the outline of the face is statistically estimated to some extent, and the accuracy of detecting the face area can be improved by using the estimated outline of the face. The contour of the face may be estimated by determining the position of the eyebrows. As described above, each part of the face image is specified. That is, the position and area of each part are determined. Incidentally, the identification of each part may be performed by a method of automatically recognizing the face image after designating the face image.

Returning to the flowchart of FIG. 1 again, after the computer 1 displays the second dialog box D2 in step S104, the user selects an emotion parameter which is a parameter for changing the facial expression. (Step S104). FIG. 4 is a diagram showing a second dialog box D2 displayed on the display. On the display 1a, a target face image is displayed as a preview image Pv together with a dialog box D2 as shown in FIG.

More specifically, the manner of selecting emotion parameters will be described. While viewing the preview image Pv, the user selects one of emotions such as “laugh”, “anger”, “surprise”, and “sadness” using the scroll bars B7 to B10, and further changes the emotion from 0 to 100. %
Input freely as a parameter in the range of. The parameter indicates the degree of emotion. For example, in the case of the parameter 0%, the facial expression does not change from the original. On the other hand, when the parameter is 100%, the facial expression is most emphasized. Therefore, if the face part is moved further, an unnatural facial expression is formed with respect to the contour of the face. However, if an unrealistic comical expression is desired, for example, it may be possible to input a parameter exceeding 100%.

In step S104, based on the parameters selected and input by the user, step S104 is executed.
At 105, the computer 1 converts the coordinates of the eye, eyebrow, and mouth regions as follows to change the expression.
Note that a coordinate system in which the X axis and the Y axis extend as shown in FIG. 5 is determined and used for the following calculations.

(1) In the case of “laughing” For each of the right eye and the left eye, coordinate conversion is performed such that only the outer corner of the eye is lowered. The coordinate conversion is performed according to the following equation. Left _{eye: if (X i, j <} X_mid_l) Y_new i, j = Y i, j -SLOPE_Y * (X i, j -
X_mid_l) right _{eye: if (X i, j>} X_mid_r) Y_new i, j = Y i, j -SLOPE_Y * (X_mi
d_l−X _i , _j ) where X_mid_l and X_mid_r indicate the x-coordinate of the horizontal center of the right-eye and left-eye regions, respectively.
j indicates x and y coordinates, respectively. SLOPE_Y is
Indicates the amount of coordinate conversion in the y direction, and is determined based on the parameter selected in step S104.

The region of the eyebrows is determined by calculation from the position information of the region of both eyes. For example, in the coordinate system shown in FIG. 6, the coordinates of the upper left, upper right, lower left, and lower right of the rectangular area surrounding the left eye are (X_l_l, Y_u_
l), (X_r_l, Y_u_l), (X_l_l, Y
_D_l) and (X_r_l, Y_d_l) assuming that the width and height of the region are Width_l and Height_l, the upper left, upper right, lower left, and lower right coordinates (X_r
l_lb, Y_u_lb), (X_r_lb, Y_u_
lb), (X_l_lb, Y_d_lb), (X_r_
lb, Y_d_lb) are calculated as follows. X_l_lb = X_l_l X_r_lb = X_r_l Y_u_lb = Y_u_l-Height_l Y_d_lb = Y_d_l-Height_l

It is estimated that an eyebrow is included in such a region, and coordinate conversion is performed so as to lower both eyebrows. The coordinate conversion is performed according to the following equation. Left _{eyebrow: Y_new i, j = Y i} , j + SLOPE_Y * (X i, j -
_{X_r_l) * (X i, j} -X_r_l) right _{eyebrow: Y_new i, j = Y i} , j + SLOPE_Y * (X i, j -
X_l_r) * (X _i , _j− X_l_r)

Further, coordinate conversion is performed such that both ends of the mouth are raised. FIG. 7 shows the area of the mouth together with the coordinate system. Assuming that the x coordinate of the center of the mouth area is X_MOUTH, it is calculated as follows. _{Y_new i, j = Y i,} j + SLOPE_Y * (X i, j -
X_MOUTH) * (X _i , _j
-X_MOUTH)

(2) In case of anger For each of the right eye and the left eye, coordinate conversion is performed such that only the outer corner of the eye is raised. The coordinate conversion is performed according to the following equation. Left _{eye: if (X i, j <} X_mid_l) Y_new i, j = Y i, j + SLOPE_Y * (X i, j -
X_mid_l) right _{eye: if (X i, j>} X_mid_r) Y_new i, j = Y i, j + SLOPE_Y * (X_mi
d_l−X _i , _j ) where X_mid_l and X_mid_r indicate the x-coordinate of the horizontal center of the right-eye and left-eye regions, respectively.
j indicates x and y coordinates, respectively. SLOPE_Y is
Indicates the amount of coordinate conversion in the y direction, and is determined based on the parameter selected in step S104.

The eyebrow area is determined by calculation from the position information of the both eye areas. For example, in the coordinate system shown in FIG. 6, the coordinates of the upper left, upper right, lower left, and lower right of the rectangular area surrounding the left eye are (X_l_l, Y_u_
l), (X_r_l, Y_u_l), (X_l_l, Y
_D_l) and (X_r_l, Y_d_l), assuming that the width and height of the region are width_l and Height_l, the upper left, upper right, lower left, and lower right coordinates (X_r
l_lb, Y_u_lb), (X_r_lb, Y_u_
lb), (X_l_lb, Y_d_lb), (X_r_
lb, Y_d_lb) are calculated as follows. X_l_lb = X_l_l X_r_lb = X_r_l Y_u_lb = Y_u_l-Height_l Y_d_lb = Y_d_l-Height_l

It is estimated that an eyebrow is included in such an area, and coordinate conversion is performed so that both eyebrows are raised. The coordinate conversion is performed according to the following equation. Left _{eyebrow: Y_new i, j = Y i} , j -SLOPE_Y * (X i, j -
_{X_r_l) * (X i, j} -X_r_l) right _{eyebrow: Y_new i, j = Y i} , j -SLOPE_Y * (X i, j -
X_l_r) * (X _i , _j− X_l_r)

Further, coordinate transformation is performed to raise both ends of the mouth. In the coordinate system of FIG. 7, when the x coordinate of the center of the mouth area is X_MOUTH, the calculation is performed as follows. Y_new _i , _j = Y _i , _j −SLOPE_Y * (X _i , _j −
X_MOUTH) * (X _i , _j
-X_MOUTH)

(3) In case of "sadness" Coordinate transformation is performed for each of the right eye and the left eye such that only the outer corner of the eye is lowered. The coordinate conversion is performed according to the following equation. Left _{eye: if (X i, j <} X_mid_l) Y_new i, j = Y i, j -SLOPE_Y * (X i, j -
X_mid_l) right _{eye: if (X i, j>} X_mid_r) Y_new i, j = Y i, j -SLOPE_Y * (X_mi
d_l−X _i , _j ) where X_mid_l and X_mid_r indicate the x-coordinate of the horizontal center of the right-eye and left-eye regions, respectively.
j indicates x and y coordinates, respectively. SLOPE_Y is
Indicates the amount of coordinate conversion in the y direction, and is determined based on the parameter selected in step S104.

The region of the eyebrows is determined by calculation from the position information of the region of both eyes. For example, in the coordinate system shown in FIG. 6, the coordinates of the upper left, upper right, lower left, and lower right of the rectangular area surrounding the left eye are (X_l_l, Y_u_
l), (X_r_l, Y_u_l), (X_l_l, Y
_D_l) and (X_r_l, Y_d_l), assuming that the width and height of the region are width_l and Height_l, the upper left, upper right, lower left, and lower right coordinates (X_r
l_lb, Y_u_lb), (X_r_lb, Y_u_
lb), (X_l_lb, Y_d_lb), (X_r_
lb, Y_d_lb) are calculated as follows. X_l_lb = X_l_l X_r_lb = X_r_l Y_u_lb = Y_u_l-Height_l Y_d_lb = Y_d_l-Height_l

It is estimated that the eyebrow is included in such an area, and coordinate conversion is performed so that both eyebrows are lowered. The coordinate conversion is performed according to the following equation. Left _{eyebrow: Y_new i, j = Y i} , j + SLOPE_Y * (X i, j -
_{X_r_l) * (X i, j} -X_r_l) right _{eyebrow: Y_new i, j = Y i} , j + SLOPE_Y * (X i, j -
X_l_r) * (X _i , _j− X_l_r)

A coordinate transformation is performed to lower both ends of the mouth.
In the coordinate system shown in FIG. 7, the x coordinate of the center of the mouth area is
Assuming that X_MOUTH, the calculation is performed as follows. _{Y_new i, j = Y i,} j + SLOPE_Y * (X i, j -
X_MOUTH) * (X _i , _j
-X_MOUTH)

(4) Case of "surprise" No coordinate conversion is performed for the eye area. That is,
Same as before.

For the eyebrow area, coordinate conversion is performed so as to widen the distance between the eye and the eyebrow. For example, in the coordinate system shown in FIG. 6, the coordinates of the upper left, upper right, lower left, and lower right of the rectangular area surrounding the left eye are (X_l_l, Y_u_l),
(X_r_l, Y_u_l), (X_l_l, Y_d_
l) and (X_r_l, Y_d_l) assuming that the width and height of the region are width_l and Height_l,
Upper left, upper right, lower left, lower right coordinates (X_l_l) of the eyebrow area
b, Y_u_lb), (X_r_lb, Y_u_l)
b), (X_l_lb, Y_d_lb), (X_r_l
b, Y_d_lb) are converted as follows. X_l_lb = X_l_l X_r_lb = X_r_l Y_u_lb = Y_u_l-Height_l * SLO
PE_Y Y_d_lb = Y_d_l-Height_l * SLO
PE_Y

The mouth performs coordinate conversion so as to spread in the Y-axis direction. In the coordinate system shown in FIG. 7, x at the center of the mouth area,
Let y coordinate be (X_MOUTH, Y_MOUTH),
X_min at the left end and X_max at the right end
Then, it is calculated as follows. Upper _{lip: if (Y i, j <} Y_MOUTH) {if (X i, j <X_MOUTH) Y_new i, j = Y i, j -SLOPE_Y * (X i, j -X_min) 0.5 else Y_new i, j = Y i _{, j -SLOPE_Y * (X_max-X} i, j) 0.5} else under _{lip: if (Y i, j>} Y_MOUTH) {if (X i, j <X_MOUTH) Y_new i, j = Y i, j + SLOPE_Y * ( ^{_{X i, j -X_min) 0.5 else}} Y_new i, j = Y i, j -SLOPE_Y * (X_max-X i, j) 0.5}

In the "surprise" coordinate transformation described above, the mouth is set to spread without being opened. However, the mouth may be opened so that the teeth and tongue can be seen. Further, in the above-described embodiment, the facial expression is changed for a person's face, but may be applied to an animal's face such as a dog or a cat. Furthermore, if a program for executing the image processing method according to the present embodiment is installed in a digital camera with a liquid crystal screen in advance, the image processing can be performed via the liquid crystal screen.

[0042]

According to the image processing method and the image processing apparatus of the present invention, a plurality of parts of the face image to be changed are determined in accordance with the set parameters, and the processing content for each determined part is determined. Determine or based on the set parameters, from among the identified parts,
Since the part to be processed and the processing content are determined, for example, the user can easily form a desired face state, and the formed face state becomes very natural.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating an image processing method for changing a facial expression displayed on a display.

FIG. 2 is a perspective view showing a computer from which a program for executing an image processing method according to the present invention has been read.

FIG. 3 is a flowchart showing a subroutine for detecting an eye E and a mouth M;

FIG. 4 is a diagram showing a second dialog box displayed on a display.

FIG. 5 is a diagram showing a coordinate system used to determine the position of a face part.

FIG. 6 is a diagram illustrating a relationship between an eye region and an eyebrow region.

FIG. 7 is a diagram showing a mouth area together with a coordinate system.

[Description of Signs] 1 Computer 1a Display 1b Mouse B1 to B10 Scroll bar D1 First dialog box D2 Second dialog box FD Floppy

Claims (10)

[Claims] 1. A step of specifying each part of a face image based on image information including a face image; a step of setting a parameter for changing a facial expression; Determining a plurality of parts of the face image to be changed and determining processing contents for each of the determined parts; and performing processing according to the processing contents determined for each of the determined parts. An image processing method comprising: 2. The image processing method according to claim 1, further comprising the step of displaying the processed face image. 3. A step of setting a second parameter indicating a degree of change in facial expression, wherein a plurality of parts of the face image to be changed are determined in accordance with the set parameters, and the determination is performed. 2. The image processing method according to claim 1, wherein the step of determining the processing content for each of the determined parts determines the processing content in accordance with the second parameter. 4. A step of displaying an image based on image information including a face image, specifying each part of the face image by designating a part of the displayed face image, and changing a facial expression Setting a parameter for performing, and, based on the set parameters, determining a part to be processed and processing contents from among the specified parts, and a part of the determined face and Changing image information corresponding to each of the parts based on processing contents. 5. The image processing method according to claim 1, wherein two or more parts of the face to be moved are determined based on the facial expression of the face image. 6. The face image according to claim 1, wherein the part of the face image is at least one of an eye, an eyebrow, and a mouth.
The image processing method according to any one of the above. 7. The face image is displayed on display means, and the step of setting the parameter and the step of determining the part are performed using area variable information displayed together with the image on the display means. The image processing method according to claim 1, wherein: 8. A computer-readable recording medium on which a program for executing the image processing method according to claim 1 is recorded. 9. A means for specifying each part of the face image based on image information including the face image; a means for setting a parameter for changing a facial expression; Means for determining a plurality of parts of the face image to be changed, and determining processing contents for each of the determined parts; means for performing processing in accordance with the processing contents determined for each of the determined parts An image processing apparatus comprising: 10. A means for displaying an image based on image information including a face image, specifying each part of the face image by designating a part of the displayed face image, and changing a facial expression Means for setting parameters for performing, a means for determining a part to be processed and processing contents from among the specified parts based on the set parameters, and a part of the determined face and Means for changing image information corresponding to each part based on processing contents.