JP2009123101A - Estimation of face image direction - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To estimate the direction of a face image in an object image. <P>SOLUTION: An image processor is provided with: an organ area detection part for detecting an area including an image of facial organ in the object image as an organ area; and a face direction estimation part for estimating the direction of the face image in the object image on the basis of the detected organ area. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for estimating the orientation of a face image in an image.

  An image processing technique for deforming an image for a digital image is known (for example, Patent Document 1). In Patent Document 1, a partial area (an area representing a cheek image) on a face image is set as a correction area, the correction area is divided into a plurality of small areas according to a predetermined pattern, and set for each small area. Image processing is disclosed in which the shape of the face image is deformed by enlarging or reducing the image at the specified magnification.

JP 2004-318204 A

  Since the above conventional image processing for deforming the shape of the face image assumes that the face image is a front-facing image, the face image orientation is not a front-facing orientation but a horizontal or vertical swing. In such a case, the processed image may become unnatural. Therefore, it is useful to estimate the orientation of the face image in the target image.

  In this specification, the direction of an image in which the face of the subject person faces upward is referred to as “upward swing”, and the direction of the image in which the face of the subject person faces downward is referred to as “downward swing”. . The direction of the image of the subject person facing to the right when viewed from the viewer side of the image (that is, the person is actually facing the left side) is called “right swing”. Refers to the direction of the image facing the left as viewed from the viewer side of the image (that is, the person is actually facing the right). That the image orientation is up or down is also referred to as “up and down”, and that the right or left is referred to as “left or right”.

  In addition, for example, when setting a region including a face image in the target image, it is useful to estimate the orientation of the face image in the target image.

  SUMMARY An advantage of some aspects of the invention is that it provides a technique capable of estimating the orientation of a face image in a target image.

  In order to solve at least a part of the above problems, the present invention can be realized as the following forms or application examples.

Application Example 1 An image processing apparatus,
An organ region detection unit that detects a region including an image of a facial organ in the target image as an organ region;
An image processing apparatus comprising: a face orientation estimating unit that estimates the orientation of a face image in the target image based on the detected organ region.

  In this image processing apparatus, a region including a facial organ image in the target image can be detected as an organ region, and the orientation of the facial image in the target image can be estimated based on the detected organ region.

[Application Example 2] The image processing apparatus according to Application Example 1,
The organ region detector detects a plurality of the organ regions,
The face orientation estimating unit estimates an orientation of a face image in the target image based on a relationship related to at least one of a position and a size between the detected organ regions.

  Since the positional relationship and the size relationship between the plurality of organ regions are considered to change according to the orientation of the face image, this image processing apparatus uses a relationship related to at least one of the position and size between the plurality of organ regions. Based on the above, the orientation of the face image in the target image can be estimated.

[Application Example 3] The image processing apparatus according to Application Example 1 or Application Example 2,
The face orientation estimation unit calculates a reference width as an index correlated with the width of the face image and a reference height as an index correlated with the height of the face image based on the detected organ region. An image processing device that estimates a face image orientation in the target image based on a ratio between the reference width and the reference height.

  The ratio between the reference width correlated to the face image width and the reference height correlated to the face image height is considered to change according to the orientation of the face image. Based on the ratio between the width and the reference height, the orientation of the face image in the target image can be estimated.

[Application Example 4] The image processing apparatus according to Application Example 3,
The face direction estimation unit is an image processing device that estimates a degree of a swing of a face image from a front direction in the target image based on a ratio between the reference width and the reference height.

  Since the ratio between the reference width and the reference height is considered to change according to the degree of swing from the front of the face image, this image processing apparatus uses the ratio between the reference width and the reference height in the target image. The degree of the swing of the face image from the front direction can be estimated.

[Application Example 5] The image processing apparatus according to Application Example 3 or Application Example 4,
The organ region detection unit includes at least a right eye region including an image of the right eye as the facial organ, a left eye region including an image of the left eye as the facial organ, and an image of the mouth as the facial organ. An image processing apparatus for detecting a mouth region as the organ region.

  In this image processing apparatus, the orientation of the face image in the target image can be estimated based on the right eye region, the left eye region, and the mouth region.

[Application Example 6] The image processing apparatus according to Application Example 5,
The face direction estimating unit calculates the reference width based on the right eye region and the left eye region, and calculates the reference height based on at least one of the right eye region and the left eye region and the mouth region; Image processing device.

  In this image processing apparatus, the reference width can be calculated based on the right eye region and the left eye region, and the reference height can be calculated based on at least one of the right eye region and the left eye region and the mouth region.

Application Example 7 The image processing apparatus according to any one of Application Examples 1 to 6, further comprising:
A face area detection unit that detects an area including a face image in the target image as a detection face area;
The face orientation estimation unit is an image processing device that estimates the orientation of a face image in the target image based on the detected organ region and the detected face region.

  Since the relationship between the organ area and the detected face area is considered to change according to the orientation of the face image, this image processing apparatus determines the orientation of the face image in the target image based on the organ area and the detected face area. Can be estimated.

Application Example 8 The image processing apparatus according to any one of Application Examples 1 to 7, further comprising:
An image processing unit that performs predetermined image processing on a predetermined region including a face image in the target image;
An image processing apparatus comprising: a process availability determination unit that determines whether the image processing can be performed based on the estimated orientation of the face image in the target image.

  In this image processing apparatus, whether or not image processing can be executed can be determined based on the estimated orientation of the face image in the target image. For example, the result is unnatural depending on the orientation of the face image. Regarding image processing, it is possible to avoid execution of image processing that results in unnatural results.

[Application Example 9] The image processing apparatus according to Application Example 8,
The predetermined image processing includes image deformation processing,
The processing availability determination unit determines that the execution of the image processing is impossible when the orientation of the face image in the estimated target image is not front-facing.

  In this image processing apparatus, when the orientation of the face image is not front-facing, it is possible to avoid execution of the image deformation process.

Application Example 10 The image processing apparatus according to any one of Application Examples 1 to 7, further comprising:
A face area setting unit configured to set a face area including a face image in the target image by a predetermined method according to the estimated orientation of the face image in the target image based on the detected organ area; Image processing device.

  In this image processing apparatus, the face area including the face image in the target image can be set by a predetermined method according to the estimated face image orientation, so that the setting accuracy of the face area is improved. Can do.

[Application Example 11] The image processing apparatus according to Application Example 10,
The face area setting unit is configured to determine at least one of the size, position, and inclination of the face area based on a relationship with the organ area determined in advance according to the orientation of the face image in the estimated target image. An image processing apparatus that identifies one area and sets an area having at least one of the identified size, position, and inclination as the face area.

  In this image processing apparatus, since at least one of the size, position, and inclination of the face area can be specified based on the relationship with the organ area determined in advance according to the estimated orientation of the face image, The area setting accuracy can be improved.

[Application Example 12] The image processing apparatus according to Application Example 11, further comprising:
A face area detection unit that detects an area including a face image in the target image as a detection face area;
The face area setting unit sets, as the face area, an area in which at least one of the size, position, and inclination of the detected face area is matched with at least one of the specified size, position, and inclination. , Image processing device.

  In this image processing apparatus, the face area can be set based on the detected face area, and the setting accuracy of the face area can be improved.

[Application Example 13] The image processing apparatus according to any one of Application Examples 3 to 6, further comprising:
A face area setting unit configured to set a face area including a face image in the target image in a predetermined method according to the estimated orientation of the face image in the target image based on the detected organ area;
The face area setting unit is an image processing apparatus that specifies a size of the face area based on the reference width and the reference height.

  In this image processing apparatus, the size of the face area can be specified based on the reference width and the reference height, and the setting accuracy of the face area can be improved.

[Application Example 14] The image processing apparatus according to Application Example 13,
The face area setting unit specifies a width of the face area by multiplying the reference width by a predetermined coefficient,
The value of the predetermined coefficient used when the estimated orientation of the face image in the target image is right-handed or left-handed is compared with the value of the predetermined coefficient used when the face-oriented. Large, image processing device.

  In this image processing apparatus, it is possible to improve the setting accuracy of the face area by changing the value of a predetermined coefficient that is multiplied by the reference width according to the orientation of the face image.

[Application Example 15] The image processing apparatus according to Application Example 13,
The face area setting unit specifies the height of the face area by multiplying the reference height by a predetermined coefficient,
The value of the predetermined coefficient used when the estimated orientation of the face image in the target image is up or down is compared with the value of the predetermined coefficient used when the face is facing forward. Large, image processing device.

  In this image processing apparatus, the setting accuracy of the face area can be improved by changing the value of a predetermined coefficient multiplied by the reference height according to the orientation of the face image.

[Application Example 16] The image processing apparatus according to any one of Application Example 10 to Application Example 15,
An image processing apparatus comprising: an image processing unit that sets a processing target area based on the set face area and performs predetermined image processing on the processing target area.

  In this image processing apparatus, a processing target area for predetermined image processing can be set based on the face area set with high accuracy, and thus more preferable image processing can be realized.

Application Example 17 The image processing apparatus according to any one of Application Example 1 to Application Example 16,
The organ area detection unit includes a reliability setting unit that sets a reliability index that represents the probability that the detected organ area is an area including an image of a facial organ,
The face orientation estimating unit estimates an orientation of a face image in the target image based only on the organ area in which the value of the reliability index is greater than a predetermined threshold among the detected organ areas. apparatus.

  In this image processing apparatus, it is possible to improve the estimation accuracy of the orientation of the face image in the target image.

  Note that the present invention can be realized in various modes. For example, an image processing method and apparatus, an image correction method and apparatus, an image printing method and apparatus, a face direction estimation method and apparatus, a face area setting method and The present invention can be realized in the form of an apparatus, a computer program for realizing the functions of these methods or apparatuses, a recording medium recording the computer program, a data signal including the computer program and embodied in a carrier wave, and the like. .

Next, embodiments of the present invention will be described in the following order based on examples.
A. First embodiment:
A-1. Configuration of image processing device:
A-2. Face shape correction printing process:
A-3. Transformation process:
B. Second embodiment:
C. Variations:

A. First embodiment:
A-1. Configuration of image processing device:
FIG. 1 is an explanatory diagram schematically showing the configuration of a printer 100 as an image processing apparatus according to the first embodiment of the present invention. The printer 100 of this embodiment is a color inkjet printer that supports so-called direct printing, in which an image is printed based on image data acquired from a memory card MC or the like. The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by, for example, a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, A printer engine 160 and a card interface (card I / F) 170 are provided. The printer 100 may further include an interface for performing data communication with other devices (for example, a digital still camera or a personal computer). Each component of the printer 100 is connected to each other via a bus.

  The printer engine 160 is a printing mechanism that performs printing based on print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, an image file including image data is stored in the memory card MC.

  The internal memory 120 stores a face shape correction printing unit 200, a display processing unit 310, and a print processing unit 320. The face shape correction printing unit 200 is a computer program for executing a face shape correction printing process described below under a predetermined operating system. The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus, messages, images, and the like on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printer engine 160, and printing an image based on the print data. The CPU 110 implements the functions of these units by reading and executing these programs from the internal memory 120.

  The face shape correction printing unit 200 includes, as program modules, a face region detection unit 210, an organ region detection unit 220, a face orientation estimation unit 230, a deformation region setting unit 240, a deformation region dividing unit 250, and a divided region deformation. Part 260 and a process availability determination part 270. In addition, the organ region detection unit 220 includes a reliability setting unit 222. The functions of these units will be described in detail in the description of the face shape correction printing process described later. Note that, as will be described later, image processing for deforming an image is performed by the deformed region dividing unit 250 and the divided region deforming unit 260. Therefore, the deformed area dividing unit 250 and the divided area deforming unit 260 can be collectively referred to as an “image processing unit”.

  The internal memory 120 also stores a dividing point arrangement pattern table 410 and a dividing point movement table 420. These contents will also be described in detail in the description of the face shape correction printing process described later.

A-2. Face shape correction printing process:
The printer 100 can execute a face shape correction printing process for correcting the face shape of the target image and printing the target image after the face shape correction. The face shape correction is a process for correcting at least a part of the face in the target image (for example, the contour shape of the face). In this embodiment, it is assumed that face shape correction for slimming the face shape is performed, and printing of a target image whose face shape is corrected to slim can be realized by the face shape correction printing process. When the memory card MC is inserted into the card slot 172 (FIG. 1) and a predetermined operation is performed by the user via the operation unit 140, the face shape correction printing process is started.

  FIG. 2 is a flowchart showing the flow of the face shape correction printing process. In step S100, the face shape correction printing unit 200 (FIG. 1) sets a target image TI to be processed. The face shape correction printing unit 200 instructs the display processing unit 310 to display a predetermined user interface for setting the target image TI on the display unit 150. FIG. 3 is an explanatory diagram illustrating an example of a user interface for setting the target image TI. In the image display field IA of the user interface shown in FIG. 3, an image is displayed based on the image data stored in the memory card MC. When a plurality of images are stored in the memory card MC, the images displayed in the image display field IA are sequentially switched according to the operation through the operation unit 140 by the user. Note that a plurality of images may be displayed in a list on the user interface for setting the target image TI based on the plurality of image data stored in the memory card MC. The face shape correction printing unit 200 sets an image selected by the user on the user interface for setting the target image TI as the target image TI.

  In step S200 (FIG. 2), the face area detection unit 210 (FIG. 1) detects the face area FA in the target image TI. Here, the face area FA is an image area in the target image TI and includes at least a partial image of the face. The face area detection unit 210 analyzes the target image TI and detects a rectangular area assumed to include a face image as the face area FA. The detection of the face area FA by the face area detection unit 210 is performed using a known detection method such as a pattern matching method using a template (see Japanese Patent Application Laid-Open No. 2006-279460). In this specification, the face area FA detected in step S200 is also referred to as “detected face area FAd”.

  FIG. 4 is an explanatory diagram illustrating an example of a detection result of the detected face area FAd. In the example of FIG. 4, since the target image TI includes two face images of the person P1 and the person P2, an area including the face image of the person P1 and an area including the face image of the person P2 The two detected face areas FAd are detected. In the present embodiment, as shown in FIG. 4, the detected face area FAd is a rectangular area including images of both eyes, nose and mouth. The face area detection unit 210 identifies the detected face area FAd by, for example, the coordinates of the four vertices of the detected face area FAd.

  In step S200 (FIG. 2), when the detection of the detected face area FAd is not successful (step S300: No), the fact is notified to the user through the display unit 150. In this case, in order to allow the user to select another image as the target image TI, the user interface (FIG. 3) for selecting the target image TI is displayed again on the display unit 150, and the target image TI is reset ( Step S100) is performed. Note that if the detection of the detected face area FAd is not successful, the processing from step S400 to S700 in FIG. 2 may be skipped and the target image TI in step S800 may be printed.

  On the other hand, when the detection of the detected face area FAd has succeeded in step S200 (FIG. 2) (step S300: Yes), the organ area detection unit 220 (FIG. 1) then displays the organ area in the target image TI. Perform detection. Here, the organ region is an image region in the target image TI and includes at least a partial image of the facial organ. In this embodiment, the right eye, the left eye, and the mouth of the subject person are set as the facial organs, and the right eye area EA (r) including the right eye image and the left eye area including the left eye image as the organ areas. Detection of EA (l) and the mouth area MA including the mouth image is performed. The organ area detection 220 analyzes the detected face area FAd in the target image TI, and converts the three rectangular areas assumed to include the right eye, left eye, and mouth images into the right eye area EA (r) and the left eye area EA, respectively. (L) Detect as mouth area MA. The detection of the organ region by the organ region detection unit 220 is executed using a known detection method such as a pattern matching method using a template, as in the detection of the face region.

  FIG. 5 is an explanatory diagram showing an example of the detection result of the organ region. As shown in FIG. 5, in the detected face area FAd corresponding to each of the person P1 and the person P2, the right eye area EA (r), the left eye area EA (l), and the mouth area MA are detected. In the present embodiment, as shown in FIG. 5, the right eye area EA (r) is a rectangular area that includes an image of the right eye of a person. Similarly, the left eye area EA (l) is a rectangular area that includes the image of the person's left eye, and the mouth area MA is a rectangular area that includes the image of the person's mouth. The organ region detection unit 220 identifies the detected organ region by, for example, the coordinates of four vertices of each organ region.

  In this embodiment, the reliability setting unit 222 (FIG. 1) sets a reliability index for each detected organ region. The reliability index is an index representing the probability that the detected organ region is a region that truly includes an image of a facial organ. In the present embodiment, the number of matches when pattern matching is performed a plurality of times while changing the template is used as the reliability index.

  In step S400 (FIG. 2), when the detection of the organ region is successful (step S500: Yes), the process proceeds to step S600. Here, in this embodiment, when the detection of the organ region is successful, all three organ regions of the right eye region EA (r), the left eye region EA (l), and the mouth region MA are detected, and all This means that the probability represented by the reliability index is larger than a predetermined threshold for the organ region. If any one of the three organ regions is not detected, or if the probability represented by any one of the three organ regions is less than a predetermined threshold, This means that the detection was not successful.

  In step S600 (FIG. 2), the face direction estimation unit 230 (FIG. 1) performs face direction estimation. The face orientation estimation is processing for estimating the orientation of the face image included in the target image TI. FIG. 6 is an explanatory diagram showing a method of face orientation estimation. In FIG. 6, the length of the line segment CL connecting the center point Ce (r) of the right eye area EA (r) and the center point Ce (l) of the left eye area EA (l) is referred to as a reference width Wr, and the mouth area MA The distance between the center point Cm and the line segment CL is referred to as a reference height Hr. As is apparent from FIG. 6, the reference width Wr is an index correlated with the width of the face image, and the reference height Hr is an index correlated with the height of the face image.

  The face orientation estimation unit 230 calculates a reference height Hr and a reference width Wr, calculates a ratio (Hr / Wr) between the reference height Hr and the reference width Wr as a determination index DI, and based on the determination index DI Estimate the direction. That is, as shown in FIG. 6, when the value of the determination index DI is not less than the threshold value T1 and less than the threshold value T2, it is estimated that the orientation of the face image is the front direction. Further, when the value of the determination index DI is less than the threshold T1, it is estimated that the orientation of the face image is upward or downward, and when the value of the determination index DI is equal to or greater than the threshold T2, the face It is estimated that the orientation of the image is right-handed or left-handed. Here, in this specification, the direction of an image in which the face of the subject person faces upward is called “upward”, and the direction of the image in which the face of the subject person faces downward is called “downward”. Call. The direction of the image of the subject person facing to the right when viewed from the viewer side of the image (that is, the person is actually facing the left side) is called “right swing”. Refers to the direction of the image facing the left as viewed from the viewer side of the image (that is, the person is actually facing the right). That the image orientation is up or down is also referred to as “up and down”, and that the right or left is referred to as “left or right”.

  When the orientation of the face image is right-handed or left-handed, the reference height Hr is hardly changed, but the reference width Wr is considered to be small compared to the case of face-to-face. Therefore, when the orientation of the face image is right-handed or left-handed, the value of the determination index DI (= Hr / Wr) is larger than when the face image is faced. On the contrary, when the orientation of the face image is upside down or downside, the reference width Wr is hardly changed, but the reference height Hr is considered to be small compared to the case of facing the front. Therefore, the value of the determination index DI (= Hr / Wr) is smaller when the orientation of the face image is upside down or downside than when it is faced up.

  The threshold T1 and the threshold T2 are statistically determined from the determination index DI (ratio between the reference width Wr and the reference height Hr) of a predetermined number of face sample images. The predetermined threshold T1 and threshold T2 are stored in a predetermined area in the internal memory 120, and the face direction estimation unit 230 uses the determination index DI of the target image TI, the threshold T1, and the threshold T2 to Estimate the direction.

  As can be seen from FIG. 6, the determination index DI can be said to be an index representing the degree of swing from the front direction of the face image. For example, of two face images estimated to be right-handed or left-handed, one face image having a larger value of the determination index DI is compared with the other face image and the degree of swing from the front direction Can be estimated to be larger. In addition, the threshold value T1 and the threshold value T2 are determined from the front direction in which the face image is strictly oriented (a state in which the face and the imaging device are facing each other) to the degree of the swinging face image. Is equivalent to setting whether or not to estimate.

  FIG. 7 is an explanatory diagram illustrating an example of a result of estimation of the front direction in the face direction estimation. As shown in FIG. 7, when the value of the determination index DI is equal to or greater than the threshold value T1 and less than the threshold value T2, the orientation of the face image is estimated to be the front direction.

  FIG. 8 is an explanatory diagram illustrating an example of a result estimated to be right swing or left swing in face orientation estimation. As shown in FIG. 8, when the value of the determination index DI is equal to or greater than the threshold T2, it is estimated that the orientation of the face image is right-handed or left-handed. In the face orientation estimation, it is possible to specify not only the right swing or the left swing but also the right swing or the left swing. Such specification can be realized, for example, by comparing the width We (r) of the right eye area EA (r) and the width We (l) of the left eye area EA (l). That is, if the width We (r) of the right eye area EA (r) is larger, it is estimated that the camera is swung right, and if the width We (l) of the left eye area EA (l) is larger, it is estimated that the camera is swung left. Is done.

  FIG. 9 is an explanatory diagram illustrating an example of a result of estimation of upward swing or downward swing in face orientation estimation. As shown in FIGS. 9A and 9B, when the value of the determination index DI is less than the threshold T1, it is estimated that the orientation of the face image is upward or downward. In the face direction estimation, it is possible not only to estimate that the face is up or down, but also to specify whether it is up or down. Such specification is performed by, for example, calculating the ratio of the width Wm of the mouth area MA to the width We of any one of the eye areas EA, and if the value is larger than a predetermined threshold, the ratio is raised. It is possible to realize by swinging down. Alternatively, the ratio of the distance Hmc from the lower side of the detected face area FAd to the center point Cm of the mouth area MA with respect to the height Hf of the detected face area FAd is calculated. If it is smaller than the threshold, it can also be realized by making a downward swing.

  In step S400 (FIG. 2), when the detection of the organ area is not successful (step S500: No), the fact that the detection of the detection face area FAd is not successful is indicated by the display unit. The user is notified through 150. In this case, in order to allow the user to select another image as the target image TI, the user interface (FIG. 3) for selecting the target image TI is displayed again on the display unit 150, and the target image TI is reset ( Step S100) is performed. If the detection of the organ region is not successful, the processing from step S600 to step S700 in FIG. 2 may be skipped and the target image TI in step S800 may be printed.

  In step S620 (FIG. 2), the process availability determination unit 270 (FIG. 1) determines whether or not a deformation process (step S700 in FIG. 2) described later can be executed. Specifically, the processability determination unit 270 determines whether or not there is at least one face image that is estimated to be the front direction in the face direction estimation (step S600 in FIG. 2). If there is at least one face image estimated to be in the front direction, it is determined that the deformation process can be performed. If no face image is estimated to be in the front direction, it is determined that the deformation process cannot be performed. The

  As will be described later, the deformation process (step S700 in FIG. 2) of the present embodiment is a process for reducing the face image in the width direction and slimming the face shape. Therefore, when such a deformation process is performed on a face image that is not front-facing, the processed face image may become unnatural. Therefore, in this embodiment, when there is no face image estimated to be in front, it is determined that the deformation process cannot be executed. Even when it is determined that the deformation process can be performed, only the face image that is estimated to be front-facing among the face images included in the target image TI is subjected to the deformation process, and is not estimated to be front-facing. The face image is not subject to deformation processing.

  In step S700 (FIG. 2), deformation processing is performed. The deformation process is a process of setting a deformation area based on the detected face area FAd and deforming an image in the deformation area. Details of the deformation process will be described later in “A-3. Deformation process”. FIG. 10 is an explanatory diagram illustrating an example of the target image TI after the deformation process. In the example of FIG. 10, the face image of the person P1 is subject to the deformation process because it is estimated to be the front direction in the face direction estimation, and the face shape is corrected more slimly than before the deformation process. On the other hand, the face image of the person P2 is not subject to the deformation process because it is estimated that the face direction is not the front direction in the face direction estimation, and is not different from that before the deformation process.

  In step S800 (FIG. 2), the face shape correction printing unit 200 (FIG. 1) prints the target image TI (see FIG. 10) after the deformation process when the deformation process is performed. If not, the print processing unit 320 is instructed to print the original target image TI. The print processing unit 320 controls the printer engine 160 to print the target image TI. The print processing unit 320 performs processing such as resolution conversion and halftone processing on the image data of the target image TI to generate print data. The generated print data is supplied from the print processing unit 320 to the printer engine 160, and the printer engine 160 executes printing of the target image TI. Note that the display of the target image TI after the deformation process may be performed on the display unit 150 before the target image TI is printed. Thereby, the user can confirm the deformation process result.

  As described above, the printer 100 according to the present embodiment can detect an organ region and estimate the orientation of the face image in the target image TI based on the detected organ region. Specifically, the printer 100 detects the right eye area EA (r), the left eye area EA (l), and the mouth area MA as organ areas, and the ratio of the reference width Wr to the reference height Hr (determination index DI). Based on the above, the orientation of the face image in the target image TI is estimated. In addition, the printer 100 according to the present exemplary embodiment can estimate the degree of the swing of the face image in the target image TI from the front direction based on the determination index DI.

  Further, since the printer 100 of the present embodiment can estimate the orientation of the face image in the target image TI, the face image that is not front-facing is excluded from the target of the deformation process (step S700 in FIG. 2). Thus, it is possible to prevent the image from being unnaturally corrected by the deformation process.

  In the printer 100 of this embodiment, a reliability index is set for each organ region, and the orientation of the face image in the target image TI is estimated based only on the organ region whose reliability index value is larger than a predetermined threshold. Therefore, a decrease in estimation accuracy can be suppressed.

A-3. Transformation process:
FIG. 11 is a flowchart showing the flow of the deformation process (step S700 in FIG. 2). In step S710, the deformation area setting unit 240 sets the deformation area TA. The deformation area TA is an area on the target image TI that is a target of deformation processing for face shape correction. FIG. 12 is an explanatory diagram showing an example of a method for setting the deformation area TA. FIG. 12 shows a detected face area FAd (see FIG. 4). A reference line RL shown in FIG. 12 is a line that defines the height direction (vertical direction) of the detected face area FAd and indicates the center in the width direction (left and right direction) of the detected face area FAd. That is, the reference line RL is a straight line that passes through the center of gravity of the rectangular detection face area FAd and is parallel to the boundary line along the height direction (vertical direction) of the detection face area FAd.

  As shown in FIG. 12, in this embodiment, the deformation area TA extends (or extends in the direction (width direction) parallel to the reference line RL and the direction (width direction) perpendicular to the reference line RL. It is set as a shortened area. Specifically, when the height direction of the detected face area FAd is Hf and the width direction is Wf, the detected face area FAd is extended by k1 · Hf upward and by k2 · Hf downward. At the same time, a region extended to the left and right by k3 · Wf is set as the deformation region TA. Note that k1, k2, and k3 are predetermined coefficients.

  When the deformation area TA is set in this way, the reference line RL, which is a straight line parallel to the contour line in the height direction of the detection face area FAd, is a straight line parallel to the contour line in the height direction of the deformation area TA. Become. The reference line RL is a straight line that divides the width of the deformation area TA in half.

  As shown in FIG. 12, the deformation area TA is set as an area that generally includes an image from the jaw to the forehead in the height direction and includes images of the left and right cheeks in the width direction. That is, in the present embodiment, the above-described coefficients k1, k2, and k3 are set in advance based on the relationship with the size of the detected face area FAd so that the deformation area TA is an area that includes an image in such a range. ing.

  In step S720 (FIG. 11), the deformation area dividing unit 250 (FIG. 1) divides the deformation area TA into a plurality of small areas. FIG. 13 is an explanatory diagram illustrating an example of a method of dividing the deformation area TA into small areas. The deformation area dividing unit 250 arranges a plurality of division points D in the deformation area TA, and divides the deformation area TA into a plurality of small areas using a straight line connecting the division points D.

  The arrangement mode of the dividing points D (the number and positions of the dividing points D) is defined by the dividing point arrangement pattern table 410 (FIG. 1). The deformation area dividing unit 250 arranges the dividing points D with reference to the dividing point arrangement pattern table 410. In this embodiment, face shape correction for slimming the face shape is performed, and the division points D are arranged in a manner corresponding to such face shape correction.

  As shown in FIG. 13, the division points D are arranged at the intersections of the horizontal division line Lh and the vertical division line Lv and at the intersections of the horizontal division line Lh and the vertical division line Lv and the outer frame of the deformation area TA. . Here, the horizontal dividing line Lh and the vertical dividing line Lv are reference lines for arranging the dividing points D in the deformation area TA. As shown in FIG. 13, in the arrangement of the dividing points D in this embodiment, two horizontal dividing lines Lh perpendicular to the reference line RL and four vertical dividing lines Lv parallel to the reference line RL are set. The The two horizontal dividing lines Lh are called Lh1 and Lh2 in order from the bottom of the deformation area TA. The four vertical dividing lines Lv are referred to as Lv1, Lv2, Lv3, and Lv4 in order from the left of the deformation area TA.

  The horizontal dividing line Lh1 is arranged below the chin image in the deformation area TA, and the horizontal dividing line Lh2 is arranged near the eye image. The vertical dividing lines Lv1 and Lv4 are arranged outside the cheek line image, and the vertical dividing lines Lv2 and Lv3 are arranged outside the eye corner image. The horizontal dividing line Lh and the vertical dividing line Lv are arranged in the deformation area TA set in advance so that the positional relationship between the horizontal dividing line Lh and the vertical dividing line Lv and the image becomes the above-described positional relationship as a result. It is executed according to the correspondence with the size.

  According to the arrangement of the horizontal dividing line Lh and the vertical dividing line Lv described above, the intersection of the horizontal dividing line Lh and the vertical dividing line Lv, and the intersection of the horizontal dividing line Lh and the vertical dividing line Lv and the outer frame of the deformation area TA In addition, the dividing point D is arranged. As shown in FIG. 13, the division points D located on the horizontal division line Lhi (i = 1 or 2) are referred to as D0i, D1i, D2i, D3i, D4i, and D5i in order from the left. For example, the dividing points D located on the horizontal dividing line Lh1 are called D01, D11, D21, D31, D41, D51. Similarly, the dividing points D located on the vertical dividing line Lvj (j = 1, 2, 3, 4) are called Dj0, Dj1, Dj2, Dj3 in order from the bottom. For example, the division points D located on the vertical division line Lv1 are called D10, D11, D12, and D13.

  As shown in FIG. 13, the arrangement of the dividing points D in the present embodiment is symmetrical with respect to the reference line RL.

  The deformation area dividing unit 250 divides the deformation area TA into a plurality of small areas by a straight line connecting the arranged dividing points D (that is, the horizontal dividing line Lh and the vertical dividing line Lv). In the present embodiment, as shown in FIG. 13, the deformation area TA is divided into 15 rectangular small areas.

  In the present embodiment, since the arrangement of the dividing points D is determined by the numbers and positions of the horizontal dividing lines Lh and the vertical dividing lines Lv, the dividing point arrangement pattern table 410 has the numbers and positions of the horizontal dividing lines Lh and the vertical dividing lines Lv. In other words, it is possible to define that

  In step S730 (FIG. 11), the divided region deformation unit 260 (FIG. 1) performs an image deformation process on the deformation region TA of the target image TI. The deformation process by the divided area deforming unit 260 is performed by moving the position of the dividing point D arranged in the deformed area TA in step S720 to deform the small area.

  The movement mode (movement direction and movement distance) of each division point D for the deformation process is determined in advance by the division point movement table 420 (FIG. 1). The divided area deforming unit 260 moves the position of the divided point D with reference to the divided point movement table 420. In this embodiment, face shape correction for slimming the face shape is performed, and the position of the dividing point D is moved in a manner corresponding to such face shape correction.

  FIG. 14 is an explanatory diagram showing an example of the contents of the dividing point movement table 420. FIG. 15 is an explanatory diagram showing an example of movement of the position of the dividing point D according to the dividing point movement table 420. FIG. 14 shows a movement mode of the position of the division point D that is defined by the division point movement table 420 and associated with face shape correction for slimming the face shape. As shown in FIG. 14, in the dividing point movement table 420, for each dividing point D, there is a moving amount along a direction (H direction) perpendicular to the reference line RL and a direction parallel to the reference line RL (V direction). It is shown. In this embodiment, the unit of movement amount shown in the dividing point movement table 420 is the pixel pitch PP of the target image TI. For the H direction, the amount of movement to the right side is represented as a positive value, the amount of movement to the left side is represented as a negative value, and for the V direction, the amount of movement upward is positive. It is expressed as a value, and the downward movement amount is expressed as a negative value. For example, the dividing point D11 is moved to the right along the H direction by a distance that is seven times the pixel pitch PP, and is moved upward along the V direction by a distance that is 14 times the pixel pitch PP. For example, the division point D22 is not moved because the movement amount is zero in both the H direction and the V direction.

  In this embodiment, the division points D (for example, the division points D10 shown in FIG. 13) located on the outer frame of the deformation area TA are set so that the boundary between the inner and outer images of the deformation area TA does not become unnatural. The position is not moved. Therefore, in the dividing point movement table 420 shown in FIG. 14, the movement mode for the dividing point D located on the outer frame of the deformation area TA is not defined.

  In FIG. 15, the division point D before the movement is indicated by a white circle, and the division point D after the movement or the division point D without the movement of the position is indicated by a black circle. Further, the divided point D after the movement is referred to as a divided point D ′. For example, the position of the dividing point D11 is moved in the upper right direction in FIG. 15, and becomes the dividing point D′ 11.

  In the present embodiment, all combinations of two division points D (for example, combinations of division points D11 and D41) that are in a symmetric positional relationship with respect to the reference line RL are the same after the movement of the division point D. The movement mode is determined so as to maintain a symmetrical positional relationship with respect to the line RL.

  For each small area constituting the deformation area TA, the divided area deforming unit 260 is an image of a small area newly defined by moving the position of the dividing point D. The image is deformed so that For example, in FIG. 15, an image of a small region (small region indicated by hatching) having vertices at division points D11, D21, D22, and D12 is divided into points D′ 11, D′ 21, D22, and D′ 12. Is transformed into an image of a small area with the vertex at.

  FIG. 16 is an explanatory diagram illustrating the concept of the image deformation processing method performed by the divided region deformation unit 260. In FIG. 16, the dividing point D is indicated by a black circle. In FIG. 16, for simplification of description, regarding the four small regions, the state before the position movement of the dividing point D is shown on the left side, and the state after the position movement of the dividing point D is shown on the right side. In the example of FIG. 16, the central division point Da is moved to the position of the division point Da ′, and the positions of the other division points D are not moved. Thereby, for example, an image of a rectangular small area (hereinafter also referred to as “pre-deformation noticeable small area BSA”) having the vertices at the division points Da, Db, Dc, Dd before the movement of the division point D is obtained from the division point Da ′. , Db, Dc, and Dd are transformed into an image of a rectangular small area (hereinafter also referred to as “the noticed small area ASA after deformation”).

  In this embodiment, a rectangular small region is divided into four triangular regions using the center of gravity CG of the small region, and image deformation processing is performed in units of triangular regions. In the example of FIG. 16, the pre-deformation attention small area BSA is divided into four triangular areas having the centroid CG of the pre-deformation attention small area BSA as one vertex. Similarly, the post-deformation attention small area ASA is divided into four triangular areas having the centroid CG ′ of the post-deformation attention small area ASA as one vertex. Then, image deformation processing is performed for each corresponding triangular area in each state before and after the movement of the dividing point Da. For example, an image of a triangular area having vertices at the division points Da and Dd and the center of gravity CG in the attention small area BSA before deformation has a vertex at the division points Da ′ and Dd and the center of gravity CG ′ in the attention small area ASA after deformation. It is transformed into an image of a triangular area.

  FIG. 17 is an explanatory diagram showing the concept of an image deformation processing method in a triangular area. In the example of FIG. 17, the image of the triangular area stu with the points s, t, u as vertices is transformed into the image of the triangular area s′t′u ′ with the points s ′, t ′, u ′ as vertices. . For the deformation of the image, the position of a certain pixel in the image of the triangular area s't'u 'after the deformation corresponds to the position in the image of the triangular area stu before the deformation, and the calculated position This is performed by using the pixel value in the image before deformation in step S4 as the pixel value of the image after deformation.

  For example, in FIG. 17, the position of the pixel of interest p ′ in the image of the triangular area s′t′u ′ after deformation corresponds to the position p in the image of the triangular area stu before deformation. The position p is calculated as follows. First, coefficients m1 and m2 for expressing the position of the target pixel p ′ by the sum of the vector s′t ′ and the vector s′u ′ as shown in the following equation (1) are calculated.

  Next, by using the calculated coefficients m1 and m2, the position p is obtained by calculating the sum of the vector st and the vector su in the triangular area stu before deformation by the following equation (2).

  When the position p in the triangular area stu before deformation coincides with the pixel center position of the image before deformation, the pixel value of the pixel is set as the pixel value of the image after deformation. On the other hand, when the position p in the triangular area stu before deformation is shifted from the pixel center position of the image before deformation, an interpolation operation such as bicubic using the pixel values of the pixels around the position p. Thus, the pixel value at the position p is calculated, and the calculated pixel value is set as the pixel value of the image after deformation.

  Image deformation from the image of the triangular area stu to the image of the triangular area s't'u 'by calculating the pixel value for each pixel in the image of the triangular area s't'u' after the deformation as described above Processing can be performed. The divided area deformation unit 260 performs a deformation process by defining a triangular area as described above for each small area constituting the deformation area TA shown in FIG. 13 and performs an image deformation process in the deformation area TA.

  Here, the face shape correction mode of the present embodiment will be described in more detail. FIG. 18 is an explanatory diagram showing a form of face shape correction in the present embodiment. In this embodiment, as described above, face shape correction is performed to make the face shape slim. In FIG. 18, the image of the deformation | transformation aspect of each small area | region which comprises deformation | transformation area | region TA is shown with the arrow.

  As shown in FIG. 18, in the face shape correction of the present embodiment, the dividing points D (D11, D21, D31, D41) arranged on the horizontal dividing line Lh1 with respect to the direction (V direction) parallel to the reference line RL. Is moved upward, while the positions of the dividing points D (D12, D22, D32, D42) arranged on the horizontal dividing line Lh2 are not moved (see FIG. 14). Therefore, the image located between the horizontal dividing line Lh1 and the horizontal dividing line Lh2 is reduced in the V direction. As described above, since the horizontal dividing line Lh1 is arranged below the chin image and the horizontal dividing line Lh2 is arranged in the vicinity immediately below the eye image, in the face shape correction of this embodiment, the face image is corrected. The image of the part from the inner jaw to the lower part of the eye is reduced in the V direction. As a result, the jaw line in the image moves upward.

  On the other hand, in the direction (H direction) perpendicular to the reference line RL, the position of the dividing point D (D11, D12) arranged on the vertical dividing line Lv1 is moved rightward and arranged on the vertical dividing line Lv4. The position of the dividing point D (D41, D42) is moved to the left (see FIG. 14). Further, of the two division points D arranged on the vertical division line Lv2, the position of the division point D (D21) arranged on the horizontal division line Lh1 is moved rightward and arranged on the vertical division line Lv3. Of the two divided points D, the position of the divided point D (D31) arranged on the horizontal dividing line Lh1 is moved to the left (see FIG. 14). Therefore, the image located on the left side of the vertical dividing line Lv1 is enlarged on the right side in the H direction, and the image located on the right side of the vertical dividing line Lv4 is enlarged on the left side. An image located between the vertical dividing line Lv1 and the vertical dividing line Lv2 is reduced or moved to the right in the H direction, and an image located between the vertical dividing line Lv3 and the vertical dividing line Lv4 is moved in the H direction. Is reduced or moved to the left. Further, the image located between the vertical division line Lv2 and the vertical division line Lv3 is reduced in the H direction with the position of the horizontal division line Lh1 as the center.

  As described above, the vertical division lines Lv1 and Lv4 are arranged outside the cheek line image, and the vertical division lines Lv2 and Lv3 are arranged outside the corner image. Therefore, in the face shape correction according to the present embodiment, the image of the portion outside the both corners of the face image is reduced in the H direction as a whole. In particular, the reduction rate is high near the jaw. As a result, the shape of the face in the image becomes thinner in the width direction as a whole.

  If the deformation | transformation aspect of the H direction mentioned above and the V direction are put together, the shape of the face in the target image TI will become slim by the face shape correction | amendment of a present Example. Note that the slim shape of the face can also be expressed as a so-called “small face”.

  Note that the small areas (hatched areas) having the divide points D22, D32, D33, and D23 shown in FIG. 18 as vertices are determined according to the arrangement method of the horizontal division lines Lh2 and the vertical division lines Lv2 and Lv3. This area includes the image. As shown in FIG. 14, since the dividing points D22 and D32 are not moved in the H direction or the V direction, the small region including the images of both eyes is not deformed. As described above, in this embodiment, the small area including the images of both eyes is not deformed, and the image after the face shape correction is made more natural and preferable.

B. Second embodiment:
FIG. 19 is an explanatory diagram schematically showing the configuration of a printer 100a as an image processing apparatus according to the second embodiment of the present invention. The printer 100a of the second embodiment is different from the printer 100 of the first embodiment shown in FIG. 1 in the programs and data stored in the internal memory 120. That is, the internal memory 120 of the printer 100a according to the second embodiment stores the face correction printing unit 202 instead of the face shape correction printing unit 200 (FIG. 1), and also includes the division point arrangement pattern table 410 and the division point arrangement pattern table 410. The point movement table 420 (see FIG. 1) is not stored. Other configurations of the printer 100a of the second embodiment are the same as those of the printer 100 of the first embodiment.

  The face correction printing unit 202 is a computer program for executing a face correction printing process described later under a predetermined operating system. The face correction printing unit 202 includes a face region detection unit 210, an organ region detection unit 220, a face orientation estimation unit 230, a face region setting unit 280, and a correction processing unit 290 as program modules. The functions of the face area detection unit 210, the organ area detection unit 220, and the face direction estimation unit 230 are the same as those in the first embodiment. The face area setting unit 280 and the correction processing unit 290 will be described in detail in the description of the face correction printing process described later.

  The printer 100a according to the second embodiment can execute face correction printing processing for performing face correction of the target image and printing of the target image after face correction. The face correction of this embodiment is a skin color correction that changes the color of the image of the skin portion of the face to a preferable color. When the memory card MC is inserted into the card slot 172 and a predetermined operation is performed by the user via the operation unit 140, the face correction printing process is started.

  FIG. 20 is a flowchart showing the flow of the face correction printing process. The processing contents from step S100 to step S600 are the same as the processing contents from step S100 to step S600 of the face shape correction printing process (FIG. 2) in the first embodiment. In the face orientation estimation (step S600 in FIG. 20) in the second embodiment, it is estimated that the orientation of the face image is any one of the front orientation, the upward swing, the downward swing, the right swing, and the left swing. Is done.

  In step S660 (FIG. 20), the face area setting unit 280 (FIG. 19) sets the face area FA including the face image in the target image TI. In this specification, the face area FA set in step S660 is also referred to as “determined face area FAs”. The face area setting unit 280 sets the determined face area FAs based on the detected organ area by a method corresponding to the orientation of the face image estimated in step S600 (FIG. 20). In the detection of the detected face area FAd in step S200 (FIG. 20), the unit shift amount (shift amount) is set in the pattern matching when the candidate areas of the detected face area FAd are sequentially set while shifting the inclination, size, and position. ) Is limited to a relatively large value, the processing speed is increased, and there may be a deviation between the detected face area FAd and the actual face image. Further, when the face image is not front-facing, there may be a difference between the detected face area FAd and the actual face image due to the fact that the face image is not front-facing. In order to correct such a shift, in the present embodiment, the determined face area FAs is set.

  FIG. 21 is an explanatory diagram illustrating a method of setting the determined face area FAs when the estimated face image orientation is the front orientation. As shown in FIG. 21, the size (width and height) of the rectangular determined face area FAs is determined based on the reference width Wr and the reference height Hr. That is, the width of the determined face area FAs is a value obtained by multiplying the reference width Wr by a predetermined coefficient c1, and the height of the determined face area FAs is a value obtained by multiplying the reference height Hr by a predetermined coefficient c2. . The position of the determined face area FAs is determined based on the midpoint BP of the line segment CL connecting the center point Ce (r) of the right eye area EA (r) and the center point Ce (l) of the left eye area EA (l). Is done. That is, the position of the determined face area FAs is divided into a straight line that divides the determined face area FAs along the width direction at a predetermined ratio R1 to R2 and a determined face area FAs along the height direction at a predetermined ratio R3 to R4. The intersection point RP with the straight line to be overlapped with the midpoint BP of the line segment CL. In the present embodiment, the predetermined ratio R1 to R2 is 1: 1. The inclination of the determined face area FAs is the same as that of the detected face area FAd.

  An area having the size, position, and inclination determined as described above is set as the determined face area FAs. When the estimated orientation of the face image is front-facing, the determined face area FAs is an area in which a deviation from the actual face image is suppressed as compared to the detected face area FAd.

  FIG. 22 is an explanatory diagram illustrating a method of setting the determined face area FAs when the estimated orientation of the face image is right-handed. As shown in FIG. 22, the size (width and height) of the rectangular determined face area FAs is determined based on the reference width Wr and the reference height Hr. That is, the height of the determined face area FAs is set to a value obtained by multiplying the reference height Hr by a predetermined coefficient c2 as in the case of the front orientation shown in FIG. The width of the determined face area FAs is a value obtained by multiplying the reference width Wr by a predetermined coefficient c1 and further by a coefficient α. Here, the coefficient α is a ratio of the width We (r) of the right eye area EA (r) to the width We (l) of the left eye area EA (l). In general, when the orientation of the face image is to the right, the width We (r) of the right eye area EA (r) is larger than the width We (l) of the left eye area EA (l). Large value. Accordingly, the coefficient (α · c1) multiplied by the reference width Wr for calculating the width of the determined face area FAs in the case of turning right is a coefficient (α · c1) multiplied by the reference width Wr in the case of facing front (see FIG. 21). Greater than c1). Accordingly, it is possible to set an area having an appropriate width as the determined face area FAs in the right-faced face image in which the reference width Wr tends to be smaller than the face image facing the front.

  Further, the position of the determined face area FAs is determined based on the midpoint BP of the line segment CL that connects the center points Ce (r) and Ce (l). That is, the position of the determined face area FAs is divided into a straight line that divides the determined face area FAs along the width direction at a predetermined ratio R5 to R6 and a determined face area FAs along the height direction at a predetermined ratio R3 to R4. The intersection point RP with the straight line to be overlapped with the midpoint BP of the line segment CL. Here, the predetermined ratio R5 to R6 is the same as the width We (r) of the right eye area EA (r) and the width We (l) of the left eye area EA (l). In general, when the orientation of the face image is right-handed, the width We (r) of the right eye area EA (r) is larger than the width We (l) of the left eye area EA (l). Larger than the value of. Therefore, in the case of turning right, the center position along the width direction of the determined face area FAs is shifted to the left side with respect to the position of the midpoint BP. Thus, the determined face area FAs is set at an appropriate position in the right-faced face image in which the area representing the right side portion of the person's face tends to be larger than the face-oriented face image. Is possible.

  The inclination of the determined face area FAs is the same as that of the detected face area FAd. An area having the size, position, and inclination determined as described above is set as the determined face area FAs. When the estimated orientation of the face image is right-handed, the determined face area FAs is less misaligned with the actual face image than the detected face area FAd. This is a region in which the shift due to the image not being front-facing is also suppressed. Note that the method of setting the determined face area FAs when the estimated face image orientation is left-handed is the same as the setting method when the face image orientation is right-handed although the left and right are reversed. is there.

  FIG. 23 is an explanatory diagram showing a method for setting the determined face area FAs when the estimated orientation of the face image is upward. As shown in FIG. 23, the size (width and height) of the rectangular determined face area FAs is determined based on the reference width Wr and the reference height Hr. That is, the width of the determined face area FAs is set to a value obtained by multiplying the reference width Wr by the predetermined coefficient c1 as in the case of the front direction shown in FIG. The height of the determined face area FAs is a value obtained by multiplying the reference height Hr by a predetermined coefficient c2 and further by a coefficient β. Here, the coefficient β is a ratio of the standard determination index DIs to the determination index DI (ratio of the reference height Hr to the reference width Wr). The standard determination index DIs is a value set in advance as the value of the determination index DI of a standard front-facing face image. Generally, when the orientation of the face image is upward, the value of the determination index DI is smaller than in the case of facing the front, so the coefficient β is a value greater than 1. Therefore, the coefficient (β · c2) to be multiplied by the reference height Hr for calculating the height of the determined face area FAs in the case of the upward swing is multiplied by the reference height Hr in the case of facing the front (see FIG. 21). Greater than the coefficient (c2) obtained. Accordingly, it is possible to set an area having an appropriate height as the determined face area FAs in the upward face image in which the reference height Hr tends to be smaller than the face image facing the front. .

  Further, the position of the determined face area FAs is determined based on the midpoint BP of the line segment CL that connects the center points Ce (r) and Ce (l). That is, the position of the determined face area FAs is divided into a straight line that divides the determined face area FAs along the width direction at a predetermined ratio R1 to R2 and a determined face area FAs along the height direction at a predetermined ratio R7 to R8. The intersection point RP with the straight line to be overlapped with the midpoint BP of the line segment CL. In the present embodiment, the predetermined ratio R1 to R2 is 1: 1.

  The inclination of the determined face area FAs is the same as that of the detected face area FAd. An area having the size, position, and inclination determined as described above is set as the determined face area FAs. Even when the estimated orientation of the face image is upward, the determined face area FAs is suppressed from being shifted from the actual face image as compared to the detected face area FAd. This is a region in which the shift due to the image not being front-facing is also suppressed.

  FIG. 24 is an explanatory diagram illustrating a method of setting the determined face area FAs when the estimated face image orientation is downward. As shown in FIG. 24, the size (width and height) of the rectangular determined face area FAs is determined based on the reference width Wr and the reference height Hr. That is, the width of the determined face area FAs is set to a value obtained by multiplying the reference width Wr by the predetermined coefficient c1 as in the case of the front direction shown in FIG. Further, the height of the determined face area FAs is set to a value obtained by multiplying the reference height Hr by a predetermined coefficient c2 and further multiplying by the coefficient β, as in the case of the upward swing shown in FIG. When the face image is in the downward direction, the coefficient β is larger than 1 because the value of the determination index DI is smaller than in the case of the upward direction, as in the case of the upward direction. Therefore, the coefficient (β · c2) to be multiplied by the reference height Hr for calculating the height of the determined face area FAs is the coefficient (c2) to be multiplied by the reference height Hr when facing the front (see FIG. 21). Larger than Accordingly, it is possible to set an area having an appropriate height as the determined face area FAs in the downward face image in which the reference height Hr tends to be smaller than the face image facing the front. .

  Further, the position of the determined face area FAs is determined based on the midpoint BP of the line segment CL that connects the center points Ce (r) and Ce (l). That is, the position of the determined face area FAs is divided into a straight line that divides the determined face area FAs in the width direction at a predetermined ratio R1 to R2, and the determined face area FAs is divided in a predetermined ratio R9 to R10 along the height direction. The intersection point RP with the straight line to be overlapped with the midpoint BP of the line segment CL. In the present embodiment, the predetermined ratio R1 to R2 is 1: 1.

  The inclination of the determined face area FAs is the same as that of the detected face area FAd. An area having the size, position, and inclination determined as described above is set as the determined face area FAs. Even when the estimated orientation of the face image is downward, the determined face area FAs is less misaligned with the actual face image as compared to the detected face area FAd. This is a region in which the shift due to the image not being front-facing is also suppressed.

  In step S680 (FIG. 20), the correction processing unit 290 (FIG. 1) sets a correction area based on the determined face area FAs set in step S660, and performs face correction on the correction area. As described above, the face correction of this embodiment is a skin color correction that changes the color of the image of the skin part of the face to a preferable color.

  The correction area to be set may be the determined face area FAs itself, or may be an area obtained by enlarging or reducing the determined face area FAs at a predetermined magnification. Since the determined face area FAs is an area in which the deviation from the actual face image is suppressed compared to the detected face area FAd, the correction area set based on the determined face area FAs Deviation from the actual face image can be suppressed. The face correction is executed by, for example, extracting a skin color pixel from the correction area and adjusting the color of the extracted pixel.

  In step S800 (FIG. 20), as in the first embodiment, the face correction printing unit 202 instructs the print processing unit 320 to print the corrected target image TI, and the print processing unit 320 causes the printer engine 160 to operate. The target image TI is printed under control.

  As described above, the printer 100a according to the second embodiment detects the organ area and uses the method according to the estimated orientation of the face image based on the detected organ area to detect the face area in the target image TI. A decision face area FAs including an image can be set. Therefore, the printer 100a according to the second embodiment can set the determined face area FAs as an area in which a deviation from an actual face image is suppressed. Further, the printer 100a according to the second embodiment sets a correction area based on the determined face area FAs, and performs face correction on the set correction area, so that the face correction process is executed on a more appropriate area. Therefore, the corrected image can be a preferable image.

C. Variations:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

C1. Modification 1:
The method of face orientation estimation (step S600 in FIGS. 2 and 20) in each of the above embodiments is merely an example, and face orientation estimation may be performed by other methods. For example, when the width We of the right eye area EA (r) and the left eye area EA (l) is simply compared, and the width We of the right eye area EA (r) is greater than a predetermined ratio, the right-hand area is The left-eye area EA (l) may be estimated to be left-handed when the width We is greater than a predetermined ratio, and to be front-facing in other cases.

  Alternatively, the width Wm of the mouth area MA and the width We of the eye area EA are simply compared. If the width Wm of the mouth area MA is larger than a predetermined ratio, If the width We is larger than a predetermined ratio, it may be estimated that the swing is downward, and that the width We is facing the front in other cases.

  Further, the face orientation may be estimated based on the relationship between the detected face area FAd and the organ area. For example, based on the position of the eye area EA in the detected face area FAd, it may be estimated that the camera is swung to the right or left, or based on the position of the mouth area MA in the detected face area FAd, It may be estimated that there is.

  Further, in the face orientation estimation in each of the above embodiments, the types of orientation of the estimated face image are three types of front orientation, left / right swing, and up / down swing, or front orientation, right swing, left swing, up swing, Five types of downward swing are employed, but two types of orientation of the face image may be employed, that is, the front direction or the direction other than the front direction. Also, the types of face image orientation are more subdivided, such as front, weak right, medium right, strong right, weak left, medium left, strong left, and so on. The type of orientation of the converted face image may be employed.

C2. Modification 2:
In each of the above embodiments, the right eye, the left eye, and the mouth are set as the facial organs, and the right eye area EA (r), the left eye area EA (l), and the mouth area MA are detected as the organ areas. However, it is possible to change which organ of the face is set as the facial organ. For example, only the right eye and the left eye may be set as facial organs, and the right eye area EA (r) and the left eye area EA (l) may be detected as organ areas. Alternatively, only the mouth may be set as the facial organ, and the mouth area MA may be detected as the organ area. In addition to the right eye, the left eye, and the mouth, or in place of at least one of the right eye, the left eye, and the mouth, other organs of the face (for example, the nose and eyebrows) are set as the organs of the face. A region including an image of such an organ may be detected.

C3. Modification 3:
In the first embodiment, the estimated orientation of the face image is used for determining whether or not the deformation process can be performed (step S620 in FIG. 2). It is also possible to determine whether or not to execute other image processing on the face image of (2).

  In addition, the condition used for determining whether or not the deformation process can be performed in the first embodiment is merely an example, and other conditions may be used. For example, in addition to the case where the estimated orientation of the face image is the front direction, it may be determined that the deformation process can be executed even when the face is swung up and down. Since the deformation process according to the present embodiment is considered to have an unnatural result, particularly when a left-right image is targeted, the determination may be performed in this way.

  Further, the present invention can be applied to the determination of whether or not to execute predetermined image processing different from the deformation processing, and the conditions used for the determination of whether or not to execute image processing are appropriately set according to the characteristics of the image processing. In other words, depending on the characteristics of the image processing, a determination condition may be adopted in which it is determined that the process cannot be executed when the estimated face image orientation is the front direction. Alternatively, a determination condition is adopted in which it is determined that the process can be executed only when the estimated orientation of the face image is left and right, and in other cases, the process is determined not to be executable. There is also.

C4. Modification 4:
The method for specifying the inclination, size, and position of the determined face area FAs in the second embodiment is merely an example, and the inclination, size, and position of the determined face area FAs may be specified using other methods. . For example, in the second embodiment, the inclination of the determined face area FAs is specified to be the same as the inclination of the detected face area FAd regardless of the orientation of the face image. The inclination may be specified as being the same as the inclination of any organ region.

  In the second embodiment, when the determined face area FAs is set, an area having the size and position specified based on the organ area is set as the determined face area FAs. However, the determined face area FAs is determined based on the organ area. Only one of the size and position of the face area FAs may be specified. In this case, as the size and position of the determined face area FAs that has not been specified, the size and position of the detected face area FAd may be employed, or a predetermined value may be employed.

  In the second embodiment, the determined face area FAs is set by a method according to the estimated orientation of the face image. For example, when it is estimated that the front face is detected, the detected face area is detected. The FAd itself may be set as the determined face area FAs.

C5. Modification 5:
The orientation of the face image estimated in each of the embodiments can be used for other purposes. For example, the estimated orientation of the face image may be associated with the image data as the attached information of the image file including the image data. In this way, for example, the estimated orientation of the face image can be used for image search.

C6. Modification 6:
In each of the embodiments described above, organ detection is performed on the detected face area FAd. However, the range in which the organ detection is performed is not necessarily limited to the detected face area FAd. For example, the organ is detected on the entire target image TI. Detection may be performed. However, if organ detection is performed on the detected face area FAd, the processing speed can be increased.

  In each of the above embodiments, the face area FA (detected face area FAd) is detected, but the face area FA need not necessarily be detected. Even when the face area FA is not detected, the organ area is detected for a predetermined range (for example, the whole) of the target image TI, and the orientation of the face image is determined based on the detected organ area. It is possible to perform estimation and setting of the determined face area FAs.

C7. Modification 7:
In each of the above embodiments, the number of matches is used as a reliability index set for each detected organ region, but another index (for example, a match rate with a template) may be used as the reliability index. . Further, it is not always necessary to set the reliability index when detecting the organ region. When the reliability index is not set for the organ area, the reliability index is not used when determining whether or not the detection of the organ area is successful. However, when the reliability index is set for the organ area and the determined face area FAs is set based only on the organ area whose reliability index value is larger than a predetermined threshold, the estimation accuracy of the orientation of the face image And a decrease in the setting accuracy of the determined face area FAs can be suppressed.

C8. Modification 8:
In each of the embodiments described above, as a condition for determining that the detection of the organ area has succeeded, all three organ areas of the right eye area EA (r), the left eye area EA (l), and the mouth area MA are detected. Although it is imposed, such a condition does not necessarily have to be imposed. It may be determined that the detection of the organ area is successful if at least an organ area necessary for estimating the orientation of the face image and setting the determined face area FAs is detected.

  Further, in each of the above embodiments, as a condition for determining that the detection of the organ region is successful, it is required that the probability represented by the reliability index is larger than a predetermined threshold for all the organ regions. There is no need to impose such conditions. Even if there is an organ region whose reliability index is smaller than the threshold value, it is determined that the detection of the organ region is successful, and the determined face region FAs is set using only the organ region whose reliability index is larger than the threshold value. Good.

C9. Modification 9:
In each of the above embodiments, the face area FA and the organ area are rectangular areas, but the face area FA and the organ area may be areas having a shape other than a rectangle.

C10. Modification 10:
In each of the embodiments described above, the face shape correction printing process (FIG. 2) and the face correction printing process (FIG. 20) by the printer 100 as the image processing apparatus have been described, but a part of the processing (for example, steps S100 to S700 in FIG. 2). May be executed by a personal computer. The printer 100 is not limited to an ink jet printer, and may be another type of printer, such as a laser printer or a sublimation printer. The present invention is generally applicable to image processing for estimating the orientation of a face image. For example, the present invention can also be applied to processing in which only the orientation of a face image is estimated by a personal computer.

C11. Modification 11:
In each of the above embodiments, a part of the configuration realized by hardware may be replaced with software, and conversely, a part of the configuration realized by software may be replaced by hardware. .

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, and the like. An external storage device fixed to the computer is also included.

1 is an explanatory diagram schematically showing the configuration of a printer 100 as an image processing apparatus in a first embodiment of the present invention. FIG. It is a flowchart which shows the flow of a face shape correction | amendment printing process. It is explanatory drawing which shows an example of the user interface for the setting of the target image TI. It is explanatory drawing which shows an example of the detection result of detection face area | region FAd. It is explanatory drawing which shows an example of the detection result of an organ area | region. It is explanatory drawing which shows the method of face direction estimation. It is explanatory drawing which shows an example of the result estimated as the front direction in face direction estimation. It is explanatory drawing which shows an example of the result estimated to be right swing or left swing in face direction estimation. It is explanatory drawing which shows an example of the result estimated to be a swing or a downward swing in face direction estimation. It is explanatory drawing which shows an example of the target image TI after a deformation | transformation process. It is a flowchart which shows the flow of a deformation | transformation process. It is explanatory drawing which shows an example of the setting method of deformation | transformation area | region TA. It is explanatory drawing which shows an example of the division | segmentation method to the small area | region of deformation | transformation area | region TA. It is explanatory drawing which shows an example of the content of the dividing point movement table. It is explanatory drawing which shows an example of the movement of the position of the dividing point D according to the dividing point movement table. 6 is an explanatory diagram showing a concept of an image deformation processing method performed by a divided region deformation unit 260. FIG. It is explanatory drawing which shows the concept of the deformation | transformation processing method of the image in a triangular area | region. It is explanatory drawing which shows the aspect of the face shape correction | amendment in a present Example. It is explanatory drawing which shows roughly the structure of the printer 100a as an image processing apparatus in 2nd Example of this invention. It is a flowchart which shows the flow of a face correction printing process. It is explanatory drawing which shows the setting method of the determination face area | region FAs in case the direction of the estimated face image is a front direction. It is explanatory drawing which shows the setting method of the determination face area | region FAs when the direction of the estimated face image is right-handed. It is explanatory drawing which shows the setting method of the determination face area | region FAs when the direction of the estimated face image is upward. It is explanatory drawing which shows the setting method of the determination face area | region FAs when the direction of the estimated face image is downward.

Explanation of symbols

100 ... Printer 110 ... CPU
DESCRIPTION OF SYMBOLS 120 ... Internal memory 140 ... Operation part 150 ... Display part 160 ... Printer engine 170 ... Card interface 172 ... Card slot 200 ... Face shape correction printing part 202 ... Face correction printing part 210 ... Face area detection part 220 ... Organ area detection part 222 ... reliability setting section 230 ... face orientation estimation section 240 ... deformation area setting section 250 ... deformation area division section 260 ... divided area deformation section 270 ... processability determination section 280 ... face area setting section 290 ... correction processing section 310 ... display processing Section 320 ... Print processing section 410 ... Division point arrangement pattern table 420 ... Division point movement table

Claims (19)

An image processing apparatus,
An organ region detection unit that detects a region including an image of a facial organ in the target image as an organ region;
An image processing apparatus comprising: a face orientation estimating unit that estimates the orientation of a face image in the target image based on the detected organ region. The image processing apparatus according to claim 1,
The organ region detector detects a plurality of the organ regions,
The face orientation estimating unit estimates an orientation of a face image in the target image based on a relationship related to at least one of a position and a size between the detected organ regions. The image processing apparatus according to claim 1 or 2,
The face orientation estimation unit calculates a reference width as an index correlated with the width of the face image and a reference height as an index correlated with the height of the face image based on the detected organ region. An image processing device that estimates a face image orientation in the target image based on a ratio between the reference width and the reference height. The image processing apparatus according to claim 3,
The face direction estimation unit is an image processing device that estimates a degree of a swing of a face image from a front direction in the target image based on a ratio between the reference width and the reference height. The image processing apparatus according to claim 3 or 4, wherein:
The organ region detection unit includes at least a right eye region including an image of the right eye as the facial organ, a left eye region including an image of the left eye as the facial organ, and an image of the mouth as the facial organ. An image processing apparatus for detecting a mouth region as the organ region. The image processing apparatus according to claim 5,
The face direction estimating unit calculates the reference width based on the right eye region and the left eye region, and calculates the reference height based on at least one of the right eye region and the left eye region and the mouth region; Image processing device. The image processing apparatus according to any one of claims 1 to 6, further comprising:
A face area detection unit that detects an area including a face image in the target image as a detection face area;
The face orientation estimation unit is an image processing device that estimates the orientation of a face image in the target image based on the detected organ region and the detected face region. The image processing apparatus according to claim 1, further comprising:
An image processing unit that performs predetermined image processing on a predetermined region including a face image in the target image;
An image processing apparatus comprising: a process availability determination unit that determines whether the image processing can be performed based on the estimated orientation of the face image in the target image. The image processing apparatus according to claim 8,
The predetermined image processing includes image deformation processing,
The processing availability determination unit determines that the execution of the image processing is impossible when the orientation of the face image in the estimated target image is not front-facing. The image processing apparatus according to claim 1, further comprising:
A face area setting unit configured to set a face area including a face image in the target image by a predetermined method according to the estimated orientation of the face image in the target image based on the detected organ area; Image processing device. The image processing apparatus according to claim 10,
The face area setting unit is configured to determine at least one of the size, position, and inclination of the face area based on a relationship with the organ area determined in advance according to the orientation of the face image in the estimated target image. An image processing apparatus that identifies one area and sets an area having at least one of the identified size, position, and inclination as the face area. The image processing apparatus according to claim 11, further comprising:
A face area detection unit that detects an area including a face image in the target image as a detection face area;
The face area setting unit sets, as the face area, an area in which at least one of the size, position, and inclination of the detected face area is matched with at least one of the specified size, position, and inclination. , Image processing device. The image processing apparatus according to claim 3, further comprising:
A face area setting unit configured to set a face area including a face image in the target image in a predetermined method according to the estimated orientation of the face image in the target image based on the detected organ area;
The face area setting unit is an image processing apparatus that specifies a size of the face area based on the reference width and the reference height. The image processing apparatus according to claim 13,
The face area setting unit specifies a width of the face area by multiplying the reference width by a predetermined coefficient,
The value of the predetermined coefficient used when the estimated orientation of the face image in the target image is right-handed or left-handed is compared with the value of the predetermined coefficient used when the face-oriented. Large, image processing device. The image processing apparatus according to claim 13,
The face area setting unit specifies the height of the face area by multiplying the reference height by a predetermined coefficient,
The value of the predetermined coefficient used when the estimated orientation of the face image in the target image is up or down is compared with the value of the predetermined coefficient used when the face is facing forward. Large, image processing device. The image processing apparatus according to any one of claims 10 to 15, further comprising:
An image processing apparatus comprising: an image processing unit that sets a processing target area based on the set face area and performs predetermined image processing on the processing target area. The image processing apparatus according to any one of claims 1 to 16,
The organ area detection unit includes a reliability setting unit that sets a reliability index that represents the probability that the detected organ area is an area including an image of a facial organ,
The face orientation estimating unit estimates an orientation of a face image in the target image based only on the organ area in which the value of the reliability index is greater than a predetermined threshold among the detected organ areas. apparatus. An image processing method comprising:
(A) detecting a region including an image of a facial organ in the target image as an organ region;
And (b) estimating a face image orientation in the target image based on the detected organ region. A computer program for image processing,
An organ region detection function for detecting a region including an image of a facial organ in the target image as an organ region;
A computer program that causes a computer to realize a face orientation estimation function that estimates the orientation of a face image in the target image based on the detected organ region.

Images