JP5338887B2 - Image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem that much resources are consumed during image deformation processing. <P>SOLUTION: An image processing apparatus capable of deforming images comprises: a candidate region setting part for setting a region including a specific image on a target image as a candidate region to be a target of deformation processing; an elimination determination part for eliminating one or more candidate regions from the target of the deformation processing so that an overlap is resolved when overlap occurs between the candidate regions; and a deformation processing part for deforming images inside the region with the candidate regions other than the candidate regions eliminated as the target. <P>COPYRIGHT: (C)2012,JPO&amp;INPIT

Description

  The present invention relates to an image processing technique capable of executing image deformation.

  An image processing technique for deforming an image for a digital image is known (see Patent Document 1). In Patent Document 1, a partial area (an area representing a cheek image) on a face image is set as a correction area, the correction area is divided into a plurality of small areas according to a predetermined pattern, and set for each small area. Image processing for deforming the shape of a face by enlarging or reducing the image at a specified magnification is disclosed.

JP 2004-318204 A

In the conventional image processing for image deformation, since the image is enlarged or reduced for each of the plurality of small regions at a magnification set for each small region, the processing is complicated. Further, the conventional image processing for image deformation is specialized in correcting cheek lines, and it is difficult to deal with various other deformation modes.
In addition, since the conventional technique simply detects a human face and corrects and deforms the detected face (a part of the face), if there are multiple faces in the image data, All faces are subject to deformation. However, always making all faces subject to deformation processing leads to an increase in the amount of calculation processing and an increase in the amount of memory used for a buffer or the like in the calculation. In particular, the increase in memory consumption and arithmetic processing associated with such deformation processing is a heavy burden on image processing devices called multifunction devices having many functions such as a print function, a copy function, and a scanner function. It hindered the operation of the equipment.

  The present invention has been made in order to solve the above-described conventional problems, and provides a technique capable of easily and efficiently realizing image processing for image deformation corresponding to various deformation modes. The purpose is to do. At the same time, it is an object of the present invention to provide a technique capable of realizing the image deformation with a small processing load and a small memory consumption and reducing the operation load of the apparatus.

In order to achieve the above object, the image processing apparatus of the present invention can perform image deformation, and includes at least a candidate area setting unit, an exclusion determination unit, and a deformation processing unit. The candidate area setting unit sets an area including the specific image on the target image as a candidate area for the deformation process, and the exclusion determination unit performs the deformation process on the set candidate area according to a predetermined exclusion condition. Determine the area to be excluded from the target. The deformation processing unit performs deformation of the image in the region for candidate regions other than the candidate region determined as the region to be excluded by the exclusion determining unit.
That is, according to the present invention, the deformation process is performed only on candidate areas other than the areas excluded based on the predetermined exclusion condition among the candidate areas of the deformation process existing on the target image. The increase in the memory consumption and the calculation processing amount accompanying the above can be appropriately suppressed.

Various conditions for exclusion adopted by the exclusion determination unit can be considered as follows.
As an example, when there is an overlap between candidate areas, the exclusion determination unit excludes one or more candidate areas from the target of deformation processing so that the overlap is eliminated. In order to perform deformation processing on both overlapping candidate areas, it is necessary to perform deformation processing on one area first and retain the image data after deformation on the one area in order to deform the other area later. And memory consumption is enormous. According to the above configuration, since the predetermined candidate area is excluded from the target of the deformation process so that the overlap between the candidate areas is eliminated, the memory consumption can be greatly suppressed.

  There are various specific methods for eliminating the overlap between candidate areas. The exclusion determination unit selects one candidate region from a plurality of candidate regions according to a predetermined selection order, and if there is another candidate region that overlaps with the selected candidate region, the other candidate region is selected. It is also possible to eliminate the overlap between candidate areas by repeatedly performing the process of excluding from the deformation process target. More specifically, when selecting one candidate area, the exclusion determination unit may select the remaining candidate areas in the descending order of the size of the specific image. In other words, since the specific image having a large size is estimated to be an image portion important for the user, the specific image having a large size is preferentially selected, and other candidate regions overlapping therewith are excluded. As a result, candidate areas that do not overlap each other and each include a relatively large specific image remain as deformation targets on the target image.

An image processing apparatus includes a detection unit that detects a plurality of face regions from a target image, and a candidate region setting unit that sets a region including one face region as a candidate region for deformation processing for each detected face region And when there is an overlap between the candidate areas, the candidate area is selected in accordance with a selection instruction input from the outside, and there is another candidate area that overlaps the selected candidate area An exclusion determination unit that excludes other overlapping candidate regions from the target of the deformation process, and a deformation processing unit that deforms an image in the region with respect to candidate regions other than the excluded candidate regions may be provided. . That is, a candidate area is selected in accordance with a selection instruction from the user, and other candidate areas that overlap the candidate area are excluded from the transformation processing targets. As a result, the candidate area selected by the user can be preferentially subjected to deformation processing, which is preferable.

  As another example, the exclusion determination unit acquires the size of the specific image and the inclination on the target image, and whether or not the acquired size falls within a size upper limit value determined according to the acquired inclination. It may be determined and the candidate area having the specific image whose acquired size exceeds the size upper limit value may be excluded from the deformation processing target. The amount of memory required for deformation of the candidate area including the specific image differs depending not only on the size of the specific image but also on the inclination of the specific image. According to the above configuration, the candidate area including the specific image exceeding the allowable size according to the inclination is not subjected to the deformation process. Therefore, it is possible to appropriately increase the memory consumption and the calculation process amount associated with the deformation process. Can be suppressed.

  Specifically, the exclusion determining unit refers to the size upper limit value defining information that defines the size upper limit value as the inclination approaches a predetermined upper limit angle, so that the acquired size falls within the size upper limit value. Judge whether it fits. Depending on the content of the deformation process performed by the deformation processing unit, the amount of memory required for deformation of the candidate area including the specific image may increase as the inclination of the specific image approaches a predetermined upper limit angle. According to the above configuration, a specific image with a large inclination that tends to consume a large amount of memory during the deformation process is easily excluded from the target of the deformation process by tightening the size upper limit value. Increase in the memory consumption and the amount of arithmetic processing associated with.

  As another example, when there is a candidate area that is at least partly extracted from the trimmed image cut out from the target image by predetermined trimming, the exclusion determination unit selects the candidate area that is at least partly excluded from the target of the deformation process. It may be excluded. When there is a candidate area that partially or entirely protrudes from the trimmed image, it is necessary to set again the candidate area that includes the specific image in order to perform the deformation process on the specific image included in the candidate area. Such re-setting processing is a heavy burden on the image processing apparatus. Therefore, as described above, candidate regions that appear at least partially from the trimmed image are excluded from the target of the deformation process, thereby suppressing an increase in memory consumption and calculation processing amount associated with the deformation process.

  Specifically, the exclusion determination unit acquires information on a plurality of aspect ratios used for image output and at least partly from an image range common to each trimmed image cut out by each trimming based on each acquired aspect ratio. Candidate areas where “” appears may be excluded from the target of deformation processing. Image output includes printing and image display. According to the above configuration, when the target image can be output with various aspect ratios, only the candidate region whose entire output result is always included can be the target of the deformation process. For this reason, there is no possibility that a candidate area in which deformation processing is performed or not performed due to a difference in aspect ratio at the time of output.

  Alternatively, the exclusion determination unit may exclude a candidate area from which at least a part of a trimmed image cut out based on a trimming instruction input from the outside is excluded from the target of deformation processing. That is, candidate areas that partially or entirely protrude from a trimmed image cut out by trimming arbitrarily designated by the user are also excluded from the target of deformation processing. As a result, it is possible to avoid candidate area resetting processing as described above, and to suppress memory consumption and processing load.

  As another example, the exclusion determination unit determines whether or not the inclination of the specific image on the target image belongs to a predetermined angle range, and performs a deformation process on a candidate region whose inclination of the specific image is outside the predetermined angle range. It may be excluded from the target. That is, a certain degree of angle range assumed as the inclination of the specific image is determined, and candidate areas related to the specific image having an inclination deviating from the range are excluded. As a result, it is possible to accurately reduce the memory consumption and the processing burden required for the deformation process, and it is possible to prevent the image portion that is not the specific image from being deformed by mistake.

  Here, when trying to transform each candidate area on the target image, if the vertical direction of the specific image of each candidate area is different, the processing load and the memory consumption increase. For this reason, the deformation processing unit may perform the deformation process on the candidate areas that have substantially the same vertical orientation of the specific image. When there are candidate areas having different vertical orientations of the specific image, the exclusion determining unit corresponds to the direction in which the number of candidate areas having the same vertical orientation of the specific image is maximized. You may exclude candidate areas other than a candidate area from the object of a deformation | transformation process. With such a configuration, the deformation process is performed by narrowing down to a candidate area where the vertical direction of the specific image is common. As a result, the memory consumption amount and the processing burden amount can be accurately reduced, and a specific image that is assumed to have a high importance on the target image can be a target for deformation.

  Furthermore, when there are a plurality of directions in which the number of candidate areas having the same vertical direction is maximized, the exclusion determination unit selects a candidate area having a specific image with the largest size among the plurality of directions. Candidate regions other than the candidate region corresponding to the upward direction of the specific image may be excluded from the transformation processing target. Further, in addition to or in place of the above configuration, when there are a plurality of directions in which the number of candidate areas having the same vertical direction is the maximum, the exclusion determination unit selects the plurality of directions according to a predetermined priority order. Candidate regions other than the candidate region in which the upper direction of the specific image corresponds to one direction may be excluded from the deformation processing target. For example, there are a plurality of directions in which the number of candidate areas having the same vertical direction is the maximum, and there are a plurality of directions corresponding to candidate areas having a specific image with the largest size among the plurality of directions. In this case, one direction is selected according to the priority order. In this way, if a double or triple determination criterion is provided, the deformation processing unit reliably performs the deformation process only on the candidate region that is substantially directed in one direction with the vertical direction of the specific image. Is possible.

  As another example, when the number of candidate areas exceeds a predetermined threshold, the exclusion determination unit selects a number of candidate areas that fall within the threshold according to the size of the specific image, and is a candidate area that is not selected. May be excluded from the target of deformation processing. Since the processing load naturally increases as the number of candidate areas subject to deformation processing increases, the number of candidate areas subject to deformation processing is set within a predetermined number (threshold). For example, if the number of candidate regions that fall within the threshold is selected in the order of the size of the specific image, the specific image that is assumed to be more important on the target image can be the target of deformation.

Here, the candidate area setting unit detects a face image on the target image as the specific image, and sets a candidate area for each face image based on the detected face image. In this way, deformation processing can be performed on candidate regions other than the candidate regions excluded according to the exclusion conditions described above, among candidate regions set based on the face image detected from the target image. .
The deformation processing unit arranges a plurality of dividing points in the candidate area to be deformed, divides the area into a plurality of small areas using a straight line connecting the dividing points, and moves the position of at least one dividing point. Then, the image may be deformed by deforming the small area. As described above, since the image can be deformed simply by placing the dividing points in the candidate area that is finally the object of deformation and moving the arranged dividing points, image deformation corresponding to various deformation modes is possible. This image processing can be realized easily and efficiently.
Furthermore, the image processing apparatus may include a printing unit that prints the target image on which the image has been deformed. In this way, it is possible to easily and efficiently realize printing of the image after image deformation corresponding to various deformation modes.

  So far, the technical idea according to the present invention has been described as an image processing apparatus. However, the invention of the image processing method including steps corresponding to the respective units included in the above-described image processing apparatus, and the above-described image processing apparatus. It goes without saying that the invention of an image processing program for causing a computer to execute a function corresponding to each means included in can be grasped.

FIG. 2 is an explanatory diagram schematically illustrating a configuration of a printer 100 as an image processing apparatus. Explanatory drawing which shows an example of the user interface containing the list display of an image. 5 is a flowchart showing a flow of face shape correction printing processing by the printer. The flowchart which shows the flow of a face shape correction process. Explanatory drawing which shows an example of the user interface for setting the type and degree of image deformation. Explanatory drawing which shows an example of the detection result of face area FA. The flowchart which shows the flow of the position adjustment process of the height direction of face area FA. Explanatory drawing which shows an example of specific area | region SA. Explanatory drawing which shows an example of the calculation method of an evaluation value. Explanatory drawing which shows an example of the selection method of evaluation object pixel TP. Explanatory drawing which shows an example of the determination method of height reference point Rh. Explanatory drawing which shows an example of the calculation method of rough inclination angle RI. Explanatory drawing which shows an example of the position adjustment method of the height direction of face area FA. The flowchart which shows the flow of the inclination adjustment process of face area FA. Explanatory drawing which shows an example of the calculation method of the evaluation value for inclination adjustment of the face area FA. Explanatory drawing which shows an example of the calculation result of the dispersion | distribution of the evaluation value about each evaluation direction. Explanatory drawing which shows an example of the inclination adjustment method of face area FA. Explanatory drawing which shows an example of the setting method of deformation | transformation area | region TA. Explanatory drawing which shows an example of the division | segmentation method into the small area | region of the deformation | transformation area | region TA. Explanatory drawing which shows an example of the content of the dividing point movement table. Explanatory drawing which shows an example of the movement of the position of the dividing point D according to the dividing point movement table 420. FIG. Explanatory drawing which shows the concept of the deformation | transformation processing method of the image by the deformation | transformation process part 260. FIG. Explanatory drawing which shows the concept of the deformation | transformation processing method of the image in a triangular area | region. Explanatory drawing which shows the aspect of the face shape correction | amendment in a present Example. Explanatory drawing which shows an example of the state of the display part 150 on which the target image TI after face shape correction | amendment was displayed. 5 is a flowchart showing a flow of corrected image printing processing. The flowchart which shows an example of the content of exclusion discrimination | determination processing. The flowchart which shows an example of the detailed content of a part of exclusion determination process. Explanatory drawing which showed a mode that the angle which the face area | region FA and the vertical reference line Vs or the horizontal reference line Hs make is acquired. The figure which showed an example of the face area size upper limit table 430. Explanatory drawing which illustrated the positional relationship of the trimming range and deformation | transformation area | region TA on the target image TI. Explanatory drawing which illustrated the predetermined angle range and deformation | transformation area | region TA on the target image TI. The flowchart which shows an example of the detailed content of a part of exclusion determination process. Explanatory drawing which illustrates the state in which the deformation | transformation area | region TA of a different direction is set to the object image TI. Explanatory drawing which illustrates the state in which the deformation | transformation area | region TA of a different direction is set to the object image TI. Explanatory drawing which illustrates the state in which the deformation | transformation area | region TA of a different direction is set to the object image TI. The flowchart which shows an example of the detailed content of a part of exclusion determination process. Explanatory drawing which illustrates the state in which the some deformation | transformation area | region TA which mutually overlaps on the target image TI is set. Explanatory drawing which illustrates the partial range of deformation | transformation area | region TA.

Examples of the present invention will be described in the following order.
(1) Schematic configuration of image processing apparatus:
(2) Face shape correction printing process:
(2-1) From target image selection to deformation area setting:
(2-2) Deformation region exclusion process:
(2-3) From deformation area division to print processing:
(3) Additional description and other examples:

(1) Schematic configuration of image processing apparatus:
FIG. 1 is an explanatory diagram schematically showing the configuration of a printer 100 as an example of an image processing apparatus according to the present invention. The printer 100 is a stand-alone type color ink jet printer compatible with so-called direct printing that prints an image based on image data acquired from a recording medium (for example, a memory card MC). In addition to the printing function, the printer 100 is a device having various functions such as a copy function (not shown) and a scanner function (not shown). In this sense, the printer 100 can also be called a so-called multifunction device.

  The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by, for example, a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, A printer engine 160, a card interface (card I / F) 170, and the like are provided. The printer 100 may further include an interface for performing data communication with another device (for example, a digital still camera). Each component of the printer 100 is connected to each other via a bus.

  The printer engine 160 is a printing mechanism that performs printing based on print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, image data as RGB data is stored in the memory card MC, and the printer 100 acquires the image data stored in the memory card MC via the card interface 170.

  The internal memory 120 stores a face shape correction unit 200, a display processing unit 310, and a print processing unit 320. The face shape correction unit 200 is a computer program for executing a face shape correction process described later under a predetermined operating system. The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus and messages on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printer engine 160, and printing an image based on the print data. The CPU 110 implements the functions of these units by reading and executing these programs from the internal memory 120.

  The face shape correction unit 200 includes a deformation mode setting unit 210, a face region detection unit 220, a face region adjustment unit 230, a deformation region setting unit 240, an exclusion determination unit 270, and a deformation region dividing unit 250 as program modules. And a deformation processing unit 260. The deformation mode setting unit 210 includes a designation acquisition unit 212. The face area adjustment unit 230 includes a reference area setting unit 232, an evaluation unit 234, and a determination unit 236. The functions of these units will be described in detail in the description of the face shape correction printing process described later.

  The internal memory 120 also stores a face area size upper limit value table 430, a division point arrangement pattern table 410, and a division point movement table 420. The contents of the face area size upper limit value table 430, the dividing point arrangement pattern table 410, and the dividing point movement table 420 will also be described in detail in the description of the face shape correction printing process described later.

(2) Face shape correction printing process:
The printer 100 prints an image based on the image data stored in the memory card MC. When the memory card MC is inserted into the card slot 172, the display processing unit 310 displays a user interface including a list display of images stored in the memory card MC on the display unit 150.

  FIG. 2 is an explanatory diagram illustrating an example of a user interface including a list display of images. The image list display is realized by using, for example, thumbnail images included in image data (image file) stored in the memory card MC. When the user selects one (or a plurality of) images and selects the normal print button in the user interface shown in FIG. 2, the printer 100 performs normal print processing for printing the selected image as usual. Run. On the other hand, in the user interface, when one (or a plurality) of images is selected by the user and the face shape correction print button is selected, the printer 100 determines the shape of the face in the image for the selected image. The face shape correction printing process is executed to correct the image and print the corrected image.

FIG. 3 is a flowchart illustrating a flow of face shape correction printing processing by the printer 100 according to the present exemplary embodiment.
In step S (hereinafter, step notation is omitted) 100, the face shape correction unit 200 (FIG. 1) executes face shape correction processing. In the face shape correction process, when a face exists in the image, it is determined whether or not to correct the face based on a predetermined condition, and when it is determined to be corrected, at least a part of the shape ( For example, it is a process of correcting the contour shape of the face and the shape of the eyes.

(2-1) From target image selection to deformation area setting:
FIG. 4 is a flowchart showing the flow of face shape correction processing in this embodiment.
In S110, the face shape correction unit 200 (FIG. 1) sets a target image TI that is a target of face shape correction processing. The face shape correcting unit 200 sets an image selected by the user in the user interface shown in FIG. 2 as the target image TI. The set image data of the target image TI is acquired by the printer 100 from the memory card MC via the card interface 170 and stored in a predetermined area of the internal memory 120.

  In S120, the deformation mode setting unit 210 (FIG. 1) sets the image deformation type and the degree of image deformation for face shape correction. The deformation mode setting unit 210 instructs the display processing unit 310 to display a user interface for setting the type and degree of image deformation on the display unit 150, and the image deformation type designated by the user through the user interface. And the degree are selected and set as the image deformation type and degree used for processing.

  FIG. 5 is an explanatory diagram showing an example of a user interface for setting the type and degree of image deformation. As shown in FIG. 5, this user interface includes an interface for setting an image deformation type. In this embodiment, for example, it is assumed that a deformation type “type A” that sharpens the shape of the face, a deformation type “type B” that increases the shape of the eyes, and the like are set in advance as options. The user specifies the type of image deformation via this interface. The deformation mode setting unit 210 sets the image deformation type designated by the user as the image deformation type used for actual processing.

  The user interface shown in FIG. 5 includes an interface for setting the degree (degree) of image deformation. As shown in FIG. 5, in this embodiment, it is assumed that three levels of strong (S), medium (M), and weak (W) are preset as options as the degree of image deformation. The user designates the degree of image deformation via this interface. The deformation mode setting unit 210 sets the degree of image deformation designated by the user as the degree of image deformation used for actual processing.

In this embodiment, the user can specify details of the deformation mode. In the user interface shown in FIG. 5, when the user selects a check box indicating that detailed specification is desired, detailed specification of the deformation mode is performed by the user.
However, in the following, the deformation type “type A” for sharpening the shape of the face is set as the image deformation type, the degree of “medium” is set as the degree of image deformation, and the user desires detailed designation. The explanation will be given assuming that there was no.

  In S130, the face area detection unit 220 (FIG. 1) detects a face area (face image) FA in the target image TI. Here, the face area FA is an image area on the target image TI and means an area including at least a partial image of the face. The face area FA corresponds to an example of the specific image described in the claims. The detection of the face area FA by the face area detection unit 220 is executed using a known face detection method such as a pattern matching method using a template (see Japanese Patent Application Laid-Open No. 2004-318204).

  FIG. 6 is an explanatory diagram showing an example of the detection result of the face area FA. As shown in FIG. 6, according to the face detection method used in the present embodiment, a rectangular area including the eye, nose, and mouth images on the target image TI is detected as the face area FA. The face area FA may be square. The reference line RL shown in FIG. 6 is a line that defines the height direction (vertical direction) of the face area FA and indicates the center in the width direction (horizontal direction) of the face area FA. That is, the reference line RL is a straight line that passes through the center of gravity of the rectangular face area FA and is parallel to the rectangular boundary line along the substantially vertical direction of the face. If the face area FA is not detected in S130, the fact is notified to the user through the display unit 150. In this case, normal printing without face shape correction may be performed, or face area FA detection processing using another face detection method may be performed again.

  Here, in general, a known face detection method such as a pattern matching method using a template does not detect in detail the position or inclination (angle) of the entire face or face part (eyes, mouth, etc.) An area that is considered to contain a face image from the target image TI is set as the face area FA. On the other hand, as will be described later, the printer 100 according to the present exemplary embodiment sets an area (deformation area TA described later) to be subjected to image deformation processing for face shape correction based on the detected face area FA. Since the face image generally has a high degree of attention by the observer, the image after the face shape correction may be unnatural depending on the position and angle relationship between the set deformation area TA and the face image. There is sex. Therefore, in this embodiment, the position adjustment and the inclination adjustment described below are performed on the face area FA detected in S130 so that more natural and preferable face shape correction is realized.

  In S140, the face area adjustment unit 230 (FIG. 1) adjusts the vertical position of the face area FA detected in S130. The vertical position adjustment of the face area FA means that the position along the reference line RL (see FIG. 6) of the face area FA is adjusted to reset the face area FA in the target image TI. .

  FIG. 7 is a flowchart showing a flow of position adjustment processing in the vertical direction of the face area FA in the present embodiment. In S141, the reference area setting unit 232 (FIG. 1) sets the reference area SA. Here, the reference area SA is an area on the target image TI and includes an image of a predetermined reference subject that is referred to when the vertical position adjustment of the face area FA is executed. In this embodiment, the reference subject is set as “eyes”, and the reference area SA is set as an area including an image of “eyes”.

  FIG. 8 is an explanatory diagram showing an example of the reference area SA. In this embodiment, the reference area setting unit 232 sets the reference area SA based on the relationship with the face area FA. Specifically, the face area FA has a size that is reduced (or enlarged) by a predetermined ratio in a direction perpendicular to the reference line RL and a direction parallel to the reference line RL. An area having a predetermined positional relationship with this position is set as the reference area SA. That is, in the present embodiment, if the reference area SA is set based on the relationship with the face area FA detected by the face area detection unit 220, the reference area SA is an area including both eye images. A predetermined ratio and a predetermined positional relationship are set in advance. The reference area SA is preferably set as small as possible so long as the image of both eyes is included so that an image that is confusing with the eye image (for example, an image of the hair) is included as much as possible.

Further, as shown in FIG. 8, the reference area SA is set as a rectangular area symmetrical to the reference line RL. The reference area SA is, on the basis of the reference line RL, the left area (hereinafter also referred to as “left divided area SA (l)”) and the right area (hereinafter also referred to as “right divided area SA (r)”). And divided. The reference area SA is set so that the image of one eye is included in each of the left divided area SA (l) and the right divided area SA (r).
In S142, the evaluation unit 234 (FIG. 1) calculates an evaluation value for detecting the position of the eye image in the reference area SA.

  FIG. 9 is an explanatory diagram illustrating an example of an evaluation value calculation method. In this embodiment, the R value (R component value) of each pixel of the target image TI as RGB image data is used for calculation of the evaluation value. This is because the difference in the R value is large between the skin image and the eye image, and thus the detection accuracy of the eye image can be improved by using the R value for calculating the evaluation value. Because it is. In addition, in the present embodiment, since the data of the target image TI is acquired as RGB data, it is possible to increase the efficiency of calculation of the evaluation value by using the R value for calculation of the evaluation value. As shown in FIG. 9, the evaluation value is calculated individually for each of the two divided areas (the right divided area SA (r) and the left divided area SA (l)).

  As shown in FIG. 9, the evaluation unit 234 includes n straight lines (hereinafter referred to as “target pixels”) orthogonal to the reference line RL in each divided area (the right divided area SA (r) and the left divided area SA (l)). Specific lines PL1 to PLn ”). The target pixel specifying lines PL1 to PLn are straight lines that equally divide the height (size along the reference line RL) of the divided regions into (n + 1). That is, the intervals between the target pixel specifying lines PL are all equal intervals s. For each of the target pixel specifying lines PL1 to PLn, the evaluation unit 234 selects a pixel (hereinafter referred to as “evaluation target pixel TP”) that is used to calculate an evaluation value from among pixels that form the target image TI.

  FIG. 10 is an explanatory diagram illustrating an example of a method for selecting the evaluation target pixel TP. The evaluation unit 234 selects, as the evaluation target pixel TP, a pixel that overlaps the target pixel specifying line PL among the pixels constituting the target image TI. FIG. 10A shows a case where the target pixel specifying line PL is parallel to the row direction (X direction in FIG. 10) of the pixels of the target image TI. In this case, the pixel on the pixel row that overlaps with each target pixel specifying line PL (the pixel marked with a circle in FIG. 10A) is selected as the evaluation target pixel TP for each target pixel specifying line PL. .

  On the other hand, depending on the detection method of the face area FA and the setting method of the reference area SA, as shown in FIG. 10B, the target pixel specifying line PL is parallel to the row direction (X direction) of the pixels of the target image TI. There may be cases where this is not possible. Even in such a case, in principle, a pixel that overlaps each target pixel specifying line PL is selected as the evaluation target pixel TP for each target pixel specifying line PL. However, for example, as in the relationship between the target pixel specifying line PL1, the pixel PXa, and the pixel PXb in FIG. 10B, a certain target pixel specifying line PL is located in the same column of the pixel matrix of the target image TI (ie, Y In the case of overlapping two pixels having the same coordinates (for example, the pixel PXb) having a shorter overlapping distance is excluded from the evaluation target pixel TP. That is, for each target pixel specifying line PL, only one pixel is selected as the evaluation target pixel TP from one column of the pixel matrix.

  When the inclination of the target pixel specifying line PL exceeds 45 degrees with respect to the X direction, the relationship between the column and the row of the pixel matrix is reversed in the above description, and one pixel from one row of the pixel matrix is reversed. Only the pixel to be evaluated TP is selected. Further, depending on the size relationship between the target image TI and the reference area SA, one pixel may be selected as the evaluation target pixel TP for a plurality of target pixel specifying lines PL.

  The evaluation unit 234 calculates an average value of R values of the evaluation target pixels TP as an evaluation value for each of the target pixel specifying lines PL. However, in this embodiment, for each target pixel specifying line PL, out of the plurality of selected evaluation target pixels TP, some pixels having a large R value are excluded from the evaluation value calculation target. Specifically, for example, when k evaluation target pixels TP are selected for a certain target pixel specifying line PL, the evaluation target pixel TP includes a 0.75k pixel having a relatively large R value. An average value of R values as an evaluation value is calculated only for pixels that are divided into two groups, one group and a second group composed of 0.25k pixels having a relatively small R value. It becomes a target. The reason why some of the evaluation target pixels TP are excluded from the evaluation value calculation target will be described later.

  As described above, in this embodiment, the evaluation value for each target pixel specifying line PL is calculated by the evaluation unit 234. Here, since the target pixel specifying line PL is a straight line orthogonal to the reference line RL, the evaluation value can be expressed as being calculated for a plurality of positions (evaluation positions) along the reference line RL. In addition, the evaluation value can be expressed as a value representing the feature of the distribution of pixel values along the direction orthogonal to the reference line RL for each evaluation position.

  In S143, the determination unit 236 (FIG. 1) detects the eye position in the reference area SA, and determines the height reference point Rh based on the detection result. First, as illustrated on the right side of FIG. 9, the determination unit 236 creates a curve representing the distribution of evaluation values (average value of R values) along the reference line RL for each divided region, and the evaluation value is a minimum value. The position along the direction of the reference line RL that takes is detected as the eye position Eh. The eye position Eh in the left divided area SA (l) is represented as Eh (l), and the eye position Eh in the right divided area SA (r) is represented as Eh (r).

  In the case of a yellow race, the portion representing the skin image in the divided region has a large R value, while the portion representing the image of the eye (more specifically, the black eye portion at the center of the eye) is considered to have a small R value. Therefore, as described above, the position along the reference line RL where the evaluation value (average value of R values) takes the minimum value can be determined as the eye position Eh. As shown in FIG. 9, the divided area may include other images having a small R value (for example, images of eyebrows and hairs) in addition to the eye image. Therefore, when the curve representing the distribution of evaluation values along the reference line RL has a plurality of minimum values, the determination unit 236 determines the position corresponding to one minimum value based on a predetermined determination criterion. The position is determined as Eh. For example, among the positions on the reference line RL corresponding to a plurality of local minimum values, the position closest to the center position of the reference line RL in the reference area SA is set as the eye position Eh. This is because, as described above, the reference area SA is set as an area as small as possible including the image of the eye, so the eye position is close to the center position of the reference line RL in the reference area SA, while the eyebrows and hair This is because it is assumed that it is located near the boundary line of each divided region.

  In addition, even if the curve is other than the position of the eye image, there is a possibility that the evaluation value is large, but the minimum value may be taken. Therefore, among the minimum values, those that are larger than a predetermined threshold may be ignored. Good. Alternatively, the position of the target pixel specifying line PL corresponding to the minimum value among the evaluation values calculated for each target pixel specifying line PL may be simply set as the eye position Eh.

In the present embodiment, the eye (the black eye part at the center of the eye), which is considered to be a part where the color difference between the face and the surroundings is relatively large, is used as a reference subject for position adjustment of the face area FA. However, since the average value of the R values as the evaluation values is calculated for a plurality of evaluation target pixels TP on the target pixel specifying line PL, for example, due to the influence of the image of the white part of the periphery of the black eye, the black eye part There is a fear that the accuracy of detection will be reduced. In the present embodiment, as described above, some evaluation target pixels TP (for example, pixels having a relatively large R value belonging to the first group described above) that are considered to have a large color difference from the reference subject are evaluated values. Is excluded from the calculation target, the reference subject detection accuracy is further improved.
Next, the determination unit 236 determines the height reference point Rh based on the detected eye position Eh.

FIG. 11 is an explanatory diagram illustrating an example of a method for determining the height reference point Rh. The height reference point Rh is a point used as a reference when adjusting the position of the face area FA in the height direction. In the present embodiment, as shown in FIG. 11, a point on the reference line RL located in the middle between the positions Eh (l) and Eh (r) of the two left and right eyes is set as the height reference point Rh. That is, the intersection of the straight line EhL (l) indicating the left eye position Eh (l) and the reference line RL, and the intersection of the straight line EhL (r) indicating the right eye position Eh (r) and the reference line RL. The midpoint is set as the height reference point Rh.
In the present embodiment, the determination unit 236 calculates a rough inclination angle of the face image (hereinafter referred to as “rough inclination angle RI”) based on the detected eye position Eh. The approximate inclination angle RI of the face image is an angle obtained by estimating how much the face image in the target image TI is inclined with respect to the reference line RL of the face area FA.

FIG. 12 is an explanatory diagram showing an example of a method for calculating the approximate inclination angle RI. As shown in FIG. 12, the determination unit 236 first determines the intersection point IP (l) between the straight line EhL (l) and the straight line that divides the width Ws (l) of the left divided region SA (l) in half, and the right division. A straight line that divides the width Ws (r) of the region SA (r) in half and an intersection IP (r) of the straight line EhL (r) are determined. Then, an angle formed by the straight line IL orthogonal to the straight line connecting the intersection point IP (l) and the intersection point IP (r) and the reference line RL is calculated as the approximate inclination angle RI.
In S144, the face area adjustment unit 230 (FIG. 1) adjusts the position of the face area FA in the vertical direction.

  FIG. 13 is an explanatory diagram showing an example of a method for adjusting the position of the face area FA in the height direction. The position adjustment of the face area FA in the height direction is performed by resetting the face area FA so that the height reference point Rh is positioned at a predetermined position in the face area FA after the position adjustment. Specifically, as shown in FIG. 13, the height reference point Rh is positioned so as to divide the vertical length Hf of the face area FA by a predetermined ratio r1 to r2. The position of the rectangular frame defining the face area FA is adjusted along the reference line RL. In the example of FIG. 13, the adjusted face area FA indicated by the solid line is reset by moving the face area FA before adjustment indicated by the broken line upward in the face.

  After the position adjustment of the face area FA, in S150 (FIG. 4), the face area adjustment unit 230 (FIG. 1) performs the inclination adjustment (angle adjustment) of the face area FA. Here, the tilt adjustment of the face area FA means that the face area FA is adjusted by adjusting the tilt of the face area FA in the target image TI to match the tilt of the face image, and the face area FA is reset. In the present embodiment, the predetermined reference subject that is referred to when the inclination adjustment of the face area FA is executed is set to “both eyes”. In the inclination adjustment of the face area FA in the present embodiment, a plurality of evaluation directions representing options of adjustment angles for inclination adjustment are set, and the evaluation reference area ESA corresponding to each evaluation direction is set as an area including the images of both eyes. . Then, an evaluation value is calculated based on the pixel value of the image in the evaluation reference area ESA for each evaluation direction, and the inclination of the face area FA is adjusted using the adjustment angle of the inclination adjustment determined based on the evaluation value.

FIG. 14 is a flowchart illustrating the flow of the inclination adjustment processing of the face area FA in the present embodiment.
FIG. 15 is an explanatory diagram illustrating an example of an evaluation value calculation method for adjusting the inclination of the face area FA.
In S151, the reference area setting unit 232 (FIG. 1) sets the initial evaluation reference area ESA (0). The initial evaluation reference area ESA (0) is an evaluation reference area ESA associated with a direction (hereinafter also referred to as “initial evaluation direction”) parallel to the reference line RL (see FIG. 13) after the position adjustment of the face area FA. is there. In this embodiment, the reference area SA (see FIG. 13) corresponding to the face area FA after the position adjustment is set as the initial evaluation reference area ESA (0) as it is. Note that the evaluation reference area ESA is not divided into two left and right areas, unlike the reference area SA at the time of position adjustment of the face area FA. The set initial evaluation reference area ESA (0) is shown at the top of FIG.

  In S152, the reference area setting unit 232 (FIG. 1) sets a plurality of evaluation directions and evaluation reference areas ESA corresponding to the respective evaluation directions. The plurality of evaluation directions are set as directions representing options of adjustment angles for tilt adjustment. In this embodiment, a plurality of evaluation direction lines EL whose angles with the reference line RL are within a predetermined range are set, and a direction parallel to the evaluation direction line EL is set as the evaluation direction. As shown in FIG. 15, a straight line determined by rotating the reference line RL counterclockwise and clockwise around the center point (center of gravity) CP of the initial evaluation reference area ESA (0) at a predetermined angle α, It is set as a plurality of evaluation direction lines EL. Note that the evaluation direction line EL whose angle with the reference line RL is φ degrees is represented as EL (φ).

  In the present embodiment, the predetermined range for the angle formed by each evaluation direction line EL and the reference line RL is set to ± 20 degrees. In this specification, the rotation angle when the reference line RL is rotated clockwise is represented by a negative value, and the rotation angle when the reference line RL is rotated counterclockwise is represented by a positive value. The reference area setting unit 232 rotates the reference line RL counterclockwise and clockwise while increasing the rotation angle within a range not exceeding α degrees, 2α degrees,..., 20 degrees, and a plurality of evaluation direction lines EL. Set. FIG. 15 shows evaluation direction lines EL (EL (α), EL (2α), EL (−α)) determined by rotating the reference line RL by α degrees, 2α degrees, and −α degrees, respectively. . The reference line RL can also be expressed as an evaluation direction line EL (0).

  The evaluation reference area ESA corresponding to the evaluation direction line EL representing each evaluation direction rotates the initial evaluation reference area ESA (0) around the center point CP at the same angle as the rotation angle at the time of setting the evaluation direction line EL. This is the area that was The evaluation reference area ESA corresponding to the evaluation direction line EL (φ) is represented as an evaluation reference area ESA (φ). FIG. 15 shows evaluation reference areas ESA (ESA (α), ESA (2α), ESA (−α)) corresponding to the evaluation direction lines EL (α), EL (2α), and EL (−α). It is shown. Note that the initial evaluation reference area ESA (0) is also treated as one of the evaluation reference areas ESA.

  In S153, the evaluation unit 234 (FIG. 1) calculates an evaluation value for each of a plurality of evaluation directions based on the pixel value of the image in the evaluation reference area ESA. In the present embodiment, the average value of the R values is used as the evaluation value in the adjustment of the inclination of the face area FA, similarly to the evaluation value in the position adjustment of the face area FA described above. The evaluation unit 234 calculates evaluation values for a plurality of evaluation positions along the evaluation direction.

  The evaluation value calculation method is the same as the evaluation value calculation method in the position adjustment of the face area FA described above. That is, as shown in FIG. 15, the evaluation unit 234 sets target pixel specifying lines PL1 to PLn orthogonal to the evaluation direction line EL in each evaluation reference area ESA, and evaluates each target pixel specifying line PL1 to PLn. The target pixel TP is selected, and an average value of R values of the selected evaluation target pixel TP is calculated as an evaluation value.

  The setting method of the target pixel specifying line PL and the selection method of the evaluation target pixel TP in the evaluation reference area ESA are different depending on whether the area is divided into right and left, but the face area FA shown in FIGS. This is the same as the method for position adjustment. As in the position adjustment of the face area FA, a part of the selected evaluation target pixels TP (for example, 0.75k pixels having a relatively large R value among the k evaluation target pixels TP). May be excluded from the evaluation value calculation target. The right side of FIG. 15 shows the distribution of the calculated evaluation values along the evaluation direction line EL for each evaluation direction.

  In addition, since the target pixel specifying line PL is a straight line orthogonal to the evaluation direction line EL, it can be expressed that the evaluation value is calculated for a plurality of positions (evaluation positions) along the evaluation direction line EL. In addition, the evaluation value can be expressed as a value representing the feature of the distribution of pixel values along the direction orthogonal to the evaluation direction line EL for each evaluation position.

  In S154, the determination unit 236 (FIG. 1) determines an adjustment angle used for adjusting the inclination of the face area FA. For each evaluation direction, the determination unit 236 calculates the variance along the evaluation direction line EL of the evaluation value calculated in S153, and selects the evaluation direction that maximizes the variance value. Then, an angle formed by the evaluation direction line EL corresponding to the selected evaluation direction and the reference line RL is determined as an adjustment angle used for tilt adjustment.

  FIG. 16 is an explanatory diagram illustrating an example of a calculation result of variance of evaluation values for each evaluation direction. In the example of FIG. 16, the variance takes the maximum value Vmax in the evaluation direction where the rotation angle is α degrees. Accordingly, α degrees, that is, a rotation angle of α degrees counterclockwise is determined as an adjustment angle used for adjusting the inclination of the face area FA.

  The reason why the angle corresponding to the evaluation direction when the variance of the evaluation values is maximized is determined as the adjustment angle used for the tilt adjustment will be described. As shown in the second row from the top in FIG. 15, in the evaluation reference area ESA (α) when the rotation angle is α degrees, the image of the center part (black eye part) of the left and right eyes is substantially the target pixel specifying line. The arrangement is aligned in a direction parallel to the PL (that is, a direction perpendicular to the evaluation direction line EL). At this time, the left and right eyebrow images are similarly arranged so as to be aligned in a direction substantially perpendicular to the evaluation direction line EL. Therefore, the evaluation direction corresponding to the evaluation direction line EL at this time is considered to be a direction that generally represents the inclination of the face image. In this case, the positional relationship between the image of the eye or eyebrow generally having a small R value and the image of the skin portion generally having a large R value is a small positional relationship of the overlapping portion along the direction of the target pixel specifying line PL. . Therefore, the evaluation value at the position of the eye or eyebrow image is relatively small, and the evaluation value at the position of the image of the skin portion is relatively large. Therefore, as shown in FIG. 15, the distribution of evaluation values along the evaluation direction line EL is a distribution with relatively large variation (amplitude is large), and the variance value is large.

  On the other hand, as shown in the uppermost stage, the third stage, and the fourth stage in FIG. 15, the evaluation reference areas ESA (0), ESA (2α), when the rotation angle is 0 degrees, 2α degrees, and −α degrees, In ESA (−α), the center portions of the left and right eyes and the images of the left and right eyebrows are not aligned in a direction perpendicular to the evaluation direction line EL, but are shifted from each other. Therefore, the evaluation direction corresponding to the evaluation direction line EL at this time does not represent the inclination of the face image. At this time, the positional relationship between the image of the eyes and eyebrows and the image of the skin portion is a large positional relationship of the portion where both overlap along the direction of the target pixel specifying line PL. Therefore, the distribution of evaluation values along the evaluation direction line EL is a distribution with relatively small variation (small amplitude) as shown in FIG. 15, and the value of variance is small.

  As described above, when the evaluation direction is close to the inclination direction of the face image, the evaluation value variance along the evaluation direction line EL is large, and the evaluation direction is far from the inclination direction of the face image. In this case, the variance of the evaluation values along the evaluation direction line EL becomes small. Therefore, if the angle corresponding to the evaluation direction when the variance of the evaluation values is maximized is determined as the adjustment angle used for the inclination adjustment, the face area in which the inclination of the face area FA matches the inclination of the face image. FA tilt adjustment can be realized.

In the present embodiment, when the calculation result of the variance of the evaluation value is the critical value of the angle range, that is, the maximum value at −20 degrees or 20 degrees with respect to the reference line RL, Since it is considered that there is a high possibility that the face inclination is not accurately evaluated, the inclination adjustment of the face area FA is not performed. In the present embodiment, the determined adjustment angle is compared with the approximate inclination angle RI calculated at the time of adjusting the position of the face area FA described above. If the difference between the adjustment angle and the approximate inclination angle RI is larger than a predetermined threshold value, it is considered that some error has occurred during the evaluation and determination in the position adjustment and inclination adjustment of the face area FA. Position adjustment and tilt adjustment are not performed.
In S155, the face area adjustment unit 230 (FIG. 1) adjusts the inclination of the face area FA.

FIG. 17 is an explanatory diagram showing an example of a method for adjusting the inclination of the face area FA. The tilt adjustment of the face area FA is performed by rotating the face area FA about the center point CP of the initial evaluation reference area ESA (0) by the adjustment angle determined in S154. In the example of FIG. 17, the adjusted face area FA indicated by the solid line is set by rotating the face area FA before adjustment indicated by the broken line by α degrees counterclockwise.
In S160 after the adjustment of the inclination of the face area FA is completed, the deformation area setting unit 240 (FIG. 1) sets the deformation area TA. The deformation area TA is an area on the target image TI and is a candidate area for image deformation processing for face shape correction. In this sense, the deformation area TA set in S160 corresponds to an example of a candidate area described in the claims.

  FIG. 18 is an explanatory diagram illustrating an example of a method for setting the deformation area TA. As shown in FIG. 18, in this embodiment, the deformation area TA extends (or shortens) the face area FA in a direction (height direction) parallel to the reference line RL and a direction (width direction) perpendicular to the reference line RL. ). Specifically, assuming that the size of the face area FA in the height direction is Hf and the width direction is Wf, the rectangle of the face area FA is extended by k1 · Hf upward and by k2 · Hf downward. At the same time, a region extended to the left and right by k3 · Wf is set as the deformation region TA. Note that k1, k2, and k3 are predetermined coefficients.

  When the deformation area TA is set in this way, the reference line RL, which is a straight line parallel to the contour line in the height direction of the face area FA, becomes a straight line parallel to the contour line in the height direction of the deformation area TA. . The reference line RL is a straight line that divides the width of the deformation area TA in half.

As shown in FIG. 18, the deformation area TA is set as an area that generally includes an image from the chin to the forehead with respect to the height direction of the face and includes images of the left and right cheeks with respect to the width direction of the face. The That is, in this embodiment, the above-described coefficients k1, k2, and k3 are set in advance based on the relationship with the size of the face area FA so that the deformation area TA is an area that includes an image in such a range. Yes. Here, it can be said that the deformation area setting unit 240 corresponds to the candidate area setting unit described in the claims. The face area detection unit 220, the face area adjustment unit 230, and the deformation area setting unit 240 described above may be collectively referred to as a candidate area setting unit.
Note that FIG. 6 and the like illustrate the case where only one human face exists in the target image TI. However, when there are a plurality of faces in the target image TI, in S130 to 160, a plurality of faces are present. For each face that is detected, the face area FA is set, the position adjustment / tilt adjustment of the face area FA, and the deformation area TA are set.

(2-2) Deformation region exclusion process:
When the deformation area TA on the target image TI is set by the processing up to S160, in S170, the exclusion determination unit 270 (FIG. 1) performs deformation for each deformation area TA according to various exclusion conditions described later. It is determined whether or not to be a target of processing, and a deformation region that is determined not to be a target of deformation processing is excluded from the target of the deformation processing. That is, in the present embodiment, not all of the deformation area TA set by the processing up to S160 is subject to image deformation described later, and does not correspond to various exclusion conditions in the set deformation area TA. Only objects are subject to image transformation. As a result, it is possible to appropriately suppress the processing load of the printer 100 and the consumption of the memory area in the internal memory 120 and the like due to image deformation.

FIG. 27 is an example of a flowchart showing the processing content of S170.
In S171, as the first exclusion determination, the exclusion determination unit 270 determines whether or not to exclude from the deformation process target for each deformation area TA according to the size restriction according to the inclination of the face area FA.

FIG. 28 is a flowchart showing details of the process in S171.
In S1711, the exclusion determination unit 270 acquires the inclination and size of the face area FA for each deformation area TA. The inclination acquired in S1711 is the magnitude of the angle formed by the vertical reference line Vs that faces the top and bottom of the target image TI or the horizontal reference line Hs that faces the horizontal direction of the target image TI and the reference line RL of the face area FA. Shall be said.

  FIG. 29 exemplifies a state in which the inclination of the face area FA is acquired for a certain deformation area TA in S1711. Here, first, it is determined whether the reference direction for obtaining the inclination is the vertical reference line Vs or the horizontal reference line Hs. In this case, the reference line RL is determined depending on which side of the vertical reference line Vs or the horizontal reference line Hs is inclined. That is, on the target image TI, when the upward direction of the face is directed substantially upward or substantially downward of the target image TI, the vertical reference line Vs serves as a reference for acquiring the inclination, and the upward direction of the face is the target. When the image TI is directed substantially to the right or substantially left, the horizontal reference line Hs serves as a reference for acquiring the inclination.

  In the example shown in FIG. 29, since the reference line RL is inclined toward the vertical reference line Vs side with respect to the horizontal reference line Hs of the target image TI, the angle β formed by the vertical reference line Vs and the reference line RL is S1711. It is detected as the inclination of the face area FA. The angle β is obtained as a positive value because it is the magnitude of the angle formed by the reference line RL and the vertical reference line Vs (or the horizontal reference line Hs). In S1711, the size of the face area FA is also acquired. As the size of the face area FA, various values such as the area of the face area FA can be used. In the present embodiment, as an example, the length of the side in the height direction (the above Hf) among the rectangular sides of the face area FA is set as the size of the face area FA. In this case, the length of Hf on the target image TI is obtained by converting into the number of pixels on the target image TI. However, the length of the side in the width direction (Wf) may be acquired as the size of the face area FA, and if the face area FA is a square as described above, one of the four sides of the face area FA is selected. The length of one side may be acquired as the size of the face area FA.

In S1712, the exclusion determination unit 270 refers to the face area size upper limit value table 430 (FIG. 1), and acquires the upper limit value of the size of the face area FA according to the acquired inclination of the face area FA.
FIG. 30 shows an example of the face area size upper limit value table 430. In the face area size upper limit value table 430, regarding the plurality of inclinations of the face area FA acquired as described above, the allowable size of the face area FA (in this embodiment, the height of the side of the face area FA in the height direction). 6 is a table in which an upper limit value of the number of pixels) is defined in advance. In the present embodiment, the memory consumption required for the deformation process of the deformation area TA including the face area FA increases as the inclination of the face area FA approaches a predetermined upper limit angle (referred to as angle βL). In view of this (the situation will be described later), a face area size upper limit value table 430 is adopted in which the upper limit value of the size is defined to be smaller as the value indicating the inclination of the face area FA approaches the angle βL. .

  If the size upper limit value corresponding to the inclination (angle β) acquired in S1711 is directly defined in the face area size upper limit value table 430, the exclusion determination unit 270 reads the directly defined upper limit value. Good. On the other hand, when the size upper limit value corresponding to the acquired inclination (angle β) is not directly defined in the face area size upper limit value table 430, an interpolation calculation or the like based on another defined upper limit value is performed. The upper limit value corresponding to the acquired inclination (angle β) may be acquired.

  In step S1713, the exclusion determination unit 270 determines whether the size of the face area FA acquired in step S1711 is within the size upper limit value acquired in step S1712. If the size exceeds the upper limit value, the exclusion determination unit 270 determines the size exceeding the upper limit value. The decision is made to exclude the deformation area TA having the face area FA from the object of deformation processing. As a result, the deformation area TA in which the memory consumption required for the deformation process increases due to the inclination (angle β) on the target image TI can be excluded from the object of the deformation process.

  Next, in S172, as a second exclusion determination, the exclusion determination unit 270 determines from the image range common to each trimmed image cut out from the target image TI by each trimming based on a plurality of aspect ratios used for image output from the deformation region. It is determined whether or not at least a part of TA protrudes, and a process of excluding at least a part of the deformation area TA that protrudes from the target of the deformation process is performed. This determination is performed for each deformation area TA that has not been determined to be excluded from the deformation processing target in S171.

  In this embodiment, the printer 100 handles the target image TI as a single size of image data in the process of performing the face shape correction process. The image data of the target image TI is, for example, QVGA (Quarter Video Graphics Array) size image data having a long side and a short side of 320 × 240 pixels (aspect ratio 4: 3). On the other hand, there are various types of printing paper that can be used for printing the target image TI in the printer 100, such as A4 size, L size, and letter size, and the aspect ratio varies depending on the type of printing paper. Therefore, when the target image TI is printed, the image data is appropriately trimmed based on the aspect ratio of the printing paper selected to be used for printing.

  FIG. 31A illustrates a case where the target image TI is trimmed based on the aspect ratio of a certain printing paper. When the target image TI is trimmed with an aspect ratio different from the aspect ratio of the target image TI, as shown in the figure, a part of the deformation area TA set on the target image TI is outside the trimming range (hatching is indicated. It is possible that the part may protrude. Since the data portion that is out of the trimming range is discarded, in order to deform the face included in the deformation area TA that partially protrudes from the trimming range in this way, all of the deformation area TA including the face is changed. It is necessary to set again so as to be within the trimming range (move or reduce the rectangle of the deformation area TA). However, such a resetting process of the deformation area TA causes an increase in processing load on the printer 100 and an increase in memory consumption.

  Therefore, as illustrated in FIG. 31B, the exclusion determination unit 270 trims each trimmed image that is cut out when the target image TI is trimmed based on the aspect ratios of all print sheets usable by the printer 100. A common image range CA common to the two is set. In FIG. 31B, for ease of explanation, it is assumed that there are two types of aspect ratios that are different from the aspect ratio of the target area TI and that are the aspect ratios of the printing paper that can be used by the printer 100. The area on the target image TI that is discarded without being cut out when trimming is performed according to the aspect ratio is hatched. That is, the area on the target image TI that is not hatched in the figure is the common image range CA. The exclusion determination unit 270 determines, for each deformation area TA to be determined, whether or not it has protruded from the common image range CA, and for a deformation area TA that partially protrudes from the common image range CA, the deformation processing target Make a decision to exclude

  As a result, only the deformation area TA that is entirely included in the common image range CA can be the target of the deformation process, so that it is not necessary to reset the deformation area TA as described above, increasing the processing load on the printer 100 and the memory consumption. An increase in quantity can be avoided. In addition, since the situation in which the face in the target image TI is not deformed due to the difference in the type of printing paper used for printing (difference in the aspect ratio of the printing paper) is eliminated, the user may feel uncomfortable. Absent.

  Next, in S173, as a third exclusion determination, the exclusion determination unit 270 determines whether the inclination on the target image TI belongs to a predetermined angle range for the deformation area TA, and does not belong to the predetermined angle range. As for, a process of excluding from the target of the deformation process is performed. This determination is performed for each deformation area TA that has not been determined to be excluded from the deformation process in S171 and 172.

FIG. 32 shows an example of the predetermined angle range. In this embodiment, the positive direction of the horizontal reference line Hs (right direction of the target image TI. 0 degree direction), the positive direction of the vertical reference line Vs (upward direction of the target image TI. 90 degree direction), and the horizontal reference line. A range of ± 15 degrees centering on the negative direction of Hs (left direction of the target image TI, 180 degree direction) is defined as the predetermined angle range. Of course, the setting of the predetermined angle range is not limited to the above numerical values.
In S173, the exclusion determination unit 270 determines whether or not the inclination of the reference line RL belongs to the predetermined angle range for the deformation area TA to be determined. In this determination, the exclusion determination unit 270 needs to recognize which side of the upper and lower sides of the reference line RL, that is, one of both ends of the reference line RL indicates the upward direction of the face.

  Here, when the face area FA is detected by a known face detection method, each of the four vertices of the rectangle of the face area FA is the upper right point of the face or the upper left point of the face. It is possible to obtain vertex position information indicating information such as whether the point is the lower right point of the face or the lower left point of the face. Therefore, for example, the face area detection unit 220 refers to the vertex position information, specifies the upper right point and the lower right point (or the upper left point and the lower left point) of the face area FA, and both ends of the reference line RL. Of the sides, the side close to the identified upper right point (upper left point) can be identified as the upward direction of the reference line RL.

  Identification of which side of the both ends of the reference line RL indicates the upward direction of the face may be performed as follows. As described with reference to FIG. 7, in S142, the evaluation unit 234 calculates an evaluation value for detecting the position of the eye image in the reference region SA. As shown in FIG. 9, the evaluation value takes one minimum value at the eye position Eh, and also takes a minimum value at a position where there are few R components other than the eyes, that is, images of eyebrows and hair. And it can be said that the eyebrows and the hairs exist above the eyes in the human face. Therefore, the determination unit 236, together with the process of determining the height reference point Rh, sees the reference line corresponding to a minimum value different from the minimum value corresponding to the eye position when viewed from the position of the height reference point Rh. A process of specifying the position side on the RL as the upward direction of the face is performed. As a result, it is specified which side of both ends of the reference line RL indicates the upward direction of the face.

In this way, since it is specified which of the two ends of the reference line RL of the deformation area TA points to the upper direction of the face, the exclusion determination unit 270 determines, for each deformation area TA to be determined, based on the determination result. Then, it is determined whether or not the angle γ formed by counterclockwise rotation between the right direction (0 degree direction) of the target image TI and the upper direction of the reference line RL belongs to a predetermined angle range as illustrated in FIG. The deformation area TA that does not belong to any predetermined angle range is determined to be excluded from the deformation process target.
In this way, by limiting the face to be deformed to some extent based on the inclination of the face on the target image TI, it is possible to avoid an increase in processing load and an increase in memory consumption in the printer 100.

  In S174, as a fourth exclusion determination, the exclusion determination unit 270 determines whether or not the deformation area TA is to be subjected to deformation processing based on the vertical orientation of the face on the target image TI. For those belonging to directions other than one direction, a process of excluding them from the target of the deformation process is performed. The determination is performed on the deformation area TA that has not been determined to be excluded from the deformation processing target in S171 to S173.

FIG. 33 is a flowchart showing details of the process in S174.
In S1741, the exclusion determination unit 270 first groups the deformation area TA on the target image TI according to the commonality of the face vertical orientation. By the processing up to S173, the upward direction of the face is directed substantially upward of the target image TI on the target image TI (the angle γ belongs to the range of ± 15 degrees above the target image TI). A group of deformation areas TA, a group of deformation areas TA in which the upward direction of the face is directed substantially to the right side of the target image TI (the angle γ belongs to the range of ± 15 degrees to the right of the target image TI), There may be a group of deformation areas TA in which the upward direction of the face is directed substantially to the left side of the target image TI (the angle γ belongs to the range of ± 15 degrees in the left direction of the target image TI). Depending on the accuracy of the processing of S173, the upper direction of the face rarely faces the lower side of the target image TI (the angle γ belongs to the range of ± 15 degrees in the negative direction of the vertical reference line Vs). There may also be a deformation area TA.

The exclusion determination unit 270 divides the deformation area TA on the target image TI into upper, lower, left, and right groups as described above according to the angle γ in which the upward direction of the reference line RL is directed.
In S 1742, the exclusion determination unit 270 determines whether or not there are a plurality of groups having the maximum number of deformation areas TA to which they belong. If there is one group having the largest number of deformation areas TA, the process proceeds to S1743, and deformation processes TA other than the deformation area TA belonging to the group having the largest number of deformation areas TA belong to the deformation process. Decide to be excluded from the target.

  In the case where a plurality of deformation areas TA are set on the target image TI as illustrated in FIG. 34, the number of deformation areas TA in which the face faces substantially upward is 2 and the maximum, and 2 or more There is no other direction (right, left, or lower side) that faces almost in common. For this reason, in S1743, a determination is made to exclude the deformation area TA facing in a direction other than the substantially upper side from the deformation processing target.

  On the other hand, if it is determined in S 1742 that there are a plurality of groups having the largest number of deformation areas TA, the exclusion determination unit 270 proceeds to S 1744 and includes each group having the largest number of deformation areas TA. Then, it is determined whether there are a plurality of groups including the deformation area TA having the largest size of the face area FA. The size of the face area FA is as described above. If there is one group including the deformation area TA having the maximum size of the face area FA, the process proceeds to S1745, where the number of deformation areas TA to which the face area FA belongs is the maximum and the size of the face area FA is the maximum. A determination is made to exclude the deformation area TA other than the deformation area TA belonging to the group including the area TA from the deformation processing target.

  As illustrated in FIG. 35, when a plurality of deformation areas TA are set on the target image TI, the number of deformation areas TA in which the face faces substantially upward and the deformation areas in which the face faces substantially right. The number of TAs is the maximum number (2), and the group of deformation areas TA in which the face faces substantially upward includes those having the largest face area FA. Therefore, in S1745, a determination is made to exclude the deformation area TA that faces in a direction other than the substantially upper side from the object of the deformation process.

  On the other hand, if there are a plurality of groups including the deformation area TA having the maximum size of the face area FA in each group having the maximum number of deformation areas TA, the exclusion determination unit 270 proceeds to S1746, A group in one direction is selected from the plurality of groups according to a predetermined priority. Then, a decision is made to exclude the deformation area TA other than the deformation area TA belonging to the selected group from the objects of deformation processing. The predetermined priority order mentioned here is the order of up, down, left and right, and at least the upper side is given the highest priority. The priority order for left, right, and lower is not particularly limited, but for example, priorities are assigned in advance such as lower, left, and right.

  As illustrated in FIG. 36, when a plurality of deformation areas TA are set on the target image TI, a deformation area TA in which the face faces substantially upward and a deformation area TA in which the face faces substantially right. And the same number of deformation areas TA with the face facing substantially left, and the deformation area TA with the face facing substantially upward, and the deformation area TA with the face facing substantially right, The size of each face area FA is the same as that of the deformation area TA in which the face is directed substantially to the left side. Therefore, in S1746, according to the above priority order, a determination is made to exclude the deformation area TA that faces in a direction other than the substantially upper side from the object of the deformation process.

  As described above, according to the process of S174, the vertical direction of the face of each deformation area TA set on the target image TI is disjoint such as substantially upper side, substantially lower side, substantially right side, and substantially left side. However, only one direction of the deformation area TA to be subjected to the deformation process is selected according to the number and size of the deformation areas TA in each direction or the priority order between the directions. Therefore, it is possible to avoid an increase in processing load and an increase in memory consumption in the printer 100. Note that the flowchart illustrated in FIG. 33 is merely an example, and the exclusion determination unit 270 does not necessarily have to perform the steps according to the flowchart. For example, after “Yes” is determined in S 1742, the determination may be shifted to S 1746 without performing the determination in S 1744. With this procedure, the process of S174 can be performed more simply.

  In S175, as a fifth exclusion determination, the exclusion determination unit 270 excludes one or more deformation areas TA from the target of the deformation process so that the overlap is eliminated when there is an overlap between the deformation areas TA. The process to do is performed. The determination is performed on the deformation area TA that has not been determined to be excluded from the deformation processing target in S171 to S174.

FIG. 37 is a flowchart showing details of the process in S175.
In S1751, the exclusion determination unit 270 selects one deformation area TA having the maximum size of the face area FA from among the deformation areas TA that are not determined to be excluded from the deformation process at that time.
In S1752, it is determined whether there is another deformation area TA that overlaps with the deformation area TA selected in the latest S1751 and that has not been determined to be excluded from the deformation process at that time. If there is another overlapping deformation area TA, the exclusion determination unit 270 proceeds to S1753 and determines to exclude all of the overlapping other deformation areas TA from the target of deformation processing.

  After S1753 or when “No” is determined in S1752, the process proceeds to S1754. In S1754, the exclusion determination unit 270 determines whether there is a deformation area TA that is not determined to be excluded from the deformation process at that time other than the deformation area TA selected in the previous S1751. If it is determined that it exists, the process returns to S1751, and the exclusion determination unit 270 newly selects a deformation area TA other than the deformation area TA selected in the past S1751, and repeats the processes after S1752. On the other hand, if there is no deformation area TA that has not been determined to be excluded from the deformation process at that time other than the deformation area TA selected in the previous S1751, the processing in FIG. 37 ends.

  As shown in FIG. 38, the process of FIG. 37 will be described as follows by taking as an example a case where a plurality of deformation areas TA are set to overlap on the target image TI. In FIG. 38, since the deformation area TA1 has the largest size of the face area FA among the deformation areas TA1 to TA6, the deformation area TA1 is first selected in S1751. Since the deformation area TA5 overlaps the deformation area TA1, the deformation area TA5 is excluded from the object of the deformation process. In the next S1751, the deformation area TA2 having the largest face area FA size is selected from the deformation areas TA not selected in the previous S1751. Since the deformation area TA3 and the deformation area TA4 overlap with the deformation area TA2, the deformation areas TA3 and TA4 are excluded from the objects of the deformation process. In the next S1751, the deformation area TA6 is selected. However, there is no other deformation area TA that is not excluded at that time and overlaps the deformation area TA6 (No in S1752). In addition to the deformation areas TA1, TA2, and TA6, since there is no deformation area TA that has not been determined to be excluded from the deformation process at that time (No in S1754), the process in FIG. 37 ends. As a result, in the example of FIG. 38, the deformation areas TA1, TA2, and TA6 remain without being excluded, and the overlap of the deformation areas TA is eliminated.

  Here, in order to perform the deformation process on both deformation areas TA that partially overlap each other on the target image TI, the above-described one area is used to deform one area first and then deform the other area. It is necessary to store the image data after the deformation separately from the original image (the target image TI in a state where none of the deformation areas TA is deformed). For this reason, the amount of memory used is enormous. In the present embodiment, the exclusion determination unit 270 selects one deformation area TA from the plurality of deformation areas TA in descending order of the size of the face area FA, and another deformation area overlapping the selected deformation area TA. By repeatedly performing the process of excluding the TA from the deformation process target, the deformation areas TA to be deformed are prevented from overlapping. Therefore, according to the present embodiment, a large amount of memory as described above due to the overlapping of the deformation areas TA, while leaving a face that is assumed to be larger and more important to the user as a target for deformation processing. Can be eliminated.

  Here, the method of eliminating the overlap of the deformation areas TA is not limited to the method described above. For example, when selecting one deformation area TA to eliminate the overlap of the deformation areas TA, the exclusion determination unit 270 does not select the size of the face area FA as a reference, but uses the deformation area TA input from the outside. It may be selected according to the selection instruction. That is, the exclusion determination unit 270 receives a selection instruction by the user's operation of the operation unit 140. In this case, the exclusion determination unit 270 instructs the display processing unit 310 to display on the display unit 150 a user interface that can arbitrarily select the deformation area TA on the target image TI, and the user performs the user interface via the user interface. The deformation area TA is selected according to the input selection instruction. When there is another deformation area TA that overlaps the selected deformation area TA, all the other deformation areas TA that overlap are determined to be excluded from the deformation processing targets. With such a configuration, it is possible to eliminate the overlap between the deformation areas TA while leaving the deformation area TA selected by the user as a target of the deformation process.

  In S176, as a sixth exclusion determination, the exclusion determination unit 270 determines whether or not the number of deformation areas TA on the target image TI exceeds a predetermined threshold value (maximum number of processes), and the threshold value. When the value exceeds the value, the number of deformation areas TA that fall within the threshold value is selected, and the non-selected deformation areas TA are determined to be excluded from the deformation processing target. The determination is performed on the deformation area TA that has not been determined to be excluded from the deformation processing target in S171 to S175. Although various numerical values can be used as the threshold value, in the present embodiment, the threshold value is set to “5” as an example in consideration of the processing capability of the printer 100.

  When the number of deformation areas TA on the target image TI exceeds the threshold value, the exclusion determination unit 270 selects, for example, the number of deformation areas TA within the threshold value in descending order of the size of the face area FA. To do. As a result, the number of deformation areas TA exceeding the processing capability of the printer 100 are targeted for deformation processing while leaving a face that is assumed to be large in size and more important to the user as a deformation processing target. It can be avoided.

  In S177, as a seventh exclusion determination, the exclusion determination unit 270 determines whether or not there is a deformation area TA in which at least a part of the trimmed image cut out based on the trimming instruction input from the outside exists. In this case, a determination is made to exclude a deformation area TA at least a part of which is from the trimmed image from being subjected to deformation processing. The determination is performed for each deformation area TA that has not been determined to be excluded from the deformation processing target in S171 to S176. In S172, the deformation area TA that protrudes from the common image range CA that remains in common when the target image TI is trimmed based on the aspect ratios of all print sheets usable by the printer 100 is excluded. Then, the deformation area TA that protrudes from the trimming range that can be arbitrarily set by the user is also excluded from the object of the deformation process.

  In step S177, the exclusion determination unit 270 acquires an external trimming instruction. In this case, when an instruction for trimming the target image TI has already been given by an input operation via the operation unit 140 by the user, an image range to be cut out by trimming is set on the target image TI based on the instruction. . Alternatively, the exclusion determination unit 270 instructs the display processing unit 310 to display a user interface for performing trimming settings for the target image TI on the display unit 150, and performs trimming input by the user via the user interface. Based on the instruction, an image range by trimming may be set on the target image TI. The exclusion determination unit 270 determines, for each deformation area TA that is a determination target, whether or not the trimming image range that has been set out of the set trimming image range, and excludes a deformation area TA that partially protrudes from the target of the deformation process. Make a decision to do.

  According to the process of S177, only the deformation area TA that is entirely included in the image range cut out by the trimming instructed by the user can be the target of the deformation process. Therefore, it is necessary to reset the deformation area TA as described above. Thus, an increase in processing load and an increase in memory consumption in the printer 100 can be avoided. However, the process of S177 is not executed when an instruction for trimming cannot be obtained from the outside.

  As described above, in the present embodiment, the first to seventh exclusion determinations described above are performed in the order described above, and the deformation area TA that is finally left without being excluded by any exclusion determination is targeted. Processing is performed. However, the order of the first to seventh exclusion determinations described above is merely an example, and the exclusion determination unit 270 can perform the processing of each exclusion determination in any order. It is also possible to carry out only a part of the first to seventh exclusion determinations.

(2-3) From deformation area division to print processing:
In S180 (FIG. 4), the deformation area dividing unit 250 (FIG. 1) divides the deformation area TA into a plurality of small areas. FIG. 19 is an explanatory diagram illustrating an example of a method of dividing the deformation area TA into small areas. The deformation area dividing unit 250 arranges a plurality of division points D in the deformation area TA, and divides the deformation area TA into a plurality of small areas using a straight line connecting the division points D. When there are a plurality of deformation areas TA, division into small areas is performed for each deformation area TA.

  The arrangement mode of the dividing points D (the number and position of the dividing points D) is defined in association with the deformation type set in S120 (FIG. 4) by the dividing point arrangement pattern table 410 (FIG. 1). The deformation area dividing unit 250 refers to the dividing point arrangement pattern table 410 and arranges the dividing points D in a manner associated with the deformation type set in S120. In the present embodiment, as described above, the deformation “type A” (see FIG. 5) for sharpening the face is set as the deformation type, and therefore the division point D is used in a manner associated with this deformation type. Is placed.

  As shown in FIG. 19, the dividing points D are arranged at the intersections of the horizontal dividing line Lh and the vertical dividing line Lv and at the intersections of the horizontal dividing line Lh and the vertical dividing line Lv and the outer frame of the deformation area TA. . Here, the horizontal dividing line Lh and the vertical dividing line Lv are reference lines for arranging the dividing points D in the deformation area TA. As shown in FIG. 19, in the arrangement of the dividing points D associated with the deformation type for sharpening the face, two horizontal dividing lines Lh perpendicular to the reference line RL and 4 parallel to the reference line RL are used. A vertical division line Lv is set. The two horizontal dividing lines Lh are called Lh1 and Lh2 in order from the bottom of the deformation area TA. The four vertical dividing lines Lv are referred to as Lv1, Lv2, Lv3, and Lv4 in order from the left of the deformation area TA.

  The horizontal dividing line Lh1 is arranged below the chin image in the deformation area TA, and the horizontal dividing line Lh2 is arranged near the eye image. The vertical dividing lines Lv1 and Lv4 are arranged outside the cheek line image, and the vertical dividing lines Lv2 and Lv3 are arranged outside the eye corner image. The horizontal dividing line Lh and the vertical dividing line Lv are arranged in the deformation area TA set in advance so that the positional relationship between the horizontal dividing line Lh and the vertical dividing line Lv and the image becomes the above-described positional relationship as a result. It is executed according to the correspondence with the size.

  According to the arrangement of the horizontal dividing line Lh and the vertical dividing line Lv described above, the intersection of the horizontal dividing line Lh and the vertical dividing line Lv, and the intersection of the horizontal dividing line Lh and the vertical dividing line Lv and the outer frame of the deformation area TA In addition, the dividing point D is arranged. As shown in FIG. 19, the division points D located on the horizontal division line Lhi (i = 1 or 2) are called D0i, D1i, D2i, D3i, D4i, and D5i in order from the left. For example, the dividing points D located on the horizontal dividing line Lh1 are called D01, D11, D21, D31, D41, D51. Similarly, the dividing points D located on the vertical dividing line Lvj (j = 1, 2, 3, 4) are called Dj0, Dj1, Dj2, Dj3 in order from the bottom. For example, the division points D located on the vertical division line Lv1 are called D10, D11, D12, and D13.

As shown in FIG. 19, the arrangement of the dividing points D in the present embodiment is symmetrical with respect to the reference line RL.
The deformation area dividing unit 250 divides the deformation area TA into a plurality of small areas by a straight line connecting the arranged dividing points D (that is, the horizontal dividing line Lh and the vertical dividing line Lv). In this embodiment, as shown in FIG. 19, the deformation area TA is divided into 15 rectangular small areas.
In the present embodiment, since the arrangement of the dividing points D is determined by the numbers and positions of the horizontal dividing lines Lh and the vertical dividing lines Lv, the dividing point arrangement pattern table 410 has the numbers and positions of the horizontal dividing lines Lh and the vertical dividing lines Lv. In other words, it is possible to define that

  In S190 (FIG. 4), the deformation processing unit 260 (FIG. 1) performs an image deformation process on the deformation area TA of the target image TI. The deformation process by the deformation processing unit 260 is performed by moving the position of the division point D arranged in the deformation area TA in S180 and deforming the small area. The process of S190 is also performed for each deformation area TA when there are a plurality of deformation areas TA.

  The movement mode (movement direction and movement distance) of the position of each division point D for the deformation process is determined based on the deformation type and the degree of deformation set in S120 (FIG. 4) by the division point movement table 420 (FIG. 1). It is determined in advance in association with these combinations. The deformation processing unit 260 refers to the dividing point moving table 420 and moves the position of the dividing point D with the moving direction and moving distance associated with the combination of the deformation type and the degree of deformation set in S120.

  In this embodiment, as described above, the deformation “type A” (see FIG. 5) for sharpening the face is set as the deformation type, and the degree of “medium” is set as the deformation degree. The position of the dividing point D is moved with the movement direction and the movement distance associated with the combination of these deformation types and deformation degrees.

  FIG. 20 is an explanatory diagram showing an example of the contents of the dividing point movement table 420. FIG. 21 is an explanatory diagram showing an example of movement of the position of the division point D according to the division point movement table 420. In FIG. 20, the movement type of the position of the dividing point D defined by the dividing point movement table 420 is associated with the combination of the deformation type for sharpening the face and the degree of deformation of the degree “medium”. The movement mode is shown. As shown in FIG. 20, in the dividing point movement table 420, for each dividing point D, the amount of movement along the direction perpendicular to the reference line RL (H direction) and the direction parallel to the reference line RL (V direction) is displayed. It is shown. In this embodiment, the unit of movement amount shown in the dividing point movement table 420 is the pixel pitch PP of the target image TI. For the H direction, the amount of movement to the right side is represented as a positive value, the amount of movement to the left side is represented as a negative value, and for the V direction, the amount of movement upward of the face is represented. Expressed as a positive value, the amount of downward movement of the face is expressed as a negative value. For example, the dividing point D11 is moved to the right along the H direction by a distance that is seven times the pixel pitch PP, and is moved upward along the V direction by a distance that is 14 times the pixel pitch PP. For example, the division point D22 is not moved because the movement amount is zero in both the H direction and the V direction.

  In the present embodiment, the division points D (for example, the division points D10 shown in FIG. 21) located on the outer frame of the deformation area TA are set so that the boundary between the inner and outer images of the deformation area TA does not become unnatural. The position is not moved. Therefore, in the dividing point movement table 420 shown in FIG. 20, the movement mode for the dividing point D located on the outer frame of the deformation area TA is not defined.

  In FIG. 21, the division point D before the movement is indicated by a white circle, and the division point D after the movement or the division point D without the movement of the position is indicated by a black circle. Further, the divided point D after the movement is referred to as a divided point D ′. For example, the position of the dividing point D11 is moved in the upper right direction in FIG. 21 to become a dividing point D′ 11.

  In the present embodiment, all combinations of two division points D (for example, combinations of division points D11 and D41) that are in a symmetric positional relationship with respect to the reference line RL are the same after the movement of the division point D. The movement mode is determined so as to maintain a symmetrical positional relationship with respect to the line RL.

  For each small area constituting the deformation area TA, the deformation processing unit 260 converts the small area image before the position movement of the dividing point D into an image of the small area newly defined by the position movement of the dividing point D. In this way, an image deformation process is performed. For example, in FIG. 21, an image of a small region (small region indicated by hatching) having vertices at the division points D11, D21, D22, and D12 is divided into points D′ 11, D′ 21, D22, and D′ 12. Is transformed into an image of a small area with the vertex at.

  FIG. 22 is an explanatory diagram showing the concept of an image deformation processing method by the deformation processing unit 260. In FIG. 22, the division point D is indicated by a black circle. In FIG. 22, for simplification of description, regarding the four small regions, the state before the position movement of the dividing point D is shown on the left side, and the state after the position movement of the dividing point D is shown on the right side. In the example of FIG. 22, the central dividing point Da is moved to the position of the dividing point Da ′, and the positions of the other dividing points D are not moved. Thereby, for example, an image of a rectangular small area (hereinafter also referred to as “pre-deformation noticeable small area BSA”) having the vertices at the division points Da, Db, Dc, Dd before the movement of the division point D is obtained from the division point Da ′. , Db, Dc, and Dd are transformed into an image of a rectangular small area (hereinafter also referred to as “the noticed small area ASA after deformation”).

  In this embodiment, a rectangular small region is divided into four triangular regions using the center of gravity CG of the small region, and image deformation processing is performed in units of triangular regions. In the example of FIG. 22, the pre-deformation attention small area BSA is divided into four triangular areas having the centroid CG of the pre-deformation attention small area BSA as one vertex. Similarly, the post-deformation attention small area ASA is divided into four triangular areas having the centroid CG ′ of the post-deformation attention small area ASA as one vertex. Then, image deformation processing is performed for each corresponding triangular area in each state before and after the movement of the dividing point Da. For example, an image of a triangular area having vertices at the division points Da and Dd and the center of gravity CG in the attention small area BSA before deformation has a vertex at the division points Da ′ and Dd and the center of gravity CG ′ in the attention small area ASA after deformation. It is transformed into an image of a triangular area.

  FIG. 23 is an explanatory diagram showing the concept of the image deformation processing method in the triangular area. In the example of FIG. 23, the image of the triangular area stu with the points s, t, and u as vertices is transformed into the image of the triangular area s′t′u ′ with the points s ′, t ′, and u ′ as vertices. . For the deformation of the image, the position of a certain pixel in the image of the triangular area s't'u 'after the deformation corresponds to the position in the image of the triangular area stu before the deformation, and the calculated position This is performed by setting the pixel value in the image before the deformation in FIG.

  For example, in FIG. 23, the position of the pixel of interest p ′ in the image of the triangular area s′t′u ′ after deformation corresponds to the position p in the image of the triangular area stu before deformation. The position p is calculated as follows. First, coefficients m1 and m2 for expressing the position of the target pixel p ′ by the sum of the vector s′t ′ and the vector s′u ′ as shown in the following equation (1) are calculated.

  Next, by using the calculated coefficients m1 and m2, the position p is obtained by calculating the sum of the vector st and the vector su in the triangular area stu before deformation by the following equation (2).

  When the position p in the triangular area stu before deformation coincides with the pixel center position of the image before deformation, the pixel value of the pixel is set as the pixel value of the image after deformation. On the other hand, when the position p in the triangular area stu before deformation is shifted from the pixel center position of the image before deformation, an interpolation operation such as bicubic using the pixel values of the pixels around the position p. Thus, the pixel value at the position p is calculated, and the calculated pixel value is set as the pixel value of the image after deformation.

  Image deformation from the image of the triangular area stu to the image of the triangular area s't'u 'by calculating the pixel value for each pixel in the image of the triangular area s't'u' after the deformation as described above Processing can be performed. The deformation processing unit 260 performs a deformation process by defining a triangular area as described above for each small area constituting the deformation area TA shown in FIG. 21, and performs an image deformation process in the deformation area TA.

  Here, the face shape correction mode of the present embodiment will be described in more detail. FIG. 24 is an explanatory diagram showing a form of face shape correction in the present embodiment. In this embodiment, as described above, the deformation “type A” (see FIG. 5) for sharpening the face is set as the deformation type, and the degree of “medium” is set as the deformation degree. In FIG. 24, the image of the deformation | transformation aspect of each small area | region which comprises deformation | transformation area | region TA is shown with the arrow.

  As shown in FIG. 24, in the face shape correction of the present embodiment, the dividing points D (D11, D21, D31, D41) arranged on the horizontal dividing line Lh1 with respect to the direction (V direction) parallel to the reference line RL. Is moved upward, while the positions of the dividing points D (D12, D22, D32, D42) arranged on the horizontal dividing line Lh2 are not moved (see FIG. 20). Therefore, the image located between the horizontal dividing line Lh1 and the horizontal dividing line Lh2 is reduced in the V direction. As described above, since the horizontal dividing line Lh1 is arranged below the chin image and the horizontal dividing line Lh2 is arranged in the vicinity immediately below the eye image, in the face shape correction of this embodiment, the face image is corrected. The image of the part from the inner jaw to the lower part of the eye is reduced in the V direction. As a result, the jaw line in the image moves upward.

  On the other hand, in the direction (H direction) perpendicular to the reference line RL, the position of the dividing point D (D11, D12) arranged on the vertical dividing line Lv1 is moved rightward and arranged on the vertical dividing line Lv4. The position of the divided point D (D41, D42) is moved to the left (see FIG. 20). Further, of the two division points D arranged on the vertical division line Lv2, the position of the division point D (D21) arranged on the horizontal division line Lh1 is moved rightward and arranged on the vertical division line Lv3. Of the two divided points D, the position of the divided point D (D31) arranged on the horizontal dividing line Lh1 is moved to the left (see FIG. 20). Therefore, the image located on the left side of the vertical dividing line Lv1 is enlarged on the right side in the H direction, and the image located on the right side of the vertical dividing line Lv4 is enlarged on the left side. An image located between the vertical dividing line Lv1 and the vertical dividing line Lv2 is reduced or moved to the right in the H direction, and an image located between the vertical dividing line Lv3 and the vertical dividing line Lv4 is moved in the H direction. Is reduced or moved to the left. Further, the image located between the vertical division line Lv2 and the vertical division line Lv3 is reduced in the H direction with the position of the horizontal division line Lh1 as the center.

  As described above, the vertical division lines Lv1 and Lv4 are arranged outside the cheek line image, and the vertical division lines Lv2 and Lv3 are arranged outside the corner image. Therefore, in the face shape correction according to the present embodiment, the image of the portion outside the both corners of the face image is reduced in the H direction as a whole. In particular, the reduction rate is high near the jaw. As a result, the shape of the face in the image becomes thinner in the width direction as a whole.

  When the deformation modes in the H direction and the V direction described above are combined, the face shape in the target image TI is sharpened by the face shape correction of this embodiment. Note that a sharp face shape can be expressed as a so-called “small face”.

  Note that the small regions (hatched regions) having the vertexes at the dividing points D22, D32, D33, and D23 shown in FIG. 24 are both eyes according to the arrangement method of the horizontal dividing lines Lh2 and the vertical dividing lines Lv2 and Lv3 described above. This area includes the image. As shown in FIG. 20, since the division points D22 and D32 are not moved in the H direction or the V direction, the small region including the images of both eyes is not deformed. As described above, in this embodiment, the small area including the images of both eyes is not deformed, and the image after the face shape correction is made more natural and preferable.

  In S200 (FIG. 4), the face shape correction unit 200 (FIG. 1) instructs the display processing unit 310 to display the target image TI after the face shape correction on the display unit 150. FIG. 25 is an explanatory diagram illustrating an example of a state of the display unit 150 on which the target image TI after face shape correction is displayed. The display unit 150 displaying the target image TI after the face shape correction allows the user to check the correction result. If the user is not satisfied with the correction result and selects the “Return” button, for example, the display unit 150 displays a screen for selecting the deformation type and the deformation degree shown in FIG. The setting is performed again. When the user is satisfied with the correction result and selects the “print” button, the following corrected image printing process is started.

  In S300 (FIG. 3), the print processing unit 320 (FIG. 1) controls the printer engine 160 to print the target image TI after the face shape correction process. FIG. 26 is a flowchart showing the flow of the corrected image printing process in the present embodiment. The print processing unit 320 converts the resolution of the image data of the target image TI after the face shape correction processing into a resolution suitable for the print processing by the printer engine 160 (S310), and the image data after the resolution conversion is converted to the printer engine 160. Is converted into ink color image data expressed in gradation with a plurality of ink colors used in printing (S320). In this embodiment, it is assumed that the plurality of ink colors used for printing in the printer engine 160 are four colors of cyan (C), magenta (M), yellow (Y), and black (K). Further, the print processing unit 320 generates dot data indicating the ink dot formation state for each print pixel by executing halftone processing based on the gradation value of each ink color in the ink color image data (S330). ), Dot data is arranged to generate print data (S340). The print processing unit 320 supplies the generated print data to the printer engine 160, and causes the printer engine 160 to print the target image TI (S350). Thereby, the printing of the target image TI after the face shape correction is completed.

  As described above, in the face shape correction printing process by the printer 100 of the present embodiment, a plurality of division points D are arranged in the deformation area TA set on the target image TI, and a straight line ( The deformation area TA is divided into a plurality of small areas using the horizontal dividing line Lh and the vertical dividing line Lv). Further, the position of the dividing point D is moved and the small area is deformed, whereby the deformation process of the image in the deformation area TA is executed. As described above, in the face shape correction printing process by the printer 100 according to the present embodiment, the image can be deformed simply by arranging the dividing point D in the deformation area TA and moving the arranged dividing point D. Image deformation corresponding to various deformation modes can be easily and efficiently realized.

  Further, according to the present embodiment, among the deformation areas TA set on the target image TI based on the face area detection, none of the above-described exclusion determinations has been determined to be excluded from the deformation processing target. Image deformation processing is performed on the deformation area TA. For this reason, the excessive processing load on the CPU 110 of the printer 100 and the memory, which has been conventionally caused by executing the deformation process on all the deformation areas TA set on the target image TI based on the face area detection. Mass consumption can be appropriately suppressed. In particular, when the printer 100 is a multifunction device as described above, the resources that can be secured for the face shape correction printing process are limited, and thus the deformation area TA to be subjected to the deformation process is limited as in this embodiment. This leads to the realization of stable operation of the printer 100.

(3) Additional description and other examples:
As described in the above description of S171, the memory consumption required for the deformation process of the deformation area TA increases as the inclination (angle β) of the face area FA approaches a predetermined upper limit angle (angle βL). explain.
FIG. 39A shows a part of the deformation area TA in which the angle β formed by the vertical reference line Vs and the reference line RL is 0 degree, and FIG. 39B shows the vertical reference line Vs and the reference line. A part of the deformation area TA in which the angle β formed by the RL is a predetermined angle (for example, an angle β2) larger than 0 degree is illustrated.

  Here, in the process of the deformation process of S190, the deformation processing unit 260 sequentially generates data of the target image TI after the deformation in units of rows (pixel rows) orthogonal to the vertical reference line Vs. That is, a single target image that has been deformed by sequentially performing, for each pixel row, a process of acquiring pixel values corresponding to each pixel of the deformed image from the target image TI (original image) before the deformation. TI is finally obtained. In the process of generating each pixel row, the printer 100 performs processing on the original image necessary for acquiring the pixel value corresponding to each pixel of the one pixel row during the process of generating one pixel row. The predetermined range of data is temporarily stored in a predetermined buffer area in the internal memory 200 or the like and used for deformation processing. Accordingly, the wider the range on the original image necessary for obtaining the pixel value corresponding to each pixel in one pixel row after the deformation, the larger the memory consumption in the printer 100.

39A and 39B, the lower left area of the deformation area TA including the dividing points D11 and D21 is shown in the center. The dividing points D11 and D21 (the same applies to the dividing points D31 and 41 on the right side of the reference line RL) are the dividing points having the longest moving distance d before and after the deformation in the deformation processing of the present embodiment (the dividing point moving table 420 is stored). Therefore, the difference in the amount of memory required for the deformation process will be described using the movement of the dividing point D having a long moving distance as an example.
For example, in the deformation process of S190, in order to generate the pixel row L1 including the division point D′ 21 shown in FIG. 39A, at least on the original image related to the division point D21 before the division point D′ 21 is moved. Data is required. Further, in order to generate the pixel row L1, since it is necessary to obtain pixel values from the original image for other pixels on the pixel row L1 other than the dividing point D′ 21, the same as the pixel row L1 in the original image. Data on the pixel row at the position and data on a pixel row that is not as far away from the dividing point D21 but is some distance from the pixel row L1 are required.

  Therefore, when the pixel row L1 is generated, the image data corresponding to the width (Vband1) from the pixel row at the same position as the pixel row L1 to the pixel row including the dividing point D21 is the data in the original image. As described above, it is temporarily stored in the buff area and used for the deformation process. On the other hand, as shown in FIG. 39B, in a situation where the deformation area TA is inclined on the target image TI, the movement of the dividing point D′ 21 is also generated when the pixel row L2 including the dividing point D′ 21 is generated. Data on the original image related to the previous division point D21 is required. Therefore, when the pixel row L2 is generated, the image data corresponding to the width (Vband2) from the pixel row at the same position as the pixel row L2 to the pixel row including the division point D21 is the data in the original image. The data is temporarily stored in the buff area and used for deformation processing.

  Whether the deformation area TA is inclined or not inclined with respect to the vertical reference line Vs, the moving distance d of each division point D does not change. However, in the deformation process of the present embodiment, as can be seen from the description of FIG. 24 and the like, most of the points that move before and after the deformation are not in a direction parallel to the reference line RL, but oblique to the reference line RL. Move in the direction of. The dividing point D21 also moves in the upper right direction within the deformation area TA, not in parallel with the reference line RL. Therefore, when the deformation area TA is inclined with respect to the vertical reference line Vs as shown in FIG. 39B, the moving direction of the dividing point D21 approaches a state parallel to the vertical reference line Vs, and the dividing points D21, The distance in the direction parallel to the vertical reference line Vs between D′ 21 becomes longer. That is, as shown in FIGS. 39A and 39B, when the deformation area TA has an inclination of 0 degree with respect to the vertical reference line Vs, the buffer necessary for generating the pixel row L1 including the dividing point D′ 21. The area may be a capacity capable of storing image data of Vband1 in the original image. However, when the deformation area TA has an inclination of β2 with respect to the vertical reference line Vs, the generation of the pixel row L2 including the dividing point D′ 21 The necessary buffer area has a capacity capable of storing image data of Vband2 having a larger bandwidth than Vband1.

  In the present embodiment, the deformation process in the deformation area TA is performed symmetrically with respect to the central reference line RL. Therefore, as shown in FIG. 39B, when the deformation area TA is tilted with respect to the vertical reference line Vs, the left and right division points D (for example, the division point D21 and the division point) that are symmetrical with respect to the reference line RL. Regarding the relationship with the point D31), one movement direction is substantially parallel to the vertical reference line Vs, and the other movement direction is substantially perpendicular to the vertical reference line Vs. However, since the size of the buffer area that must be secured for the image transformation process must correspond to the required maximum amount, eventually, the size of FIG. In the case of b), the size of the required buffer area becomes larger when the deformation process is performed. In the above description, the relationship between the reference line RL and the vertical reference line Vs has been described. The same applies to the relationship between the reference line RL and the horizontal reference line Hs. That is, the division points D11 and 21 are more when the angle β between the reference line RL and the horizontal reference line Hs is generated than when the reference line RL is parallel to the horizontal reference line Hs. This is because the moving direction of (or the dividing points D31, 42) approaches a state parallel to the vertical reference line Vs.

  Thus, in this embodiment, even if the size of the face area FA is the same, the movement direction of the dividing point D before and after the deformation is caused by the inclination of the face area FA (the deformation area TA) on the target image TI. In other words, the more parallel the vertical reference line Vs or the horizontal reference line Hs is, the more memory is required for the deformation process. Therefore, in this embodiment, as described above, the closer the inclination (angle β) of the face area FA is to the upper limit angle βL, the smaller the upper limit value of the size of the face area FA that can be subjected to deformation processing is. This is intended to reduce memory consumption.

  The value of the upper limit angle βL can be defined based on the moving direction in the deformation area TA for the dividing point D having a long moving distance before and after the deformation. For example, based on the movement distance of the dividing point D21 defined in the dividing point movement table 420 in the H direction (direction orthogonal to the reference line RL) and the V direction (direction parallel to the reference line RL), The moving direction in the deformation area TA is specified, and the angle formed by the moving direction of the dividing point D21 and the V direction can be set as the upper limit angle βL. In addition, since the moving direction of each division point D in the deformation area TA may vary depending on the degree of deformation set in S120, the upper limit angle βL (face area size upper limit value table 430 defining the upper limit angle βL and the like). May be defined for each degree of deformation set in S120.

  Alternatively, the value of the upper limit angle βL may be set in consideration of the processing of S173 (third exclusion determination) performed after S171. In the present embodiment, after the third exclusion determination, the angle γ formed by the positive direction of the horizontal reference line Hs and the upward direction of the reference line RL is equal to the positive direction of the horizontal reference line Hs and the positive direction of the vertical reference line Vs. Only the deformation area TA belonging to the range of ± 15 degrees centering on the negative direction of the direction and the horizontal reference line Hs remains as the object of the deformation process. Therefore, the numerical value of 15 degrees may be used as the upper limit angle βL. In the situation where the upper limit angle βL is set to 15 degrees, if a value larger than 15 degrees is detected as the angle β in S1711 (FIG. 28), the deformation area TA in which the angle β is detected will be described. There is no need to continue the first exclusion determination any further. In this case, the deformation area TA in which the angle β is detected is determined to be excluded from the deformation target by the third exclusion determination.

  Further, even when the vertical direction of the face on the target image TI is varied, the range on the original image necessary for acquiring the pixel value corresponding to each pixel of the pixel row after deformation is enlarged. The burden on the deformation process increases. For example, a case where another deformation area TA (not shown) exists on the pixel row L1 shown in FIG. 39A at the right side or the left side of the deformation area TA shown in FIG. Suppose. If the direction above the other deformation area TA is the same as the direction above the deformation area TA shown in FIG. 39A (the direction above both deformation areas TA is an abbreviation of the target image TI). The range on the original image necessary for acquiring the pixel value of each pixel in the pixel row L1 is approximately a pixel row L1 and a pixel composed of a plurality of pixel rows below the pixel row L1. Limited to a bunch of lines. However, if the upward direction of the another deformation area TA is directed substantially downward, substantially right, or substantially left, the tendency of the movement direction of the pixels in the area differs for each deformation area TA. Therefore, the range on the original image necessary for obtaining the pixel value of each pixel in the pixel row L1 tends to be enlarged above and below the pixel row L1, and as a result, many buffers are required. . Therefore, in the present embodiment, as described above, the orientation of the face of the deformation area TA to be subjected to the deformation process is substantially unified (fourth exclusion determination), and the vertical direction of the face on the target image TI varies. This prevents an increase in memory consumption due to the above.

  As yet another embodiment, the exclusion determination unit 270 may detect available free space in the memory in the printer 100 one by one during the first to seventh exclusion determinations. If it is determined that the detected free space is sufficient and sufficient to perform the deformation process for all the deformation areas TA that are not excluded from the deformation process at that time, the process of S170 is in progress. And all the deformation areas TA that are not excluded from the deformation processing target at that time may be set as processing targets after S180.

The present invention is not limited to the above-described embodiments, and can be implemented in various modes without departing from the scope of the invention.
In the above-described embodiment, the face shape correction printing process by the printer 100 as the image processing apparatus has been described. For example, the face shape correction printing process is executed by the personal computer, and only the printing process (S300) is performed. May be executed by the printer. The printer 100 is not limited to an ink jet printer, and may be another type of printer, such as a laser printer or a sublimation printer.

DESCRIPTION OF SYMBOLS 100 ... Printer, 110 ... CPU, 120 ... Internal memory, 140 ... Operation part, 150 ... Display part, 160 ... Printer engine, 170 ... Card interface, 172 ... Card slot, 200 ... Face shape correction part, 210 ... Deformation mode setting , 212: designation acquisition unit, 220: face region detection unit, 230 ... face region adjustment unit, 232 ... reference region setting unit, 234 ... evaluation unit, 236 ... determination unit, 240 ... deformation region setting unit, 250 ... deformation region Division unit 260 ... Deformation processing unit 270 ... Exclusion determination unit 310 ... Display processing unit 320 ... Print processing unit 410 ... Division point arrangement pattern table 420 ... Division point movement table 430 ... Face area size upper limit value table

Claims (1)

An image processing apparatus capable of performing image transformation,
A detection unit for detecting a plurality of face regions from the target image ;
A candidate area setting unit that sets an area including one face area as a candidate area for transformation processing for each detected face area ;
When there is an overlap between candidate areas, a candidate area is selected according to a selection instruction input from the outside , and if there is another candidate area that overlaps with the selected candidate area, the other overlap An exclusion discriminating unit for excluding the candidate region from the target of the deformation process;
A deformation processing unit that performs deformation of an image in a region for candidate regions other than the excluded candidate region,
An image processing apparatus comprising:

Images