JP2012103941A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize an image processing device that performs correction to the frontal eye line with natural facial expression.SOLUTION: After an around-eye image MG is cut out from a frontal-eye-line face image SG obtained by photographing in a state where a user is looking straight at a camera (imaging means 12), when a deviated-eye-line face image HG obtained by photographing the face in a state where the user is gazing at a display screen of display means 14 is acquired, if the photographing distance in photographing the deviated-eye-line face image HG is smaller than a predetermined distance, the cut-out around-eye image MG is combined with the acquired deviated-eye-line face image HG and they are displayed on the screen of the display means 14, so that correction to the frontal eye line with natural facial expression can be performed. Also, since composition with the deviated-eye-line face image HG is performed using the around-eye image MG including a facial contour, even when the photographing distance in photographing the around-eye image MG is different from the photographing distance in photographing the deviated-eye-line face image HG, thanks to many feature points, correction to the frontal eye line with natural facial expression can be performed by scaling.

Description

  The present invention relates to an image processing apparatus and an image processing method suitable for use in, for example, a portable terminal having a photographing function.

  In a portable terminal that embodies a videophone using a photographing function, a self-portrait camera and a display unit are arranged at different locations on the same plane, so that the line of sight of callers does not match. As a technique for eliminating the disagreement between the line of sight of the callers, for example, in Patent Document 1, the image data for the normal eye prepared in advance is enlarged or reduced according to the orientation of the face estimated from the image signal to be transmitted, There is disclosed a technique for correcting the line of sight by replacing the enlarged and reduced normal eye image data with the eye part of the face region extracted from the image signal.

  In recent years, a portable terminal of a type that realizes a function called a hand mirror using a photographing function has been developed. As shown in FIG. 10, the hand mirror function is that the camera C for self-portrait and the display unit D are arranged in different places on the same plane, and the face taken by the camera C is displayed on the screen of the display unit D. This is a display function and can be used as a hand mirror to check the makeup of the makeup.

JP 2000-137789 A

  As described above, in the hand mirror function, since the arrangement positions of the camera C and the display unit D are separated from each other, for example, in the example illustrated in FIG. 10, the user's face looking at the display screen of the display unit D is displayed. When taking a picture with the camera C, it is taken as a face that faces downward, that is, a face with a downturn, and is taken as an unnatural expression. Such adverse effects become more prominent as the position of the user's face approaches the display screen of the display unit D.

  Therefore, it is conceivable to correct the line of sight by applying the technique disclosed in the above-mentioned Patent Document 1, but simply enlarge the image data prepared in advance according to the orientation of the face estimated from the image signal. However, there is a problem that it is not possible to correct the normal vision line of the natural expression simply by replacing the enlarged and reduced normal vision image data with the eyes of the face area extracted from the image signal.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an image processing apparatus, an image processing method, and a program capable of correcting a natural expression to the normal line of sight.

  In order to achieve the above object, according to the first aspect of the present invention, a normal-view facial image acquisition unit that acquires a normal-view facial image by photographing a face of a normal view line, and a normal-view facial image acquired by the normal-view facial image acquisition unit. An eye-cutting unit that cuts out an eye part of the eye and generates an eye image; a biased face image acquisition unit that captures a biased face image by taking a face that is deviated from a state in which the line of sight is normal; Eye correction means for correcting the eye line in the eccentric face image by combining the eye image generated by the eye clipping means with the eye part in the eccentric face image acquired by the eye face image acquisition means. It is characterized by that.

  In the invention described in claim 2, a normal-view face image acquisition unit that acquires a normal-view face image by photographing a face with a normal view line, and a pupil portion of both eyes in the normal-view face image acquired by the normal-view face image acquisition unit A pupil cut-out means for generating a pupil image by cutting out an eye, a biased face image acquisition means for acquiring a biased face image by photographing a biased face deviating from a state in which the line of sight is normal, and acquisition of the biased face image Eye correction means for correcting the line of sight in the eccentric face image by synthesizing the pupil image generated by the pupil cutting means with the pupil portion of both eyes in the eccentric face image acquired by the means. It is characterized by.

  In the invention described in claim 3, the image is obtained by an eccentric face image acquiring unit that acquires an eccentric face image by photographing an eccentric face that deviates from a state in which the line of sight is normal. The determination means for determining the size of the eyeball and the rotation angle of the eyeball from the pupil in the biased face image, and according to the eyeball size and the rotation angle of the eyeball determined by the determination means, A pupil image generating means for generating a pupil image with a normal line of sight by performing rotation correction on the pupil, and combining the pupil image generated by the pupil image generating means with the binocular position in the eccentric face image Eye-line correcting means for correcting.

  In the invention according to claim 4 that is dependent on claim 3, the eye line correction unit measures the diameter of the pupil in the eccentric face image, and the pupil image generation unit performs the measurement according to the measured pupil diameter. Enlarging / reducing means for enlarging / reducing the size of the generated normal-eyeline pupil image, and synthesizing means for synthesizing the normal-eyeline pupil image enlarged / reduced by the enlarging / reducing means to the binocular position in the eccentric face image It is characterized by providing.

In the invention according to claim 5, which is dependent on any one of claims 1 to 3, the distance measuring means for measuring the shooting distance to the subject, and the distance measuring means when the eccentric face image is captured. And a control unit that executes the line-of-sight correction unit when the distance of the shooting distance is less than a predetermined distance, and that disables the line-of-sight correction unit when the shooting distance is equal to or greater than the predetermined distance. To do.

  In the invention according to claim 6, a normal face image acquisition step of acquiring a normal face image by photographing the face of the normal line of sight, and an eye part in the normal face image acquired by the normal face image acquisition step is cut out. Acquired by an eye clipping step that generates an eye image, an eccentric face image acquisition step that captures an eccentric face image that deviates from a state where the line of sight is normal, and acquires an eccentric face image, and the eccentric face image acquisition step. And a eye line correcting step of correcting the eye line in the eccentric face image by synthesizing the eye image generated by the eye cutting step with the eye part in the eccentric face image.

  In the invention according to claim 7, a normal-view face image acquisition step of capturing a normal-view face image by acquiring a normal-view face, and a binocular pupil portion in the normal-view image acquired by the normal-view image acquisition step A pupil cutting step for generating a pupil image by cutting out the image, a biased face image acquisition step for acquiring a biased face image by photographing a biased face that deviates from a state in which the eye is viewed normally, and acquiring the biased face image And a line of sight correction step of correcting the line of sight in the eccentric face image by synthesizing the pupil image generated in the pupil cutting step with the pupil portion of both eyes in the eccentric face image acquired in the step. It is characterized by.

  In the invention according to claim 8, the image is obtained by an eccentric face image acquisition step of acquiring an eccentric face image by photographing an eccentric face deviating from a state in which the line of sight is normal, and the eccentric face image acquisition step. The determination step of determining the size of the eyeball and the rotation angle of the eyeball from the pupil in the biased face image, and the size of the eyeball determined by the determination step and the rotation angle of the eyeball, A pupil image generation step for generating a pupil image with a normal line of sight by performing rotation correction on the pupil, and combining the pupil image generated by the pupil image generation step with a binocular position in the eccentric face image And a line of sight correction step for correcting.

  In the invention according to claim 9, which is dependent on claim 8, the eye line correcting step measures the diameter of the pupil in the eccentric face image and performs the pupil image generating step according to the measured pupil diameter. An enlargement / reduction step for enlarging / reducing the size of the generated pupil image of the normal line of sight, and a combining step for combining the pupil image of the normal line of sight enlarged / reduced by the enlargement / reduction step with the binocular position in the eccentric face image It is characterized by providing.

  In the invention according to claim 10 that depends on any one of claims 6 to 9, a distance measuring step for measuring a shooting distance to a subject and a distance measuring step when an eccentric face image is captured. A control step for executing the eye-correction step if the distance is less than a predetermined distance, and for not executing the eye-correction step if the imaging distance is greater than or equal to the predetermined distance. And

  In the present invention, it is possible to correct to a normal visual line of a natural expression.

1 is a block diagram illustrating an overall configuration of a photographing apparatus 100 according to a first embodiment. It is a flowchart which shows the operation | movement of the eye-gaze correction process by 1st Embodiment. It is a figure for demonstrating operation | movement of the eye-gaze correction process by 1st Embodiment. It is a block diagram which shows the whole structure of the imaging device 100 by 2nd Embodiment. It is a flowchart which shows the operation | movement of the eye-gaze correction process by 2nd Embodiment. It is a figure for demonstrating operation | movement of the eye-gaze correction process by 2nd Embodiment. It is a block diagram which shows the whole structure of the imaging device 100 by 3rd Embodiment. It is a flowchart which shows the operation | movement of the eye-gaze correction process by 3rd Embodiment. It is a figure for demonstrating operation | movement of the eye-gaze correction process by 3rd Embodiment. It is a figure for demonstrating background art.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
(1) Configuration FIG. 1 is a block diagram showing an overall configuration of a photographing apparatus 100 including an image processing apparatus according to the first embodiment. In this figure, the distance measuring means 11 is configured by an active method, for example, and measures the distance to the subject. When the distance measuring unit 11 is configured by a known passive method for detecting a phase difference or contrast, the distance is measured based on image data (imaging data) output from the imaging unit 12 described later. The imaging means 12 is composed of an individual imaging element such as a CCD, and generates image data obtained by photographing a subject that is focused based on the photographing distance measured by the distance measuring means 11.

  The control means 13 is composed of a CPU, a ROM, a RAM, and the like, and implements the functions of the eye cutout means 131 and the composition means 132 according to the gist of the present invention. That is, the eye cut-out means 131 cuts out the eye image MG from the user's orthographic face image SG (see FIG. 3) taken by the image pickup means 12. The orthographic face image SG indicates an image obtained by imaging the face of the user in a state where the camera (imaging means 12) is viewed normally. As shown in FIG. 3, the eye image MG refers to an image of the eye area including a part of the face outline and a part of the nose in the orthographic face image SG photographed by the imaging unit 12.

  The eye image MG cut by the eye cutting means 131 is stored and saved in a temporary storage means (RAM) not shown. In the synthesizing unit 132, the eye read from the temporary storage unit (RAM) to the eccentric face image HG (see FIG. 3) acquired by photographing the face of the user looking at the display screen of the display unit 14 with the imaging unit 12. The image MG is synthesized to generate a line-of-sight correction image. The prejudice face image HG refers to an image obtained by photographing a prejudice face deviating from a state in which the line of sight is viewed normally. The display unit 14 is composed of, for example, a color LCD panel, and displays a line-of-sight correction image output from the control unit 13 (compositing unit 132) on the screen.

(2) Operation Next, the operation of the first embodiment will be described with reference to FIGS. FIG. 2 is a flowchart showing the eye-line correction process performed by the control means 13 including the eye-cutting means 131 and the synthesizing means 132. When the line-of-sight correction process is executed, the control unit 13 advances the process to step S101, and instructs the user to look at the camera (imaging unit 12) from the front. The instruction to the user may be an aspect in which the instruction content is displayed on the screen as characters on the display unit 14, or may be an aspect in which the instruction content is voice-guided.

  Subsequently, in step S102, an eye-capturing process is performed in which a state in which the user views the camera (imaging unit 12) is viewed and an orthographic face image SG is acquired. In this eye-photographing process, when a state in which the user's face is normally viewed with respect to the camera is detected, in addition to a mode in which the normal-view facial image SG is automatically captured, the normal-view facial image SG is captured by a user operation (shutter button depression) It does not matter as a mode to do. The imaging range may be the entire face of the user or only the eye area of the face. When the acquired orthographic face image SG is supplied to the control means 13, as shown in FIG. 3, the eye cutout means 131 cuts out the eye image MG from the orthographic face image SG and stores it in the temporary storage means (RAM). To do.

  When the eye image MG has been cut out from the normal-view face image SG, the process proceeds to step S103, and a switching notification for switching to the above-described shooting mode for the hand mirror function is displayed on the display unit 14. Subsequently, in the next step S104, a face in a state where the user is staring at the display screen of the display unit 14 according to the switching notification, that is, an eccentric face deviating from a state where the line of sight is viewed from the right side is photographed to obtain an eccentric face image HG. get. Specifically, for example, in the example illustrated in FIG. Subsequently, in step S105, it is determined whether or not the shooting distance measured by the distance measuring unit 11 is less than a predetermined distance when the eccentric face image HG is shot.

  If the shooting distance is greater than or equal to the predetermined distance, the line of sight cannot be identified from the resolution of the captured image, or the line of sight does not deviate enough to correct the line of sight, so the determination result in step S105 is “NO”. This processing is finished. On the other hand, if the shooting distance is less than the predetermined distance, it is necessary to correct the line of sight, so the determination result in step S105 is “YES”, and the flow proceeds to step S106.

  In step S106, as shown in FIG. 3, the eye image MG read from the temporary storage means (RAM) by the synthesizing unit 132 is synthesized with the eccentric face image HG to generate a line-of-sight correction image, and the generated eye line is generated. The corrected image is displayed on the screen of the display means 14 and the present process ends. In order to synthesize the eye image MG with the eccentric face image HG, the eye image MG is adjusted so that the deviation of the contour line is minimized from the face contour, nose position, eye size, pupil size, and the like. The image is enlarged and reduced, overwritten and combined with the eccentric face image HG, and displayed on the display unit 14.

  As described above, in the first embodiment, after the user cuts out the eye image MG from the normal-view face image SG obtained by photographing the camera (imaging unit 12) with the front view, the user displays the display screen of the display unit 14. When the image of the face being stared is photographed to obtain the eccentric face image HG, if the photographing distance at the time of photographing the oblique face image HG is less than a predetermined distance, Since the cut-out eye image MG is synthesized and displayed on the display unit 14, it can be corrected to a normal visual line of sight with a natural expression. In addition, since the eye image MG including the face outline is combined with the eccentric face image HG, the shooting distance when the eye image MG is shot and the shooting distance when the bias face image HG is shot are calculated. Even if they are different, since there are many feature points, it can be corrected to a normal visual line of sight by scaling up and down.

  In the first embodiment described above, when the eye image MG is synthesized with the eccentric face image HG, the outline of the face is referred to. However, since the noise information such as hair is included in some cases, the face is not necessarily included. There is no need to refer to the contour of In addition, when the eye image MG is synthesized with the eccentric face image HG, the position of the nose is referred to, but it is not always necessary to refer to this. Similarly, in this embodiment, when the eye image MG is cut out from the orthographic face image SG, the eye area including a part of the face outline and a part of the nose shape is cut out, but the present invention is not limited thereto. The eye image MG of only the eye that does not include a part of the face or nose may be cut out.

[Second Embodiment]
Next, a second embodiment will be described with reference to FIGS.
(1) Configuration FIG. 4 is a block diagram showing an overall configuration of a photographing apparatus 100 including an image processing apparatus according to the second embodiment. In this figure, the same number is attached | subjected to the component which is common in 1st Embodiment mentioned above, and the description is abbreviate | omitted. The second embodiment is different from the first embodiment described above in that the control unit 13 includes a pupil cutting unit 133 and a synthesizing unit 134 including a pupil diameter measuring unit 134a and a white-eye painting unit 134b. . Hereinafter, a configuration different from the first embodiment will be described.

  The pupil cutout means 133 cuts out the pupil image EG from the user's orthographic face image SG (see FIG. 6) taken by the imaging means 12. The orthographic face image SG indicates an image obtained by imaging the face of the user in a state where the camera (imaging means 12) is viewed normally. As illustrated in FIG. 6, the pupil image EG indicates a binocular pupil image in the orthographic face image SG photographed by the imaging means 12. The pupil image EG cut by the pupil cutting unit 133 is stored and saved in a temporary storage unit (RAM) not shown.

  The pupil diameter measuring unit 134a includes a pupil diameter in an eccentric face image HG (see FIG. 6) acquired by photographing the face of the user looking at the display screen of the display unit 14 with the imaging unit 12, and pupil cutting unit. The diameter of the pupil in the pupil image EG cut out by 133 is measured. The white eye painting unit 134b paints the pupil in the eccentric face image HG with white eyes. A texture image prepared in advance is used for painting. In the synthesizing unit 134, the pupil of the pupil image EG is determined according to an enlargement / reduction ratio corresponding to the ratio of the pupil diameter in the eccentric face image HG measured by the pupil diameter measuring unit 134a and the pupil diameter in the pupil image EG. The enlarged and reduced pupil image EG is combined with the binocular position in the eccentric face image HG in which the eyes of both eyes are filled with white eyes by the white eye painting unit 134b. Needless to say, it is desirable to synthesize the pupil at the approximate center of the eye.

(2) Operation Next, the operation of the second embodiment will be described with reference to FIGS. FIG. 5 is a flowchart illustrating the eye-line correction process performed by the control unit 13 including the pupil cutting unit 133 and the synthesizing unit 134 including the pupil diameter measuring unit 134a and the white-eye painting unit 134b. When this process is executed, the control means 13 advances the process to step S201, and instructs the user to look at the camera (imaging means 12) from the front. The instruction to the user may be an aspect in which the instruction content is displayed on the screen as characters on the display unit 14, or may be an aspect in which the instruction content is voice-guided.

  Subsequently, in step S202, a pupil photographing process is performed in which a state in which the user views the camera (imaging unit 12) is photographed to obtain a stereographic face image SG (see FIG. 6). In this pupil photographing process, when a state in which the user's face is normally viewed with respect to the camera is detected, the normal facial image SG is captured by a user operation (shutter button depression) in addition to a mode in which the normal facial image SG is automatically captured. It does not matter as a mode to do. The imaging range may be the entire face of the user or only the eye area of the face. When the acquired orthographic face image SG is supplied to the control means 13, as shown in FIG. 6, the pupil cutting means 133 cuts out the pupil image EG from the orthographic face image SG and stores it in the temporary storage means (RAM). To do.

  When the extraction of the pupil image EG from the normal-view face image SG is completed, the process proceeds to step S203, and a switching notification indicating that the photographing mode for the hand mirror function is switched is displayed on the display unit 14 on the screen. Subsequently, in the next step S204, a face in a state where the user is gazing at the display screen of the display unit 14 according to the switching notification, that is, an eccentric face deviating from a state in which the line of sight is viewed from the right side is photographed. (See FIG. 6). Specifically, for example, in the example illustrated in FIG. Subsequently, in step S205, it is determined whether or not the photographing distance measured by the distance measuring unit 11 is less than a predetermined distance when the eccentric face image HG is photographed.

  If the shooting distance is greater than or equal to the predetermined distance, the line of sight cannot be identified from the resolution of the captured image, or the line of sight does not deviate enough to correct the line of sight, so the determination result in step S205 is “NO”. Thus, the present process is finished without correcting the line of sight. On the other hand, if the shooting distance is less than the predetermined distance, it is necessary to correct the line of sight, so the determination result in step S205 is “YES”, and the flow proceeds to step S206.

  In step S206, the pupils of both eyes in the eccentric face image HG are filled with white eyes using a texture image prepared in advance. In step S207, the diameter of the pupil in the eccentric face image HG and the diameter of the pupil in the pupil image EG are measured, and the pupil in the pupil image EG is determined according to the enlargement / reduction ratio corresponding to the ratio between the measured diameters. The pupil image EG of the enlarged and reduced pupil is synthesized with the binocular position in the eccentric face image HG in which the eyes of both eyes are filled with white eyes in the above-described step S205, and the display means 14 Display on the screen.

  As described above, in the second embodiment, after the user cuts out the pupil image EG from the normal-view face image SG obtained by photographing the camera (imaging unit 12) with the front view, the user displays the display screen of the display unit 14. When a face in a state of staring is photographed to obtain an oblique face image HG, if the photographing distance when the oblique face image HG is photographed is less than a predetermined distance, both eyes in the oblique face image HG The pupil of the image is filled with white eyes using a texture image prepared in advance, the diameter of the pupil in the eccentric face image HG and the diameter of the pupil in the pupil image EG are measured, and the ratio of both measured diameters is determined. According to the enlargement / reduction ratio, the size of the pupil in the pupil image EG is enlarged / reduced, and the pupil image EG of the enlarged / reduced pupil is converted into the binocular position in the eccentric face image HG in which the eyes of both eyes are filled with white eyes. Is displayed on the display means 14 so that it can be corrected to the natural line of sight. That. In addition, by correcting to the normal line of sight of a natural expression, it is possible to realize a hand mirror function in which the face of the user who is looking normal is displayed on the screen of the display unit 14.

  In the present embodiment, since only the pupil is cut out and synthesized, the synthesis area is minimized, so that it is possible to correct the eyesight to a more natural expression. Further, in the synthesis, it is necessary to consider the relationship between the pupil and the eyelid. In other words, when the eyelid covers the pupil, it is possible to perform synthesis by masking the eyelid to make the white eye transparent and concealing the eyelid. In such a situation, it may be considered that the display unit 14 is not viewed and the composition processing in step S207 is not performed.

[Third Embodiment]
Next, a third embodiment will be described with reference to FIGS.
(1) Configuration FIG. 7 is a block diagram showing an overall configuration of a photographing apparatus 100 including an image processing apparatus according to the third embodiment. In this figure, the same number is attached | subjected to the same component as 1st Embodiment mentioned above, and the description is abbreviate | omitted. The third embodiment is different from the first embodiment described above in that the control unit 13 includes a synthesizing unit 137 including an eyeball size and rotation angle determining unit 135, a pupil rotation correcting unit 136, and a white eye filling unit 137a. There is. Hereinafter, a configuration different from the first embodiment will be described.

  The eyeball size and rotation angle determination means 135 rotates the eyeball size and the eyeball from the pupil in the eccentric face image HG acquired by photographing the user's face looking at the display screen of the display means 14 by the imaging means 12. Determine the angle. This point will be described with reference to FIG. FIG. 9 is a diagram in which a line having the maximum width in the horizontal direction of the pupil has a horizontal diameter L, and a line having the maximum width in the vertical direction from the center of the horizontal diameter L is set to a vertical radius R. When the line of sight faces downward and becomes biased (oblique), the lateral diameter L of the pupil is not different from the normal vision state, but the vertical radius R in the biased state is smaller than the vertical radius R in the normal vision state. Assuming that the eyeball diameter is proportional to the pupil diameter, the size of the eyeball can be determined from the lateral diameter L of the pupil. Further, when it is considered that the eyeball has rotated as a sphere when changing from the normal vision state to the biased state, the rotation angle can be calculated from the ratio between the horizontal diameter L and the vertical radius R.

  The pupil rotation correction unit 136 corrects the pupil in the normal vision state based on the eyeball size and the rotation angle determined by the eyeball size and rotation angle determination unit 135. That is, as illustrated in FIG. 9, an angle in which the center of the lateral diameter L of the pupil is the approximate pupil center position, and the triangle T (or fan-shaped) image having the pupil center position as a vertex is in consideration of the eyeball diameter and the rotation angle. A pupil image in a normal vision state is generated by performing rotation correction. The white eye painting unit 137a paints the pupil in the eccentric face image HG with white eyes. A texture image prepared in advance is used for painting. In the synthesizing unit 137, the orthographic face image generated by the eyeball size and rotation angle determining unit 135 and the pupil rotation correcting unit 136 is converted into an eccentric face in which the eyes of both eyes are filled with white eyes by the white eye painting unit 137 a. The image is synthesized at the binocular position in the image HG.

(2) Operation Next, the operation of the third embodiment will be described with reference to FIGS. FIG. 8 is a flowchart showing the eye correction processing performed by the control unit 13 including the synthesizing unit 137 including the eyeball size and rotation angle determining unit 135, the pupil rotation correcting unit 136, and the white eye filling unit 137a. When this process is executed, in step S301, the control means 13 captures an image of a face in which the user is staring at the display screen of the display means 14, that is, an eccentric face deviated from a state in which the line of sight is not normal. A face image HG is acquired.

  When the eccentric face image HG is acquired, the process proceeds to step S302, and as shown in FIG. 9, the eyeball size and rotation angle determination unit 135 rotates the eyeball size and eyeball from the pupil in the eccentric face image HG. Determine the angle. Subsequently, in step S303, angle rotation correction is performed to correct the normal vision pupil based on the eyeball size and rotation angle determined in step S302, thereby generating a normal vision pupil image. Next, in step S304, it is determined whether or not the shooting distance measured by the distance measuring unit 11 is less than a predetermined distance when the eccentric face image HG is shot.

  If the shooting distance is greater than or equal to the predetermined distance, the line of sight cannot be identified from the resolution of the captured image, or the line of sight does not deviate enough to correct the line of sight, so the determination result in step S304 is “NO”. This processing is finished. On the other hand, if the shooting distance is less than the predetermined distance, the determination result in Step S304 is “YES”, and the process proceeds to Step S305. In step S305, the eyes of both eyes in the eccentric face image HG are filled with white eyes using a texture image prepared in advance. In step S306, the diameter of the pupil in the eccentric face image HG is measured, and the size of the pupil image in the normal vision state generated in step S303 is enlarged and reduced according to the measured pupil diameter. The reduced pupil image in the normal vision state is synthesized on the binocular position in the eccentric face image HG in which the eyes of both eyes are filled with white eyes and displayed on the display unit 14.

  As described above, in the third embodiment, when a user faces the display screen of the display unit 14 while photographing the face and acquires the eccentric face image HG, the eyeball is viewed from the pupil in the eccentric face image HG. And a rotation angle of the eyeball are determined, and an angle rotation correction is performed to correct the pupil in the normal vision state based on the determined eyeball size and rotation angle, thereby generating a normal vision pupil image. Then, if the shooting distance when shooting the eccentric face image HG is less than a predetermined distance, the eyes of both eyes in the eccentric face image HG are filled with white eyes using a texture image prepared in advance. The diameter of the pupil in the eccentric face image HG is measured, the size of the pupil image in the normal vision state is enlarged / reduced according to the measured diameter of the pupil, and the enlarged / reduced pupil image in the normal vision state is converted into the pupil of both eyes. Is combined with the positions of both eyes in the biased face image HG filled with white eyes and displayed on the display means 14, so that it can be corrected to a normal viewing line of a natural expression. In addition, by correcting to the normal line of sight of a natural expression, it is possible to realize a hand mirror function in which the face of the user who is looking normal is displayed on the screen of the display unit 14.

  In the third embodiment, the maximum width line in the horizontal direction of the pupil is the horizontal diameter L, the maximum width line in the vertical direction is the vertical radius R, and the center of the horizontal diameter L of the pupil is substantially the same. Although the processing is simplified with the pupil center position, strictly speaking, since the pupil center is located below the lateral diameter L, the corrected image may be distorted depending on the photographing angle. In such a case, the pupil center position may be corrected downward according to the size of the eyeball and the rotation angle, and a triangle T (or fan-shaped) image having this as a vertex may be acquired.

  Note that when the pupil image is combined with the eccentric face image HG, it is necessary to consider the relationship between the pupil and the eyelid. That is, when the eyelid in the eccentric face image HG covers the pupil, it is possible to perform synthesis by masking the eyelid to make the white eye portion transparent and hide the eyelid. In a situation where the eyes are covered with the eyes, it may be considered that the display unit 14 is not viewed and the combining process in step S207 is not performed.

  In addition, although the white eye is painted with the texture image prepared in advance by the white eye painting unit 137a, instead of this, the triangle T (or fan shape) including the white eye portion is included when the pupil is rotationally corrected. Then, the rotation may be corrected and composition may be performed in consideration of the eye lid. Furthermore, in the above-described embodiment, the rotation angle correction is performed in consideration of the eyeball diameter. However, the present invention is not limited to this, and the pupil having a flatness ratio determined by the ratio between the horizontal diameter L and the vertical radius R of the pupil is circular. It is good also as an aspect corrected. In the case of such correction, it is not necessary to correct with the triangle T (or fan shape) and develop it all around.

  In addition, in the first to third embodiments described above, for the purpose of simplifying the description, the bind-off eyes are illustrated as an example of the bias that deviates from the state where the line of sight is viewed from the front, but the gist of the present invention is limited to the cover-up eyes. Needless to say, the present invention can be applied to a deviation that deviates in one of the vertical and horizontal directions from a state in which the line of sight is viewed normally.

The following are additional notes regarding the present invention.
(Appendix 1)
On the computer,
An orthographic face image acquisition step of capturing the orthographic face image by acquiring the orthographic face image;
Eye cut-out step of cutting out the eye part in the normal face image acquired in the normal face image acquiring step and generating an eye image;
A biased face image acquisition step of capturing a biased face image by photographing a biased face that deviates from a state in which the line of sight is normal.
A line of sight correction step of correcting the line of sight in the eccentric face image by synthesizing the eye part image generated by the eye part cutting step with the eye part of the part of the eccentric face image acquired by the step of acquiring the eccentric face image. A program characterized by being executed.

(Appendix 2)
On the computer,
An orthographic face image acquisition step of capturing the orthographic face image by acquiring the orthographic face image;
A pupil cutting step of generating a pupil image by cutting out the pupil portion of both eyes in the orthographic face image acquired in the orthographic face image acquiring step;
A biased face image acquisition step of capturing a biased face image by photographing a biased face that deviates from a state in which the line of sight is normal.
A line of sight correction that corrects the line of sight in the eccentric face image by synthesizing the pupil image generated in the pupil cutting step with the pupil part of both eyes in the side of the eccentric face image acquired in the step of acquiring the eccentric face image. A program characterized by causing steps to be executed.

(Appendix 3)
On the computer,
A biased face image acquisition step of capturing a biased face image by photographing a biased face that deviates from a state in which the line of sight is normal.
A determination step of determining the size of the eyeball and the rotation angle of the eyeball from the pupil in the biased face image acquired in the biased face image acquisition step;
A pupil image generation step of generating a pupil image of the normal line of sight by performing rotation correction on the pupil in the eccentric face image according to the size of the eyeball determined in the determination step and the rotation angle of the eyeball;
A program for performing a line of sight correction step of correcting the line of sight by combining the pupil image generated in the pupil image generation step with the binocular position in the eccentric face image.

(Appendix 4)
The eye correction step includes
An enlargement / reduction step of measuring the diameter of the pupil in the prejudice face image, and enlarging / reducing the size of the pupil image of the normal line of sight generated in the pupil image generation step according to the measured pupil diameter,
The program according to claim 3, further comprising: a combining step of combining the pupil image of the normal line of sight enlarged and reduced in the enlargement / reduction step with the binocular position in the eccentric face image.

(Appendix 5)
A distance measuring step for measuring the shooting distance to the subject;
If the photographing distance measured in the distance measuring step is less than a predetermined distance when an eccentric face image is captured, the eye line correcting step is executed. On the other hand, if the photographing distance is not less than the predetermined distance, the eye line correction is performed. The program according to any one of appendices 1 to 3, further comprising: a control step that disables the step.

DESCRIPTION OF SYMBOLS 11 Distance measuring means 12 Imaging means 13 Control means 131 Eye segment cutting means 132 Compositing means 133 Pupil clipping means 134 Combining means 134a Pupil diameter measuring means 134b White eye filling means 135 Eyeball size and rotation angle determining means 136 Pupil rotation correcting means 137 Compositing Means 137a White-eye painting means 14 Display means 100 Imaging device

Claims (10)

A normal face image acquisition means for capturing a normal line of sight and acquiring a normal face image;
Eye segment cutting means for generating an eye image by cutting out an eye part in the orthographic face image acquired by the orthographic face image acquiring means;
A biased face image acquisition means for capturing a biased face image by photographing a biased face deviating from a state in which the line of sight is normal;
Eye correction means for correcting the eye in the eccentric face image by combining the eye image generated by the eye clipping means with the eye portion in the eccentric face image acquired by the eccentric face image acquiring means. An image processing apparatus comprising: A normal face image acquisition means for capturing a normal line of sight and acquiring a normal face image;
Pupil cutting means for cutting out the pupil part of both eyes in the orthographic face image acquired by the orthographic face image acquiring means and generating a pupil image;
A biased face image acquisition means for capturing a biased face image by photographing a biased face deviating from a state in which the line of sight is normal;
A line of sight correction that corrects the line of sight in the eccentric face image by synthesizing the pupil image generated by the pupil cutting unit with the pupil part of both eyes in the side of the eccentric face image acquired by the side of the eccentric face image. And an image processing apparatus. A biased face image acquisition means for capturing a biased face image by photographing a biased face deviating from a state in which the line of sight is normal;
Determination means for determining the size of the eyeball and the rotation angle of the eyeball from the pupil in the eccentric face image acquired by the eccentric face image acquisition means;
A pupil image generating unit that performs rotation correction on the pupil in the eccentric face image according to the size of the eyeball determined by the determining unit and the rotation angle of the eyeball, and generates a pupil image of the normal line of sight;
An image processing apparatus comprising: eye correction means for correcting eyes by synthesizing the pupil image generated by the pupil image generation means with the binocular position in the eccentric face image. The line of sight correction means includes
An enlargement / reduction unit that measures the diameter of the pupil in the prejudice face image and enlarges / reduces the size of the pupil image of the normal line of sight generated by the pupil image generation unit according to the measured pupil diameter;
The image processing apparatus according to claim 3, further comprising: a combining unit configured to combine the pupil image of the normal line of sight enlarged / reduced by the enlarging / reducing unit with a binocular position in the eccentric face image. Ranging means for measuring the shooting distance to the subject;
If the photographing distance measured by the distance measuring unit is less than a predetermined distance when the eccentric face image is captured, the eye line correcting unit is executed. On the other hand, if the photographing distance is not less than the predetermined distance, the eye line correction is performed. The image processing apparatus according to claim 1, further comprising a control unit that disables the unit. An orthographic face image acquisition step of capturing the orthographic face image by acquiring the orthographic face image;
An eye clipping step for generating an eye image by cutting out the eye part of the eye viewing face image acquired by the eye viewing face image acquiring step;
A biased face image acquisition step of capturing a biased face image by photographing a biased face that deviates from a state in which the line of sight is normal.
A line of sight correction step of correcting the line of sight in the eccentric face image by synthesizing the eye part image generated by the eye part cutting step with the eye part of the part of the eccentric face image acquired by the step of acquiring the eccentric face image. An image processing method comprising: An orthographic face image acquisition step of capturing the orthographic face image by acquiring the orthographic face image;
A pupil cutting step of generating a pupil image by cutting out the pupil portions of both eyes in the orthographic face image acquired by the orthographic face image acquiring step;
A biased face image acquisition step of capturing a biased face image by photographing a biased face that deviates from a state in which the line of sight is normal.
A line of sight correction that corrects the line of sight in the eccentric face image by synthesizing the pupil image generated by the pupil cutting step with the pupil part of both eyes in the side of the eccentric face image acquired by the step of acquiring the eccentric face image. An image processing method comprising the steps of: A biased face image acquisition step of capturing a biased face image by photographing a biased face that deviates from a state in which the line of sight is normal.
A determination step of determining the size of the eyeball and the rotation angle of the eyeball from the pupil in the eccentric face image acquired by the eccentric face image acquisition step;
A pupil image generating step of generating a pupil image of the normal line of sight by performing rotation correction on the pupil in the eccentric face image according to the size of the eyeball determined by the determining step and the rotation angle of the eyeball;
An eye processing method comprising: a eye line correcting step of correcting the eye line by synthesizing the pupil image generated in the pupil image generating step with a binocular position in the eccentric face image. The eye correction step includes
An enlargement / reduction step of measuring the diameter of the pupil in the prejudice face image, and enlarging / reducing the size of the pupil image of the normal line of sight generated by the pupil image generation step according to the measured pupil diameter,
The image processing method according to claim 8, further comprising: a combining step of combining the pupil image of the normal line of sight enlarged / reduced by the enlargement / reduction step with the binocular position in the eccentric face image. A distance measuring step for measuring the shooting distance to the subject;
If the photographing distance measured by the distance measuring step is less than a predetermined distance when the eccentric face image is captured, the eye line correcting step is executed. On the other hand, if the photographing distance is not less than the predetermined distance, the eye line correction is performed. The image processing method according to claim 6, further comprising: a control hand step that does not execute the step.

Images