JP2005202477A - Method for deciding top-and-bottom direction of face image and image recording device and image reproducing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately decide the top-and-bottom direction of an image including the face of a figure as an object. <P>SOLUTION: A face image including the face of a figure is acquired (S2), and the eyes and mouth are detected from the acquired face image (S4), and the top-and-bottom direction of the face image is decided based on the position of the mouth corresponding to the positions of the eyes in the face image (S8). The face image whose top-and-bottom direction is decided is rotated (S12), and stored (S16). The face image may not be rotated, and the top-and-bottom direction and the image may be stored. The rotated face image is displayed while the acquired top-and-bottom directions of the face image are various. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a face image top / bottom direction determination method, an image recording apparatus, and an image reproduction apparatus, and more particularly to a face image top / bottom direction determination method for determining a top / bottom direction of a face image including a human face as a subject, and the top / bottom direction determination method The present invention relates to an image recording apparatus and an image reproducing apparatus using the.

  Conventionally, as a technique for detecting a face from an input image obtained by photographing a person as a subject, the input image is divided into small regions, and eye candidate regions and candidate regions between the eyes are detected from luminance characteristics of each small region, The area between the eyes is determined from the mutual positional relationship between the eye candidate area and the candidate area between the eyes, and the face area is determined so as to include the area between the eyes.

  Japanese Patent Application Laid-Open No. H10-228667 describes a camera that controls the photographing function of a camera such as an aperture and a focus alignment based on a face recognition result. Specifically, the brightness of each block grouped in units of multiple pixels is either low brightness (estimated as pupil or hair), medium brightness (estimated face part other than pupil or hair), or high brightness (estimated as background). The face recognition target is only a block group in which the low-brightness area and the middle-brightness area are adjacent to each other and its periphery, and the skin color area is extracted and the edge is detected in this face recognition target. The face area is determined by performing a matching process with the template image of the face.

Patent Documents 2 and 3 describe an image including characters, in which the vertical direction is determined based on the level of character recognition rate.
JP 2001-330882 A JP-A-8-274980 JP-A-6-103410

  However, it has been difficult to accurately determine the vertical direction of the face image based on the recognition of only the face and eyes for various photographing opportunities.

  Patent Document 1 describes what recognizes a face based on eyes and hair, and suggests which top / bottom direction can be derived by determining which top / bottom direction based on what face part. There is no description to do. Those described in Patent Documents 2 and 3 cannot determine the vertical direction of an image that does not include characters.

  The present invention has been made in view of such circumstances, and a method of determining a top / bottom direction of a face image that can accurately determine the top / bottom direction of an image obtained by photographing a person's face as a subject. It is an object of the present invention to provide a recording apparatus and an image reproducing apparatus.

  In order to achieve the object, the invention according to claim 1 includes a step of acquiring a face image including a human face as a subject, a step of detecting eyes and a mouth from the face image, and an eye in the face image. And a step of determining the top-to-bottom direction of the face image based on the position of the mouth with respect to the position.

  With this configuration, the eyes and mouth are detected from the face image, and the top and bottom direction is determined based on the position of the mouth with respect to the position of the eyes, so that the top and bottom direction of the image can be determined accurately and easily.

  According to a second aspect of the present invention, in the first aspect of the invention, the step of determining the top-and-bottom direction includes the line segment connecting the eyes in the triangle formed by the eyes and the mouth as the base, the mouth as the vertex, and the base The top-and-bottom direction is determined by determining whether the apex is at the top, bottom, left, or right position.

  According to a third aspect of the present invention, there is provided an image acquisition means for acquiring a face image including a human face as a subject, a detection means for detecting eyes and a mouth from the face image, and a mouth position relative to the position of the eyes in the face image. Direction determining means for determining the top and bottom direction of the face image based on the position, image rotating means for rotating the face image based on the determined top and bottom direction, and recording the rotated face image on a predetermined recording medium An image recording means is provided.

  According to a fourth aspect of the present invention, there is provided an image acquisition means for acquiring a face image including a human face as a subject, a detection means for detecting eyes and a mouth from the face image, and a mouth position relative to the position of the eyes in the face image. The apparatus includes a direction determination unit that determines the top and bottom direction of the face image based on a position, and an image recording unit that records the top and bottom direction on a predetermined recording medium in association with the face image together with the face image. .

  According to a fifth aspect of the present invention, there is provided an image acquisition means for acquiring a face image including a human face as a subject, a detection means for detecting eyes and a mouth from the face image, and a mouth position relative to an eye position in the face image. Direction determining means for determining the top and bottom direction of the face image based on the position, image rotating means for rotating the face image based on the determined top and bottom direction, and image display means for displaying the rotated image It has a configuration.

  According to the present invention, it is possible to accurately determine the top-and-bottom direction of an image obtained by photographing including a human face as a subject.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

  FIG. 1A is a front view of a digital camera to which the present invention is applied, and FIG. 1B is a rear view of the digital camera.

  1A and 1B, the digital camera 1 includes a photographing lens 12 for photographing a subject, an image display device 15 for displaying an image obtained by photographing, and a photographing range for visual confirmation. Finder 17, a mode switch 111 for switching between the shooting mode and the reproduction mode, a zoom switch 112 for changing the magnification of the image, and a release switch 113 for instructing the start of shooting.

  FIG. 2 is a block diagram illustrating an example of a specific internal configuration of the digital camera 1.

  In FIG. 2, a photographic lens 12 mainly includes a plurality of optical lenses (a zoom lens 121 and a focus lens 122), an iris 123 for adjusting the amount of exposure light, and a zoom for changing a photographic magnification by changing a focal length. A zoom motor 124 that moves the position of the lens 121, a focus motor 125 that moves the position of the focus lens 122 for focusing, an iris motor 126 that changes the aperture value of the iris 123, and the subject image as an analog image signal A solid-state imaging device 127 for outputting is configured.

  The motor driver 13 includes a zoom motor driver 131 that electrically drives the zoom motor 124, a focus motor driver 132 that electrically drives the focus motor 125, and an iris driver 133 that electrically drives the iris motor 126. ing.

  The analog processing unit 14 samples and holds the analog signal output from the solid-state image sensor 127 for each pixel and amplifies it, and then converts it into a digital signal.

  The external recording device 16 stores images and other data.

  The control unit 10 mainly includes a key controller 101, an image signal processing unit 102, a buffer memory 103, an image display controller 104, a media controller 105, a system controller 106, an AF / AE calculation unit 1061, a face area detection unit 1062, and a direction determination unit. 1063, an image rotation unit 1064, an image compression / decompression unit 1065, and an internal memory 107.

  The key controller 101 monitors the mode selector switch 111, the zoom switch 112, and the release switch 113, and detects an instruction input from the user.

  The image signal processing unit 102 performs image processing such as gamma correction on the image converted into a digital signal by the analog processing unit 14.

  The buffer memory 103 buffers an image to be output to the image display device 15.

The image display controller 104 controls the image display device 15 to display an image on the image display device 15. The media controller 105 writes an image to the external recording device 16 and an image from the external recording device 16. Are to be read.

  The system controller 106 performs overall control of the entire digital camera 1.

  The AF / AE calculation unit 1061 performs automatic focus adjustment (AF) calculation and automatic exposure (AE) calculation.

  The face area detection unit 1062 performs face detection and eye and mouth detection in the subject image acquired by the solid-state imaging device 14 (or the media controller 105).

  The direction determination unit 1063 determines the top-to-bottom direction of the image based on the position of the mouth with respect to the eye position when the face area detection unit 1062 detects a face. Here, the “top-to-bottom direction” is a direction in which the direction with the crown is “top” and the direction with the jaw is “ground”. In the relationship between the eyes and the mouth, the center point of the right eye and the left eye is “heaven”, and the center point of the mouth is “ground”.

  By the way, if the ear is detected from the facial parts and the position of the ear is used as a reference for determining the top and bottom direction, the ear may be hidden in the hair. In such a case, accurate detection of the ear is not possible. Since it is difficult, it is difficult to accurately determine the vertical direction. In addition, among the facial parts, the eyebrows may be hidden in the hair as well, and it is difficult to accurately determine the top and bottom direction. In addition, hair has various styles depending on the individual, and it is difficult to accurately determine the top and bottom direction. In addition, detection of the nose and nostrils may be difficult depending on individual differences in the nose shape, the shooting angle, and the lighting conditions, and accurate determination of the top-and-bottom direction becomes difficult. The eyes (especially pupils) and the mouth (especially lips) of the face parts (ears, eyebrows, hair, nose) are reliably and easily detected from the face image by comparison with the surrounding parts. In addition, it is easy to select a representative point, and it is excellent as a criterion for determining the vertical direction. Therefore, in the present embodiment, the top-and-bottom direction is determined based on the pupils and lips.

  The image rotation unit 1064 rotates the image based on the top / bottom direction determined by the direction determination unit 1063.

  The image compression / decompression unit 1065 performs image compression and image expansion by a predetermined compression and expansion method such as JPEG (Joint Photographic coding Experts Group).

  The internal memory 107 stores information necessary for executing each process such as an image recording process and an image reproduction process, and an image to be processed at each process.

  Next, the flow of image recording processing in the digital camera 1 will be described.

  When the shooting mode is selected by the mode switch 111, the image of the subject imaged on the light receiving surface of the solid-state image sensor 127 is output as an analog signal from the solid-state image sensor 127, and is converted into a digital signal by the analog processing unit 14. After that, the data are sequentially stored in the internal memory 107. The images sequentially stored in the internal memory 107 are subjected to predetermined signal processing such as gamma correction by the image signal processing unit 102, and then transferred to the image display device 15 through the buffer memory 103 and controlled by the image display controller 104. In addition to looking through the viewfinder 17, the user can check the shooting range of the subject by looking at the image displayed on the image display device 15.

  When the release switch 113 is pressed in such a photographing mode, the AF / AE calculation unit 1061 performs automatic focus adjustment calculation and automatic exposure calculation, and automatic focus adjustment processing and automatic exposure processing based on such calculation are performed. In this state, images acquired by the solid-state image sensor 127 are sequentially stored in the internal memory 107.

  In such an aspect (image recording aspect) of recording the image stored in the internal memory 107 in the external recording device 16, firstly, an aspect in which the top and bottom direction of the image is determined and the image is rotated and recorded, Second, there is an aspect in which the top / bottom direction information indicating the top / bottom direction is recorded together with the image without rotating the image.

  FIG. 3 is a flowchart showing a main part of an example of image recording processing in the first image recording mode (a mode in which an image is rotated and recorded).

  When the image acquired by the solid-state imaging device 127 in a state where the focus adjustment process and the exposure process are performed is stored in the internal memory 107 (S2), the face area, the eye area, And each area | region of a mouth is detected (S4). Specifically, as shown in FIG. 4, a face area 210 that is a face portion is determined from the entire area 200 of the image based on contrast, and the face area 210 is a plurality of pixels each composed of a plurality of pixels. Divide it into blocks. Each block is provided with a code for identifying each block. Next, in the face area 210, the boundary between the red area and the periphery (for example, skin color area), that is, the outline of the lips is detected. In FIG. 5, the first boundary line 221, the second boundary line 222, the third boundary line 223, and the fourth boundary line 224 indicate the boundary between the lips and the surrounding area. The mouth area 220 is a block group including a block including the first boundary line 221 and the fourth boundary line 224 and a block surrounded by both the boundary lines 221 and 214, and is represented by a code of each block. A region composed of a plurality of blocks of 25th to 27th, 31st to 37th, 39th to 65th, and 67th to 73rd corresponds. In the mouth area 220, the leftmost end 225 (57th block) and the rightmost end 226 (56th block) of the boundary lines 221 to 224 are determined to be lip edges and stored in the internal memory 107. Also, a block (52nd block) at the approximate center of both edges 225 and 226 is determined as the center point of the mouth area and stored in the internal memory 107.

  Whether the mouth is open or closed by tracing the boundary lines 221 to 224 from one edge (for example, the left edge 225) and counting the number of the boundary lines. Can also be determined. For example, as shown in FIG. 5, if there are four boundary lines, the mouth is open, and if there are three boundary lines, the mouth is closed. That is, the mouth area can be accurately detected regardless of opening and closing of the mouth. In the case of an open mouth, the mouth region may be detected more accurately by confirming the presence of teeth or tongue.

  If there is a detected face area, first, the vertical direction of the image is determined (S8). Specifically, for a total of two points selected one by one from each eye area and one point selected from the mouth area, the line segment connecting the two points of the eye area is the base, the point of the mouth area is the vertex, and the base The top-to-bottom direction of the image is determined by determining whether the vertex is in the top, bottom, left, or right position. For example, the position of the block where the center point 231 of the left pupil is the representative point of the left eye region, the position of the block where the center point 232 of the right pupil is the representative point of the right eye region, and both edges 225 and 226 of the lips The position of the block at the approximate center is taken as the representative point of the mouth area. Then, it is determined whether the representative point (vertex) of the mouth is above, below, left, or right with respect to the line segment (base) connecting the representative points of both eyes.

  Next, the rotation angle of the face image is obtained (S10), and the face image is rotated (S12). Specifically, when the mouth point (vertex) is below the line segment (bottom) connecting the points of both eyes as shown in FIG. 6A, t = 0 is set as shown in FIG. 6B. Thus, t = 1 at the right, t = 2 at the top as shown in FIG. 6 (c), and t = -1 at the left as shown in FIG. 6 (d). The obtained rotation angle α = (π / 2) × t is obtained, and the image is rotated α. That is, when the mouth position is down as shown in FIG. 6A, the image is rotated by 0 degree (that is, not rotated), and when the mouth position is right as shown in FIG. When the mouth position is upward as shown in FIG. 6C, the image is rotated 180 degrees clockwise, and when the mouth position is left as shown in FIG. Is rotated 90 degrees counterclockwise (ie, rotated 270 degrees clockwise).

  Regarding the determination of up, down, left, and right, for example, as shown in FIG. 7, the left eye representative point 231 (which is the center point of the left pupil) and the right eye representative point 232 (which is the center point of the right pupil). The angle of the mouth representative point 227 (which is the center point of both edges of the lips) with respect to the center point 233 at the center is obtained. If it is −45 degrees or more and less than 45 degrees, it is above 45 degrees or more and less than 135 degrees If it is right, 135 degrees or less and less than 225 degrees, and if it is 225 degrees or more and less than 315 degrees, it is determined to be left.

  The rotated image is subjected to data compression by a predetermined compression method such as JPEG by the image compression / decompression unit 1065 (S14), and is stored in the external recording device 16 under the control of the media controller 105 (S16). .

  It should be noted that the image determined to have no face area in step S6 is subjected to data compression without being rotated (S14) and stored in the external recording device 16 (S16).

  When the playback mode is selected by the mode switch 111, an image stored in the external recording device 16 (an image rotated based on the vertical direction) is transferred from the external recording device 16 under the control of the media controller 105. It is taken out and temporarily stored in the internal memory 107. The image stored in the internal memory 107 is subjected to data expansion by a predetermined method by the image compression / decompression unit 1065, and is displayed on the image display device 15 through the buffer memory 103 under the control of the image display controller 104.

  FIG. 8 is a flowchart showing a main part of an example of image recording processing in the second image recording mode (a mode in which the top-and-bottom direction information is recorded together with the image).

  In FIG. 8, when the image acquired by the solid-state image sensor 127 is stored in the internal memory 107 (S2), the face area, the eye area, and the mouth area are respectively detected from the entire area of the image. (S4) After determining the presence or absence of the face area (S6), if there is a face area, the vertical direction of the image is determined (S8) and the vertical direction information indicating the vertical direction of the image is generated (S8). S10).

  Specifically, for example, when the mouth position (vertex) is below the line segment (base) connecting the positions of both eyes as shown in FIG. 6A, t = 0 is set, and FIG. As shown in FIG. 6, t = 1 at the right, t = 2 at the top as shown in FIG. 6C, t = −1 at the left as shown in FIG. The indicated information (t) is the top / bottom direction information. As shown in FIG. 7, the angle of the mouth representative point 227 (which is the center point of both edges of the lips) with respect to the center point 233 of the left eye representative point 231 and the right eye representative point 232 may be used as the top-and-bottom direction information. .

  The face image is subjected to data compression by a predetermined compression method such as JPEG by the image compression / decompression unit 1065 (S24), and is stored in the external recording device 16 under the control of the media controller 105. Here, the top / bottom direction information is associated with each image and stored in the external recording device 16 (S26). For example, the top / bottom direction information is written in the header area of the file storing the image. Specifically, for example, when the file is in the Exif (Exchangeable image format) format, the top-and-bottom direction information is written in the header area of the file as one of the tag information. In addition to the aspect in which the face image and the top and bottom direction information are recorded in the same file as described above, there is a mode in which the face image and the top and bottom direction information are recorded in separate files.

  When the playback mode is selected by the mode selector switch 111, an image stored in the external recording device 16 (an unrotated image) is taken out from the external recording device 16 under the control of the media controller 105 together with the top / bottom direction information. And temporarily stored in the internal memory 107. The image stored in the internal memory 107 is subjected to data decompression by a predetermined method by the image compression / decompression unit 1065 and rotated by the image rotation unit 1064 based on the top / bottom direction information. Then, the image is displayed on the image display device 15 under the control of the image display controller 104.

  Specifically, as shown in FIG. 6A, when the position of the mouth is down (t = 0), the image is rotated by 0 degree (that is, not rotated), and as shown in FIG. When the position is right (t = 1), the image is rotated 90 degrees clockwise. When the mouth position is up (t = 2) as shown in FIG. 6C, the image is rotated 180 degrees clockwise. As shown in FIG. 6D, when the position of the mouth is left (t = −1), the image is rotated 90 degrees counterclockwise.

  As described above, the case where the top and bottom direction is determined when recording an image has been described as an example. However, the present invention is not limited to this, and the top and bottom direction is determined and the image is rotated when the image is reproduced. You may do it.

  FIG. 9 is a flowchart showing a main part of an example of processing for detecting the top and bottom direction and rotating the image when reproducing the image stored in the external recording device 16.

  When the playback mode is selected by the mode switch 111, an image (an unrotated image) stored in the external recording device 16 is acquired from the external recording device 16 under the control of the media controller 105, and the internal memory 107. (S42). The image stored in the internal memory 107 is subjected to data expansion by a predetermined method by the image compression / decompression unit 1065 (S44).

  Next, the face area, the eye area, and the mouth area are respectively detected from the entire area of the image (S46), and the presence / absence of the face area is determined (S48). Determines the vertical direction of the image (S50), obtains the rotation angle of the face image (S52), and rotates the face image (S54).

  Specifically, when the mouth position (vertex) is below the line segment (bottom) connecting the positions of both eyes, as shown in FIG. 6A, t = 0, as shown in FIG. 6B. Thus, t = 1 at the right, t = 2 at the top as shown in FIG. 6 (c), and t = -1 at the left as shown in FIG. 6 (d). The obtained rotation angle α = (π / 2) × t is obtained, and the image is rotated α. That is, in the case shown in FIG. 6 (a), the image is rotated by 0 degree (that is, not rotated), and in the case shown in FIG. 6 (b), the image is rotated by 90 degrees clockwise. In the case shown in FIG. 6, the image is rotated 180 degrees clockwise, and in the case shown in FIG. 6D, the image is rotated 90 degrees counterclockwise.

  The rotated image is displayed on the image display device 15 through the buffer memory 103 under the control of the image display controller 104 (S56).

  When determining the vertical direction, the approximate center of the pupil is the representative point of the eye, and the approximate center of both edges of the lips is the representative point of the mouth, based on these three points (representative points of both eyes and the mouth). However, the present invention is not limited to this. For example, the left edge of the left eye and the right edge of the right eye may be the representative points of each eye, and the right edge of the left eye and the left edge of the right eye may be the representative points of each eye. The same effect can be obtained by directly determining the top-to-bottom direction from a total of 4 points including the representative points (2 points) of both eyes and both edges (2 points) of the lips.

  4 to 6 describe the case where the face is photographed from the front, but when the face is photographed from the side, the image is obtained from a total of two points, one point for one eye region and one point for the lip region. You may make it determine the top-and-bottom direction.

  Further, the case where the image is classified into the top, bottom, left, and right cases has been described as an example, but the present invention is not limited to this. For example, in FIG. 7, the image may be rotated by the angle based on the angle of the representative point of the mouth with respect to the center point 233 of the left eye and the right eye. Specifically, the image is rotated so that the representative point of the mouth is just below according to the angle. The four corners of the rotated image are trimmed and removed, and the image is enlarged as necessary.

  As described above, the case where the present invention is applied to a digital camera has been described as an example. However, the present invention is not limited to this, and may be applied to other image processing apparatuses that handle a face image including a human face as a subject. Needless to say. For example, you may apply to the image processing apparatus only for image reproduction which does not have an image recording function.

1 is a block diagram showing an internal configuration of an image processing apparatus according to an embodiment of the present invention. External view of digital camera A flowchart showing a first mode of image recording processing Explanatory drawing used for explanation of eye and mouth detection Explanatory drawing used for explanation of detection of mouth position Explanatory drawing used for explanation of rotation of face image Explanatory drawing used for explanation of angle of face image Flowchart showing a second mode of image recording processing Flowchart showing one aspect of image reproduction processing

Explanation of symbols

  DESCRIPTION OF SYMBOLS 15 ... Image display apparatus, 16 ... External recording device, 1062 ... Face area | region detection part (detection means), 104 ... Image display controller, 105 ... Media controller (image recording means, image acquisition means), 1063 ... Direction determination part, 1064 ... Image rotator, 107 ... Internal memory, 127 ... Solid-state imaging device (image acquisition means)

Claims (5)

Obtaining a face image including a human face as a subject;
Detecting eyes and mouth from the face image;
Determining the top-to-bottom direction of the face image based on the position of the mouth with respect to the position of the eyes in the face image;
A method of determining the top / bottom direction of a face image, comprising:   The step of determining the top-to-bottom direction includes a line segment connecting the eyes in a triangle composed of eyes and a mouth as a base, the mouth as a vertex, and the vertex is located above, below, left, or right with respect to the bottom. The method of determining a top / bottom direction of a face image according to claim 1, wherein the top / bottom direction is determined by determining whether the top / bottom direction is vertical. Image acquisition means for acquiring a face image including a human face as a subject;
Detecting means for detecting eyes and mouth from the face image;
Direction determining means for determining a top-to-bottom direction of the face image based on a mouth position with respect to an eye position in the face image;
Image rotating means for rotating the face image based on the determined top-and-bottom direction;
Image recording means for recording the rotated face image on a predetermined recording medium;
An image recording apparatus comprising: Image acquisition means for acquiring a face image including a human face as a subject;
Detecting means for detecting eyes and mouth from the face image;
Direction determining means for determining a top-to-bottom direction of the face image based on a mouth position with respect to an eye position in the face image;
Image recording means for recording the top and bottom direction on a predetermined recording medium in association with the face image together with the face image;
An image recording apparatus comprising: Image acquisition means for acquiring a face image including a human face as a subject;
Detecting means for detecting eyes and mouth from the face image;
Direction determining means for determining a top-to-bottom direction of the face image based on a mouth position with respect to an eye position in the face image;
Image rotating means for rotating the face image based on the determined top-and-bottom direction;
Image display means for displaying the rotated image;
An image reproducing apparatus comprising:

Images