JPH08138024A - Picture direction discriminating method - Google Patents

Abstract

PURPOSE: To rotate a picture into a correct image to output the picture by discriminating the direction of the picture in accordance with positional relations or hands and legs to the face to automatically discriminate the direction of the picture including the face of a person. CONSTITUTION: A scanner 2 optically reads the color original picture on a film 1 and obtains B(blue), G(green), and R(red) values of respective picture elements by color separation. These values are amplified by an amplifier 3 and are converted into digital data by an A/D converter 4 and are inputted to a CPU 5. The CPU 5 executes processings for discrimination of the direction or true picture. That is, a race candidate area as the part where the face of a person is photographed is extracted from the picture, and hand and leg candidate areas as parts where hands and legs of the person are photographed are extracted. The direction of the picture is discriminated in accordance with positional relations of hand and leg candidate areas to the face candidate area. For example, it is discriminated that the direction of the picture is correct when hand and leg candidate areas exist under the face candidate area.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for determining the orientation of an image of a negative film, a positive film or the like in which a person is photographed by a camera.

[0002]

2. Description of the Related Art When handling an image taken by a camera or the like, it is sometimes necessary to determine the direction of the image, that is, whether the image is vertical or horizontal. For example, when displaying an image on the display, it is desirable that the upper part of the image is displayed on the upper part of the display regardless of the direction in which the image is read. Further, even when the image processing for changing the brightness of a part of the image is applied, it becomes easier to perform the processing if the upper and lower sides and the right and left sides of the image are known.

Conventionally, it has been difficult to automatically determine the orientation of an image, and a person has made the determination. For example, when displaying an image on a display, the read image is once displayed, and the user views the displayed image and then rotates the image on the display to display the image in the correct direction. .

Further, Japanese Patent Laid-Open No. 53-78874 discloses a method for automatically determining the orientation of an image.
This method is a method of determining the orientation of an image based on the contrast difference between the top, bottom, left and right of the image. For example, in a photograph of a landscape, the sky is at the top of the image and the ground is at the bottom, so the image This method utilizes the fact that the contrast difference between the upper part and the lower part becomes large, and the contrast difference between the left part and the right part becomes small.

[0005]

As described above, conventionally, there has been a problem that a person must determine the orientation of an image.

Further, the method for automatically determining the orientation of an image disclosed in Japanese Patent Laid-Open No. 53-78874 has a problem of low accuracy. This is because, in a general image, there are many images in which the left and right have a greater contrast difference than the upper and lower, and in the case of such an image, the up / down and left / right determination of the image is erroneous.

The present invention has been made in consideration of the above points. The orientation of an image showing a person's face is automatically determined without depending on the person's determination, and a normal image is formed when the image is output. The object is to enable the image to be rotated and output.

[0008]

In order to achieve the above object, the present invention determines a direction of an image from a positional relationship between a face, a hand, and a foot in a method of determining the direction of an image showing a human face. I did it.

Also, the orientation of the image is determined from the positional relationship of the face constituent parts.

Further, in the method of determining the orientation of an image in which at least two human faces are shown, the orientation of the image is determined based on the positional relationship between the faces of a plurality of persons.

[0011]

As described above, according to the present invention, the face of a person shown in the image is extracted, the limbs and the constituent parts of the face are extracted, and the orientation of the image is determined based on the positional relationship between them.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram of an image processing apparatus using the image orientation determining method according to the present invention.

The film 1 is a film on which a color original image is recorded and may be a negative film or a positive film. The scanner 2 optically reads the original color image on the film 1, color-separates it, and outputs B for each pixel.
(Blue), G (green), R (red) values can be obtained. The BGR value is amplified by the amplifier 3, converted into digital data by the A / D converter 4, and input to the CPU 5. The CPU 5 executes each processing for determining the orientation of the image, which will be described later.

FIG. 2 is a flowchart of the first embodiment of the image orientation determining method according to the present invention.

First, a face candidate area in which a person's face is photographed is extracted from an image (A-1), and then a hand / foot candidate area in which a person's limbs are photographed is extracted. (A-2).

Then, the orientation of the image is determined from the positional relationship between the face candidate area and the hand / foot candidate area (A-3). For example, as shown in FIG. 3, if the hand / foot candidate area is below the face candidate area, the image is determined to be in the correct orientation.

The face candidate area may be extracted by the face extraction method previously filed by the applicant of the present application as Japanese Patent Application No. 6-196735. Here, an example thereof will be described.

FIG. 4 is a flow chart of a first embodiment of the processing for extracting the face candidate area in step (A-1) of FIG.

First, from the BGR value of each pixel obtained by the scanner 2, the brightness, hue, saturation, chromaticity, (B-
R), (GR), and color feature quantities (B-
1) If the feature amounts of these colors are within a predetermined range, it is determined that the target pixel is a skin color (B-2).
Of course, the predetermined range of the feature amount differs between the case of extracting the skin color from the BGR value of the negative image and the case of extracting the skin color from the BGR value of the positive image. Further, it may be possible to determine whether the color is a skin color by using the obtained color feature amount as an input value of the neural network. When extracting the skin color from the positive image, it is sufficient to extract the skin color directly from the positive photometric value, and when extracting the skin color from the negative image, the negative photometric value may be converted to positive to extract the skin color, The skin color may be directly extracted from the negative photometric value.

Next, edge extraction is performed on the image composed of pixels determined to be flesh color (B-3). As an edge extraction method, for example, with the target pixel as the center,
The brightness average of the pixels is calculated, and if the difference between the average value and the brightness of the target pixel is larger than a predetermined value, the target pixel is set as an edge pixel, and binarization is performed depending on whether the pixel is an edge pixel. Here, the image thus obtained is referred to as an edge image.

In step (B-4), first, the size,
Create multiple elliptical or circular face templates with different major / minor axis ratios. These face templates may be created in advance and stored in the storage device 6.
The face template is binarized depending on whether it is the outline of an ellipse or a circle. Since the actual contour of the face is not an exact ellipse or circle, the contour of the face template is a few pixels,
Preferably, the width of 2 to 3 pixels may be provided to increase the degree of matching with the actual contour of the face.

In step (B-4), subsequently, the degree of matching between the edge image and the face template is obtained. An existing method is used to obtain the matching degree. For example, the matching degree m (u, v) is obtained by the method represented by the equation 1.

[0024]

[Equation 1] In Expression 1, f represents a target image, t represents a face template, and S represents a range of t (x, y). f ', t'
Represents the average of f (x + u, y + v) and t (x, y) in S, respectively.

With such a method, matching is performed using several types of face templates, the best matching template is found for the target pixel (B-4), and the matching degree is equal to or more than a predetermined threshold value. In this case (B-5), the area surrounded by the best matching face template centering on the target pixel is determined to be the face candidate area (B-
6).

Further, in order to reduce the number of calculations, a rough scan is first performed and a face template of a certain size is set to 1
You can shift pixels pixel by pixel or every other pixel,
The optimum face candidate area may be determined by obtaining the matching degree and applying templates of different sizes only to the target pixel having the matching degree of a predetermined value or more.

FIG. 5 is a flowchart of the processing for extracting the hand / foot candidate area in step (A-2) of FIG.

In step (C-1), a hand / foot template is created, and the edge image obtained in step (B-3) in FIG. 4 is processed in the same manner as in the process in step (B-4) in FIG. Match the hand / foot template and find the hand / foot template with the highest degree of matching.

If the matching degree is equal to or higher than a predetermined threshold value (C-2), the area surrounded by the best matching hand / foot template with the target pixel as the center is the hand / foot candidate area. (C-3).

In this embodiment, the hand / foot is extracted based on the degree of matching with the hand / foot template. However, the present invention is not limited to this, and circumscribes the candidate area by utilizing the fact that the hand or foot is elongated. You may extract by the value of the long side / short side of the rectangle.

By the way, it is sufficient to regard a portion determined to be a face candidate area by the first embodiment of the process for extracting a face candidate area shown in FIG. 4 as a face, but a portion which is not a face in some images. May be determined to be a face candidate area. Therefore, hereinafter, a method of further narrowing down a plurality of portions determined as face candidate areas in the first embodiment and surely extracting a portion that is actually a face will be described.

FIG. 6 is a flow chart of a second embodiment of the processing for extracting the face candidate area in step (A-1) of FIG.

First, an area with eyes (eye candidate area) is estimated and set in the face candidate area determined by the first embodiment of the processing for extracting the face candidate area shown in FIG. 4 (D-
1). As an example of this estimation method, the upper half and the lower half of the face candidate area are set as eye candidate areas in case the image orientation is vertical, while the face candidate area is set to face orientation in case the image orientation is horizontal. The right and left halves of the candidate area are set as eye candidate areas. That is, for the face candidate area, the upper half, the lower half,
The right half and the left half are set as eye candidate areas.

Next, a histogram regarding lightness is created in each eye candidate area (D-2). here,
A case where the lightness is represented by 8 bits will be described (lightness 0: 0, lightness 100: 255). First, the brightness range (0-2
55) is equally divided into eight, and the appearance frequency in each lightness region is obtained. Since the brightness of the black eye is low, the histogram has peaks in at least the brightness of the black eye and the brightness of the skin color. Although a histogram of brightness is created here, the present invention is not limited to this, and saturation, hue, and chromaticity may be used instead of brightness.

As one of the histogram shape recognition methods thus created, there is the following method. First, the maximum peak (first peak) is searched for in the lightness range (for example, 8 bits or more 96 or more) considered to be a skin color region.
Next, the maximum peak (second peak) is searched in the lightness range lower than the predetermined value. This predetermined value is determined empirically by photometrically measuring several eye samples using a measuring instrument. The value of (frequency of second peak) / (frequency of first peak) is calculated, and this value is set as a feature amount (D-3).

If this feature amount is within the predetermined range (D-
4) It is determined that eyes are present, and it is determined that the shape of this histogram represents a face. The predetermined range in step (D-4) differs depending on the size of the eye candidate area, so the optimum range must be obtained for each case.

In this way, the judgment is made for each eye candidate area, and if there is an area judged as an eye, it is judged that the face candidate area is a face (D-5), and it is judged as an eye. If there is no region to be identified, it is determined that it is not a face.

In the second embodiment shown in FIG. 6, a one-dimensional histogram of lightness is created, but the present invention is not limited to this, and for example, a two-dimensional histogram centered on lightness and hue is created. Then, the peak in the area representing the skin color is the first peak, and the peak in the area representing the black eye is the second peak.
The determination may be performed by the same method as that of the embodiment, or a two-dimensional histogram of lightness and saturation or a two-dimensional histogram of hue and saturation may be used.

FIG. 7 is a flowchart of the second embodiment of the image orientation determining method according to the present invention.

First, the eye size is estimated based on the size of the face candidate area determined by the first embodiment of the process for extracting the face candidate area shown in FIG. 4, and an eye template is created (E -1). It is more preferable to prepare both eyeglasses and eyeglasses as eye templates.

Then, the target image in the face candidate area is binarized depending on whether it is an image showing eyes (E-2). As this method, a black pixel representing an eye may be detected by color information such as lightness, saturation, and hue, but the following method is preferable. (1) First, the average of the luminosity of the flesh color pixels in the face candidate area is calculated. (2) If the difference from this average value (of course, the pixel representing the eye is smaller) is greater than or equal to the threshold value, it is determined to be a black pixel representing the eye.

By doing so, it is possible to cover the difference in the brightness of the black pixels of the eyes due to the difference in the brightness of the face.

With respect to the face candidate area binarized in step (E-2), each eye template is scanned in the vertical direction and the horizontal direction to obtain the highest matching degree, which is used as the feature amount ( E-3).

If this feature amount is equal to or larger than the threshold value (E-4), it is determined that the face candidate area is a face (E
-5), The orientation of the image is determined from the position of the portion matched with the eye template in the face candidate area (E-6). For example, if the position of the eyes is at the bottom of the face candidate area, it is determined that the orientation of the image is upside down.

According to the second embodiment shown in FIG. 7, the orientation of the image can be determined even when the subject wears glasses, and highly accurate determination can be performed. .

Further, the orientation of the image can be determined even if the hand or foot is not shown in the image.

Although the image orientation is determined by detecting the eyes in the second embodiment shown in FIG. 7, the present invention is not limited to this. For example, hair is detected in addition to the eyes and the image is detected. May be determined. The details will be described below.

When the eye position is determined, the hair position is estimated from the eye position information. Specifically, as shown in FIG.
For example, when both eyes are in the upper half of the face candidate area, the boundary area above the line connecting both eyes is estimated as the hair candidate area, and the black pixels indicating the hair are counted in this area. If both eyes are in the right half of the face candidate area, the boundary area to the right of the line connecting both eyes is estimated to be the hair candidate area.

If the ratio of hair pixels (the number of black pixels / area of hair candidate area) is greater than or equal to the threshold value, it is determined that there is hair, and this face candidate area is determined to be a face, and the orientation of the image is determined. To judge. Here, if hair is detected after the size of the face candidate area is expanded by several pixels, the detection accuracy is further improved.

In this case, the orientation of the image can be determined depending on whether the hair region is in the upper half, the lower half, the left half, or the right half of the face. effective. That is, as shown in FIG. 8, when the width of the face candidate area is w, the height is h, and the position of the pixel representing the hair is (x, y), the orientation of the image is determined as in Expression 2. .

[0051]

## EQU00002 ## If y.gtoreq.h / 2, the correct orientation; if h / 2> y.gtoreq.0, upside down; if x.ltoreq.w / 2, the part that should be on top is It is on the left; if w / 2 <x ≦ w, the part that should be on the top is on the right.

Furthermore, the present invention may detect the orientation of the image by detecting the neck in addition to the eyes. The details will be described below.

When the eye position is determined, the neck position is estimated from the eye position information. As shown in FIG. 8, for example, when both eyes are in the upper half of the face candidate area, the lower 1/3 is set as the neck candidate area. It may be limited to a wider area such as the lower half.

Next, the size of the face candidate area is expanded by 3 to 4 pixels, and the skin-colored pixels representing the neck are counted in the area in contact with the neck candidate area. If the ratio of the number of neck pixels (count number / area of face candidate area) is greater than or equal to a threshold value, it is determined to be a neck, and this face candidate area is determined to be a face and the orientation of the image is determined.

Furthermore, the present invention may detect the mouth in addition to the eyes to determine the orientation of the image. The details will be described below.

First, the mouth position is estimated from the eye position information. For example, when the positions of both eyes are both in the upper half of the face, it is estimated that the position of the mouth is in the lower half of the face. If both eyes are in the right half of the face, it is estimated that the mouth is in the left half of the face. Further, if the position of the mouth is estimated by the distance from the midpoint of the line segment connecting both eyes, as is usually done, higher detection accuracy can be obtained.

Next, with respect to the mouth candidate area, it is determined from the color information whether or not a mouth is present. Specifically, for example, the BGR signal is represented by 8 bits (0 to 255), GR is calculated for the entire face candidate area, and the average value thereof is obtained. Next, GR is similarly calculated for the mouth candidate area, the difference between the calculated value and the previously calculated average value is calculated, and if this difference is equal to or greater than the threshold value, it is counted as a mouth pixel. To do.

Then, the ratio of the mouth pixels (the number of pixels counted as the mouth pixels / the total number of pixels in the face candidate area) is obtained, and this is set as the second feature amount.

If the second feature amount is within the predetermined range,
The face candidate area is determined to be a mouth, and the face orientation is determined as well as the face. Each threshold may be empirically obtained using a measuring device. Also, G-
Instead of R, other color information such as lightness, hue, saturation,
Chromaticity may be used.

According to this embodiment, it is possible to know which part of the face area the mouth is located in and to determine the orientation of the image. There is also an effect that the orientation of the image can be determined with higher accuracy.

FIG. 9 is a flow chart of the third embodiment of the image orientation determining method according to the present invention.

The present embodiment is an image orientation determination method that can be applied to an image in which a plurality of faces are shown. In an image in which a plurality of humans are shown, faces are normally arranged in the lateral direction. It is a thing.

First, a representative point is obtained for each of a plurality of face candidate areas determined by the above-described various face candidate area extraction processing (F-1). As a representative point,
For example, the center point of the face candidate area may be used as shown in FIG.

Next, in order to determine whether the plurality of representative points are widely scattered in the horizontal direction or widely in the vertical direction, the horizontal distribution range and the vertical distribution range of the representative points are calculated (F-2). ).

Then, when the horizontal distribution range is larger than the vertical distribution range, it is determined that the orientation of the image is the correct orientation (F-4), and otherwise 90 degrees or 27 degrees.
It is determined that the direction has been rotated by 0 degrees (F-4).

Incidentally, in the case of an image in which many persons are shown as shown in FIG. 11B, the difference between the horizontal distribution range and the vertical distribution range is large, so that the orientation of the image can be judged well. However, as shown in FIG. 11A, in the case of an image containing only a small number of people, the difference between the horizontal distribution range and the vertical distribution range is small, and the orientation of the image may be erroneously determined. Therefore, when the difference between the horizontal distribution range and the vertical distribution range is small and is less than or equal to a predetermined value, based on the position of the internal eyes or mouth, or hair or neck, for any face candidate area. If the orientation of the image is determined or the orientation of the image is determined based on the positional relationship between the face candidate area and the hand / foot candidate area,
The accuracy can be increased.

Further, in the third embodiment shown in FIG. 9, the central point of the face candidate area is used as the representative point, but pixels representing internal parts of the face such as eyes and mouth may be used as the representative point.

In the third embodiment shown in FIG. 9, the orientation of the image is judged based on the relationship between the horizontal distribution range and the vertical distribution range of the representative points, but the present invention is not limited to this. Obtain standard deviations of multiple representative points in the horizontal and vertical directions,
It is also possible to determine the orientation of the image by comparing their sizes. Further, instead of the standard deviation, another statistic such as variance may be used.

Further, instead of the representative point, a plurality of pixels in the face candidate area may be used to obtain the distribution in the vertical and horizontal directions from the distribution of pixels in the face candidate area to determine the orientation of the image. Also in this case, the standard deviation may be used, or another statistical amount such as variance may be used instead of the standard deviation.

The third embodiment shown in FIG. 9 is similar to that shown in FIG.
The accuracy can be further increased by combining the first embodiment shown in FIG. 9 and the second embodiment shown in FIG.

Here, an example combined with the second embodiment shown in FIG. 7 will be described.

For example, in the case where a plurality of persons are photographed, when the orientation of the image is determined based on the eye position in the face candidate area according to the second embodiment shown in FIG. It is conceivable that the orientation of the determined image may differ depending on whether or not the determination has been made. In such a case, more accurate determination can be performed by further performing the third embodiment shown in FIG. Specifically, it can be determined by the following procedure. (1) The orientation of the image is determined based on the position of the eyes in the face candidate area according to the second embodiment shown in FIG. At this time,
If n humans are shown, the ratio is (the ratio of the image orientation is correct = the number of people determined to have the correct image orientation / n), and the ratio of the image orientations in other directions is the same. And define it. (2) The orientation of the image is determined by the third embodiment shown in FIG. (3) In the case where the orientation of the image is determined to be the correct orientation by the third embodiment shown in FIG. 9 and the ratio of the orientation of the image obtained in (1) is the correct orientation is equal to or greater than a predetermined value. , It is determined that the orientation of the image is correct.

[0073]

As described above, according to the present invention,
It is possible to automatically determine the orientation of an image showing a person's face without depending on the person's determination. Therefore, when outputting an image, the image can be rotated and output so as to be a normal image.

Further, according to the second embodiment shown in FIG. 7, it is possible to determine the orientation of an image even if it does not show the limbs of a person.

Further, according to the third embodiment shown in FIG. 9, it is possible to determine the orientation of an image in which a plurality of persons are shown.

[Brief description of drawings]

FIG. 1 is a block diagram of an image processing apparatus using an image orientation determination method according to the present invention.

FIG. 2 is a flowchart of a first embodiment of an image orientation determination method according to the present invention.

FIG. 3 is a diagram illustrating a positional relationship between a face candidate area and a hand / foot candidate area.

FIG. 4 is a flowchart of a first embodiment of a process of extracting a face candidate area in step (A-1) of FIG.

5 is a flowchart of a process of extracting a hand / foot candidate region in step (A-2) of FIG.

FIG. 6 is a flowchart of a second embodiment of a process of extracting a face candidate area in step (A-1) of FIG.

FIG. 7 is a flowchart of a second embodiment of the image orientation determination method according to the present invention.

FIG. 8 is a diagram illustrating a method for estimating a hair yellow pond region.

FIG. 9 is a flowchart of a third embodiment of the image orientation determination method according to the present invention.

FIG. 10 is a diagram illustrating representative points of a face candidate area.

FIG. 11 is a diagram illustrating a relationship between a horizontal distribution range and a vertical distribution range with respect to a representative point of a face candidate area. FIG. 11A is an image in which only a small number of people are shown in the horizontal distribution range and the vertical distribution range. This is an example of an image with a small difference from the distribution range, and (b) is an example of an image in which many people are shown and in which the difference between the horizontal distribution range and the vertical distribution range is large.

[Explanation of symbols]

 1 Film 2 Scanner 3 Amplifier 4 A / D Converter 5 CPU 6 Storage Device

Claims (12)

[Claims] 1. A method for determining an orientation of an image showing a face of a person, wherein the orientation of the image is determined based on a positional relationship between constituent parts of the body. 2. As a positional relationship of the constituent parts of the body,
The image orientation determination method according to claim 1, wherein the positional relationship between the face, the hand, and the foot is used. 3. As a positional relationship of the constituent parts of the body,
The image orientation determination method according to claim 1, wherein the positional relationship between the face constituent portions is used. 4. The image orientation determination method according to claim 3, wherein a positional relationship between both eyes is used as a positional relationship between the face constituent parts. 5. The image orientation determination method according to claim 3, wherein the positional relationship between the face and hair is used as the positional relationship between the constituent parts of the face. 6. The image orientation determining method according to claim 3, wherein the positional relationship between the face and the neck is used as the positional relationship between the constituent parts of the face. 7. The image orientation determination method according to claim 3, wherein the positional relationship between the face and the mouth is used as the positional relationship between the constituent parts of the face. 8. An image orientation determination method, characterized in that at least two or more of the image orientation determination methods according to claims 2, 4, 5, 6 and 7 are combined. 9. A method for determining the orientation of an image in which at least two human faces are shown, wherein the orientation of the image is determined based on the positional relationship between the faces of the plurality of persons. Method. 10. The image orientation determining method according to claim 9, wherein the representative points of the respective faces are obtained, and the orientation of the image is determined from the shape in which the respective representative points are distributed. 11. Claims 2, 4, 5, 6, 7 and 10.
10. The image orientation determination method according to claim 9, wherein at least one of the image orientation determination methods according to claim 9 is combined. 12. An image orientation determining method according to claim 1, wherein the orientation of the image is determined, and then the image is rotated so that the orientation of the image becomes a normal image. A characteristic image processing device.