JP2005250778A - Vertical direction decision of image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide technology capable of automatically deciding the vertical direction of an image. <P>SOLUTION: First, by analyzing a pixel value of the color image, prescribed image characteristic information showing a characteristic of the color image is acquired. The vertical direction of the color image is decided by use of a decision rule using the image characteristic information. As the image characteristic information, a plurality of pieces of information such as person area information, sky/ground area information, linear information, rod-like object information can be used. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for automatically determining the top and bottom of an image.

  With the widespread use of digital cameras and camera-equipped mobile phones, the digitization of photographs is rapidly progressing. For this reason, as for photo albums, the number of users who prefer and use “electronic albums” that make up albums on computers tends to increase.

  In order to arrange photographic images on an electronic album, it is necessary to perform an operation for correctly orienting the images. However, the conventional electronic album does not have a function of determining the top and bottom of the image, forcing the user to perform a cumbersome task of manually correcting the image direction to the correct direction.

  Patent Document 1 discloses a method for detecting the orientation of an original in a copying machine. However, this method detects the orientation of a document from the orientation of characters, and cannot be applied to a photographic image (non-character image) that is a material of an electronic album.

JP 2003-143375 A

  The above-described problems are not limited to electronic albums, but are common to other image processing software such as image management software and image processing apparatuses.

  The present invention has been made to solve the above-described conventional problems, and an object thereof is to provide a technique capable of automatically determining the top and bottom of an image.

In order to achieve at least a part of the above object, an apparatus according to the present invention is an image processing apparatus for determining a vertical direction of a color image,
A feature information acquisition unit that acquires predetermined image feature information representing features of the color image by analyzing pixel values of the color image;
A top and bottom direction determination unit that determines the top and bottom direction of the color image using a determination rule that uses the image feature information; and
It is characterized by having.

  According to this apparatus, since the top / bottom direction of the color image is determined using the determination rule using the image feature information, it is possible to automatically determine the top / bottom of the image.

  The top / bottom direction determination unit may determine the probability together with the top / bottom direction of the color image, and may adopt a top / bottom direction with a high probability.

  According to this configuration, it is possible to determine the vertical direction more accurately.

The feature information acquisition unit acquires person area information representing a person area existing in the color image as the image feature information,
The top and bottom direction determination unit may determine the top and bottom direction of the color image according to the top and bottom direction of the person area.

  In this configuration, when the image includes a person, the top-and-bottom direction can be determined according to the direction of the person area, so that the top-and-bottom direction of the image can be determined so that the person faces correctly.

  The feature information acquisition unit may detect a skin color area existing in the color image as the person area.

  In this configuration, a person can be detected from the skin color area.

  The feature information acquisition unit may change the range of pixel values recognized as skin color according to the average luminance of the color image.

  In general, the skin color pixel value in an image is affected by ambient brightness at the time of shooting. Therefore, according to the above configuration, the parameter for detecting the flesh color region is changed according to the average luminance (brightness) of the image, so that the flesh color can be detected more accurately.

  The top / bottom direction determination unit may use the presence / absence of a body region adjacent to the skin color region for the top / bottom direction determination.

  With this configuration, the top-and-bottom direction can be determined more accurately.

  The feature information acquisition unit may detect a face area of a person existing in the color image as the person area.

  In this configuration, a person can be detected from the face area.

  The feature information acquisition unit may extract a skin color area existing in the color image and select a skin color area to be the face area from an aspect ratio of the extracted skin color area.

  With this configuration, the face area can be detected more accurately.

  The feature information acquisition unit selects the face area candidate from the aspect ratio of the skin color area, and selects an area including a facial organ from the face area candidates as the face area. Also good.

  With this configuration, the face area can be detected more accurately.

  The feature information acquisition unit may select an area that is a skin color area and includes a facial organ as the face area.

  Also with this configuration, the face area can be accurately detected.

  The top / bottom direction determination unit may use the positional relationship of the facial organs in the face region for the top / bottom direction determination.

  With this configuration, the top-and-bottom direction can be determined more accurately.

  The top and bottom direction determination unit may use the presence or absence of a hair region adjacent to the face region for the top and bottom direction determination.

  Also with this configuration, the top-and-bottom direction can be accurately determined.

  The feature information acquisition unit may acquire sky region information representing a sky region existing in the color image as the image feature information.

  In this configuration, when the image includes the sky, the top / bottom direction can be determined based on the sky region information. Therefore, the top / bottom direction of the image can be determined so that the sky faces up correctly.

The feature information acquisition unit
(I) Perform day / night determination of the color image,
(Ii) When it is determined that the color image is an image taken in the daytime, the color image is an area composed of pixels brighter than the average luminance of the color image, and an end portion of the color image And an area that is a color belonging to any one of blue, white, and gray may be detected as the sky area.

  With this configuration, it is possible to correctly detect the sky region in the case of a daytime image.

The feature information acquisition unit
(I) dividing the color image into a plurality of divided regions;
(Ii) obtaining frequency characteristics of each segmented region by performing frequency analysis on each segmented region;
(Iii) You may make it detect the division area | region which has a frequency characteristic mainly having a low frequency component as said sky area | region.

  In general, since there are almost no objects in the sky region, there are few high-frequency components and mainly low-frequency components. Therefore, according to the above configuration, it is possible to correctly detect the sky region.

  The top-and-bottom direction determination unit may determine that an image end portion close to a place where the sky region is present most widely is set as the upper side.

  With this configuration, it is possible to accurately determine the vertical direction.

  The feature information acquisition unit may acquire ground area information representing a ground area existing in the color image as the image feature information.

  With this configuration, when the image includes a ground area, the top / bottom direction can be determined based on the ground area information, so that the top / bottom direction of the image can be determined so that the ground area exists correctly below.

The feature information acquisition unit
(I) Perform day / night determination of the color image,
(Ii) When it is determined that the color image is an image taken at night, an area brighter than the average luminance of the color image or a high luminance part having a high luminance equal to or higher than a predetermined luminance An existing dark area may be detected as the ground area.

  In a night image, there is a light in the ground area, so there is a high possibility that a bright area will be the ground area. Street lights and building lights are high-intensity portions scattered in the image. Therefore, with the above configuration, the ground area in the night image can be accurately detected.

The feature information acquisition unit
(I) dividing the color image into a plurality of divided regions;
(Ii) obtaining frequency characteristics of each segmented region by performing frequency analysis on each segmented region;
(Iii) You may make it detect the division area which has a frequency characteristic mainly having a high frequency component as a ground area.

  In general, since there are a large number of objects in the ground area, high frequency components are mainly used. Therefore, it is possible to correctly detect the ground area by the above configuration.

The feature information acquisition unit
(I) Perform day / night determination of the color image,
(Ii) If it is determined that the color image is an image taken at night,
For the first image obtained by dividing the color image into two in the vertical direction and the second image divided into two in the horizontal direction, the luminance contrast of the two divided regions is calculated as the image feature information, respectively.
The top-and-bottom direction determination unit may determine a region having a higher luminance as a lower side in a division direction in which the luminance contrast is higher.

  Generally, in night images, the ground tends to be brighter than the sky. Therefore, the top and bottom direction can be correctly determined by the above configuration.

  The feature information acquisition unit acquires, as the image feature information, straight line information representing a straight line over the entire color image, and the top and bottom direction determination unit includes the straight line between the vertical direction and the horizontal direction of the image. A direction with a larger angle may be determined as the top-and-bottom direction.

  In this configuration, the top-and-bottom direction can be determined using a straight line that traverses the entire image, such as a horizontal line or a horizon line.

The top and bottom direction determination unit
(I) Considering the straight line as a horizon or horizon,
(Ii) When it is determined that the longitudinal direction of the color image and the direction of the straight line are within a predetermined angle range from parallel, the current top-to-bottom relationship may be determined to be correct.

  With this configuration, the top-to-bottom direction can be correctly determined when the image includes a horizontal line or a horizon line.

The feature information acquisition unit further acquires, as the image feature information, sky region information representing a sky region existing in the color image,
The top and bottom direction determination unit
(I) Considering the straight line as a horizon or horizon,
(Ii) The direction in which the sky region exists with respect to the horizon or horizon may be determined as upward.

  Even with this configuration, it is possible to correctly determine the top-to-bottom direction when the image includes a horizontal line or a horizon line.

The top and bottom direction determination unit
(I) Considering the straight line as a horizon or horizon,
(Ii) When only one high-brightness area having a predetermined luminance or higher exists in the vicinity of the straight line, the direction in which the high-brightness area exists with respect to the straight line may be determined as upward.

  With this configuration, for example, when the image is an evening image and the sun is present near the horizon or the horizon, the top-to-bottom direction can be correctly determined.

  The feature information acquisition unit may acquire vanishing line information representing vanishing lines composed of three or more converging straight lines existing in the color image as the image feature information.

  With this configuration, it is possible to correctly determine the vertical direction when the image includes a vanishing line.

  The top and bottom direction determination unit may compare the luminance contrasts on both sides of the vanishing line and determine that the brighter side is the upper side.

  Since there is often a sky above the vanishing line, the top and bottom direction can be correctly determined by the above configuration.

  The feature information acquisition unit may acquire bar-shaped object information representing a bar-shaped object existing in the color image as the image feature information.

  With this configuration, the top-and-bottom direction can be correctly determined when the image includes a bar-like object such as a building.

The feature information acquisition unit
(I) separating the color image into a plurality of color regions based on color information;
(Ii) A color area having an aspect ratio equal to or greater than a predetermined value among the plurality of color areas may be detected as the rod-like object.

  With this configuration, it is possible to easily detect a rod-like object.

Alternatively, the feature information acquisition unit
(I) generating an edge image by spatially differentiating the color image;
(Ii) detecting a closed region included in the edge image;
(Iii) You may make it detect the closed area | region whose aspect ratio is more than predetermined value as the said rod-shaped object.

  Also with these configurations, the rod-shaped object can be easily detected.

The feature information acquisition unit further acquires, as the image feature information, sky region information representing a sky region existing in the color image,
The top-and-bottom direction determination unit determines that a direction along the longitudinal direction of the bar-shaped object and where the overlap exists is upward when one end in the longitudinal direction of the bar-shaped object overlaps the sky region. You may do it.

  When a bar-like object such as a building stands in the vertical direction, it is normal that the upper part is related to the sky region. Therefore, the top and bottom direction can be correctly determined by the above configuration.

The image feature information acquisition unit acquires a plurality of types of image feature information,
The top and bottom direction determining unit may determine the top and bottom direction of the color image using the plurality of types of image feature information.

  In this configuration, since the top-and-bottom direction is determined using a plurality of pieces of image feature information, the determination can be performed more accurately.

The top and bottom direction determination unit
(I) determining the vertical direction of the color image and its probability using each type of image feature information,
(Ii) The vertical direction of the color image may be determined by combining the vertical direction obtained by using each type of image feature information and the probability thereof.

  With this configuration, it is possible to perform the top-and-bottom determination more accurately.

  It should be noted that the present invention can be realized in various modes. For example, an image top-and-bottom direction determination method and apparatus, an electronic album creation method and apparatus, an image processing method and apparatus for an electronic album, and their The present invention can be realized in the form of a computer program for realizing the function of the method or apparatus, a recording medium on which the computer program is recorded, and the like.

Next, embodiments of the present invention will be described in the following order based on examples.
A. Outline of apparatus configuration and processing of embodiment:
B. Details of the top / bottom direction determination process:
C. Variation:

A. Outline of apparatus configuration and processing of embodiment:
FIG. 1 is an explanatory diagram showing an image processing system as an embodiment of the present invention. This image processing system includes a digital camera 100, a computer 200, and a color printer 300. The computer 200 includes an image processing unit 210 that executes an electronic album creation process. The image processing unit 210 may be provided in the digital camera 100 or may be provided in the color printer 300.

  The image processing unit 210 includes a user interface unit 212, an image group selection unit 214, an image feature information acquisition unit 216, a top and bottom direction determination unit 218, a page allocation unit 220, an album generation unit 222, and an image data output unit. 224. A plurality of images used as materials for the electronic album are stored in the image storage unit 230. These images are usually photographic images, but images acquired by a scanner or computer graphics may be used as material for the album. As will be described later, the user interface unit 212 (FIG. 1) has a function as a rule selection unit used when the user selects a top / bottom determination rule. The function of each component of the image processing unit 210 is realized by a computer program.

  FIG. 2 is a flowchart illustrating a processing procedure in the embodiment. In step S100, the user designates various conditions regarding creation of an electronic album. This designation is performed using a user interface screen displayed on the display device by the user interface unit 212.

  FIG. 3 is an explanatory diagram showing an example of a user interface screen for specifying album creation conditions. This user interface screen includes a field F1 for inputting an album name, a plurality of buttons BT for selecting a top / bottom determination rule, and an image selection area ISA for selecting an image to be included in the album. ing.

  The top / bottom determination rule is a rule used when determining the top / bottom of an image. Each rule performs top-and-bottom determination based on different image characteristics. In the example of FIG. 3, rules using four characteristics of “person”, “sky / ground”, “straight line”, and “bar-shaped object” are displayed in a selectable manner. The rule based on “person” is a rule for analyzing an image, extracting a region that is a person as an image feature, and determining the top and bottom based on the person region. The rule based on “sky / ground” is a rule for analyzing the image, extracting a sky or ground area as an image feature, and determining the sky based on the sky / ground area. The rule based on “straight line” is a rule for analyzing the image, extracting one or more straight lines included in the image as image features, and determining the top and bottom based on the straight line. The rule based on the “bar object” is a rule for analyzing the image, extracting a bar object area such as a building as an image feature, and determining the top and bottom based on the bar object area. In addition, as a top-and-bottom determination rule, it is possible to select an arbitrary plurality of rules as selection targets.

  As can be understood from this example, the plurality of top and bottom determination rules determine the top and bottom of the image using different image characteristics. In addition, as an image characteristic, what is obtained from analyzing the pixel value of an image and what is acquired from attached information created at the time of photographing each image such as a subject position can be used. However, it is preferable that the plurality of top and bottom determination rules include a rule that uses at least an image characteristic obtained by analyzing a pixel value of an image.

  The “all” options in the button BT of FIG. 3 are rules for performing final top and bottom judgment by combining individual top and bottom judgments based on each feature. Even when a plurality of types of image features (for example, “person” and “sky / ground”) are selected, final top and bottom determination is performed by combining individual top and bottom determinations based on the respective features.

  Among the eight images G1 to G8 shown in the image selection area ISA of FIG. 3, the first image G1 and the sixth image G6 are on the left side of the image. Therefore, these images G1 and G6 are determined to have the left end in the top direction in the top / bottom determination described later.

  When the album creation conditions are specified, the top and bottom direction of each image and its probability are determined in steps S200, S300, S400, and S500 according to the selected top and bottom judgment rule. At this time, the image group selection unit 214 (FIG. 1) acquires a plurality of images selected in the image selection area ISA from the image storage unit 230. Then, the image feature information acquisition unit 216 acquires image feature information of each image, and the top / bottom direction determination unit 218 determines the top / bottom direction and probability of the image using the image feature information. Details of steps S200, S300, S400, and S500 will be described later.

  In step S600, the top and bottom direction determination unit 218 performs final top and bottom determination by combining the results of top and bottom determination based on a plurality of rules. This process will also be described later. When only one top / bottom determination rule is selected in step S100, the result determined by the rule is used as it is.

  In step S700, the page assignment unit 220 assigns an image to each page of the album. At this time, the image is rotated as necessary so that the orientation of each image faces the correct top-and-bottom direction determined in step S600.

  When the assignment of images to each page is completed in this way, in step S800, the album generation unit 222 generates album image data representing the album image. FIG. 4 shows an example of the generated album image. In this example, the first image G1 and the sixth image G6 are arranged such that the images are rotated so as to face the correct top-and-bottom direction.

  As described above, in this embodiment, image features are extracted by analyzing an image, and the top and bottom of the image is determined based on the features. Therefore, the image is arranged in the correct orientation without any user operation. be able to. In particular, since the top / bottom determination is comprehensively performed using the top / bottom determination based on a plurality of features, the top / bottom determination can be performed with high accuracy.

B. Details of the top / bottom direction determination process:
Hereinafter, the top / bottom direction determination processing will be described in the following order.
B1. Top / bottom direction determination procedure based on person area:
B2. Top-to-bottom direction determination procedure based on sky and ground area:
B3. Top and bottom direction determination procedure based on a straight line:
B4. Top-and-bottom direction determination procedure based on a bar-shaped object:
B5. Comprehensive top and bottom judgment procedure:

B1. Top / bottom direction determination procedure based on person area:
FIG. 5A is a flowchart illustrating the top-and-bottom determination procedure based on the person area. In step S202, the image feature information acquisition unit 216 extracts a person region from a color image to be processed (hereinafter referred to as a “processing target image”). FIG. 5B shows the detailed procedure of step S202.

  In step S262, a skin color region is extracted from the color image. In this case, first, the image data is converted into an HSV (Hue / Saturation / Luminance) color space, and an average luminance Vave of the color image is calculated. Then, for each predetermined range of the average luminance Vave, an appropriate range of skin color pixel values is set and skin color pixels are extracted. Specific examples of the skin color pixel value range are as follows (when the H component range is 0 to 359 and the S and V component ranges are 0 to 255).

(A1) Skin color pixel value range when 120 ≦ Vave:
H component: 10 <H <40
S component: S> 60
V component: V> 150
However, regarding a pixel having a V component of 210 or more, regardless of the value of the S component, a pixel having an H component of 10 <H <45 is defined as a flesh color pixel. This is because it is more appropriate to ignore the S (saturation) component and make a determination only with the H (hue) component for a pixel with extremely high luminance, that is, a pixel considered to be a highlight.

(A2) Skin color pixel value range when 90 ≦ Vave <120:
H component: 0 <H <40
S component: S> 60
V component: V> 100
However, regarding a pixel having a V component of 210 or more, regardless of the value of the S component, a pixel having an H component of 10 <H <45 is defined as a flesh color pixel.

(A3) Skin color pixel value range when Vave <90:
H component: 0 <H <30 or 350 <H <360
S component: 30 <S <200
V component: V> 80
However, for a pixel having a V component of 210 or more, a pixel of 10 <H <35 is defined as a skin color pixel regardless of the value of the S component.

  In this way, if the average luminance Vave (or average brightness) of the image is divided into a plurality of ranges and the range of skin color pixel values is set for each range, depending on whether the entire image is bright or dark, An appropriate skin color pixel range can be set. As a result, skin color pixels can be detected more accurately.

  A set of skin color pixels thus detected is extracted as a skin color region. At this time, only an image area gathered in a state where a certain number (for example, 10 pixels or 1% or more of the total number of pixels) or more skin color pixels are in contact may be extracted as the skin color area. If the skin color area includes facial organs (brows, eyes, nose, mouth, ears) and the like, the area including these organs may be extracted as one skin color area. . The organ of the face is also called “face element”.

  FIG. 6A shows an example of the skin color area FA extracted from the first image G1. In this example, an area including a facial organ is extracted as one skin color area FA.

  It should be noted that any other method can be employed as the skin color pixel determination method. For example, regardless of the values of the S component and the V component, when the H component has a value within a predetermined range, it can be detected as a skin color pixel. As the color space, a color space such as a CIELab color space or a YIQ color space may be used.

  In step S264 of FIG. 5B, the aspect ratio R of each extracted skin color area is calculated. As shown in FIG. 6B, the aspect ratio R is a ratio of the long side length Wlong and the short side length Wshort of the rectangle including the skin color region. In step S266, when the aspect ratio R is within a predetermined range (1.5 to less than 3 in the present embodiment), the skin color area is regarded as a human face area, and the face area (or “person” A flag indicating that the area is “)” is attached. The processes in steps S264 and S266 are executed for all skin color regions in the processing target image (step S268).

  In the present embodiment, the face area is extracted as a person area, but the person area may be extracted by a method other than this. For example, it is possible to extract a wide area including some of a plurality of elements (head, torso, limbs) constituting a person as a person area.

  When the extraction of the person area is thus completed, the top / bottom direction determination unit 218 determines whether or not the person area exists in the processing target image in step S204 of FIG. 5A. If the person area does not exist, it is determined in step S206 that the top / bottom direction of the processing target image is unknown, and the process proceeds to step S600 to wait for comprehensive top / bottom direction determination processing. In this case, the determination result that “the direction of the top and bottom is unknown” in step S206 means that the top and bottom direction cannot be determined based on the person area, and there is a possibility that the top and bottom determination using other features can be performed.

  On the other hand, when a person area exists in the processing target image, the top / bottom direction of the image is determined according to the top / bottom direction of the person area. Specifically, in step S208, the image feature information acquisition unit 216 detects a facial organ in the person region. For the facial organ extraction process, various processes such as a process using a neural network, a process using a color change or luminance change in an image, or a process using a shape in an image can be used. In this embodiment, it is assumed that eyes and mouth are detected as facial organs.

  The top / bottom direction determination unit 218 executes the following processing in steps S210 to S242 depending on whether an eye or a mouth is detected as a facial organ.

  If both eyes and mouth can be detected (step S210), the top-to-bottom direction of the image is determined from the positional relationship of the eyes and mouth with respect to the face region (step S212). For example, as in the example of FIG. 6A, when both eyes and the mouth are detected, the vertical direction is determined by a straight line connecting the middle position of both eyes and the mouth. That is, the direction from the mouth toward the middle position of both eyes is the celestial direction. When one eye and mouth are detected, the top and bottom direction is determined by a straight line connecting the eyes and mouth. In step S214, the determination probability is determined from the detection result of the eyes and mouth. For example, if both eyes and mouth can be detected, the determination probability is determined to be 100%, and if one eye and mouth can be detected, the determination probability is set to 75%.

  If only the eyes can be detected without detecting the mouth (step S220), the top-to-bottom direction of the image is determined from the position of the eyes with respect to the face region (step S222). For example, the direction from the end close to the eyes to the opposite end of the face region is determined as the ground direction (downward). In this case, since only eyes can be detected, the determination probability is set to 50% in step S224. This determination probability is set to a smaller value than when both eyes and mouth are detected (step S214).

  When eyes cannot be detected and only the mouth can be detected (step S230), the vertical direction (upward direction) of the image is determined from the position of the mouth with respect to the face area (step S232). For example, the direction from the end close to the mouth to the opposite end of the face region is determined as the top direction (upward). Since the mouth is located closer to the edge of the face than the eyes, the determination probability is set to a slightly higher determination probability (for example, 60%) than when only the eyes can be detected.

  If neither eyes nor mouth can be detected (step S240), the top-to-bottom direction is unknown and the process proceeds to step S250. When the top / bottom direction is unknown, the determination probabilities for the four sides of the image are all set to 0%. In addition, the determination probability is set to 0% for the sides that are not determined to be in the vertical direction (the vertical direction or the horizontal direction) in steps S212, S222, and S232. Such setting of the determination probability is the same in processing using other features.

  Instead of making the top / bottom direction unknown in step S240, the side closer to the face region may be determined as the top direction among the two sides intersecting the straight line along the longitudinal direction of the face region. In this case, the determination probability is set to a lower value (for example, 30%) than when eyes or a mouth can be detected.

  In step S250, it is determined whether or not the processing related to all the person areas in the processing target image has been completed. When the processing is completed, the process proceeds to step S600 (final top / bottom determination) in FIG.

  When different determination results are obtained in a plurality of person areas, the determination results based on the person areas can be obtained by combining the determination results of the person areas. For example, the determination result having the highest determination probability among a plurality of determination results can be employed. In addition, when there are a plurality of person regions determined to have the same top-and-bottom direction, the determination probability may be modified to increase the determination probability in that direction. For example, when there are two person regions having the same first vertical direction and a determination probability of 75%, the determination probability may be corrected to 90%. At this time, if the probability is 75% of the second vertical direction determined based on another person area, the first vertical direction and the determination probability of 90% are adopted as the final determination result.

  FIG. 6C shows an image rotated according to the vertical direction detected by the above-described processing. Normally, the direction parallel to one of the four sides of the rectangular image is the top-and-bottom direction.

  By the way, as in the example of FIG. 6D, the angle θ between the sky direction Fup determined from the person area and the direction Gup parallel to one side of the image may show a considerably large value. In this case, for example, among the four side directions Gup, Gdown, Gright, and Gleft, the one closest to the top direction Fup of the face region can be selected as the top direction of the image. However, the downward direction Gdown of the image may be excluded from the candidates for the top direction. This is because it is extremely rare to shoot with the camera turned upside down.

  In this way, in the top / bottom direction determination process based on the person area, the processing target image is analyzed to extract the person area, and the top / bottom direction of the image is determined from the top / bottom direction of the person area. Can be determined. In particular, if the top / bottom direction is determined from the position of the facial organ in the person area, the top / bottom direction can be determined more accurately.

  In the above description, the top-and-bottom direction is determined according to the position of the facial organ in the face area. Instead, the position of the face area and the hair portion (hair area) adjacent to the face area is determined. It is also possible to determine the top and bottom direction from the relationship. It is also possible to determine the top-and-bottom direction from the positional relationship between the skin color region (face region) and the body part (body region) adjacent to the skin color region. The hair part and the body part can be determined using, for example, shape template matching using typical shapes of these parts.

B2. Top-to-bottom direction determination procedure based on sky and ground area:
FIG. 7 is a flowchart showing the top / bottom determination procedure based on the sky / ground area. In step S302, the image feature information acquisition unit 216 obtains the average luminance of the processing target image, and performs day / night determination according to the average luminance. Specifically, when the average luminance of the image is equal to or higher than a predetermined threshold (for example, 110), it is determined that the image is captured in the daytime, and when it is less than the threshold, it is determined that the image is captured at night. Note that day / night determination may be performed based on shooting time information attached to image data at the time of shooting.

  FIG. 8A shows the second image G2 of FIG. This image G2 is an image taken in the daytime and includes a bright sky region.

  If it is determined that the image is a daytime image, the processes in and after step S306 are executed. First, in step S306, the image feature information acquisition unit 216 flags pixels that are brighter than the average luminance of the entire image. In step S308, the pixels that are flagged and are continuous are connected. Hereinafter, an area where pixels are connected in this manner is referred to as a “connected area”. In step S310, the color of the connected area is checked, and if the connected area has a predetermined color belonging to any of blue, white, and gray, it is flagged again as a sky area candidate (also called “sky area candidate”). Do. The range of pixel values belonging to blue, white, and gray can be set as follows, for example.

Blue: 200 <H <240, S is full range, V> (1.5 times the average luminance Vave)
White: H is the entire range, S <10, V> (1.5 times the average luminance Vave)
Gray: H is the entire range, S <30, V> (1.5 times the average luminance ave)

  For example, (i) when 70% or more of pixels in a connected region are classified as a color belonging to any one of blue, white, and gray, as a criterion for determining whether or not to flag a certain region as an empty region candidate, Alternatively, (ii) some criteria such as when the average color of the connected region is classified into a color belonging to any of blue, white, and gray can be adopted. FIG. 8B shows an empty area SA (empty area candidate) extracted with respect to the image G2.

  When the sky region candidate is detected in this way, in step S312, the top / bottom direction determination unit 218 sets an end (side) where the sky region candidate is widest among the four sides of the image above the image (top direction). ). In this case, when the sky region candidate is in contact with a plurality of sides, the length of the sky region candidate on each side is calculated, and the side having the largest ratio of the length of the sky region candidate to the length of each side is calculated. Is determined to be above the image. In the example of FIG. 8B, the upper side of the image G2 is determined to be above the image (upward direction).

  In step S314, when the sky region candidate covers only one side, the determination probability is set to 100%. In addition, when the upward direction is determined based on the length of the empty area candidate on each side, the determination probability is set to 75%.

  Thus, for a daytime image, if the sky direction of the image is determined according to which of the four sides of the image is the longest, the sky direction can be determined fairly accurately. Is possible.

  On the other hand, if it is not determined that the image is a day image in step S304, it is determined in step S320 whether the image is a night image. Specifically, for example, when the average luminance Vave of the entire image is less than 90, it can be determined that the night. Alternatively, the image may be divided into nine equal parts, and the determination may be made as night when the average luminance Vave of the central block is less than 90. If neither the daytime image nor the nighttime image is determined, it is determined in step S330 that the top-and-bottom direction is unknown, and the process proceeds to step S600 (total top-and-bottom determination processing).

  FIG. 9A shows the third image G3 of FIG. Since this image G3 is dark overall and low in luminance, it is determined that the image G3 is an image taken at night. Note that white dots in the image G3 are bright objects such as street lamps, building windows, and automobile headlights.

  If it is determined in step S320 that the image is a night image, the image feature information acquisition unit 216 equally divides the processing target image into four blocks in step S322, and obtains the average luminance of each block. FIG. 9B shows a state in which the image G3 is divided into four blocks B1 to B4. In step S324, as shown in FIG. 9C, an upper area Aupper composed of two blocks B1 and B2 above the image and a lower area Alower composed of two blocks B3 and B4 below. Luminance contrast CT1 is calculated. The luminance contrast CT1 is a ratio of the average luminance Vave (Aupper) in the upper area Aupper and the average luminance Vave (Alower) in the lower area Alower. Similarly, as shown in FIG. 9 (D), a left area Aleft composed of two blocks B1 and B3 on the left side of the current image direction, and two blocks B2 and B4 on the right side. The luminance contrast CT2 = Vave (Aleft) / Vave (Aright) between the right areas Aright is calculated.

  In step S322, the top / bottom direction determination unit 218 compares the luminance contrasts CT1 and CT2 and determines the division direction that gives a larger contrast as the top / bottom direction of the image. Furthermore, among the top-and-bottom directions, the one with higher luminance is determined to be downward (the ground direction). In the examples of FIGS. 9C and 9D, since the up and down division contrast CT1 of FIG. 9C is larger, it is determined that the vertical direction of the current image is the vertical direction. Moreover, since the brightness is higher in the vertical direction of the image, it is determined that the current lower direction is the ground direction. In the present specification, the “dividing direction” means a direction perpendicular to the dividing line.

  In step S328, when the brightness contrast value is sufficiently large (for example, 2 or more), the determination probability is set to 100%, and in other cases, the determination probability is set to 75%.

  In this way, in the top / bottom direction determination process based on the sky / ground area, the sky area is extracted in the case of a daytime image, and the top / bottom direction is determined according to the position of the sky area in the image. The top and bottom direction can be determined. In the case of a night image, the vertical direction is determined using the vertical and horizontal luminance contrasts in the image, so that the vertical direction can also be correctly determined for a night image. . However, without determining whether the image is a daytime image or a night image, the sky region in the image may be extracted, and the top-and-bottom direction may be determined from the position of the sky region in the image.

  In addition, when an empty area (empty area candidate) is extracted, other areas may be recognized as the ground area. Alternatively, even when the sky area cannot be recognized, an area having a specific color or shape (for example, a vast area of brown or green) may be extracted as the ground area. In this case, it is possible to determine the top / bottom direction according to the position of the ground region in the image.

  FIG. 10 shows a modification of the processing after step S320 in FIG. In this processing procedure, steps S322 to S328 in FIG. 7 are replaced by steps S340 to S344, and other procedures are the same as those in FIG.

  In step S340, the image feature information acquisition unit 216 equally divides the processing target image into four blocks, and in each block, the number of pixels H1 to H4 having a luminance higher than the average luminance (“high luminance pixel number”). Count). FIGS. 11A and 11B show a state in which the number of high luminance pixels H1 to H4 related to the image G3 and the four blocks B1 to B4 is detected. The number of block divisions can be set to any value other than 4. However, it is preferable to set the vertical division number and the horizontal division number of the image to the same value.

  In step S342, the top / bottom direction determination unit 218 compares the high-luminance pixel numbers H1 to H4 of the respective blocks, and determines the direction in which the block having the largest value of the high-luminance pixel number is downward (the ground direction). In the case of FIG. 11B, the lower side or the right side of the image in contact with the fourth block B4 is determined as the lower side (the ground direction) of the image. This is because a place where there is a light in a night image is likely to be close to the ground. In this determination, it is preferable to determine that the lower side or the right side of the image that touches the block B3 having the second high luminance pixel number is the lower side. FIG. 11C shows a case where the block is divided into nine blocks B1 to B9. In this case, since the number of high luminance pixels H8 of the eighth block B8 is the largest, the lower side of the image in contact with the block B8 is determined as the downward direction of the image.

  In step S344, the difference between the maximum value Hmax and the minimum value Hmin of the number of high-luminance pixels is obtained. If this difference is large, the determination probability is set to 100%, and if it is small, the determination probability is set to 75%. Then, the process proceeds to step S600 (total top / bottom determination process). The reason why the determination probability is set to a higher value when (Hmax−Hmin) is large is that the distinction between the ground and the sky is considered to be clearer in this case.

  FIG. 12 shows another modification of the processing after step S320 in FIG. In this processing procedure, steps S322 to S328 in FIG. 7 are replaced with steps S350 to S354, and other procedures are the same as those in FIG.

  In step S350, the image feature information acquisition unit 216 equally divides the processing target image into 16 blocks, performs frequency analysis in each block, and flags a block having a large high frequency component.

  FIG. 13 shows the result of frequency analysis performed on 16 blocks of the image G3. Here, the low frequency component is the main component in the eight blocks on the upper side of the image, and the high frequency component is the main component in the eight blocks on the lower side. Here, whether the high frequency component or the low frequency component is the main component is determined by dividing the entire frequency region into two at a predetermined boundary frequency, for example, and summing the components in the high frequency region (or the maximum value). ) And the sum (or maximum value) of the components in the low frequency region. In this modification as well, the number of block divisions can be set to any value other than 16.

  In step S352, the top / bottom direction determination unit 218 regards an area where many high-frequency blocks exist as a ground area, and determines an end (side) of the image close to that area as the lower side (ground direction) of the image. Specifically, for example, it is possible to detect a ground area composed of one or more high-frequency blocks adjacent to each other and determine the side of the image closest to the ground area as the lower side of the image. .

  In step S354, the determination probability is set to 80% when the high-frequency block is in contact with the end (side) determined to be below the image in step S352, and the determination probability is set to 60% otherwise. Set. The reason why a higher determination probability is set when the high-frequency block is in contact with the edge determined to be below the image is that there is a high possibility that the ground area can be detected more correctly.

  As can be understood from the above-described modified examples of FIGS. 7, 10, and 12, for night images, it is possible to determine the top-to-bottom direction of the images by using the luminance or frequency analysis results of the divided blocks. is there. However, for night images, the ground region can be detected by various methods other than those described above. For example, an area brighter than the average brightness of the processing target image may be detected as a ground area, or a dark area in which high-intensity portions having high brightness equal to or higher than a predetermined brightness are scattered may be detected as a ground area. Good.

  Note that the top and bottom determination using the frequency analysis of FIG. 12 can be applied not only to a night image but also to a day image. In the daytime image, since the low frequency component is the main component in the sky region, it is possible to select a region or block mainly containing the low frequency component as the sky region (empty region candidate). Therefore, regardless of whether it is a daytime image or a nighttime image, it is possible to determine an area mainly including low-frequency components as an empty area and an area mainly including high-frequency components as a ground area.

B3. Top and bottom direction determination procedure based on a straight line:
14A and 14B are flowcharts showing the top and bottom determination procedure based on a straight line. In step S402, the image feature information acquisition unit 216 detects a straight line in the processing target image. The straight line in the image is obtained by, for example, spatially differentiating the processing target image to obtain the edge amount of each pixel, and generating an image (edge image) composed of the edge amount of each pixel. It can be obtained by performing a Hough transform. The spatial differentiation of the image can be performed using a differential filter such as a Laplacian filter.

  In step S404, the top / bottom direction determination unit 218 determines whether or not a straight line is detected. Here, the straight line means a line segment having a predetermined length (for example, 100 pixels) or more. If a straight line is not detected, it is determined in step S440 that the top / bottom direction is unknown, and the process proceeds to step S600 (total top / bottom determination processing). On the other hand, if a straight line is detected, it is determined in step S406 whether the straight line is a straight line extending in the vertical or horizontal direction of the image. When there is no straight line extending in the vertical direction or the entire horizontal direction, the process proceeds to a convergence straight line determination process (described later) shown in FIG. 14B.

  On the other hand, when there is a straight line over the entire image, the top / bottom direction determination unit 218 executes the processing after step S410 in FIG. 14A. FIG. 15 shows an example of an image including a straight line (also referred to as a “penetrating straight line”) over the entire image. This image G4 includes a horizontal line, and this horizontal line is determined as a through straight line TSG. Note that the images G2, G5, and G6 shown in FIG. 3 also include straight lines, but do not include through straight lines.

  In step S410, first, it is determined whether or not the longitudinal direction of the image matches the direction of the through straight line. This determination can be made based on whether or not the angle formed between the longitudinal direction of the image and the direction of the penetrating straight line is equal to or less than a predetermined value (for example, 20 degrees). If there is a long straight line that coincides with the longitudinal direction of the image, it is often considered that the straight line is a horizontal line or a horizon line. Therefore, when these two directions coincide with each other, in step S412, it is determined that the current image is directly above, the determination probability is set to 75%, and the process proceeds to step S600 (total top / bottom determination processing). This is because it is extremely rare to take a picture by turning the camera upside down when taking a landscape image. On the other hand, if the two directions do not match, the image feature information acquisition unit 216 detects an empty area in step S420. This process is the same as that described in step S310 of FIG.

  If the sky area can be detected in step S420, the direction in which the sky area is detected is determined to be upward in step S422, the determination probability is set to 75%, and the process proceeds to step S600 (total top / bottom determination processing). However, as in steps S312 and S314 in FIG. 7, the determination probability may be set from the relationship between the sky region and the side (edge) of the image.

  On the other hand, if an empty area cannot be detected in step S420, it is determined in step S430 whether a high luminance area exists in the vicinity of the through straight line. Here, the “high luminance region” means, for example, a region where a predetermined number (for example, 20 pixels) or more of pixels having a V value in the HSV color space that is equal to or larger than a predetermined value (for example, 200). . For example, when a sunset image is taken, the sun is recognized as a high brightness area. If a high-luminance area is detected in the vicinity of the through-line, in step S432, the side of the image closer to the high-luminance area than the through-line is determined as the upper side, and the determination probability is set to 75%. Move to (Comprehensive top / bottom judgment processing). When two or more high-luminance areas are detected, there is a high possibility that the object is due to specular reflection of the object, and it may be determined that there is no high-luminance area. If a high brightness area cannot be detected in the vicinity of the through straight line in step S430, it is determined in step S440 that the top / bottom direction is unknown, and the process proceeds to step S600 (total top / bottom determination processing).

  In the convergence straight line determination process shown in FIG. 14B, first, in step S450, the image feature information acquisition unit 216 determines whether or not three or more straight lines are detected. If three or more straight lines are detected, a set of three or more straight lines that converge to approximately one point (referred to as a “converging straight line group” or “disappearing line group”) is searched in step S452. FIG. 16A shows the original image G5, and FIG. 16B shows a straight line group CSG that converges at the convergence point CP. The convergence point CP of the convergence straight line group CSG may have a certain allowable range.

  If three straight lines are not detected in step S450, or if a convergent straight line group is not detected in step S452, it is determined that the vertical direction is unknown, and the process proceeds to step S600 (total vertical determination process).

  On the other hand, when the convergence straight line group is detected in step S452, the image feature information acquisition unit 216 calculates the luminance contrast on both sides of the convergence straight line group. Specifically, for example, as shown in FIG. 16C, the average luminance in each of the two regions A1 and A2 sandwiching the convergence straight line group CSG is calculated, and the ratio of the average luminance is calculated as the luminance contrast. Then, the side of the image in which the region with the higher luminance exists is determined to be upward (top direction), the determination probability is set to 60%, and the process proceeds to step S600 (total top / bottom determination processing). The reason why the area with higher luminance is determined as the upper direction of the image is that a higher luminance is more likely to be closer to the sky. Note that the setting probability in this case may also be set according to the relationship between the higher brightness area and the side (edge) of the image, as in steps S312 and S314 of FIG.

  Thus, in the top / bottom direction determination process based on a straight line, when there is a penetrating straight line as shown in FIG. 15, the top / bottom of the image is correctly determined according to the direction of the penetrating straight line and the positional relationship between the penetrating straight line and the sky region. can do. Therefore, it is possible to correctly determine the top and bottom of the image when there is a through straight line such as a horizontal line or a horizon line. Further, when the convergence straight line group exists, the top and bottom of the image can be correctly determined according to the luminance on both sides of the convergence straight line group.

  When the image includes a penetrating straight line, instead of the above-described determination, the direction in which the angle formed by the penetrating straight line is larger among the two directions of the vertical and horizontal directions of the image is determined as the vertical direction. You may do it. For example, when the direction of the penetrating straight line is close to the horizontal direction of the image, it is determined that the vertical direction of the image is the top-and-bottom direction, and the original image direction is correctly facing the top and bottom. On the other hand, when the direction of the penetrating straight line is close to the vertical direction of the image, the horizontal direction of the image is the vertical direction. In this case, the celestial direction or the ground direction may be determined using other image characteristics (for example, left and right contrast of the image).

  When the image includes a convergence line group, the top and bottom may be determined based on the direction of the convergence line group instead of the above determination. Usually, as in the example of FIG. 16, the convergence point CP of the convergence line group is often directed upward in the image. Therefore, it is possible to determine the vertical direction or the horizontal direction of the image as the top-and-bottom direction based on the direction in which the convergence line group is directed. Also in this case, the celestial direction or the ground direction may be determined using other image characteristics (for example, image contrast).

B4. Top-and-bottom direction determination procedure based on a bar-shaped object:
FIG. 17 is a flowchart showing the top / bottom determination procedure based on the bar-shaped object. A bar-like object means a long subject such as a building.

  In step S502, the image feature information acquisition unit 216 performs region division processing (region separation processing) using the color information of each pixel of the processing target image. Specifically, for example, the H component of the HSV color space is divided into a plurality of predetermined ranges in advance, and the region composed of pixels having the H component in each range is extracted to perform region division. Is possible. In this case, when the number of pixels in a certain area is extremely small (for example, 10 pixels or less), the area may be absorbed by a larger surrounding area.

  It should be noted that the area division can also be performed using other methods. For example, it is possible to obtain an edge image by spatially differentiating the processing target image and to perform region division by detecting a closed region in the edge image.

  In step S504, the image feature information acquisition unit 216 performs flagging as a bar-shaped object for an area having an aspect ratio of a predetermined value (for example, four times) or more. FIG. 18A shows an example of an image G6 including a rod-shaped object SOB. As shown in FIG. 18B, the aspect ratio R is a ratio of the long side length Wlong and the short side length Wshort of the rectangle including the rod-shaped object SOB.

  In step S506, the top / bottom direction determination unit 218 determines whether one end of the bar-shaped object overlaps the sky region. The empty area extraction processing is the same as that described in step S310 in FIG. As shown in FIG. 18A, when one end of the bar-shaped object overlaps (or is in contact with) the empty area, the direction of the overlapping end is determined to be upward (top direction), and the determination probability is 75% is set (steps S508 and S510). If there is a next rod-like object, the process returns from step S512 to step S506, and the processes after step S506 are repeated. On the other hand, if both ends of the rod-shaped object do not overlap the sky area in step S506, or if there is no sky area, it is determined that the top / bottom direction is unknown, and the process proceeds to step S600 (total top / bottom determination processing). Move.

  In the case of the image G6 in FIG. 18A, since the left end of the rod-shaped object SOB overlaps with the empty area, it is determined that the left side of the image is the sky direction. FIG. 18C shows an image rotated according to the determined vertical direction.

  Thus, when there is a bar-like object such as a building in the image, there is a high possibility that the longitudinal direction is the vertical direction. Therefore, as described above, it is possible to determine the top-and-bottom direction fairly accurately using the rod-like object.

B5. Comprehensive top and bottom judgment procedure:
FIG. 19 is a flowchart illustrating the procedure of the comprehensive top / bottom determination process. The top / bottom direction determination unit 218 uses the top / bottom direction determination result using each feature selected in FIG. 3 to add a determination probability to each end (each side) of the image (steps S602 and S604). Specifically, the determination probabilities calculated using the features are added to the upper side, the lower side, the left side, and the right side of the image. Note that, as described above, in the determination process using each feature, when the top-to-bottom direction is unknown, all determination probabilities for the four sides are set to 0%. In addition, the determination probability is set to 0% for the sides that are not determined to be the celestial direction or the ground direction. Note that the addition of the determination probability is performed with respect to either the celestial direction or the ground direction. In this case, when there is a side determined as the ground direction, it is considered that the opposite side is determined as the top direction. In this embodiment, the determination probability is added with respect to the sky direction.

  When all of the determination probabilities have been added, in step S606, the end (side) having the highest determination probability is finally determined as upward (upward direction), and the result is output in step S608. As the result output, the result of rotating the image so that the end (side) finally determined to be upward may be output, or the respective ends are numbered in advance, May be output.

  In this way, in this embodiment, the top and bottom direction and its determination probability are determined for each of the plurality of features, the determination probability is added for each side of the image, and the final top and bottom direction is determined using the determination probability after the addition. Since it is determined, it is possible to determine the vertical direction with high accuracy. In addition, since the user can arbitrarily select the type of feature used for the top / bottom direction determination from predetermined options, it is possible to proceed by excluding features that seem to have little relationship in advance. As a result, it is possible to obtain the determination result with higher accuracy and in a shorter time.

  In the above-described comprehensive top / bottom determination process, instead of adding the determination probabilities for each side, the side that indicates the highest value among the determination probabilities obtained using the features is represented by the top / bottom direction of the image (top or bottom direction). It may be determined as (ground direction). However, if the determination probability is added, there is a possibility that the correct top-and-bottom direction can be determined more reliably.

C. Variation:
In addition, this invention is not restricted to the said Example and embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

C1. Modification 1:
In the above embodiment, the HSV color space is used as the color space, but it is possible to perform processing using any other color space.

C2. Variation 2:
In the above-described embodiment, the determination probability is calculated at the same time when determining the vertical direction using each feature. However, the determination probability is not necessarily calculated.

C3. Variation 3:
In the above embodiment, four types of image features are used: a person region, a sky region / ground region, a straight line, and a rod-like object. However, only a part of these are used as image features. It is also possible to use various other features. However, since it is expected that the person area and the sky area are detected in many photographic images, it is preferable to use at least these two types of features.

C4. Variation 4:
In the above embodiment, the method and apparatus for creating an electronic album has been described. However, the determination of the vertical direction of an image according to the present invention is not limited to the creation of an electronic album, but can be used for other purposes.

1 is an explanatory diagram showing an image processing system as one embodiment of the present invention. The flowchart which shows the process sequence in an Example. Explanatory drawing which shows the example of the user interface screen for album creation condition designation | designated. Explanatory drawing which shows an example of an album image. The flowchart which shows the top-and-bottom determination procedure based on a person area. The flowchart which shows the detailed procedure of step S202. Explanatory drawing which shows the content of the top-and-bottom determination process based on a person area. The flowchart which shows the top-and-bottom determination procedure based on the sky / ground area. Explanatory drawing which shows the content of the top-and-bottom determination process based on the sky and ground area regarding a daytime image. Explanatory drawing which shows the content of the top-and-bottom determination process based on the sky and the ground area regarding a night image. The flowchart which shows the modification of the top-and-bottom determination procedure based on the sky / ground area. Explanatory drawing which shows the content of the process of FIG. The flowchart which shows the other modification of the top-and-bottom determination procedure based on the sky and the ground area. Explanatory drawing which shows the content of the process of FIG. The flowchart which shows the top-and-bottom determination procedure based on a straight line. The flowchart which shows the procedure of a convergence straight line determination. Explanatory drawing which shows the example of the image containing the straight line over the whole image. Explanatory drawing which shows the example of the image containing a convergence straight line group. The flowchart which shows the top-and-bottom determination procedure based on a rod-shaped object. Explanatory drawing which shows the content of the top-and-bottom determination process based on a rod-shaped object. The flowchart which shows the procedure of a general top-and-bottom determination process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Digital camera 200 ... Computer 210 ... Image processing part 212 ... User interface part 214 ... Image group selection part 216 ... Image feature information acquisition part 218 ... Top and bottom direction determination part 220 ... Page allocation part 222 ... Album generation part 224 ... Image data Output unit 230 ... Image storage unit 300 ... Color printer

Claims (34)

An image processing apparatus for determining a vertical direction of a color image,
A feature information acquisition unit that acquires predetermined image feature information representing features of the color image by analyzing pixel values of the color image;
A top and bottom direction determination unit that determines the top and bottom direction of the color image using a determination rule that uses the image feature information; and
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The top / bottom direction determination unit determines the probability together with the top / bottom direction of the color image, and adopts a top / bottom direction with a high probability. The image processing apparatus according to claim 1 or 2,
The feature information acquisition unit acquires person area information representing a person area existing in the color image as the image feature information,
The top / bottom direction determination unit is an image processing apparatus that determines the top / bottom direction of the color image according to a top / bottom direction of the person area. The image processing apparatus according to claim 3,
The feature information acquisition unit is an image processing apparatus that detects a skin color area existing in the color image as the person area. An image processing apparatus according to claim 4,
The feature information acquisition unit is an image processing apparatus that changes a range of pixel values recognized as skin color according to an average luminance of the color image. An image processing apparatus according to any one of claims 3 to 5,
The top-and-bottom direction determination unit is an image processing apparatus that uses the presence or absence of a body region adjacent to the skin color region for the top-and-bottom direction determination. The image processing apparatus according to any one of claims 3 to 6,
The feature information acquisition unit is an image processing apparatus that detects a face area of a person existing in the color image as the person area. The image processing apparatus according to claim 7,
The image processing apparatus, wherein the feature information acquisition unit extracts a skin color area existing in the color image, and selects a skin color area to be the face area from an aspect ratio of the extracted skin color area. The image processing apparatus according to claim 8, wherein
The feature information acquisition unit selects the face area candidate from the aspect ratio of the skin color area, and selects an area including a facial organ as the face area from the face area candidates. apparatus. The image processing apparatus according to claim 7,
The image processing apparatus, wherein the feature information acquisition unit selects an area that is a skin color area and includes a facial organ as the face area. The image processing apparatus according to claim 10 or 9, wherein
The top-to-bottom direction determination unit is an image processing device that uses a positional relationship of the facial organs in the face region for the top-to-bottom direction determination. An image processing apparatus according to any one of claims 7 to 11,
The top and bottom direction determination unit is an image processing apparatus that uses the presence or absence of a hair region adjacent to the face region for the top and bottom direction determination. The image processing apparatus according to claim 1 or 2,
The feature information acquisition unit is an image processing apparatus that acquires sky region information representing a sky region existing in the color image as the image feature information. The image processing apparatus according to claim 13,
The feature information acquisition unit
(I) Perform day / night determination of the color image,
(Ii) When it is determined that the color image is an image taken in the daytime, the color image is an area composed of pixels brighter than the average luminance of the color image, and an end portion of the color image An image processing apparatus that detects an area that is present in a color and belongs to any one of blue, white, and gray as the sky area. The image processing apparatus according to claim 13,
The feature information acquisition unit
(I) dividing the color image into a plurality of divided regions;
(Ii) obtaining frequency characteristics of each segmented region by performing frequency analysis on each segmented region;
(Iii) An image processing apparatus that detects a segmented region having frequency characteristics mainly including a low-frequency component as the sky region. The image processing apparatus according to any one of claims 13 to 15,
The top / bottom direction determination unit is an image processing device that determines an image edge portion close to a place where the sky region is present most widely as an upper side. The image processing apparatus according to claim 1 or 2,
The feature information acquisition unit is an image processing device that acquires ground region information representing a ground region existing in the color image as the image feature information. The image processing device according to claim 17,
The feature information acquisition unit
(I) Perform day / night determination of the color image,
(Ii) When it is determined that the color image is an image taken at night, an area brighter than the average luminance of the color image or a high luminance part having a high luminance equal to or higher than a predetermined luminance An image processing apparatus for detecting an existing dark area as the ground area. The image processing device according to claim 17,
The feature information acquisition unit
(I) dividing the color image into a plurality of divided regions;
(Ii) obtaining frequency characteristics of each segmented region by performing frequency analysis on each segmented region;
(Iii) An image processing apparatus that detects a segmented area having frequency characteristics mainly including a high-frequency component as a ground area. The image processing apparatus according to claim 1 or 2,
The feature information acquisition unit
(I) Perform day / night determination of the color image,
(Ii) If it is determined that the color image is an image taken at night,
For the first image obtained by dividing the color image into two in the vertical direction and the second image divided into two in the horizontal direction, the luminance contrast of the two divided regions is calculated as the image feature information, respectively.
The top-and-bottom direction determination unit is an image processing apparatus that determines a region having a higher luminance as a lower side in a division direction in which the luminance contrast is higher. The image processing apparatus according to claim 1 or 2,
The feature information acquisition unit acquires, as the image feature information, straight line information representing a straight line over the entire color image,
The top-to-bottom direction determination unit is an image processing apparatus that determines a top-to-bottom direction as a direction that has a larger angle with the straight line between a vertical direction and a horizontal direction of the image. The image processing apparatus according to claim 21, wherein
The top and bottom direction determination unit
(I) Considering the straight line as a horizon or horizon,
(Ii) An image processing apparatus that determines that the current top-to-bottom relationship is correct when it is determined that the longitudinal direction of the color image and the direction of the straight line are within a predetermined angle range from parallel. The image processing apparatus according to claim 21, wherein
The feature information acquisition unit further acquires, as the image feature information, sky region information representing a sky region existing in the color image,
The top and bottom direction determination unit
(I) Considering the straight line as a horizon or horizon,
(Ii) An image processing apparatus that determines that the direction in which the sky region is present is upward with respect to the horizon or the horizon. The image processing apparatus according to claim 21, wherein
The top and bottom direction determination unit
(I) Considering the straight line as a horizon or horizon,
(Ii) An image processing apparatus that, when only one high-brightness area having a predetermined luminance or more exists in the vicinity of the straight line, determines that the direction in which the high-brightness area exists with respect to the straight line is upward. The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the feature information acquisition unit acquires vanishing line information representing vanishing lines composed of three or more converging straight lines existing in the color image as the image feature information. The image processing apparatus according to claim 25, wherein
The top-and-bottom direction determination unit is an image processing apparatus that compares luminance contrasts on both sides of the vanishing line and determines that the brighter side is the upper side. The image processing apparatus according to claim 1 or 2,
The feature information acquisition unit is an image processing apparatus that acquires, as the image feature information, bar-shaped object information representing a bar-shaped object existing in the color image. The image processing apparatus according to claim 27, wherein
The feature information acquisition unit
(I) separating the color image into a plurality of color regions based on color information;
(Ii) An image processing apparatus that detects a color area having an aspect ratio equal to or greater than a predetermined value among the plurality of color areas as the rod-like object. The image processing apparatus according to claim 27, wherein
The feature information acquisition unit
(I) generating an edge image by spatially differentiating the color image;
(Ii) detecting a closed region included in the edge image;
(Iii) An image processing apparatus that detects a closed region having an aspect ratio equal to or greater than a predetermined value as the rod-like object. 30. The image processing apparatus according to claim 27, wherein:
The feature information acquisition unit further acquires, as the image feature information, sky region information representing a sky region existing in the color image,
The top-and-bottom direction determination unit determines that a direction along the longitudinal direction of the bar-shaped object and where the overlap exists is upward when one end in the longitudinal direction of the bar-shaped object overlaps the sky region. , Image processing device. The image processing apparatus according to claim 1 or 2,
The image feature information acquisition unit acquires a plurality of types of image feature information,
The top-to-bottom direction determination unit is an image processing apparatus that determines the top-to-bottom direction of the color image using the plurality of types of image feature information. The image processing apparatus according to claim 31, wherein
The top and bottom direction determination unit
(I) determining the vertical direction of the color image and its probability using each type of image feature information,
(Ii) An image processing apparatus that determines the vertical direction of the color image by combining the vertical direction obtained using each type of image feature information and the probability thereof. A method for determining the top and bottom of a color image,
(A) obtaining predetermined image feature information representing features of the color image by analyzing pixel values of the color image;
(B) determining the vertical direction of the color image using a determination rule using the image feature information;
An image top-and-bottom determination method comprising: A computer program for determining the top and bottom of a color image,
A feature information acquisition function for acquiring predetermined image feature information representing features of the color image by analyzing pixel values of the color image;
A top / bottom direction determination function for determining the top / bottom direction of the color image using a determination rule using the image feature information;
A computer program that causes a computer to realize

Images