JP2006350642A - Image processing device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique that determines the vertical direction by focusing on information in an image other than characters and brightness at four corners. <P>SOLUTION: People are mostly on the ground, floor or the like, where they normally take different positions on the horizontal direction. In an image having a plurality of persons, the plurality of persons tend to be arranged side by side so that face images (skin regions) are disposed horizontally rather than vertically. Hence, the skin regions in the image are extracted, the dispersion of the skin regions is computed in two orthogonal directions, and the vertical direction is determined from the computations. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for determining a vertical direction in an image obtained by photographing a subject.

  In a photographing apparatus such as a digital camera, the shape of an image obtained by photographing (hereinafter referred to as “frame shape”) is usually predetermined. As a result, the user usually selects how to hold the photographing apparatus according to the subject to be photographed. As for the posture, when the short side of the frame shape is parallel or substantially parallel to the top-and-bottom direction, it is “horizontal”, and when the short side is parallel or substantially parallel to the horizontal direction, “vertical posture”. I will call it. The user usually selects either horizontal or vertical.

  In general, an image capturing apparatus normally stores an image as it is. As a result, when the stored image is simply displayed on a personal computer (hereinafter referred to as “PC”) or the like, the image captured in the vertical orientation is displayed with the horizontal direction set to the vertical direction. That is, the subject image is displayed in a rotated state from the state actually viewed by the photographer (user).

  When the subject image is displayed in a rotated state, it is usually difficult to see. For this reason, it is desirable to rotate the state to the original state. However, it can be said that instructing the user to rotate the rotation is not desirable because it means that it takes time to display an image in an appropriate state. For this reason, conventionally, image processing for determining the vertical direction of an image has been performed. Examples of conventional image processing apparatuses that perform image processing for determining the vertical direction include those described in Patent Documents 1 to 3.

  In the conventional image processing apparatus described in Patent Document 1, the top-to-bottom direction is determined by recognizing characters in an image. In the conventional image processing apparatuses described in Patent Documents 2 and 3, focusing on the fact that the upper part of an image taken in a general environment is bright, and comparing the brightness of the four corners of the image, the upper side ( (Vertical direction) is determined. As an imaging device that can perform the determination, there are a device equipped with a sensor for detecting the top and bottom (gravity sensor) and a device that has prepared a shutter button for a vertical posture and a horizontal posture (Patent Document 4).

  In the method using the character recognition technology, it is impossible to determine the vertical direction unless characters are shown in the image. In landscape photos and portraits, it is normal that characters are not shown. In addition, in the method that focuses on the brightness of the four corners of the image (photo), the brightness of the subject located at the four corners greatly affects the determination result of the top and bottom direction. Condition). Due to the restriction, the vertical direction cannot always be determined accurately. For this reason, it is considered necessary to pay attention to things other than the brightness of characters and four corners in order to accurately determine the vertical direction.

In the photographing apparatus described in Patent Document 4, the top and bottom direction can always be accurately determined by adding the detection result of the sensor or information indicating the shutter button used for photographing to the image. However, for that purpose, it is necessary to correspond to special information that the photographing apparatus stores together with the image. This means that when a stored image is displayed by another device, the device must be provided with a mechanism for corresponding to the special information. From this, it is considered desirable to determine the vertical direction from information in the image.
JP-A-8-336038 Japanese Patent Laid-Open No. 2000-24184 JP 2000-184271 A JP 2004-343283 A

  An object of the present invention is to provide a technique for determining the top and bottom direction by paying attention to information in an image other than characters and brightness of four corners.

  The image processing apparatus of the present invention is premised on performing image processing for determining the top-to-bottom direction in an image obtained by photographing a subject, and includes an image acquisition unit that acquires an image, and an image acquired by the image acquisition unit. Skin area recognition means for recognizing the skin area of a human image photographed as a subject, and direction determination means for determining the top-to-bottom direction based on the recognition result of the skin area by the skin area recognition means.

  The direction determining means calculates a variance value for the skin area recognized by the skin area recognizing means for each direction parallel to the edge of the image, and determines one of the directions parallel to the edge from the calculation result. It is desirable to determine one as the vertical direction. It is also desirable to calculate a variance value in consideration of the size of the skin area recognized by the skin area recognition means.

  The program of the present invention has a function for realizing the means included in the image processing apparatus.

  The present invention recognizes a skin area of a human image photographed as a subject in an image, and determines the top / bottom direction in the image based on the recognition result. A person usually has at least a face exposed. In many cases, people are in places where many people such as the ground and the floor can be present at one time, but in such places, it is usual to secure places at different positions in the horizontal direction. For this reason, in an image obtained by photographing a plurality of persons, the distribution of the skin area constituting the person image in the image often tends to depend on the top-to-bottom direction. Therefore, paying attention to the skin region existing in the image, the top-and-bottom direction in the image can be determined with high accuracy.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration of an image processing apparatus according to the present embodiment.
This image processing apparatus is realized by loading an image processing program according to the present embodiment on a PC. As shown in FIG. 1, a CPU 101 that controls the entire apparatus, a memory 102 that includes a ROM storing a RAM or BIOS used by the CPU 101 for work, an auxiliary storage device 103 that is a hard disk device, for example, and the like (not shown) A communication interface (I / F) 104 for performing communication via a communication network, an input interface (I / F) 105 for connection with various input devices such as a keyboard and a pointing device such as a mouse, and a display device for connection Output interface (I / F) 106, a medium driving device 107 that can access the portable recording medium MD, and a bus 108 that connects the units 101 to 107 to each other.

  The image processing program is stored in, for example, the auxiliary storage device 103, the recording medium MD mounted on the medium driving device 107, or an external device that can communicate via the communication I / F 104. The program is read out on the memory 102 and started up, whereby the image processing apparatus according to the present embodiment is realized. An image obtained by photographing with a photographing device such as a digital camera is acquired by the medium driving device 107 on which the recording medium MD is mounted or the communication I / F 104.

  FIG. 3 is a diagram for explaining the distribution of the face area when there are a plurality of human images in the image. FIG. 4 is a diagram for explaining an example of distribution of face areas in an image taken with a plurality of people as subjects. FIG. 4 shows three examples 1 to 3 in FIGS. 4 (a) to 4 (c). 3 and 4, the face area is indicated by a circle, and the range where the face area exists is indicated by a broken line. Here, with reference to FIG. 3 and FIG. 4, the determination method of the vertical direction will be specifically described.

  A person usually secures a place where he is and stays there. People are often on the ground, floors, etc. (hereinafter collectively referred to as “the ground surface” for convenience), but it is common to secure places at different positions in the horizontal direction on such ground surfaces. It is. Therefore, as shown in FIGS. 3 and 4, in an image (photograph) in which a plurality of persons are shown, the plurality of persons are arranged side by side, and the face area is vertical (top and bottom). It tends to exist in a horizontal direction rather than a direction. This tendency becomes stronger as the number of people increases (FIG. 4). The determination of the vertical direction is made paying attention to this.

  Specifically, a flesh-color region that is a flesh-color in the image is extracted, and its variance is examined in two orthogonal directions. One variance value is relatively high and the other value is relatively low. In this case, the direction with the lower value is determined as the top-and-bottom direction (FIG. 3). Since the photographing apparatus normally shoots in a vertical orientation or a horizontal orientation, the two orthogonal directions include a direction parallel to the short side (hereinafter referred to as a “short side direction”) and a long side. A parallel direction (hereinafter referred to as “long side direction”) is adopted (FIG. 3).

  Since people often look up or face down, it is not unlikely that the accuracy of the judgment result will be reduced when the face shape is recognized and the judgment is made in consideration of the direction. It is done. For this reason, in the present embodiment, the skin color region is extracted to make a determination focusing on its size and position. By not considering the shape and direction of the face, it is possible to determine the top-and-bottom direction with high accuracy regardless of the difference in the face direction of the person. Further, the determination can be performed with a smaller calculation amount.

  FIG. 2 is a diagram showing a functional configuration of the image processing apparatus according to the present embodiment. As shown in FIG. 2, the image processing apparatus includes an image input unit 201, a face recognition unit 202, a face position variance calculation unit 203, and a vertical / horizontal determination unit 204.

  The image input unit 201 inputs an image taken by a photographing apparatus or the like. The input image is data indicating a pixel value for each pixel. The configuration illustrated in FIG. 1 is realized by, for example, the CPU 101, the memory 102, the auxiliary storage device 103, the communication I / F 104, and the medium driving device 107.

For example, the face recognition unit 202 extracts a skin color area from the image input by the image input unit 201, and generates information indicating the position (here, the center of gravity position) and the size of each extracted skin color area. The face position variance calculation unit (hereinafter abbreviated as “dispersion calculation unit”) 203 receives the information for each skin color area from the face recognition unit 202 and calculates the variance value of the skin color area for each of the short side direction and the long side direction. To do. In the calculation, assuming that the center of gravity (average position) of the skin color region over the entire screen (image) is G and the variance is σ ² , these are the size (number of pixels) f _i of each skin color region _i , and its center of gravity. The position {x _i , y _i } is used as follows.

G = {G _x , G _y } = {Σf _i * x _i , Σf _i * y _i } / Σf _i (1)
σ ² = {σ ² _x , σ ² _y } = {Σf _i * (x _i −G _x ) ² , Σf _i * (y _i −G _y ) ² } / Σf _i
(2)
Here, G _x and σ ² _x represent the centroid position and dispersion value in the long side direction, respectively, and similarly G _y and σ ² _y represent the centroid position and dispersion value in the short side direction, respectively.

  In the extraction of the skin color area, it may be possible to extract an area in which other parts such as hands and feet are shown in addition to the face area in which the face is shown. In the body part, the face is most likely (exposed) and the next most likely is the hand. When comparing a face with a hand, the face area is usually larger. Therefore, in the present embodiment, the variance value of the skin color area is calculated by weighting the size of the skin color area. As a result, the influence of the skin color region having a small area such as the hand region on the calculated variance value is further reduced, and the variance value is calculated in such a manner that the face region is more important.

The vertical / horizontal discrimination unit 204 receives the variance values σ ² _x and σ ² _y calculated by the variance calculation unit 203 for each direction, and discriminates the vertical direction. For example, if the value of σ ² _y / σ ² _x is larger than a predetermined value TH (> 1), the short side direction is the vertical (vertical) direction, and the value of σ ² _x / σ ² _y is larger than the predetermined value TH. In other words, the long side direction is the top (vertical) direction, and if neither of these relations is satisfied, the discrimination is impossible.

  The image input by the image input unit 201 is automatically displayed by a rotation operation according to the determination result by the vertical / horizontal determination unit 204. Thereby, the image can be displayed in an appropriate state without the user giving an instruction to rotate the image. For this reason, an image in an appropriate state for the user can be viewed more quickly.

  In the configuration shown in FIG. 1, the face recognition unit 202, the variance calculation unit 203, and the vertical / horizontal discrimination unit 204 are all realized by, for example, the CPU 101, the memory 102, and the auxiliary storage device 103. The image rotation operation is performed by the CPU 101 using the memory 102, and the image after the rotation operation is displayed by being output to the display device via the output I / F 106.

  The units 201 to 4 illustrated in FIG. 2 are realized when the CPU 101 executes the top / bottom direction determination process illustrated in FIG. 5. Next, the determination process will be described in detail with reference to FIG. The determination process itself is realized by the CPU 101 reading, for example, the image processing program from the auxiliary storage device 103 to the memory 102 and executing it.

  First, in step 501, an image is input from the location designated by the user via the communication I / F 104 or the medium driving device 107. In the next step 502, attention is paid to one pixel in the input image, and it is determined whether or not the pixel value is within a range that can be regarded as skin color. If the pixel value is within the range that can be regarded as skin color, the determination is yes, and after setting 1 as another value of the pixel in step 503, the process proceeds to step 505. Otherwise, the determination is no, 0 is set as another value of the pixel in step 504, and the process proceeds to step 505.

  In step 505, it is determined whether or not all the pixels have been examined. If there remains a pixel that has not been checked whether or not the pixel value is within the range that can be regarded as skin color, the determination is no, the process returns to step 502, and the target pixel is changed to one of the remaining pixels. Then, it is determined whether or not the pixel value is within a range that can be regarded as skin color. Otherwise, the determination is yes and the process moves to step 506.

  By repeatedly executing the processing loop formed in steps 502 to 505 until the determination in step 505 is YES, an image binarized from the image input in step 501 depending on whether it is within the range that can be regarded as skin color is obtained. Generated. In the binarized image, the value of a pixel that can be regarded as skin color is 1, and the value of a pixel that is not so is 0. In step 506 and subsequent steps, processing for the generated binarized image is performed.

  In the binarized image, it is not clear how pixels having a pixel value of 1 constitute a skin color area. From this, first, in step 506, grouping and labeling processing for individually extracting the skin color regions is performed. By executing the processing, the same label is assigned to the pixels that are considered to constitute the same skin color region, and the pixels are grouped into the same group.

In step 507 following step 506, the size (number of pixels) f _i and the barycentric position {x _i , y _i } are calculated for each group (extracted skin color region), and the average of the skin color regions is calculated after the calculation. The position G is calculated by equation (1). In the next step 508, the dispersion values σ ² _x and σ ² _y are calculated by the equation (2) for each of the long side direction and the short side direction. Step 509 then proceeds.

In step 509, it is determined whether the value of σ ² _y / σ ² _x is greater than a predetermined value TH. If the value is larger than the predetermined value TH, the determination is YES, the process proceeds to step 510, the short side direction (indicated as “vertical direction” in the figure) is set as the top-and-bottom direction, and the series of processing ends. Otherwise, the determination is no and the process moves to step 511.

In step 511, it is determined whether or not the value of σ ² _x / σ ² _y is greater than a predetermined value TH. If the value is larger than the predetermined value TH, the determination is YES, the process proceeds to step 512, the long side direction (indicated as “horizontal direction” in the figure) is set as the top-and-bottom direction, and the series of processing ends. Otherwise, the determination is no, the process proceeds to step 513, the top-to-bottom direction is set as unknown, and the series of processes is terminated.

  The above-described top and bottom finding process is executed for each image. Thereby, when displaying a plurality of images at once, the top-and-bottom direction is determined for each image, and the determination result is reflected in the display of the image. The image input unit 201 shown in FIG. 2 is realized by executing the above step 501. Similarly, the face recognition unit 202 is the above steps 502 to 506, the variance calculation unit 203 is the above steps 507 and 508, and the vertical and horizontal discrimination unit. 204 is realized by executing the above steps 509 to 513.

  FIG. 6 is a flowchart of the grouping and labeling process executed as step 506. Next, the labeling process will be described in detail with reference to FIG. In this labeling process, starting from the pixel in the upper left corner of the image, the same label (number) is assigned to the pixels whose connected (adjacent) pixel values are 1 for all the pixels in the raster scan order. Is done.

  First, in step 601, initialization is performed. As a result of the initialization, 1 is substituted into a variable N for managing the label to be assigned, and the labels of all the pixels are reset, that is, 0 is set as the label. In the next step 602, it is determined whether or not the pixel value of the pixel A of interest is zero. If the pixel value is 0, the determination is yes and the process proceeds to step 608; otherwise, the determination is no and the process proceeds to step 603.

  Here, a method of labeling pixels will be specifically described with reference to FIG. In FIG. 7, each frame represents a pixel. A pixel of interest is indicated by A, and four pixels on the left, upper left, upper and upper right among the eight neighboring pixels are indicated by B to E, respectively. Of course, any of these four pixels may not exist depending on the position of the pixel of interest A.

  In the labeling to the four pixels B to E in the vicinity of the target pixel A, no label is assigned to any pixel (case 1), and the same label is assigned to one or more pixels (case 2). There are three cases where different labels are assigned to a plurality of pixels (case 3). Labeling to the pixel of interest A is performed by assigning a label that has not been assigned in the case 1, assigning the same label as the neighboring pixel in the case 2, and assigning the smallest label among the neighboring pixels in the case 3. . In steps 603 to 607, processing for performing such labeling is executed.

  First, in step 603, it is determined whether or not a label is assigned to any of the neighboring pixels B to E. If no label is assigned to any of the pixels, the determination is NO. Next, in step 604, the value of the variable N is assigned to the target pixel A as a label, and the value of the variable N is further incremented. Transition. Otherwise, the determination is yes and the process moves to step 605.

  In step 605, it is determined whether all labels assigned to the neighboring pixels B to E are the same. If there are a plurality of different labels as the labels, the determination is no and the process proceeds to step 606, where the smallest label among the plurality of labels is assigned to the target pixel A, and among the neighboring pixels B to E. A minimum label is newly assigned to a pixel to which a minimum label is not assigned. Thereafter, the process proceeds to step 608. On the other hand, if this is not the case, that is, if all the labels are the same, the determination is YES, and after assigning the label to the pixel of interest A in step 607, the process proceeds to step 608.

  In step 608, it is determined whether the pixel of interest A is the last (lower right corner) pixel of the image. If the target pixel A is the last pixel, the determination is yes, and the series of processing ends here. Otherwise, the determination is no, and in step 609, the target pixel A is changed to the next pixel in the raster scan order, and then the process returns to step 602. Thereby, labeling is performed on all the pixels having a pixel value of 1.

  In the grouping and labeling process, the process as described above is executed. As a result, the same label is assigned to the pixels constituting the same skin color area and grouped. Thereby, the size of the skin color area is calculated by counting the number of pixels for each different label. The barycentric position X is calculated using the coordinate position for each pixel.

In the present embodiment, the variance value σ ² is calculated in order to evaluate the distribution state of the skin color region, but another index may be employed. Specifically, a range (width) in which the skin color area exists, a shape thereof, and the like may be employed. Alternatively, only the skin color region in which a specific part of the body (for example, a face) is captured may be extracted, and the vertical direction may be determined. When only a specific skin color region is extracted, weighting according to the size may not be performed. The image for determining the top-and-bottom direction may not be an image obtained by photographing (including digitization by a scanner or the like here). For example, it may be an image generated by computer graphics.

It is a figure which shows the structure of the image processing apparatus by this Embodiment. It is a figure which shows the function structure of the image processing apparatus by this Embodiment. It is a figure explaining distribution of a face field in case a plurality of human figures exist in a picture. It is a figure explaining the example of distribution of the face area in the picture photoed with a plurality of people as a subject. It is a flowchart of a top-and-bottom direction determination process. It is a flowchart of a grouping and labeling process. It is a figure explaining the labeling method to a pixel.

Explanation of symbols

101 CPU
102 Memory 103 Auxiliary Storage Device 104 Communication Interface 105 Input Interface 106 Output Interface 107 Medium Drive Device 108 Bus

Claims (4)

In an image processing apparatus for determining a top-and-bottom direction in an image obtained by photographing a subject,
Image acquisition means for acquiring the image;
A skin area recognition means for recognizing a skin area of a person image photographed as a subject in the image acquired by the image acquisition means;
Based on the recognition result of the skin area by the skin area recognition means, direction determination means for determining the top and bottom direction,
An image processing apparatus comprising: The direction determining means calculates a variance value for the skin area recognized by the skin area recognizing means for each direction parallel to the edge of the image, and determines one of the directions parallel to the edge from the calculation result. One is determined to be the top and bottom direction,
The image processing apparatus according to claim 1. The direction determination means calculates a variance value in consideration of the size of the skin area recognized by the skin area recognition means;
The image processing apparatus according to claim 2. A program for causing an image processing apparatus to determine a top-to-bottom direction in an image obtained by photographing a subject,
A function of acquiring the image;
A function of recognizing a skin area of a human image captured as a subject in an image acquired by the function of acquiring;
Based on the recognition result of the skin area by the recognition function, the function of determining the top and bottom direction,
A program to realize

Images