JP2012128521A - Image processing device, control method of image processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device which accurately determines whether a photograph is a group photograph, a control method of the image processing device and a program.SOLUTION: An image is acquired (S300), face images existing in the acquired image are analyzed (S301), determination is made as to whether each face image existing in the analyzed image satisfies the conditions of a group photograph (S302 to S307), and based on the determination result, determination is made as to whether it is a group photograph (S308, S309).

Description

  The present invention relates to an image processing apparatus for a photographed image, a control method for the image processing apparatus, and a program.

In recent years, an image such as a photograph is analyzed, and metadata such as a keyword is added to be used for searching. Specifically, it has been proposed to detect a face in a photograph, assign a keyword depending on the number of detected faces, and determine “group photo” when there are three or more people. (Patent Document 1).
In addition, it has been proposed to prevent printing such that characters, lines, and images included in the original drawing information cannot be read when the image is subjected to reduced layout processing. (Patent Document 2).

JP 2006-350552 A JP 2006-185307 A

However, when a group photo is determined based on the number of faces, it may be determined as a group photo depending on the face that happens to be in the photo. For this reason, a keyword of a group photo may be assigned and searched.
Also, when reducing the image, the reduction rate was limited in order to identify the face if it was a group photo, but the image would not be too small due to an untargeted face that happened to appear in the photo. The reduction rate was limited and the size was larger than necessary.

  Accordingly, an object of the present invention is to accurately determine whether a photograph is a group photograph.

  In order to achieve the above object, an image processing apparatus according to an aspect of the present invention has the following arrangement. That is, an acquisition unit for acquiring an image, an analysis unit for analyzing a face image present in the image acquired by the acquisition unit, and each of the face images existing in the image analyzed by the analysis unit satisfy the group photo condition. A face image determining unit that determines whether the image is satisfied, and a group photo determining unit that determines whether the photo is a group photo based on a determination result by the face image determining unit.

  According to the present invention, it is accurately determined whether a photo is a group photo.

It is a block diagram which shows the structure which concerns on embodiment of this invention. It is an application example which concerns on embodiment of this invention. It is a flowchart which shows the process sequence which concerns on embodiment of this invention. It is a flowchart which shows the process sequence which concerns on embodiment of this invention.

<Embodiment 1>
Embodiments of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of an image processing apparatus according to the present invention. The image processing apparatus includes a data processing unit 115, which is a CPU, a storage unit 111, a display unit 116, an input unit 113, and a network I / F unit 117.

  The data processing unit 115 includes an image analysis unit 118, an image determination unit 119, a group photograph determination unit 120, and the like, and controls the processing of a flow to be described later. Details of the image analysis unit 118 will be described later with reference to FIG. 4, and details of processing performed by the image determination unit 119 and the group photograph determination unit 120 will be described later with reference to steps S302 to S309 in FIG.

  Although not shown, the image processing apparatus includes a ROM and stores a computer program and data in which a control procedure by the data processing unit 115 is stored.

  Although not shown, the image processing apparatus includes a RAM, stores a control program for processing by the data processing unit 115, and provides a work area for various data when the data processing unit 115 executes various controls. To do.

  Although not shown, the image processing apparatus in FIG. 1 is connected to another image processing apparatus and a network.

  Next, a processing flow of image processing according to the present embodiment will be described with reference to FIG. The following processing is controlled by the data processing unit 115.

  3 includes an image acquisition process (S300), an image analysis process (S301), a face image determination process (S302 to S307), and a group photo determination process (S308, S309). The face image determination processing is further performed by subject number determination processing (S302), gaze determination processing (S303), camera eye determination processing (S304), shooting condition determination processing (S305), face area determination processing (S306), and face interval determination processing. (S307). Here, the order of each determination is made in order from the determination with the highest priority. The group photo determination process further includes a group photo setting process (S308) and a group photo rejection setting process (S309).

  An operation example of the flow of FIG. 3 will be described. In the image acquisition process (S300), a target image is acquired from the storage unit 111 in which an image such as a photograph read from a camera or the like is stored. Next, in the image analysis process (S301), the target image is analyzed to detect a face, a face part, a line of sight, and the like. Details will be described later.

  Hereinafter, a description will be given of performing face image determination processing and performing group photo determination from the determination result. First, face image determination processing is performed to determine whether or not the detected face image satisfies the group photo condition. In the subject number determination process (S302), it is determined whether the number of face images existing in the image detected in S301 is equal to or greater than a threshold value. For example, if the number is less than three, the target image is not a group photo and the process proceeds to S309. If there are three or more, the process proceeds to S303. Next, in the line-of-sight determination process (S303), the ratio of the line of sight detected in the face detected in S301 is determined. For example, if the ratio of the line of sight that can be identified among the recognized faces can be identified by 70% or more, the process proceeds to the camera eye determination process (S304). If it is less than 70%, the camera gaze determination process is skipped and the process proceeds to S305.

  Next, in the camera line-of-sight determination process (S304), it is determined whether or not the line of sight of each face image detected in S301 is facing the camera direction. For example, if 90% or more of the determined lines of sight are facing the camera direction, it is determined that the target image is a group photo and the process proceeds to S308. Otherwise, the process proceeds to S305.

  Next, in shooting condition determination processing (S305), determination is made from the shooting conditions of the target image. For example, the focal length is 50 mm or less. In addition, the aperture is F8 or more. If such a condition is satisfied, the process proceeds to the next determination (S306). Otherwise, it is determined that the target image is not a group photo, and the process proceeds to S309.

  Next, in the face area determination process (S306), the area of each face image detected in S301 is obtained and determined by its uniformity. For example, if it is determined whether the area of all the faces is within ± 30% of the average value of the face areas and the condition is satisfied, the process proceeds to the next determination (S307). Otherwise, it is determined that the target image is not a group photo, and the process proceeds to S309.

  Next, in the face interval determination process (S307), the centroid of each face image detected in S301 is obtained, and the determination is made by the uniformity in the distance between the centroids. For example, if it is determined whether or not the distance between the centers of gravity of all adjacent faces is within ± 30% of the average distance between the faces, and the condition is satisfied, the target image is a group photo Determine and proceed to S308. Otherwise, it is determined that the target image is not a group photo, and the process proceeds to S309.

  In the group photo setting process (S308), it is set in the storage unit 111 that the target image is a group photo. In the group photograph rejection setting process (S309), it is set in the storage unit 111 that the target image is not a group photograph.

The processing flow of FIG. 4 is a detailed processing flow of the image analysis processing (S301) of FIG.
The processing flow of FIG. 4 includes face detection processing (S401), face detection determination processing (S402), face contour detection processing (S403), face part determination processing (S404), face orientation determination processing (S405), and face orientation. Front determination process (S406), black eye part is center determination process (S407), face orientation is left determination process (S408), white eye part determination process (S409) to the left of the black eye part, and white eye part determination process to the right of the black eye part (S410) and a line-of-sight setting process (S411) for the camera.

  As the face detection technique, the face part discrimination technique, the line-of-sight discrimination technique, etc., publicly known techniques may be used as disclosed in, for example, JP2009-059257A and JP2008-011457A.

Details of each process constituting the image analysis process will be described below.
The face detection process analyzes the image data, detects whether or not there is a person in the subject in the target image data, and if there is a person in the subject in the target image data, the position and size of the face portion in the person. Detecting etc. It should be noted that the range of the face portion of the person who is the subject (hereinafter referred to as the subject person) can be detected from the main color of the face portion.

  The contour detection process is a process for performing a sharpness emphasis process on the image data to detect the contour shape of the subject in the target image data. A T-shaped shadow composed of a horizontally long shadow by the eyebrows and eyes and a vertically long shadow by the nose is extracted from the image data. This is obtained by extracting and processing only luminance information from image data, for example. By detecting the face portion of the subject person in this manner and extracting only the portion having a color close to the skin color from the color information, the detailed range of the face portion of the subject person can be detected.

  The face part discrimination process is a process for discriminating the positions of various face parts such as the eye part, the eyebrow part and the nose part in the face part of the subject person detected by the face detection process.

  A discrimination method based on color, which is one of the methods for determining the positions of various facial parts of a subject person by facial part discrimination processing, will be described. First, a strong contrast portion that runs in the vertical direction near the center of the face portion of the subject person is determined to be the nose portion. Furthermore, it is possible to determine that the white part existing near the upper part of the left and right side parts of the nose part is a white eye part. Then, the black part surrounded by the white part determined to be the white eye part can be determined to be the black eye part (hereinafter, the part composed of the white eye part and the black eye part is referred to as the eye part). ). In this way, the position of the face of the subject person, the position of the nose, the position of the white eye, the position of the black eye, etc. in the image can be determined.

  The face orientation determination process can detect the shadow of the nose on the face by analyzing the brightness on the dividing line when the face is divided into two in the vertical direction. If the shadow is to the left of the midpoint, the face is determined to face left, and if the shadow is to the right of the midpoint, the face is determined to face right. As described above, the orientation of the face of the subject person can be determined.

  A method for detecting the direction of the line of sight of the subject person using the brightness distribution in the eye will be described in detail. The line-of-sight discrimination can detect the white-eye portion of the subject person as a peak having a predetermined width and brighter than the surroundings, and can also detect the direction of the line-of-sight of the subject person based on the degree of deviation of the black-eye portion.

Hereinafter, detection of the direction of the line of sight of the subject person based on the degree of deviation of the black eye part will be described in detail.
First, when the black-eye part is present on both the left and right sides, that is, when the subject person is facing the front, there are two brightness peaks due to the white-eye part, which are sandwiched between the two peaks. There is a dark part due to the black eye.

  Further, when the black eye portion is biased to the left side, that is, when the subject person is looking at the left side, there is only one brightness peak due to the white eye portion, and the peak is from the midpoint. Is also on the right.

  Further, when the black eye portion is biased to the right side, that is, when the subject person is looking to the right side, there is only one brightness peak due to the white eye portion, and the peak is from the midpoint. Is also on the left.

  As described above, since the orientation of the face of the subject person and the direction of the line of sight can be detected by the brightness distribution pattern in the face and eyes, it is determined whether or not the subject person is looking at the camera. be able to.

  An operation example will be described based on the processing flow of FIG. In the face detection process (S401), the image data is analyzed, it is determined whether or not there is a person in the subject of the image data, and if there is a person in the subject of the image data, the face portion is detected.

  In the face detection determination (S402), it is determined whether or not the face portion has been successfully detected. If not detected, the process ends. If detected, the process proceeds to S403. In the face contour detection process (S403), the face contour detected in S401 is detected.

In the face part discrimination process (S404), each part is discriminated from the face contour detected in S403.
In the face orientation determination process (S405), the face orientation is determined from each part of the face determined in S404.

  In the face direction determination process (S406), the result of the face direction determination performed in S405 determines whether the face part is facing the camera (whether it is facing the front). If it faces the front of the subject, the process proceeds to S407. If not, the process proceeds to S408.

  In the determination of the center of the black eye (S407), it is determined in S404 whether or not the white eye exists on both the left and right sides of the black eye (whether the black eye is the center). If it is in the center, it is determined that the line of sight is facing the camera and the process proceeds to S411. If it is not the center, the process ends with the line of sight not facing the camera.

  If the face orientation is determined to be left (S408), it is determined whether or not the face portion of the subject faces the left side as a result of the face orientation determination performed in S405. If it turns to the left, it progresses to S409. If it turns to the right, it progresses to S410.

  In the white eye portion determination process (S409) to the left of the black eye portion, it is determined in S404 whether the white eye portion exists on the right side of the black eye portion. If it is facing, it is determined that the line of sight is facing the camera and the process proceeds to S411. Otherwise it ends.

  In the white eye determination process (S410) to the right of the black eye, the face is facing right and the line of sight is facing left. In this state, it is determined that the subject person is looking at the camera, and the process proceeds to S411. If not, exit.

  In the line-of-sight setting process for the camera (S411), it is stored in the storage device that the line of sight is facing the camera. The line-of-sight discrimination for the camera is performed for the number of subject persons in the image.

  As described above, after determining whether or not a photo is a group photo, when it is determined that the photo is a group photo, as a next step, metadata (keyword) of the group photo may be added to the photo. Furthermore, using this metadata, for example, a photo to which group photo metadata is assigned may be preferentially presented as a search result from a plurality of photos, or may be determined as a representative photo.

  As described above, the present invention accurately determines whether or not a photograph is a group photograph. Therefore, the user can efficiently search and select.

<Embodiment 2>
In the second embodiment, an example will be described in which a photo determined to be a group photo limits the reduction ratio. When laying out photos, they may be reduced and placed. It is embodiment regarding the reduction rate in that case. Since the apparatus configuration and the process up to the group photo determination are the same as those in the first embodiment, a description thereof will be omitted.

  As a specific application example, an example in which the photograph of FIG. 2A is laid out as shown in FIG. 2B will be described. If it is determined that it is not a group photo, 201 can be reduced to a size such as 203 where the face cannot be identified. If it is determined as a group photo, the reduction rate is limited so that the face does not become an indistinguishable size like 202 in 204. At this time, for example, the face closest to the average of the face areas obtained in S306 of FIG. 3 may be restricted so as not to be smaller than a predetermined area.

  By accurately determining whether or not it is a group photo in this way, when the photo is reduced, the group photo is displayed in a size that allows the face to be identified. It becomes possible to display small according to the instruction.

<Other embodiments>
In the first embodiment, the image determination unit performs a plurality of determination processes from determinations with a high degree of importance. However, the combinations of the group photo determinations and the order of the determinations are not necessarily the same as those in the first embodiment. . Further, the determination may be made comprehensively by adding importance to each determination. As an example, principal component analysis, which is one method of multivariate analysis, can be used. Specifically, a covariance matrix between a plurality of determination scores is generated from learning data. By obtaining a solution to the eigenvalue problem of this covariance matrix, the dimensions are degenerated and a small number of uncorrelated synthesis scores are generated. Based on this composite score, it can be determined whether the target image is a group photo.

  Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the gist of the present invention.

  The present invention can take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

An acquisition means for acquiring an image;
Analyzing means for analyzing a face image present in the image obtained by the obtaining means;
Face image determination means for determining whether each of the face images analyzed by the analysis means satisfies a group photo condition;
An image processing apparatus comprising: a group photo determination unit that determines whether the photo is a group photo based on a determination result by the face image determination unit.   The image processing apparatus according to claim 1, further comprising a limiting unit that limits a reduction rate of the image when the group photo determination unit determines that the image is a group photo.   The image processing apparatus according to claim 2, wherein the limiting unit limits a reduction rate so that a face image in the image does not have an unidentifiable size.   The image processing apparatus according to claim 2, wherein the limiting unit limits the reduction ratio so that an average value of face images in the image does not become a predetermined size or less. . The face image determination means includes
5. The image processing apparatus according to claim 1, further comprising a comparison unit configured to compare whether or not the number of face images of the image is equal to or greater than a predetermined threshold value. The face image determination means includes
The image processing apparatus according to claim 1, further comprising line-of-sight determination means for determining whether the line-of-sight of each of the face images faces a camera direction.   The image processing apparatus according to claim 1, wherein the face image determination unit further includes a face area determination unit that determines the uniformity of the area of the face image of each of the face images. .   8. The face image determination unit according to claim 1, further comprising a face interval determination unit that determines a uniformity of a distance between each of the face images and an adjacent face image. 9. Image processing device. The face image determination means includes
The image processing apparatus according to claim 1, further comprising a shooting condition determination unit that determines a shooting condition of the image.   10. The image processing apparatus according to claim 1, wherein when the group photograph determination unit determines that the group photograph is a group photograph, the group photograph metadata is added to the image. An acquisition step in which the acquisition means acquires an image;
An analyzing step for analyzing a face image present in the image acquired in the acquiring step;
A face image determination unit that determines whether the face image analyzed in the analysis step satisfies a group photograph condition;
A method for controlling an image processing apparatus, comprising: a group photograph determination unit that determines whether the group photograph is a group photograph based on a determination result of the face image determination step.   The program for making a computer perform each process of the control method of the image processing apparatus of Claim 11.

Images