JP2010074312A - Image processing apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing technique for preventing the face of an object in an image from missing, and preventing the face from being not detected when executing a distortion aberration correction processing. <P>SOLUTION: This image processing apparatus includes: an image processing means for performing image processing accompanied with the shape change of the outer edge of image data; a face detecting means for detecting a face area included in the image data; a determining means for determining a parameter of the image processing; a comparing means for comparing face detection results about the image data before and after image processing by the parameter of the image processing determined by the determining means; and a determining means for determining whether to execute the image processing in accordance with the comparison results by the comparing means. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for digital image data.

  In recent years, due to the improvement in performance of digital cameras, the resolution of captured images has become very high, and it has become possible to obtain high-quality photographic images that are not inferior to silver halide photographs. The photographed image can be viewed using an image processing application installed in a personal computer or the like, or subjected to image editing processing.

  In general, geometric distortion caused by lens distortion occurs in an image taken by a camera. In order to correct this distortion, an image processing application for correcting aberration using information at the time of shooting such as a shooting lens has also appeared. By performing such distortion correction processing, a more uniform and high-quality image can be obtained.

  2A and 2B show distortion aberration correction processing. FIG. 2A shows a case where an image distorted in a barrel shape is corrected, and FIG. 2B shows a case where an image distorted in a pincushion shape is corrected.

Since the distortion correction process is a correction involving movement of the pixel position, the shape of the outer edge of the image also changes. On the other hand, the outer edge shape of the image is generally rectangular. In view of this, it is common to trim a peripheral image by trimming the image to the maximum size rectangle or the aspect ratio of the photographed camera after the distortion correction processing (see, for example, Patent Documents). 1).
JP 2007-129487 A

  However, if a photograph of a human face in the periphery is trimmed after distortion correction processing, the geometric distortion of the lens is corrected correctly, but the subject's face may be missing or Sometimes everything is cut off. For example, when there is a face of the subject at the edge of the screen, a part of the face may be lost due to this trimming (see FIG. 3).

  The present invention is made in view of the above problems, and realizes an image processing technique for preventing a face of a subject in an image from being lost or being unable to be detected when executing distortion correction processing.

  In order to solve the above problems, an image processing apparatus according to the present invention includes an image processing unit that performs image processing accompanied by a change in shape of an outer edge of image data, a face detection unit that detects a face area included in image data, In accordance with the determination means for determining image processing parameters, the comparison means for comparing the face detection results for the image data before and after the image processing by the image processing parameters determined by the determination means, and the comparison result by the comparison means Determining means for determining whether or not to execute the image processing.

  The image processing method of the present invention determines an image processing step for performing image processing accompanied by a shape change of the outer edge of the image data, a face detection step for detecting a face area included in the image data, and parameters for the image processing. A determination step for comparing, a comparison step for comparing face detection results for image data before and after image processing using the determined image processing parameters, and whether or not to execute the image processing according to the comparison result by the comparison step A determination step of determining whether or not.

  According to the present invention, it is possible to prevent the face of the subject in the image from being lost or the face from becoming undetectable when performing the distortion correction process.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

  The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention.

  In FIG. 1, a CPU 101 performs overall control of the overall system operation, and executes a program stored in the primary storage unit 102.

  The primary storage unit 102 is mainly a memory, and reads and stores programs stored in the secondary storage unit 103.

  The secondary storage unit 103 corresponds to, for example, a hard disk. The capacity of the primary storage unit 102 is smaller than the capacity of the secondary storage unit 103, and programs and data that cannot be stored in the primary storage unit 102 are stored in the secondary storage unit 103. Data that needs to be stored for a long time is also stored in the secondary storage unit 103. A program that realizes the processing procedure of the present embodiment is stored in the secondary storage unit 103, read to the primary storage unit 102 when the program is executed, and executed by the CPU 101.

  The input device 104 corresponds to, for example, a mouse or a keyboard that accepts a user operation, and is used to send an interrupt signal to a program or the like.

  The output device 105 corresponds to, for example, a monitor or a printer as a display device.

  The reading device 106 writes / reads image data generated by an image sensor such as a CCD directly or indirectly to / from the primary storage unit 102 or the secondary storage unit 103.

  The characteristic data supply unit 107 has a memory (not shown). This memory stores information on lens characteristics corresponding to various lenses, specifically, correction coefficients corresponding to photographing lenses. In distortion aberration correction processing described later, a correction function is determined in advance, a correction amount read from the characteristic data supply unit 107 is applied to the correction function, a correction amount is calculated, and the image is deformed according to the correction amount. The image processing is accompanied by the shape change of the outer edge of the image. The lens characteristics are not limited to the memory of the characteristic data supply unit 107, and data stored in the secondary storage unit 103 or an external storage device (not shown) may be referred to.

  FIG. 4 exemplifies a user interface (GUI) of the image processing apparatus (hereinafter referred to as an image processing application) in the present embodiment.

  In FIG. 4, the processing target image is fit-displayed on the image display unit 401 (the processing target image is enlarged and displayed to the maximum within a range where the entire region of the processing target image is displayed in the window). The processing target image is designated by a method such as dragging and dropping a file on the image display unit 401 from the file management system of the operating system. It is also possible to display an image obtained as a result of performing some image processing on the processing target image.

  When a distortion aberration correction execution button (hereinafter, execution button) 402 is pressed, a distortion aberration correction process is executed, and an image of the correction result is displayed on the image display unit 401. When distortion correction processing is executed, the shape of the outer edge of the corrected image may not be rectangular. In that case, the result image may be cut out in a rectangular area inscribed in the outer edge of the result image, and the cut out image may be used as the result image (see FIG. 2).

  When the reset button 403 is pressed, an image before the distortion aberration correction process is performed is displayed on the image display unit 401.

  When ending the image processing application, an end button 404 is pressed.

  Here, processing when the execution button 402 is pressed will be described with reference to the flow of FIG.

  In step S <b> 501, first, image data is read and displayed on the image display unit 401.

  In step S502, face detection is performed on the read image data, and detection information is recorded. The information recorded here includes the position and size of each detected face. For example, when four faces are detected, four face IDs as shown in the table of FIG. 6 are recorded as face detection information.

  Various techniques are known as techniques for detecting a face. For example, there is a method using learning represented by a neural network. In addition, there is a technique for identifying parts having physical shape features such as eyes, nose, mouth, and facial contour from image information using template matching. In addition, there is a technique that uses a statistical analysis by detecting feature amounts of image information such as skin color and eye shape. Furthermore, it is determined whether the immediately preceding face area is near the detected position, the color near the face area is determined to take into account the color of the clothes, or the face near the center of the screen is used for face identification. There is a method of setting a low threshold value.

  In S503, a distortion aberration correction parameter (hereinafter, correction parameter) is determined.

  The distortion aberration correction process of the present embodiment is performed by a known method disclosed in, for example, Japanese Patent Laid-Open No. 6-165024. That is, the characteristic data is obtained from the characteristic data supply unit 107 using the optical information at the time of photographing, and applied to a predetermined correction function to determine the correction amount for the image height (distance from the image center). In this embodiment, the correction function is a cubic function with the image height as a parameter.

  In order to perform distortion correction processing, identification information for identifying a lens, focal length information for photographing for specifying a zoom position, and optical information for photographing such as an F value are necessary. In the present embodiment, these pieces of information are stored in a MakerNote tag defined by the Exif standard. In order to further improve the correction accuracy, other information such as subject distance information may be used in addition to the above information.

  Next, in S504, it is determined whether or not the face of the image detected before the correction is missing in the corrected image.

  A method for determining whether or not a face is missing will be described with reference to FIGS.

  As described with reference to FIG. 2, the distortion correction process is a correction that involves a movement of the pixel position, and thus the shape of the outer edge of the image also changes. On the other hand, the shape of the outer edge of the image is generally rectangular, and after distortion correction processing, the image is trimmed to the maximum size rectangle or the aspect ratio of the photographed camera, and the peripheral image is cut out. .

  That is, as shown in the original image of FIG. 7, by performing distortion correction processing on an image having a width W and a height H, an image a as a correction result is obtained. As a result, the width of the corrected image is Wa and the height is Ha, and the new correction result image is a rectangle surrounded by the start point (xd1, yd1) and the end point (xd2, yd2). Hereinafter, this rectangular area is referred to as an effective image area.

  In the present embodiment, a rectangle having the maximum size is used for the effective image area. However, the effective image area may be an area that is trimmed in the same manner as the aspect ratio of the photographed camera, or may be trimmed later by the user.

  Here, the process of determining whether or not a face is missing will be described with reference to the flow of FIG.

  In step S <b> 801, a distortion aberration correction function is applied to the face detection data to obtain the corrected face position coordinates.

  In step S802, if the corrected face position coordinates are within the effective image area, the process advances to step S803. If even a part of the coordinates of the corrected face position is outside the effective image area, the process proceeds to S805, and it is determined that the face is missing by the distortion correction process.

  In step S803, the process from step S801 is repeated until the correction function is applied to all faces. If the correction function is applied to all faces, the process proceeds to step S804, and it is determined that the face is not missing by the distortion correction process.

  With the above processing, it can be determined whether or not the face in S504 is missing.

  If the face is not missing, the process proceeds to S505, and distortion correction processing is executed.

  If the face is missing, the process proceeds to S506, and the distortion correction process is not executed.

  The image processing application executes the processing of FIG. 5 when the execution button 402 is pressed. If it is determined that the face is missing, a dialog is displayed to warn the user by a message such as “The subject's face is missing.” And the process is stopped. Or, when a dialog box such as “The face of the subject is missing. Do you want to cancel the process?” Is displayed and the user confirms whether to execute the distortion correction process, and the user still wants to execute the distortion correction process. Only the process may be performed. If the face is missing, the correction parameter may be changed in a direction in which the degree of reflection of the distortion correction process in S503 is reduced, and the process based on the flow of FIG. 8 may be executed again.

  In this embodiment, the distortion aberration correction process is exemplified, but it is also effective for a process of cutting out an effective image area from an image rotated by a predetermined angle.

  As described above, when the subject's face is missing from the effective image area due to the distortion correction process, the subject's face can be prevented from being cut out by not executing the distortion correction process.

  Furthermore, in this embodiment, since it is possible to determine whether or not the face is missing before performing distortion correction processing for all pixels in the image, it can be determined at high speed.

[Second Embodiment]
In the first embodiment, after performing distortion correction processing on an image, it is determined whether or not the face position is missing from the effective image area, and it is determined whether or not to perform distortion correction processing according to the result. ing. However, in practice, there are cases where a face can be detected even when a part of the face is missing. Thus, in the present embodiment, a process for determining whether or not face detection is possible and determining a correction parameter will be described.

  The present embodiment can be realized by the apparatus configuration shown in FIG. 1 and the GUI shown in FIG.

  FIG. 9 is a flowchart illustrating processing when the execution button 402 is pressed by the image processing apparatus according to the second embodiment.

  In step S <b> 901, first, an image is read and displayed on the image display unit 401.

  In step S902, face detection is performed on the read image data, and information shown in FIG. 6 is recorded.

  Here, the start point and size of each face ID are defined as the face area of each face ID.

  In step S903, the initial value of the correction parameter is determined.

  The distortion correction processing in this embodiment is also executed by the method disclosed in Japanese Patent Laid-Open No. 6-165024.

  Further, identification information for identifying a lens, photographing focal length information for specifying a zoom position, and photographing optical information such as an F value, which are necessary for performing lens aberration correction processing, are MakerNotes defined by the Exif standard. Store in tags.

  The initial value of the correction parameter is determined by the above method.

  In step S904, distortion correction processing is performed on the entire original image data.

  In step S905, face detection processing is performed on the image after distortion aberration correction processing, and the start point and size of each face ID are recorded in a table as shown in FIG. 6 as the detection result after correction. This table is held separately from the table before correction.

In step S906, the face detection results before and after correction are compared. When the face detection result after correction is within a predetermined threshold with respect to the face detection result before correction as a comparison result, the face detection result does not change significantly before and after the correction, so distortion aberration correction processing using the current correction parameters It is determined that the face will not be lost, and the process is terminated.
Here, the determination method of the face detection result in S906 will be described with reference to the flow of FIG.

  In step S1001, first, the face ID is initialized by inputting 0.

  In step S1002, for the face (ID) coordinates, the corrected face ID coordinates are obtained using the correction parameters. The coordinates obtained here are the start point and size of the face ID.

  In S1003, it is searched whether or not the face area obtained in S905 exists within the threshold value of the start point and the size of the corrected face ID obtained in S1002.

  Here, as the threshold value, an area that is 30% of the size of the face ID after correction obtained in S1002 is set.

  As a result of searching in S1002, if the face area obtained in S905 does not exist, it is determined that the amount of change is large, and the process proceeds to S1006.

  If a face area exists, the process proceeds to S1004. If correction has been performed for all face IDs before correction, the process proceeds to S1007. If not, the process proceeds to S1005, and the ID is incremented by 1 and the process proceeds to S1002. Return.

  If the amount of change is large in S1006, the face detection result will change greatly, and the process advances to S907.

  In S907, the parameter is changed so as to reduce the correction parameter by 10%, and the distortion aberration correction process in S904 is performed again.

  In S1003, the search range of the face detection result before and after the current correction is set to 30% of the size of the face ID, but if a range larger than this is searched, another face is included as a search result. Because there is a possibility. Therefore, the search range may be expanded depending on the number of faces.

  Further, the amount of correction parameter reduction in S907 is uniformly 10%, but this is because there is a lens with a large change amount or a small lens depending on the type of lens, so that it is not necessary to consider the difference between lenses. 10%.

  As another method, the reduction amount of the correction parameter may be changed depending on the amount of change with reference to the lens type and correction parameter table.

  In the present embodiment, all the image data is used for the face detection process. However, thinned image data may be used to speed up the process.

  According to the above embodiment, the correction parameter can be determined so that the result of face detection does not change greatly before and after the distortion correction process.

[Other Embodiments]
The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, it goes without saying that this can also be achieved by the computer (CPU or MPU) of the system or apparatus reading and executing the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, nonvolatile semiconductor memory card, ROM, or the like can be used. . Further, the functions of the above-described embodiments may be realized by executing the program code read by the computer.

  However, there is a case where an OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Needless to say, it is included.

  Furthermore, the program code read from the storage medium may be written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Needless to say.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a figure explaining a distortion aberration correction process. It is a figure explaining the subject by a distortion aberration correction process. It is a figure which illustrates the user interface by the image processing application of this embodiment. It is a flowchart which shows the determination process of the distortion aberration correction process result by 1st Embodiment. It is a figure which illustrates the table of the face detection information by this embodiment. It is a figure explaining the distortion aberration correction process by 1st Embodiment. It is a flowchart which shows the determination process of the distortion aberration correction process result by 1st Embodiment. It is a flowchart which shows the determination process of the face detection result by 2nd Embodiment. It is a flowchart which shows the determination process of the face detection result by 2nd Embodiment.

Claims (9)

Image processing means for performing image processing with a change in shape of the outer edge of the image data;
Face detection means for detecting a face area included in the image data;
Determining means for determining parameters of the image processing;
Comparison means for comparing face detection results for image data before and after image processing according to image processing parameters determined by the determining means;
An image processing apparatus comprising: determination means for determining whether or not to execute the image processing according to a comparison result by the comparison means.   The image processing apparatus according to claim 1, wherein the image processing is distortion aberration correction processing.   The image processing apparatus according to claim 1, wherein the image processing is processing of rotating an image by a predetermined angle.   The comparison means determines whether a face position detected from image data before the image processing is included in a predetermined effective image area after the image processing. The image processing apparatus described.   The determination unit does not execute the image processing when a face position detected from image data before the image processing is not included in a predetermined effective image area after the image processing. Item 5. The image processing apparatus according to Item 4.   The determination means outputs a warning to that effect when it is determined that a part of the face detected by the face detection means is missing after the image processing by executing the image processing. The image processing apparatus according to claim 4. Imaging means for imaging a subject and generating image data;
The image processing apparatus according to claim 1, further comprising display means for displaying image data picked up by the image pickup means. An image processing step for performing image processing with a change in shape of the outer edge of the image data;
A face detection step of detecting a face area included in the image data;
A determining step for determining parameters of the image processing;
A comparison step of comparing face detection results for image data before and after image processing according to the determined image processing parameters;
A determination step of determining whether or not to execute the image processing according to a comparison result of the comparison step.   A program for causing a computer to execute the image processing method according to claim 8.

Images