JP2011034311A - Image processor and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for improving the speed and the precision of face detection/face recognition processing. <P>SOLUTION: An image processor estimates a dot region 54 under consideration by using a remarkable map 53S obtained by integrating a plurality of featured value maps 52Fc, 52Fh, and 52Fs about a through-image 51 (step Sa to Sc). The image processor sets a parameter to be used for face detection/face recognition processing, for example, a parameter indicating whether a region is a processing object region by using the region 54 under consideration (step Sd). For example, only a region 56 among regions 55 to 57 is set as the object of processing. Then, the image processor performs the face detection/face recognition processing to the region 56 (step Se). Thus, a face detection region 58 is obtained. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method, and more particularly to a technique for improving the speed and accuracy of face detection / face recognition processing.

  2. Description of the Related Art Conventionally, there is a process for detecting a person's face or recognizing a specific person's face from an image obtained by photographing with a camera (hereinafter referred to as a photographed image). Hereinafter, such processing is referred to as face detection / face recognition processing.

  For example, Patent Documents 1 to 6 disclose methods that can realize various image processing such as AF (Automatic Focus) processing and AE (Automatic Exposure) processing using the results of face detection / face recognition processing.

JP 05-37940 A JP 05-41830 A Japanese Patent Laid-Open No. 05-53041 Japanese Patent Laid-Open No. 06-303491 Japanese Patent Laid-Open No. 06-217187 JP 2006-254229 A

However, there are restrictions on the processing capacity and processing time that can be installed in a small camera. For this reason, when the conventional methods disclosed in Patent Documents 1 to 6 are applied to such a small camera, the speed of face detection / face recognition processing is slowed down. As a result, the user may have troubles such as trouble in photographing operation and missed shutter timing. In particular, as the number of faces to be detected or recognized increases, the probability of occurrence of a problem increases.
In addition, when the conventional methods disclosed in Patent Documents 1 to 6 are applied to a small camera with limited processing capability and processing time, the accuracy of face detection / face recognition processing cannot be obtained sufficiently. As a result, there may be a problem that erroneous recognition, erroneous detection, or in the worst case, face recognition or detection itself becomes impossible.

  Even a small camera with limited processing capability and processing time is required to improve the speed and accuracy of face detection / face recognition processing. However, the conventional methods disclosed in Patent Documents 1 to 6 cannot sufficiently meet such a demand.

  Therefore, the present invention has been made in view of such conventional problems, and an object thereof is to improve the speed and accuracy of face detection / face recognition processing.

  According to a first aspect of the present invention, for an input image including a main subject, an estimation unit that estimates an attention point region using a saliency map based on a plurality of feature amounts extracted from the input image; Using the attention point area estimated by the estimation unit, a setting unit that sets a parameter related to subject detection processing for detecting the main subject from the input image, and using the parameter set by the setting unit, An image processing apparatus is provided that includes a detection unit that executes subject detection processing.

  According to the second aspect of the present invention, the detection unit detects, as the subject detection process, a face of a person as the main subject or performs a face process of recognizing a specific person's face Providing equipment.

  According to a third aspect of the present invention, the parameter is a parameter indicating whether or not to be the target of the face processing, and the setting unit, when the target point region satisfies a predetermined condition, the target point region Is included in the face processing target, and if the target point area does not satisfy the predetermined condition, a second setting is performed to exclude the target point area from the face processing target, and the detection is performed. When the first setting is performed by the setting unit, the face detection process is performed on the attention point region, and when the second setting is performed by the setting unit, the attention is performed. Provided is an image processing device that prohibits execution of the face detection processing for a point region.

  According to a fourth aspect of the present invention, the parameters include the size or target range of the face area detected by the face processing, and the size of the area from which feature points are extracted from the face detected by the face detection processing. Alternatively, an image processing apparatus having a resolution or a combination of two or more of these is provided.

  According to a fifth aspect of the present invention, there is provided the image processing apparatus, wherein the parameter is a parameter related to the accuracy of the face processing.

  According to a sixth aspect of the present invention, the parameter is an order of the face processing, and the setting unit sets the order for each of the one or more attention point regions according to a predetermined condition. And the detection unit provides an image processing device that sequentially executes the face processing for each of the one or more attention point regions according to the order set by the setting unit.

  According to a seventh aspect of the present invention, for an input image including a main subject, an estimation step for estimating a point of interest area using a saliency map based on a plurality of feature amounts extracted from the input image; A setting unit that sets a parameter related to subject detection processing for detecting the main subject from the input image using the attention point region estimated by the processing of the estimation step, and the parameter set by the processing of the setting step And a detection step of executing the subject detection process.

  According to the present invention, the speed and accuracy of face processing (face detection / face recognition processing) can be improved.

It is a hardware block diagram of the image processing apparatus which concerns on one Embodiment of this invention. It is a figure explaining the outline of the face detection / face recognition process in one Embodiment of this invention, and its pre-process, Comprising: It is a figure which shows an example of a specific process result. It is a figure explaining the outline of the face detection / face recognition process in one Embodiment of this invention, and its pre-process, Comprising: It is a figure which shows an example of a specific process result. It is a figure explaining the outline of the face detection / face recognition process in one Embodiment of this invention, and its pre-process, Comprising: It is a figure which shows an example of a specific process result. It is a figure explaining the outline of the face detection / face recognition process in one Embodiment of this invention, and its pre-process, Comprising: It is a figure which shows an example of a specific process result. It is a flowchart which shows an example of the flow of the imaging | photography mode process in one Embodiment of this invention. It is a flowchart which shows an example of the flow of the imaging | photography mode process in one Embodiment of this invention. It is a flowchart which shows the detailed example of the flow of an attention point area | region estimation process among the imaging | photography mode processes in one Embodiment of this invention. It is a flowchart which shows an example of the flow of the feature-value map creation process among the imaging | photography mode processes in one Embodiment of this invention. It is a flowchart which shows another example of the flow of the feature-value map creation process in the imaging | photography mode processes in one Embodiment of this invention. It is a flowchart which shows the detailed example of the flow of the face detection / face recognition process in the imaging | photography mode process in one Embodiment of this invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus 100 according to an embodiment of the present invention. The image processing apparatus 100 can be configured by a digital camera, for example.

  The image processing apparatus 100 includes an optical lens device 1, a shutter device 2, an actuator 3, a CMOS sensor 4, an AFE 5, a TG 6, a DRAM 7, a DSP 8, a CPU 9, a RAM 10, a ROM 11, and a liquid crystal display controller. 12, a liquid crystal display 13, an operation unit 14, a memory card 15, a distance measuring sensor 16, and a photometric sensor 17.

  The optical lens device 1 is composed of, for example, a focus lens and a zoom lens. The focus lens is a lens for forming a subject image on the light receiving surface of the CMOS sensor 4.

  The shutter device 2 includes, for example, shutter blades. The shutter device 2 functions as a mechanical shutter that blocks a light beam incident on the CMOS sensor 4. The shutter device 2 also functions as a diaphragm that adjusts the amount of light flux incident on the CMOS sensor 4. The actuator 3 opens and closes the shutter blades of the shutter device 2 according to control by the CPU 9.

  The CMOS (Complementary Metal Oxide Semiconductor) sensor 4 is composed of, for example, a CMOS type image sensor. A subject image is incident on the CMOS sensor 4 from the optical lens device 1 through the shutter device 2. Therefore, the CMOS sensor 4 photoelectrically converts (photographs) the subject image at regular intervals according to the clock pulse supplied from the TG 6, accumulates the image signal, and sequentially outputs the accumulated image signal as an analog signal.

  An analog image signal is supplied from the CMOS sensor 4 to the AFE (Analog Front End) 5. Therefore, the AFE 5 performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal in accordance with the clock pulse supplied from the TG 6. As a result of various signal processing, a digital signal is generated and output from the AFE 5.

  A TG (Timing Generator) 6 supplies a clock pulse to the CMOS sensor 4 and the AFE 5 at regular intervals according to control by the CPU 9.

  A DRAM (Dynamic Random Access Memory) 7 temporarily stores a digital signal generated by the AFE 5 and image data generated by the DSP 8.

  A DSP (Digital Signal Processor) 8 performs various image processing such as white balance correction processing, γ correction processing, and YC conversion processing on the digital signal stored in the DRAM 7 under the control of the CPU 9. As a result of various image processing, image data composed of a luminance signal and a color difference signal is generated. Hereinafter, such image data is referred to as frame image data, and an image expressed by the frame image data is referred to as a frame image.

  A CPU (Central Processing Unit) 9 controls the operation of the entire image processing apparatus 100. A RAM (Random Access Memory) 10 functions as a working area when the CPU 9 executes each process. A ROM (Read Only Memory) 11 stores programs and data necessary for the image processing apparatus 100 to execute each process. The CPU 9 executes various processes in cooperation with a program stored in the ROM 11 using the RAM 10 as a working area.

  The liquid crystal display controller 12 converts frame image data stored in the DRAM 7 or the memory card 15 into an analog signal under the control of the CPU 9 and supplies the analog signal to the liquid crystal display 13. The liquid crystal display 13 displays a frame image as an image corresponding to the analog signal supplied from the liquid crystal display controller 12.

  The liquid crystal display controller 12 converts various image data stored in advance in the ROM 11 or the like into analog signals under the control of the CPU 9 and supplies the analog signals to the liquid crystal display 13. The liquid crystal display 13 displays an image corresponding to the analog signal supplied from the liquid crystal display controller 12. For example, in the present embodiment, image data of information that can specify various scenes (hereinafter referred to as scene information) is stored in the ROM 11. Therefore, various scene information is appropriately displayed on the liquid crystal display 13 as will be described later with reference to FIG.

  The operation unit 14 receives operations of various buttons from the user. The operation unit 14 includes a power button, a cross button, a determination button, a menu button, a release button, and the like. The operation unit 14 supplies the CPU 9 with signals corresponding to various button operations received from the user. The CPU 9 analyzes the user's operation content based on the signal from the operation unit 14 and executes processing according to the operation content.

  The memory card 15 records frame image data generated by the DSP 8. The distance measuring sensor 16 detects the distance to the subject under the control of the CPU 9. The photometric sensor 17 detects the luminance (brightness) of the subject under the control of the CPU 9.

  As an operation mode of the image processing apparatus 100 having such a configuration, there are various modes including a shooting mode and a reproduction mode. However, for the sake of simplicity, only the processing in the shooting mode (hereinafter referred to as shooting mode processing) will be described below. In the following, it is assumed that the subject of the shooting mode processing is mainly the CPU 9.

Next, in the shooting mode processing of the image processing apparatus 100 in FIG. 1, pre-processing until a parameter is set using a region of interest based on the saliency map, and face detection / face recognition processing using the parameter The outline of will be described.
FIG. 2 is a diagram for explaining the outline of the face detection / face recognition process and its pre-process, and shows an example of a specific process result.

When starting the shooting mode, the CPU 9 of the image processing apparatus 100 in FIG. 1 continues shooting by the CMOS sensor 4 and temporarily stores frame image data sequentially generated by the DSP 8 in the DRAM 7. Hereinafter, a series of processes of the CPU 9 is referred to as through imaging.
Further, the CPU 9 controls the liquid crystal display controller 12 and the like to sequentially read out each frame image data recorded in the DRAM 7 at the time of through imaging, and display a corresponding frame image on the liquid crystal display 13. Hereinafter, a series of processes of the CPU 9 is referred to as through display. A frame image displayed as a through image is referred to as a through image.
In the following description, it is assumed that a through image 51 shown in FIG. 2 is displayed on the liquid crystal display 13 by through imaging and through display. However, the through image 51 shown in FIG. 2 is a diagram of an actual frame image for simplicity of expression.

In this case, in step Sa, the CPU 9 executes, for example, the following process as the feature amount map creation process.
That is, for the frame image data corresponding to the through image 51, the CPU 9 can create a plurality of types of feature amount maps from contrasts of a plurality of types of feature amounts such as color, orientation, and luminance. A series of processing until a predetermined one type of feature amount map among the plurality of types is created is referred to herein as feature amount map creation processing. A detailed example of each feature amount map creation process will be described later with reference to FIG. 9 and FIG.
For example, in the example of FIG. 2, the feature map 52Fc is created as a result of the multi-scale contrast feature map creation process of FIG. 10A described later. In addition, as a result of processing for creating a feature map of the center-surround color histogram of FIG. 10B described later, a feature map 52Fh is created. Further, as a result of the color space distribution feature quantity map creation processing of FIG. 10C, a feature quantity map 52Fs is created.

In step Sb, the CPU 9 obtains a saliency map by integrating a plurality of types of feature amount maps. For example, in the example of FIG. 2, feature maps 52Fc, 52Fh, and 52Fs are integrated to obtain a saliency map 53S.
The process of step Sb corresponds to the process of step S45 of FIG.

In step Sc, the CPU 9 uses the saliency map to estimate an image area (hereinafter, referred to as a point-of-interest area) that is likely to attract human visual attention from the through image. For example, in the example of FIG. 2, the attention point region 54 is estimated from the through image 51 using the saliency map 53S.
The process of step Sc corresponds to the process of step S46 of FIG.

  In addition, a series of processes from the above steps Sa to Sc will be hereinafter referred to as an attention point area estimation process. The attention point region estimation process corresponds to the process in step S2 of FIG. Details of the attention point region estimation process will be described later with reference to FIGS.

Next, in step Sd, the CPU 9 executes, for example, the following process as the face detection / face recognition parameter setting process based on the attention point area.
That is, the CPU 9 sets parameters relating to face detection / face recognition processing using the attention point area. The parameter to be set is not particularly limited as long as it relates to face detection / face recognition processing.

For example, in the example of FIG. 2, the setting symmetry parameter is a parameter indicating whether or not to be a target of face detection / face recognition processing.
That is, the CPU 9 sets, for example, the area 56 shown in FIG. 2 as a target for face detection / face recognition processing. This is because the area 56 includes an attention point area that satisfies a predetermined condition among the attention point areas 54, that is, a predetermined surrounding area of such an attention point area. Note that the predetermined condition is not particularly limited, and for example, a condition regarding the size, position, shape, or aspect ratio of the region, or a combination of any two or more thereof can be adopted. .
For example, the CPU 9 excludes the area 57 and the area 58 shown in FIG. 2 from the face detection / face recognition processing targets. This is because the region 55 is a region that has not been estimated as the attention point region 54. This is because the region 57 is a region that does not satisfy the predetermined condition in the attention point region 54.
In this way, the CPU 9 can exclude the area 57 and the area 58 from the face detection / face recognition process. More generally, for example, the CPU 9 can exclude a target area of a predetermined range at the outer peripheral edge of the visual field area from being subject to face detection / face recognition processing. For example, the CPU 9 can exclude an attention point area having a size smaller than a predetermined size from the face detection / face recognition processing target. For example, even when the area is large, the CPU 9 also applies a face detection / face recognition process to a target point area corresponding to an aspect ratio whose aspect ratio is far from the aspect ratio of the face, or an elongated shape or line shape. Can be outside.
As a result, the CPU 9 can focus face detection / face recognition processing on a region that is particularly likely to attract human visual attention in the attention point region. Further, the CPU 9 can execute the face detection / face recognition process in a short time by reducing the size, the processing amount, the processing time, and the like for the target region.
The process of step Sd in the example of FIG. 2 corresponds to the processes of steps S4 and S5 of FIG.

Thus, when the parameter is set in the process of step Sd, the process proceeds to step Se. In step Se, the CPU 9 executes face detection / face recognition processing using the set parameters. As a result, a region including the face of the detected or recognized person is obtained. Hereinafter, such an area is referred to as a face detection area.
For example, in the example of FIG. 2, the face detection / face recognition process is performed only on the region 56 set as the target region among the regions 55 to 57 in the through image 51. In other words, execution of face detection / face recognition processing is prohibited for the regions 55 and 57 excluded from the target region. As a result, a face detection area 58 is obtained.
The process of step Se corresponds to the process of step S6 of FIG.

  FIG. 3 is a diagram for explaining the outline of the face detection / face recognition processing and the preprocessing thereof, and is a diagram illustrating an example different from the example of FIG. 2 as a specific processing result.

  3 is basically the same as that in FIG. 2, only the processing result shown in FIG. 3 will be described. In the following description, it is assumed that, for example, a through image 61 shown in FIG. 3 is displayed on the liquid crystal display 13 by through imaging and through display. However, the through image 61 shown in FIG. 3 is a diagram of an actual frame image for simplicity of expression.

  In this case, as a result of the process of step Sa, for example, in the example of FIG. That is, the feature map 62Fc is created as a result of the multi-scale contrast feature map creation process of FIG. In addition, as a result of processing for creating a feature map of a center-surround color histogram in FIG. Further, as a result of the color space distribution feature quantity map creation processing of FIG. Next, as a result of the processing in step Sb, these feature amount maps 62Fc, 62Fh, 62Fs are integrated to obtain a saliency map 63S. Then, as a result of the process of step Sc, the attention point areas 64, 65, 66 are estimated using the saliency map 63S.

In step Sd, face detection / face recognition parameter setting processing based on the point-of-interest area is executed, and parameters relating to face detection / face recognition processing are set. For example, in the example of FIG. 3, the setting symmetry parameter is a parameter related to the order of face detection / face recognition processing, specifically, for example, an execution order or a scanning order.
That is, for example, the CPU 9 sets the execution order of face detection / face recognition processing according to the size and position of the target point area. Specifically, for example, as shown in FIG. 3, the largest attention point region 64 is set as the first region in the face detection / face recognition processing execution order. The next largest attention point area 65 is set as the second area in the order of execution of face detection / face recognition processing. The smallest point-of-interest area 66 is set as the third area in the execution order of face detection / face recognition processing.
The process of step Sd in the example of FIG. 3 corresponds to the process of step S3 of FIG.

  In step Se, face detection / face recognition processing is executed using the set parameters. For example, in the example of FIG. 3, first, face detection / face recognition processing is performed on the attention point area 64, and as a result, a face detection area 71 is obtained. Second, face detection / face recognition processing is performed on the attention point area 65, and as a result, a face detection area 72 is obtained. Third, face detection / face recognition processing is performed on the attention point region 66, and as a result, a face detection region 73 is obtained.

  4 and 5 are diagrams for explaining the outline of the face detection / face recognition processing and the preprocessing thereof, and are diagrams illustrating examples of specific processing results different from the examples of FIGS. 2 and 3. .

  4 and 5 are basically the same as those in FIG. 2, and only the processing results shown in FIGS. 4 and 5 will be described. Further, through image pickup and through display, the through image 81 shown in FIG. 4 is displayed on the liquid crystal display 13 in the description of the example of FIG. 4, and the through image 111 shown in FIG. Suppose that However, the through image 81 shown in FIG. 4 and the through image 111 shown in FIG. 5 are diagrams of actual frame images for simplicity of expression.

  In this case, as a result of the process of step Sa, for example, in the example of FIGS. 4 and 5, the following feature amount maps are created.

  That is, in the example of FIG. 4, a feature map 82Fc is created as a result of the multi-scale contrast feature map creation process of FIG. 10A described later. Also, as a result of the feature amount map creation process of the center-surround color histogram of FIG. 10B described later, a feature amount map 82Fh is created. Further, as a result of the color space distribution feature quantity map creation processing of FIG. 10C, a feature quantity map 82Fs is created.

  On the other hand, in the example of FIG. 5, a feature map 112Fc is created as a result of the multi-scale contrast feature map creation process of FIG. 10A described later. Further, as a result of processing for creating a feature map of the center-surround color histogram of FIG. 10B described later, a feature map 112Fh is created. In addition, as a result of the feature map creation processing of the color space distribution in FIG. 10C, the feature map 112Fs is created.

  Next, in the example of FIG. 4, as a result of the process of step Sb, the feature amount maps 82Fc, 82Fh, and 82Fs are integrated to obtain the saliency map 83S. Then, as a result of the process of step Sc, the attention point region 84 is estimated using the saliency map 83S.

  On the other hand, in the example of FIG. 5, as a result of the processing in step Sb, the feature amount maps 112Fc, 112Fh, and 112Fs are integrated to obtain the saliency map 113S. Then, as a result of the processing in step Sc, a plurality of attention point regions 114 are estimated using the saliency map 113S.

  In step Sd, face detection / face recognition parameter setting processing based on the point-of-interest area is executed, and parameters relating to face detection / face recognition processing are set. For example, in the examples of FIGS. 4 and 5, the setting symmetry parameter is a parameter related to face feature data (feature points) in the face detection / face recognition processing. Specifically, for example, the extraction method, the number of extractions, the collation method, and the like for facial feature data are set as parameters to be set. Therefore, the CPU 9 sets various parameters related to facial feature data, for example, in accordance with the size and position of the attention point area.

  Specifically, for example, in the example of FIG. 4, the number of extracted attention point regions 84 is one and below a certain number (a small number), and the attention point region 84 occupying in the visual field region is larger than a predetermined size. In such a case, the CPU 9 sets the feature data extraction region in a slightly wide region 85 such as “upper forehead to under the chin”, for example. Further, the CPU 9 uses, for example, nine feature points for the face contour, 2 × 5 feature points for the left and right eyebrows, 2 × 9 feature points for the left and right eyes, three feature points for the nose, A total of 48 feature points of 8 feature points are set on the lips. For example, the CPU 9 sets the degree of collation evaluation processing to a degree of “slightly detailed”.

On the other hand, for example, in the example of FIG. 5, the number of attention point regions 114 exceeds a certain number (a large number), and each attention point region 114 is dispersed in a wide area. Moreover, each attention point area | region 114 is smaller than predetermined size. In such a case, the CPU 9 sets the feature data extraction area to a slightly narrow area 115 such as “above the eyebrows to under the lips”. Further, the CPU 9 sets, for example, a total of 24 feature points that are smaller than the example of FIG. 4 as the feature data 116 of the main part. Note that the deduction points of the feature points are not particularly limited to the example of FIG. For example, the CPU 9 sets the degree of the collation evaluation process to a degree of “slightly rough”.
Note that the processing in step Sd in the examples of FIGS. 4 and 5 corresponds to the processing in step S205 in FIG. 11 described later.

In step Se, face detection / face recognition processing is executed using the set parameters.
For example, in the example of FIG. 4, the feature data 86 is extracted from the area 85 shown in FIG. 4, and a slightly more detailed collation process is executed. As a result, the face detection area 87 is obtained.
On the other hand, for example, in the example of FIG. 5, the feature data 116 is extracted from the region 115 shown in FIG. 5 and a slightly rough collation process is executed. As a result, a plurality of face detection regions 117 are obtained.

  The outline of the face detection / face recognition processing and the preprocessing performed by the image processing apparatus 100 has been described above with reference to FIGS. Next, with reference to FIGS. 6 to 11, the entire photographing mode process including the face detection / face recognition process and its pre-process will be described.

  6 and 7 are flowcharts showing an example of the flow of shooting mode processing.

  When the user performs a predetermined operation for selecting a shooting mode on the operation unit 14, the shooting mode process is started when the operation is performed. That is, the following processing is executed.

  In step S1, the CPU 9 performs through imaging and through display.

  In step S <b> 2, the CPU 9 estimates a target point region by executing a target point region estimation process. The outline of the point-of-interest area estimation process has been described above as the description of steps Sa to Sc in FIGS. 2 to 5, and details thereof will be described later with reference to FIG.

  In step S3, the CPU 9 sets a predetermined one of the attention point areas estimated in the process of step S2 as a process target area. Here, parameters relating to the order of face detection / face recognition processing may be included as parameters relating to face detection / face recognition processing in step S6 described later. In such a case, as described in the example of FIG. 3, the CPU 9 sets the order for each attention point area according to the size and position of each attention point area estimated in the process of step S <b> 2. To do. Then, the processing target areas are sequentially set according to this order.

In step S4, the CPU 9 evaluates the processing target area. In step S5, the CPU 9 determines whether or not the processing target area satisfies a predetermined condition based on the evaluation result.
That is, in the example of FIG. 6, it is assumed that a parameter regarding whether or not to be a target area for face detection / face recognition processing is included as a parameter related to face detection / face recognition processing in step S6 described later. Yes. Accordingly, the predetermined condition here is not particularly limited as described above with reference to FIG. 2, but for example, the condition regarding the size, position, shape, and aspect ratio of the region, or Any combination of two or more of them can be employed.

  When the processing target area does not satisfy the predetermined condition, a setting is made to exclude the processing target area from the target of face detection / face recognition processing. For this reason, it is determined NO in the process of step S5, the face detection / face recognition process of step S6 is not executed for the processing target area, the process returns to step S3, and the subsequent processes are performed. Executed. That is, another attention point area is newly set as a process target in the process of step S3, and the processes after step S4 are executed.

  On the other hand, when the processing target area satisfies a predetermined condition, the processing target area is set as a target for face detection / face recognition processing. For this reason, it determines with it being YES by the process of step S5, and a process progresses to step S6. In step S6, the CPU 9 performs face detection / face recognition processing on the processing target area. Details of the face detection / face recognition processing will be described later with reference to FIG.

  In step S7, the CPU 9 determines whether or not a face detection area exists. When one or more face detection areas are obtained by the face detection / face recognition process in step S6, it is determined as YES in step S7, and the process proceeds to step S8. In step S8, the CPU 9 sets an AF (Automatic Focus) frame in each of the one or more face detection areas. Thereby, a process progresses to step S9. On the other hand, if the face detection area is not obtained by the face detection / face recognition process in step S6, it is determined as NO in step S7, the process in step S8 is not executed, and the process proceeds to step S9. Proceed to

  In step S9, the CPU 9 determines whether or not the number of face detection areas is equal to or greater than a threshold value. If the number of face detection areas is greater than or equal to the threshold, it is determined as YES in step S9, and the process proceeds to step S11. However, the process after step S11 is mentioned later.

  On the other hand, if the number of face detection areas is less than the threshold value, it is determined as NO in step S9, and the process proceeds to step S10. In step S <b> 10, the CPU 9 determines whether the processing target area is the last attention point area. If the processing target area is not the last target point area, that is, if there is an area that has not been set as the processing target area among the target point areas estimated in the process of step S2, NO is determined in step S10. After the determination, the process is returned to step S3, and the subsequent processes are repeated. That is, another attention point area is newly set as a process target in the process of step S3, and the processes after step S4 are executed.

In this way, each target point area estimated in the process of step S2 is sequentially set as a process target area, and the loop process of steps S3 to S10 is repeatedly executed. After that, when the number of face detection areas is less than the threshold value and the last attention point area is set as the processing target and the processes in steps S3 to S9 are executed, it is determined that the next process in step S10 is YES. Then, the process proceeds to step S11.
Further, when the number of face detection regions is equal to or greater than the threshold value during the loop processing of steps S3 to S10, as described above, it is determined as YES in the processing of step S9, and the processing is performed in step S11. Proceed to

  In step S11, the CPU 9 executes an AF process (autofocus process) so that the AF frame set in the process of step S8 falls within the depth of field.

In step S12, the CPU 9 determines whether or not the release button is half pressed.
If the user has not pressed the release button halfway, it is determined as NO in step S12, the process returns to step S1, and the subsequent processes are repeated. That is, until the user half-presses the release button, the loop process of steps S1 to S12 is repeatedly executed.

  Thereafter, when the user presses the release button halfway, it is determined as YES in Step S12, and the process proceeds to Step S13 in FIG.

  In step S13, the CPU 9 executes a photometric process.

  In step S14, the CPU 9 determines whether or not the release button is fully pressed.

  If the user has not fully pressed the release button, it is determined as NO in Step S14, and the process proceeds to Step S22. In step S22, the CPU 9 determines whether or not the release button has been released. If the user's finger or the like is released from the release button, it is determined as YES in step S22, and the shooting mode process ends. On the other hand, if the user's finger or the like is not released from the release button, it is determined as NO in step S22, the process returns to step S14, and the subsequent processes are repeated. That is, as long as the release button is half-pressed, the loop process of steps S14NO and S22NO is repeated.

  Thereafter, when the user fully presses the release button, it is determined as YES in Step S14, and the process proceeds to Step S15. In step S15, the CPU 9 executes an AWB (Automatic White Balance) process (auto white balance process). In step S16, the CPU 9 executes an AE (Automatic Exposure) process (automatic exposure process). That is, the aperture, exposure time, strobe conditions, and the like are set based on photometric information obtained by the photometric sensor 17 and photographing conditions.

  In step S <b> 17, the CPU 9 controls the TG 6, the DSP 8, and the like, and executes exposure and photographing processing based on photometric information and photographing conditions by the photometric sensor 17. By this exposure and photographing processing, the subject image photographed by the CMOS sensor 4 according to the photographing conditions and the like is stored in the DRAM 7 as frame image data. Hereinafter, such frame image data is referred to as captured image data, and an image expressed by the captured image data is referred to as a captured image.

  In step S18, the CPU 9 controls the DSP 8 and the like, and executes correction and change processing of captured image data. That is, the CPU 9 appropriately performs various image processes necessary for correction or change on the captured image data based on the face detection area, the attention point area, the imaging conditions, and the like.

  In step S19, the CPU 9 controls the liquid crystal display controller 12 and the like to execute a review display process for the photographed image. In step S20, the CPU 9 controls the DSP 8 and the like to execute the compression encoding process of the photographed image data. As a result, encoded image data is obtained. Therefore, in step S21, the CPU 9 executes a process for storing and recording encoded image data. As a result, the encoded image data is recorded in the memory card 15 or the like, and the photographing mode process is completed.

  Next, a detailed example of the attention point area process in step S2 (steps Sa to Sc in FIGS. 2 to 5) in the shooting mode process will be described.

As described above, in the attention point region estimation process, a saliency map is created for estimating the attention point region. Therefore, for example, the Treisman feature integration theory or the saliency map by Itti and Koch et al. Can be applied to the attention point region estimation process.
As for the feature integration theory of Treisman, “AM Treisman and G. Gelade,“ A feature-integration theory of attention ”, Cognitive Psychology, Vol. 12, No. 1, pp. 97-136, 1980. Please refer to.
For the saliency map by Itti and Koch et al., “L. Itti, C. Koch, and E. Niebur,“ A Model of Salientity-Based Visual Attention for Rapid Scen Analysis ”, IE Et. , VOL.20, No11, November 1998. ”.

  FIG. 8 is a flowchart showing a detailed example of the flow of attention point region estimation processing when applying the Triisman feature integration theory and the saliency map by Niti and Koch et al.

  In step S41, the CPU 9 inputs frame image data obtained by through imaging as processing target image data.

In step S42, the CPU 9 creates a Gaussian resolution pyramid. Specifically, for example, the CPU 9 sets the processing target image data {pixel data at the position of (x, y)} to I (0) = I (x, y), and sequentially repeats the Gaussian filter processing and the downsampling processing. Execute. As a result, a set of hierarchical scale image data I (L) (for example, Lε {0... 8}) is generated. This set of hierarchical scale image data I (L) is called a Gaussian resolution pyramid. Here, when the scale L = k (here, k is any integer value from 1 to 8), the scale image data I (k) is a reduced image of 1/2 ^k (when k = 0). Original image).

  In step S43, the CPU 9 starts each feature amount map creation process. That is, the CPU 9 can create a plurality of types of feature amount maps from the contrast of a plurality of types of feature amounts such as color, azimuth, and luminance for the processing target image data. A series of processing until a predetermined one type of feature amount map among the plurality of types is created is referred to herein as feature amount map creation processing. A detailed example of each feature amount map creation process will be described later with reference to FIG. 9 and FIG.

  In step S44, the CPU 9 determines whether or not all the feature map creation processing has been completed. If even one of the feature map creation processes has not been completed, it is determined as NO in step S44, and the process returns to step S44 again. That is, the determination process in step S44 is repeatedly executed until all the process of each feature amount map creation process is completed. When all the feature value map creation processes are completed and all feature value maps are created, it is determined as YES in Step S44, and the process proceeds to Step S45.

  In step S45, the CPU 9 obtains a saliency map S (Saliency Map) by combining the feature amount maps with a linear sum.

  In step S <b> 46, the CPU 9 estimates a region of interest from the processing target image data using the saliency map S. That is, generally, a person who is a main subject and an object which is a subject to be photographed (objects) are considered to have higher saliency than a background area. Therefore, the CPU 9 recognizes a region having high saliency from the processing target image data using the saliency map S. Then, based on the recognition result, the CPU 9 estimates an area that is likely to attract human visual attention, that is, an attention point area. When the attention point region is estimated in this way, the attention point region estimation process ends. That is, the process of step S2 in FIG. 6 ends, and the process proceeds to step S3. In the example of FIGS. 2 to 5, the series of steps Sa to Sc ends, and the process proceeds to step Sd.

  Next, a specific example of each feature amount map creation process will be described.

  FIG. 9 is a flowchart illustrating an example of a flow of a feature amount map creation process for luminance, color, and directionality.

  FIG. 9A shows an example of a luminance feature amount map creation process.

  In step S61, the CPU 9 sets each pixel of interest from each scale image corresponding to the processing target image data. For example, assuming that each pixel of interest cε {2, 3, 4} is set, the following description will be given. Each pixel of interest cε {2, 3, 4} is a pixel set as a calculation target on the scale image data I (c) of scale cε {2, 3, 4}.

  In step S62, the CPU 9 obtains the luminance component of each scale image of each pixel of interest cε {2, 3, 4}.

  In step S63, the CPU 9 calculates the luminance component of each scale image of the peripheral pixel s = c + δ of each pixel of interest. The peripheral pixel s = c + δ of each target pixel is a pixel existing around the target pixel (corresponding point) on the scale image I (s) of the scale s = c + δ, for example, when δε {3, 4}. Say.

  In step S64, the CPU 9 obtains the luminance contrast at each target pixel cε {2, 3, 4} for each scale image. For example, the CPU 9 obtains an inter-scale difference between each target pixel cε {2, 3, 4} and a peripheral pixel s = c + δ (for example, δε {3,4)} of each target pixel. Here, when the target pixel c is referred to as “Center” and the peripheral pixel s of the target pixel is referred to as “Surround”, the obtained inter-scale difference can be referred to as a luminance Center-Surround inter-scale difference. The difference between the center-surround scales of the luminance has a property of taking a large value when the target pixel c is white and the peripheral pixel s is black or vice versa. Therefore, the luminance Center-Surround scale difference represents the luminance contrast. Hereinafter, such luminance contrast is described as I (c, s).

  In step S65, the CPU 9 determines whether or not there is a pixel that is not set as the target pixel in each scale image corresponding to the processing target image data. If such a pixel exists, it is determined as YES in step S65, the process returns to step S61, and the subsequent processes are repeated.

  That is, the processing of steps S61 to S65 is performed on each pixel of each scale image corresponding to the processing target image data, and the luminance contrast I (c, s) of each pixel is obtained. Here, when each target pixel cε {2, 3, 4} and the surrounding pixel s = c + δ (for example, δε {3,4)} are set, in one process of steps S61 to S65, (3 types of pixel of interest c) × (2 types of peripheral pixel s) = 6 luminance contrasts I (c, s) are obtained. Here, the aggregate of the entire image of the luminance contrast I (c, s) obtained for the predetermined c and the predetermined s is hereinafter referred to as a feature amount map of the luminance contrast I. As a result of repeating the loop processing of steps S61 to S65, six types of feature maps of luminance contrast I are obtained. In this way, when six feature maps of luminance contrast I are obtained, it is determined NO in step S65, and the process proceeds to step S66.

  In step S <b> 66, the CPU 9 creates a luminance feature amount map by normalizing and combining the feature amount maps of the luminance contrast I. As a result, the luminance feature amount map creation process ends. Hereinafter, the feature map of luminance is described as FI in order to distinguish it from other feature maps.

  FIG. 9B shows an example of color feature amount map creation processing.

  Compared with the luminance feature quantity map creation processing of FIG. 9A, the processing flow of the color feature quantity map creation processing of FIG. 9B is basically the same, and only the processing target is different. That is, each process of steps S81 to S86 in FIG. 9B is a process corresponding to each of steps S61 to S66 in FIG. 9A, and the processing target of each step is only different from that in FIG. 9A. Therefore, the description of the processing flow of the color feature amount map creation processing in FIG. 9B is omitted, and only the processing target will be briefly described below.

That is, the processing target in steps S62 and S63 in FIG. 9A is a luminance component, whereas the processing target in steps S82 and S83 in FIG. 9B is a color component.
Further, in the process of step S64 in FIG. 9A, the luminance Center-Surround scale difference is obtained as the luminance contrast I (c, s). On the other hand, in the process of step S84 in FIG. 9B, the difference between the center and surround scales of the hue (R / G, B / Y) is obtained as the hue contrast. Of the color components, a red component is indicated by R, a green component is indicated by G, a blue component is indicated by B, and a yellow component is indicated by Y. Hereinafter, the hue contrast for the hue R / G is described as RG (c, s), and the hue contrast for the hue B / Y is described as BY (c, s).
Here, in accordance with the above example, it is assumed that there are three types of the target pixel c and two types of the peripheral pixel s. In this case, as a result of the loop processing in steps S61 to S65 in FIG. 9A, six types of feature map of luminance contrast I were obtained. On the other hand, as a result of the loop processing in steps S81 to S85 in FIG. 9B, six kinds of hue contrast RG feature quantity maps and six kinds of hue contrast BY feature quantity maps are obtained.
Finally, a luminance feature amount map FI was obtained in the process of step S66 of FIG. 9A. On the other hand, a color feature amount map is obtained in step S86 in FIG. 9B. Hereinafter, the color feature map is described as FC in order to distinguish it from other feature maps.

  FIG. 9C illustrates an example of a directional feature amount map creation process.

  Compared with the luminance feature quantity map creation processing of FIG. 9A, the flow of processing of the directional feature quantity map creation processing of FIG. 9C is basically the same, and only the processing target is different. That is, each process of steps S101 to S106 in FIG. 9C is a process corresponding to each of steps S61 to S66 in FIG. 9A, and the processing target of each step is only different from that in FIG. 9A. Therefore, the description of the flow of the directional feature amount map creation process in FIG. 9C will be omitted, and only the processing target will be briefly described below.

That is, the processing target of steps S102 and S1023 is a direction component. Here, the direction component means an amplitude component in each direction obtained as a result of convolution of the Gaussian filter φ with the luminance component. The direction here refers to the direction indicated by the rotation angle θ existing as a parameter of the Gaussian filter φ. For example, four directions of 0 °, 45 °, 90 °, and 135 ° can be adopted as the rotation angle θ.
Further, in the process of step S104, the directional Center-Surround scale difference is obtained as the directional contrast. Hereinafter, the directional contrast is described as O (c, s, θ).
Here, in accordance with the above example, it is assumed that there are three types of the target pixel c and two types of the peripheral pixel s. In this case, as a result of the loop processing in steps S101 to S105, six feature amount maps of the directional contrast O are obtained for each rotation angle θ. For example, when four directions of 0 °, 45 °, 90 °, and 135 ° are employed as the rotation angle θ, 24 (= 6 × 4) directional contrast O feature amount maps are obtained. .
Finally, a directional feature map is obtained in the process of step S106. Hereinafter, the directional feature quantity map is described as FO in order to distinguish it from other feature quantity maps.

  For more detailed processing contents of the feature amount map creation processing of FIG. 9 described above, for example, “L. Itti, C. Koch, and E. Niebur,“ A Model of Saliency-Based Visual Attention for Rapid Scene Analysis ”. , IEEE Transactions on Pattern Analysis and Machine Intelligence, VOL.20, No11, November 1998. ”.

  Note that the feature map creation processing is not particularly limited to the example of FIG. For example, as the feature quantity map creation process, it is possible to employ a process of creating each feature quantity map using each feature quantity of lightness, saturation, hue, and motion.

  Further, for example, as the feature amount map creation processing, it is also possible to employ processing for creating each feature amount map using each feature amount of multi-scale contrast, Center-Surround color histogram, and color space distribution. .

  FIG. 10 is a flowchart illustrating an example of a multi-scale contrast, a Center-Surround color histogram, and a color space distribution feature amount map creation process.

FIG. 10A shows an example of a multi-scale contrast feature map creation process.
In step S121, the CPU 9 obtains a feature map of multiscale contrast. As a result, the multi-scale contrast feature map creation processing is completed.
Hereinafter, the feature scale map of multi-scale contrast is described as Fc so as to be distinguished from other feature map.

  FIG. 10B illustrates an example of a feature map creation process for a Center-Surround color histogram.

  In step S141, the CPU 9 obtains a color histogram of the rectangular area and a color histogram of the peripheral outline for each different aspect ratio. The aspect ratio itself is not particularly limited, and for example, {0.5, 0.75, 1.0, 1.5, 2.0} can be adopted.

  In step S142, the CPU 9 obtains a chi-square distance between the color histogram of the rectangular area and the color histogram of the peripheral contour for each different aspect ratio. In step S143, the CPU 9 obtains a color histogram of a rectangular area where the chi-square distance is maximum.

In step S <b> 144, the CPU 9 creates a feature map of the Center-Surround color histogram using the color histogram of the rectangular area having the maximum chi-square distance. Thus, the center-surround color histogram feature quantity map creation processing is completed.
Hereinafter, the feature amount map of the Center-Surround color histogram is described as Fh so as to be distinguished from other feature amount maps.

  FIG. 10C illustrates an example of a feature map creation process for a color space distribution.

  In step S161, the CPU 9 calculates the horizontal variance for the color space distribution. In step S162, the CPU 9 calculates the vertical variance for the color space distribution. In step S163, the CPU 9 obtains the spatial dispersion of the colors using the horizontal dispersion and the vertical dispersion.

In step S164, the CPU 9 creates a feature amount map of color space distribution using the spatial dispersion of colors. As a result, the feature map creation process of the color space distribution is completed.
Hereinafter, the feature map of the color space distribution is described as Fs so as to be distinguished from other feature maps.

  For more detailed processing contents of the feature amount map creation processing of FIG. 10 described above, for example, “T. Liu, J. Sun, N. Zheng, X. Tang, H. Sum,“ Learning to Detect A Salient Object ”. ", CVPR07, pp. 1-8, 2007."

  Next, a detailed example of the face detection / face recognition process in step S6 in the shooting mode process of FIG. 6 will be described.

  FIG. 11 is a flowchart showing a detailed example of the flow of face detection / face recognition processing.

  In step S201, the CPU 9 sets a face detection dictionary based on the face detection dictionary information. It is assumed that the face detection dictionary information is stored in advance in, for example, the ROM 11 in FIG.

  In step S202, the CPU 9 determines whether or not personal recognition of the face is performed. When only face detection is performed, it is determined as NO in step S202, and the process proceeds to step S205. However, the processing after step S205 will be described later.

  On the other hand, if personal recognition of the face is also performed, it is determined as YES in step S202, and the process proceeds to step S203. In step S203, the CPU 9 determines whether or not a face for personal recognition is registered. If the face to be recognized is not registered, it is determined as NO in step S203, and the process proceeds to step S205. However, the processing after step S205 will be described later.

  On the other hand, if a face for personal recognition is registered, it is determined as YES in step S203, and the process proceeds to step S204. In step S <b> 204, the CPU 9 sets a face recognition dictionary based on personal face recognition dictionary information of the face. Note that it is assumed that dictionary information for personal recognition of a face is stored in advance in, for example, the ROM 11 in FIG.

  Thus, when the process of step S204 is executed, the process proceeds to step S205. As described above, when it is determined as NO in step S202 or when it is determined as NO in step S203, the process proceeds to step S205. In step S205, the CPU 9 sets parameters for face detection / face recognition based on the attention point area. Note that the attention point area here is not only the attention point area set as the processing target area in the process of step S3 in FIG. 6, but also each attention point area estimated in the process of step S2 as necessary. Including. For example, the parameters related to the facial feature data described above as examples in FIGS. 4 and 5, specifically, for example, the extraction method, the number of extractions, the collation method, and the like are set in the process of step S205.

  In step S <b> 206, the CPU 9 inputs frame image data obtained by through imaging as processing target image data.

  In step S207, the CPU 9 extracts a face detection area from the frame image corresponding to the processing target image data. In step S208, the CPU 9 extracts facial feature data from at least a part of the face detection area. The parameters relating to the extraction of the facial feature data are the parameters described as examples in FIGS. 4 and 5 and are set in the process of step S205 as described above.

  In step S209, the CPU 9 determines whether or not personal recognition of the face is performed. If personal recognition of the face is also performed, it is determined as YES in step S209, and the process proceeds to step S210. In step S210, the CPU 9 collates the face feature data with the face recognition dictionary. As a result, a collation evaluation value is obtained.

  On the other hand, when only face detection is performed, NO is determined in step S209, and the process proceeds to step S211. In step S211, the CPU 9 collates the face feature data with the face detection dictionary. As a result, a collation evaluation value is obtained.

  In this way, when the process of step S210 or S211 is executed and the collation evaluation value is obtained, the process proceeds to step S212. In step S212, the CPU 9 determines whether or not the collation evaluation value is greater than or equal to the collation threshold value.

  When the collation evaluation value is less than the collation threshold value, it is determined as NO in Step S212, and the process proceeds to Step S213. In step S213, the CPU 9 determines whether or not a predetermined error condition is satisfied. If the predetermined error condition is not satisfied, NO is determined in step S213, the process returns to step S206, and the subsequent processes are repeated. On the other hand, if the predetermined error condition is satisfied, it is determined as YES in step S213, and the face detection / face recognition process is ended. That is, in this case, it is determined that the face detection area is not obtained by the face detection / face recognition process in step S6 of FIG. 6 and NO is determined in step S7, and the process proceeds to step S9. become.

  On the other hand, when the collation evaluation value is equal to or greater than the collation threshold value, it is determined as YES in Step S212, and the process proceeds to Step S214. In step S214, the CPU 9 determines the collation result. That is, information such as a face identification ID, a face position, and a collation evaluation value is determined as information related to the face detection area and stored in the DRAM 7 or the RAM 10. As a result, the face detection / face recognition process ends. That is, in this case, assuming that the face detection area is obtained by the face detection / face recognition process in step S6 of FIG. 6, it is determined YES in the process of step S7, and the process proceeds to step S8. Become.

  As described above, the CPU 9 of the image processing apparatus 100 according to the present embodiment uses the saliency map based on a plurality of feature amounts extracted from the input image for the input image (through image) including the main subject. Thus, it has a function of estimating the attention point region. The CPU 9 has a function of setting parameters relating to face detection / face recognition processing using the estimated attention point region. The CPU 9 has a function of executing face detection / face recognition processing using the set parameters.

  As described above, the CPU 9 can execute the attention point region estimation process and the face detection / face recognition process in a tandem manner or in a parallel manner. Thus, the CPU 9 can appropriately set parameters using the attention point region, and as a result, the speed and accuracy of the face detection / face recognition processing can be improved.

  For example, the CPU 9 can execute the face detection / face recognition process only on the target point area that satisfies the predetermined condition. As a result, the CPU 9 can narrow down the face search, the feature data to be extracted, the collation range, and the like according to the attention point area, or can exclude the attention point area that can be recognized as processing unnecessary from the processing target. As a result, the speed and accuracy of face detection / face recognition processing can be improved.

  Further, for example, the CPU 9 sets parameters of each process (face detection process, feature data extraction process, collation evaluation process, etc.) in the face detection / face recognition process according to the size, position, shape, etc. Can be adjusted appropriately. Further, for example, the CPU 9 can make settings so that the face detection / face recognition process is executed in more detail for a point of interest region with a large size or a region of interest with a high possibility of face detection. On the other hand, the CPU 9 can perform setting so that the face detection / face recognition process is performed more roughly for the attention point area having a small size or the attention point area having a low possibility of face detection. As a result, face detection / face recognition processing is performed efficiently, and the overall processing time is shortened. That is, the overall speed and accuracy of face detection / face recognition processing can be improved.

  As described above, the parameters to be set need only be parameters relating to face detection / face recognition processing, and various parameters can be adopted. Although it may overlap with the above-described example, a few examples of parameters that can be adopted are listed below. For example, it is possible to employ a parameter indicating whether or not to be a target of face detection / face recognition processing. For example, the size or target range of the face area detected or recognized by the face detection / face recognition processing, the size or resolution parameter of the area where feature points (feature data) are extracted from the face, or these two Combinations of the above can be employed. For example, parameters relating to the accuracy of face detection / face recognition processing can be employed. For example, parameters relating to the order of face detection / face recognition processing can be employed.

Moreover, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.
For example, in the above-described embodiments, the image processing apparatus to which the present invention is applied has been described as an example configured as a digital camera. However, the present invention is not particularly limited to digital cameras, and can be applied to electronic devices in general. Specifically, for example, the present invention is applicable to a video camera, a portable navigation device, a portable game machine, and the like.

  Further, for example, the present invention can be widely applied not only to face detection / face recognition processing but also to detection processing for detecting a main subject from an input image. In this case, the CPU 9 has a function of setting a parameter related to the detection process using the estimated attention point region. Thus, for example, when the attention point area has a predetermined size, contour shape, aspect ratio within a predetermined range, the CPU 9 prohibits the human face detection process, Various processes such as detection (identification) of an object other than a face can be executed.

  The series of processes described above can be executed by hardware or can be executed by software.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.
Although not shown, the recording medium including such a program is not only constituted by a removable medium distributed separately from the apparatus main body in order to provide a program to the user, but also in a state of being incorporated in the apparatus main body in advance. It consists of a recording medium provided to the user. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 11 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  DESCRIPTION OF SYMBOLS 100 ... Image processing apparatus, 1 ... Optical lens apparatus, 2 ... Shutter apparatus, 3 ... Actuator, 4 ... CMOS sensor, 5 ... AFE, 6 ... TG, 7 ··· DRAM, 8 ... DSP, 9 ... CPU, 10 ... RAM, 11 ... ROM, 12 ... Liquid crystal display controller, 13 ... Liquid crystal display, 14 ... Operation unit, 15 ... Memory card, 16 ... Ranging sensor, 17 ... Photometric sensor

Claims (7)

An estimation unit that estimates a region of interest using an saliency map based on a plurality of feature amounts extracted from the input image for an input image including a main subject;
A setting unit that sets a parameter related to subject detection processing for detecting the main subject from the input image using the attention point region estimated by the estimation unit;
A detection unit that executes the subject detection process using the parameters set by the setting unit;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the detection unit detects, as the subject detection process, a face of a person as the main subject or recognizes a face of a specific person. The parameter is a parameter indicating whether or not to be a target of the face processing,
The setting unit performs a first setting to include the target point region in the face processing target when the target point region satisfies a predetermined condition, and when the target target region does not satisfy the predetermined condition, Perform a second setting to exclude the point area from the face processing target,
When the first setting is performed by the setting unit, the detection unit performs the face processing on the attention point region, and when the second setting is performed by the setting unit, The image processing apparatus according to claim 2, wherein execution of the face processing on the attention point region is prohibited. The parameter is a parameter relating to the size or target range of a face area detected or recognized by the face processing, the size or resolution of the area from which a feature point is extracted from the face, or a combination of two or more thereof. The image processing according to claim 2. The image processing according to claim 2, wherein the parameter is a parameter related to accuracy of the face processing. The parameter is a parameter related to the order of the face processing,
The setting unit performs setting to attach the order to each of the one or more attention point regions according to a predetermined condition,
The image processing apparatus according to claim 2, wherein the detection unit sequentially executes the face processing on each of the one or more attention point regions according to the order set by the setting unit. An estimation step for estimating a region of interest using an saliency map based on a plurality of feature amounts extracted from the input image for an input image including a main subject;
A setting step for setting a parameter relating to a subject detection process for detecting the main subject from the input image using the attention point region estimated by the processing of the estimation step;
A detection step of executing the subject detection process using the parameters set by the setting step;
An image processing method including: