JP5836602B2 - Image processing apparatus and image processing apparatus control method - Google Patents

Description

  The present invention relates to an image processing apparatus for determining a shooting scene and a control method for the image processing apparatus.

For example, there are various scenes such as a portrait, a landscape, and a night view taken by an imaging apparatus such as a video camera. Conventionally, in order to obtain optimal images in these shooting scenes, the camera analyzes the subject information at the time of shooting to automatically determine what scene was shot, and according to the determined scene Some perform optimal image processing.
In such a system, for example, “blue sky scene”, “natural green scene”, “evening scene” and the like are determined from information at the time of shooting, and blue sky blue, natural green, sunset scene are determined according to the determination result. Processing such as enhancing the saturation of the orange is performed.

  However, in order to perform these scene determination processes with high accuracy, advanced calculation processes are required, and if a plurality of scene determination processes are performed simultaneously, the calculation load becomes very heavy.

  For example, Patent Document 1 discloses a technique for determining whether or not correction is necessary, such as sunset determination, color covering determination, and contrast determination, and correcting corresponding items only when it is determined that correction is necessary. . According to this technique, since it is determined whether or not a predetermined correction is necessary and correction is performed only when necessary, the load of correction processing is reduced.

Japanese Patent Laid-Open No. 11-298736

However, in the above-mentioned Patent Document 1, even if it is determined whether or not correction is necessary (that is, scene detection processing) and only necessary correction is performed, a plurality of scene detection processing must be performed at the same time each time. There remains a problem that a calculation load is applied during movie shooting.
Accordingly, an object of the present invention is to provide an image processing apparatus capable of effectively detecting a scene while reducing a calculation load due to scene detection.

In view of the above problems, an image processing apparatus according to the present invention corresponds to the image data from among a plurality of preset scenes and image acquisition means for acquiring image data of a plurality of frames. A scene detection unit that identifies a scene to be set, a condition setting unit that automatically sets a plurality of image acquisition conditions or a plurality of image processing conditions according to a scene identification result by the scene detection unit, and a setting by the condition setting unit Designation means for designating image acquisition conditions or image processing conditions for automatically performing processing, and the conditions set by the condition setting means are a first scene, a first scene of the plurality of preset scenes, The first condition in which the unique setting is made in each of the second scene and the third scene, and the unique setting different from the other scene is made in the first scene. Wherein a second condition that the same setting is down and third scenes are made, and the scene detecting means, when the first condition is designated by the designation unit, corresponding to the first scene whether determined in a predetermined cycle, determining whether the first condition is not specified, and if the second condition is specified, corresponding to the first scene by said specifying means The period of performing is shorter than the predetermined period.

According to a control method of an image processing apparatus of the present invention, an image acquisition step for acquiring image data of a plurality of frames, and the image data from among a plurality of preset scenes as described in claim 11 A scene detection step for identifying a scene to be performed, a condition setting step for automatically setting a plurality of image acquisition conditions or a plurality of image processing conditions in accordance with the scene identification result of the scene detection step, and the condition setting step. A setting step of specifying an image acquisition condition or an image processing condition in which the setting is automatically performed, and the condition set in the condition setting step is a first of the plurality of preset scenes The first condition in which a unique setting is made in each of the scene, the second scene, and the third scene, and the unique condition that is different from the other scenes in the first scene A second condition in which the same setting is made in the second scene and the third scene, and in the scene detection step, the first condition is designated in the designation step , It is determined at a predetermined cycle whether or not it corresponds to the first scene , and when the first condition is not specified and the second condition is specified by the specifying step, the first condition is the period to determine whether corresponds to a scene, characterized in that shorter than the predetermined period.

  According to the present invention, the processing that prioritizes the detection speed of scene detection and the processing that prioritizes the number of detected scenes according to the shooting conditions is performed, thereby effectively reducing the computation load due to scene detection and corresponding to scene detection. Correction processing and shooting control.

1 is a block diagram illustrating an overall configuration of a digital camera 114 according to a first embodiment. It is the figure which showed the detection scene cycle in 1st Embodiment. It is the figure which showed the conceptual diagram of the detection cycle of each scene by background art. It is the figure which showed the table | surface of the detection cycle of each scene in 1st Embodiment. It is the figure which showed the table | surface which put together the option of the automatic / fixed mode in each function in 1st Embodiment. It is the figure which showed the correspondence of the function in which the process corresponding to each detection scene and each scene in 1st Embodiment is performed. It is the figure which showed the example of the histogram used for the brightness correction process in 1st Embodiment. It is the figure which showed the contrast correction curve used for the brightness correction process in 1st Embodiment. It is the figure which showed the flow of the blue sky scene detection process in 1st Embodiment. It is the figure which showed the flow of the natural green scene detection process in 1st Embodiment. It is the figure which showed the flow of the evening scene detection process in 1st Embodiment. It is the figure which showed the flow of the person scene detection process in 1st Embodiment. It is the figure which showed the operation | movement flow of the whole scene detection process in 1st Embodiment. It is the figure which showed the blue sky determination area | region on UV color space in 1st Embodiment. It is the figure which showed the natural green determination area | region on UV color space in 1st Embodiment. It is the figure which showed the evening scene determination area | region on UV color space in 1st Embodiment. It is the figure which showed the detection scene cycle when not detecting the person scene in other embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing an overall configuration of a digital camera 114 as an example of an image processing apparatus according to an embodiment of the present invention.
In FIG. 1, 100 is a photographing lens, 101 is an aperture, 102 is a shutter, 103 is an image sensor that converts an optical image into an electrical signal, and 104 is an analog / digital (hereinafter referred to as analog / digital) that converts an analog signal output of the image sensor 103 into a digital signal. A / D) converter. 100 to 104 are image acquisition means. The timing generator 105 supplies clock signals and control signals to the image sensor 103 and the A / D converter 104 to control their operations. The timing generation unit 105 is controlled by the memory control unit 106 and the system control unit 113.

  The image processing unit 108 performs development processing on 12-bit RGB Bayer array image data (generally known RAW data) from the A / D converter 104 or the memory control unit 106. This development processing includes white balance (WB) correction processing, color interpolation processing for converting RGB Bayer array signals into RGB 3 plane signals, gamma correction processing, color conversion matrix processing, brightness correction processing, color correction processing, and the like. It is configured. Details of this development processing will be described later.

The image processing unit 108 (scene detection means) also detects scenes such as a blue sky scene, a natural green scene, a sunset scene, and a person scene by analyzing the image data. The scene detection result is reflected on correction values used for WB correction, brightness correction, and color correction by the system control unit 113 (condition setting unit), and is used for the correction processing in the image processing unit 108. It is done. Details of this scene detection processing will be described later.
Each of the WB correction, color correction, and brightness correction processing functions has two modes: an automatic control mode that performs optimum correction for the detected shooting scene, and a mode that performs fixed correction regardless of the shooting scene. Is provided. The two modes, automatic and fixed, can be switched for each correction function by the user operating the operation unit 115 (designating means). Details of the automatic / fixed mode will be described later.

The memory control unit 106 controls the A / D converter 104, the timing generation unit 105, the image processing unit 108, and the memory 107. As a result, the digital data A / D converted by the A / D converter 104 is written into the memory 107 via the image processing unit 108, the memory control unit 106, or directly via the memory control unit 106.
A memory 107 is a memory for storing captured moving images. The memory 107 is also used as a work area for the system control unit 113 and the image processing unit 108.
The external storage device 109 is a detachable external recording medium such as a CF card or an SD card. The moving image data temporarily recorded in the memory 107 is finally recorded in the external storage device 109.

The exposure control unit 110 controls the aperture 101 and the shutter 102. The exposure control of the exposure controller 110 is provided with two types of modes: an automatic control mode in which the digital camera 114 automatically performs optimum correction and a fixed mode in which the user arbitrarily determines exposure. The two modes, automatic and fixed, can be switched by operating the operation unit 115 by the user. Details of the automatic / fixed mode will be described later.
The distance measurement control unit 111 controls the focusing of the lens 100.
The focus control of the distance measurement control unit 111 is provided with two types of modes: a mode in which the digital camera 114 automatically controls focusing and a fixed mode in which the user sets an arbitrary focus position. The two modes, automatic and fixed, can be switched by operating the operation unit 115 by the user. Details of the automatic / fixed mode will be described later.

The system control unit 113 controls the operation of the entire digital camera 114.
The shooting start / end button 112 is a switch for starting / ending a shooting operation.
The above is the outline of the overall configuration of the digital camera 114 according to the embodiment of the present invention.

Hereinafter, the operation of the digital camera 114 in the present embodiment will be described.
First, the details of the development processing, scene detection processing, and automatic / fixed mode of each function described above will be described in order.

(Detailed explanation of development processing)
First, details of the development processing performed by the image processing unit 108 will be described.
In the development process, a WB correction process, a color interpolation process for converting an RGB Bayer array signal into an RGB 3 plane signal, a gamma correction process, a color conversion matrix process, a brightness correction process, and a color correction process are sequentially performed.
First, in the WB process, correction is performed according to the following equation 1 for each of R, G, and B pixels of a 12-bit RGB Bayer array image.

Here, WBgainR, WBgainG, and WbgainB are WB correction values. A method for calculating the WB correction value will be described later.

Next, the RGB Bayer array signal after WB correction is converted into an RGB3 plane signal by color interpolation processing using a known method.
Then, gamma correction processing is performed on the signal after color interpolation. The gamma correction curve used here is obtained in advance and stored in the memory 107.

  Next, the RGB signal after the gamma correction process is converted into an 8-bit Y (luminance), U, V (color difference) signal by a color conversion matrix process of the following formula 2.

Here, γR, γG, and γB are R, G, and B after gamma correction, and a to i are arbitrary floating point numbers.

Next, brightness correction processing is performed on the Y value of the YUV signal. Details of this brightness correction processing will be described later.
Finally, color correction is performed by color correction processing, and all development processing is completed. This color correction process is performed by a generally known three-dimensional lookup table (3DLUT) process. Details of the color correction process will be described later.

(Detailed explanation of scene detection processing)
Next, details of the scene detection process performed by the image processing unit 108 will be described. In this embodiment, a blue sky scene, a natural green scene, an evening scene, and a person scene are detected.
Note that the scene detection process described below is performed by analyzing an image of an arbitrary frame among a plurality of frames taken with moving images.

<Blue sky scene detection method>
Hereinafter, the operation of the blue sky scene detection process will be described with reference to the flowchart of FIG.
In step S801, one frame of 12-bit RAW data obtained by moving image shooting is divided into blocks of n × m (n and m are integers) by the image processing unit 108, and R, The average value of the G and B color pixels is calculated. In this embodiment, n and m are 8.

  In step S802, the R, G, B average value of each block is subjected to provisional development processing by the above-described white balance correction processing, gamma correction processing, and color conversion matrix processing. Here, the YUV value of each 8 × 8 block is calculated as a result of the temporary development processing.

In step S803, the ratio of the blue sky block to all the blocks in the screen is calculated. Whether or not the block is a blue sky block is determined by defining a blue sky determination region (third region) in the UV color space and counting the number of blocks entering the region.
An example of a blue sky determination area on the UV color space is shown at 1301 in FIG. In this embodiment, the blue sky determination area on the UV color space is the same regardless of the Y value, but the blue sky determination area on the UV color space may be different for each Y value.
It should be noted that the blue sky determination area in the UV color space is calculated in advance so that the UV value of the blue sky portion of the blue sky image taken at various places, seasons, and times after the provisional development processing in step S802 is included. It is recorded in the memory 107.

  Next, a blue sky scene is detected in accordance with the ratio of the number of blocks entering the blue sky determination area in the UV color space (hereinafter referred to as the blue sky block ratio) occupying the total number of blocks (8 × 8 = 64 in this embodiment). . In this embodiment, a blue sky scene is determined when the blue sky block ratio is 50% or more.

<Natural green scene detection method>
Hereinafter, with reference to the flowchart of FIG. 9, the operation of a natural green scene detection process such as a plant will be described.
In step S901, one frame of 12-bit RAW data obtained by moving image shooting is divided into n × m (n and m are integers) areas (blocks) by the image processing unit 108, and R, G in each divided area is divided. , B The average value of the pixels of each color is calculated. As in step S801, n and m are 8 in this embodiment.

  In step S902, the R, G, B average values of each block are subjected to provisional development processing by the above-described white balance correction processing, gamma correction processing, and color conversion matrix processing. Here, the YUV value of each 8 × 8 block is calculated as a result of the temporary development processing.

In step S903, the ratio of natural green blocks to all blocks in the screen is calculated. The determination of whether or not the block is a natural green block is performed by defining a natural green determination region (second region) in the UV color space and counting the number of blocks that have entered the region.
An example of a natural green determination region on the UV color space is shown at 1401 in FIG. In this embodiment, the natural green determination area in the UV color space is the same regardless of the Y value, but the natural green determination area in the UV color space may be different for each Y value.
Note that the natural green determination region in the UV color space is calculated in advance so that the UV value of the green portion of the images of various plants that have undergone the temporary development processing in step S902 is included, and the memory 107 Keep a record.

  Next, according to the ratio of the number of blocks that have entered the natural green determination area in the UV color space (hereinafter, the ratio of natural green blocks) to the total number of blocks (8 × 8 = 64 in the present embodiment), Perform green scene detection. In the present embodiment, a natural green scene is determined when the natural green block ratio is 50% or more.

<Evening scene detection method>
Hereinafter, the operation of the sunset scene detection process in which the sunset sky is reflected will be described with reference to the flowchart of FIG.
In step S1001, one frame of 12-bit RAW data obtained by moving image shooting is divided into n × m (n and m are integers) areas (blocks) by the image processing unit 108, and R, G in each divided area is divided. , B The average value of the pixels of each color is calculated. As in step S801, n and m are 8 in this embodiment.

  In step S1002, the R, G, and B average values of each block are subjected to provisional development processing by the above-described white balance correction processing, gamma correction processing, and color conversion matrix processing. Here, the YUV value of each 8 × 8 block is calculated as a result of the temporary development processing.

In step S1003, the ratio of evening scene blocks in the screen is calculated. The determination of whether or not the block is an evening scene block is performed by defining an evening scene determination area (first area) in the UV color space and counting the number of blocks entering the area.
An example of an evening scene determination area on the UV color space is shown at 1501 in FIG. In the present embodiment, the sunset scene determination area on the UV color space is the same regardless of the Y value, but the sunset scene determination area on the UV color space may be different for each Y value.
Note that the sunset scene determination area in the UV color space is calculated in advance so that the UV values of the orange portions of various sunset scenes (sunset sky) subjected to the temporary development processing in step S1002 are included. Keep a record.

Next, the ratio of the number of blocks entering the evening scene determination area on the UV color space (hereinafter referred to as a natural green block ratio) occupying the total number of blocks (8 × 8 = 64 in the present embodiment) is calculated.
Further, in order to increase the accuracy of sunset detection, it is determined in step S1004 whether or not a color that does not exist in the sunset scene other than orange in the sunset or blue in the sky is included, and a color that does not exist in the sunset scene is included. In this case, it is difficult to determine that the scene is an evening scene.
Specifically, the green and red regions 1502 and 1503 on the UV color space in FIG. 15 are defined as color regions that do not exist in the evening scene. Then, the number of blocks that enter the color region that does not exist in the evening scene with respect to the 8 × 8 block YUV data is counted.

Next, the ratio of the number of blocks in a color area that does not exist in the sunset scene on the UV color space in the total number of 8 × 8 block YUV data blocks (8 × 8 = 64 in this embodiment) (hereinafter, the ratio of blocks other than the sunset scene) Is calculated.
Finally, in step S1005, if the evening scene block ratio is 50% or more and the non-evening scene block ratio is 10% or less, it is determined as an evening scene.

<Person scene detection method>
Hereinafter, with reference to the flowchart of FIG. 11, the operation of the human scene detection process in which the human face is captured will be described.
In step S1101, whether or not there is a face in the image is detected by a generally known pattern matching process for one frame of 12-bit RAW data that has been shot. It is assumed that the face pattern data is recorded in the memory 107 in advance.
In the present embodiment, the face pattern of a person is detected, but it may be configured to detect animals, buildings, vehicles, etc. in addition to the person. In either case, the pattern data of the subject to be detected is recorded in the memory 107, and the pattern matching process is performed.

In step S1102, it is determined whether a person has been detected. If no person is detected in step S1101, it is determined that the scene is not a person scene, and the process ends.
If a person is detected in step S1101, detailed information is acquired in steps S1103 to S1106 described below.
In step S1103, first, the detected face size and face position coordinates in the screen are calculated.

In step S1104, an average RGB value in the area corresponding to the 12-bit RAW face area is calculated. Then, the temporary development is performed and the YUV value in the face area is calculated by the same method as the temporary development processing performed in step S802 for detecting the blue sky.
In step S1105, the brightness of the face area is acquired from the Y value of the YUV value calculated in step S1104.
Finally, in step S1106, the color of the face area is acquired with the UV value of the YUV value calculated in step S1104.
The above is the scene detection processing method performed by the image processing unit 108.
Note that, for example, a blue sky scene and a person scene may be detected in duplicate.

(Detailed explanation of automatic / fixed mode of each function)
Next, details regarding the two modes of auto / fix in each function described above will be described.
FIG. 4 shows a table summarizing the choices of the two modes of automatic / fixed in each function described above. FIG. 5 shows the correspondence between each detected scene and a function that performs processing specialized for each scene when each function is “automatic”.
In FIG. 4, there are “automatic” and “no” as user choices for brightness correction (image processing condition, second condition).
Here, when the brightness correction is set to “automatic”, the image processing unit 108 corrects brightness and contrast suitable for the captured image.
Brightness and contrast correction values are calculated by the following method.

First, a luminance histogram is calculated by the image processing unit 108 with respect to a photographed image. This histogram is calculated by counting the Y value for each value of 0 to 255 for each pixel of the 8-bit YUV image developed by the image processing unit 108. An example of a histogram of an image photographed dark is shown at 601 in FIG. As described above, when the histogram is biased to the dark side, bright correction is performed using the contrast correction curve 701 in FIG. FIG. 7 shows the input Y value on the horizontal axis and the output Y value on the vertical axis.
By applying such a contrast correction curve to the image, the image is corrected brightly. For reference, the histogram shape after brightness correction is shown at 602 in FIG.
In the present embodiment, the method of correcting the brightness by contrast correction is shown. However, the brightness may be corrected by applying a uniform gain to the RGB signals.

The brightness and contrast correction values calculated in this way are subjected to the following processing according to the scene detection result of the image processing unit 108.
For example, when an evening scene (first scene) is detected, the brightness correction amount is reduced or not corrected in order to leave an evening scene atmosphere. When a person is detected, the brightness correction amount is further adjusted so that the brightness corresponding to the face area of the person is appropriate with respect to the correction calculated from the histogram. Specifically, an appropriate target value of the face Y value is determined in advance, and the brightness correction amount is adjusted so that the Y value corresponding to the face area becomes the target value.
When a blue sky or a natural green scene (second scene, third scene) is detected, the processing corresponding to the scene is not performed, and the brightness and contrast correction amount calculated using the histogram are used as they are. use.

These relationships are shown in the brightness correction column of FIG. In FIG. 5, the item “◯”, that is, “Evening scene” and “Human scene” uses a histogram in which the brightness correction value is calculated for all scenes when the brightness correction is “automatic”. In addition to the conventional method, the scene-specific processing is performed.
On the other hand, when the brightness correction “not performed” in FIG. 5 is set, the brightness and contrast of the image are not corrected.

Next, in FIG. 4, there are “automatic” and “manual” as user options for WB (image processing condition, second condition).
When WB “automatic” is set here, for example, the correction value is calculated by Equation 3 from the average RGB data of the entire screen of the captured image.

In addition, the following processing is performed on the WB correction value calculated in this way according to the scene detection result of the image processing unit 108.
For example, when an evening scene (first scene) is detected, correction of Expression 4 is further performed on the result of Expression 3 so that the redness of the sunset is more emphasized.

Here, a floating point with Kr = 1.0 or more and a floating point with Kb = 1.0 or less.
According to Expression 4, the R pixel is larger and the B pixel is smaller, so that an image having a stronger redness than that obtained by applying the WB correction value calculated by Expression 3 can be obtained.

When a person is detected, a color corresponding to the face area of the person is detected, and the WB correction value calculated by Equation 1 is further corrected so that the face color of the person becomes an appropriate skin color.
When a blue sky and a natural green scene (second scene, third scene) are detected, the processing corresponding to the scene is not performed, and the WB correction amount calculated by Expression 1 is used as it is.
These relationships are shown in the WB column of FIG. FIG. 5 shows that the item “◯”, that is, “Evening scene” and “Human scene” are performed by a method specific to the scene other than the method using Formula 3 performed in common for all scenes.

On the other hand, when the WB “manual” in FIG. 5 is set, a fixed correction value that allows an appropriate white balance to be obtained under a light source set by the user (for example, a color temperature of about 5200 K for “sunlight”). Applies. The fixed WB correction value set by the user is calculated in advance and recorded in the memory 107.
Next, in FIG. 4, there are “automatic” and “standard” as user options for color correction (image processing condition, first condition).
Here, when color correction “standard” is set, fixed color reproduction is performed regardless of the scene. Specifically, a fixed 3DLUT is applied to the 3DLUT processing applied by the color correction processing unit of the image processing unit 108. This fixed 3DLUT may be, for example, an LUT having the same input / output (that is, color conversion processing is not performed).
When the color correction is set to “automatic”, the color is corrected to a color suitable for the scene detected by the image processing unit 108 using a 3DLUT.

For example, when a blue sky scene (second scene) is detected, a 3DLUT is applied so that the blue saturation is higher than when “standard” is set, and the blue sky is more vividly emphasized.
When a natural green scene (second scene) is detected, a 3DLUT is applied so that the green saturation is higher than when “standard” is set, and natural green is emphasized more vividly.
When an evening scene (first scene) is detected, a 3DLUT is applied so that the saturation of the orange is higher than when “standard” is set, and the orange of the sunset is more vividly emphasized.
When a person scene is detected, the skin color is corrected to a translucent skin color by applying a 3DLUT that makes the hue of the skin color closer to red, higher brightness, and lower saturation than when “standard” is set.
These relationships are shown in the column of color correction in FIG. When the color correction is set to “automatic” as indicated by “◯” in all items, unique processing different from the standard is performed in each scene. A 3DLUT suitable for each scene is created in advance and recorded in the memory 107. Of course, it may be generated each time a scene is detected, but the calculation load increases accordingly.

Next, in FIG. 4, there are “automatic” and “manual” as user options for exposure control (image acquisition conditions).
Here, when the exposure control “manual” is set, shooting is performed at the speed of the aperture 101 and the shutter 102 arbitrarily set by the user.
When the exposure control “automatic” is set, the system control unit 113 sets an exposure control amount so that the brightness at the center of the image is appropriate according to the brightness of the image processed by the image processing unit 108. decide. The exposure control unit 110 controls the speed of the diaphragm 101 and the shutter 102 according to the exposure control amount, and controls the exposure amount of light incident on the image sensor 103.
When detection information such as a person's face is obtained by the scene detection process, the exposure control amount is calculated so that the brightness of the detected face area is appropriate, not the brightness of the center of the image. .
When a blue sky, natural green, or sunset scene is detected, the processing corresponding to the scene is not performed, and an exposure is set so that the brightness at the center of the image is appropriate as described above.
These relationships are shown in the exposure column of FIG. In the item of “◯” in FIG. 5, that is, “person scene”, when the exposure control “automatic” is set, the above-described exposure control that optimizes the brightness of the central portion of the image is not a scene specific. It shows that it becomes exposure control.

Next, in FIG. 4, there are “automatic” and “manual” as user options for focus control (image acquisition conditions).
Here, when the focus control “manual” is set, shooting is performed at a focus position arbitrarily set by the user.
When the focus control “automatic” is set, for example, the focus is controlled by contrast autofocus (hereinafter referred to as contrast AF) processing so that the subject at the center of the screen is automatically focused.
In this contrast AF processing, focus control is performed by the distance measurement control unit 111 so that the contrast of the target portion of the image becomes the highest. Specifically, the distance measurement control unit 111 analyzes the contrast of the image processed by the image processing unit 108 while gradually changing the focus in accordance with an instruction from the system control unit 113.
When detection information such as a person's face is obtained by scene detection processing, contrast AF processing is performed so that the face position is in focus.
When a blue sky, natural green, or sunset scene is detected, processing corresponding to the scene is not performed, and control is performed so that the subject at the center of the image is in focus as described above.
These relationships are shown in the column “Focus” in FIG. In FIG. 5, in the item “◯”, that is, “person scene”, when the focus control “automatic” is set, the focus control is specialized for the scene, not the control for focusing on the subject at the center of the image described above. It is shown that.
The above is a detailed description of the automatic / fixed mode of each function.

Here, the entire scene detection process at the time of moving image shooting, which is a feature of the present embodiment, will be described.
As described above, for example, in a moving image with a frame rate of 30 frames / second, as shown in FIG. 2, detection of three scenes of “blue sky scene”, “natural green scene”, and “evening scene” once every two frames. When all processing is performed, the calculation load becomes very heavy. As a result, not all detection processes can be completed within the 12-frame period, and in the worst case, movie shooting is forcibly terminated. Even if the detection process is completed in 12 frames, it is possible that a large calculation load is imposed on the scene detection and other problems such as the inability to perform other processes in parallel may occur.
Therefore, in this embodiment, priority is given to the number of scene detections for scene detection depending on whether the functions of brightness correction, WB, color correction, exposure control, and focus control are set in either the automatic mode or the fixed mode. Select whether to give priority to scene detection speed. In other words, when the function for generating the correction process or control specific to the detected scene is automatically set, priority is given to the number of scene detections, and otherwise, priority is given to the scene detection speed.
With reference to the flowchart of FIG. 12, an operation flow of the entire scene detection process of the digital camera 114 at the time of moving image shooting in the present embodiment will be described.
In the setting of automatic / fixed processing of each function, it is assumed that the automatically set functions are processed in parallel as tasks different from the operation of the flowchart of FIG. 12 at a predetermined time period (cycle). Specifically, when the brightness correction, WB, color correction, exposure, and focus functions are set in the automatic mode, the functions are processed in parallel in different tasks. For example, when a moving image is shot at a frame rate of 30 frames / second, in this embodiment, the correction value is calculated in the following cycle.

Brightness correction ・ ・ ・ Once in 12 frames White balance ・ ・ ・ Once in 12 frames Color correction ・ ・ ・ Once in 12 frames Exposure ・ ・ ・ Once in 4 frames Focus ・ ・ ・ Once in 2 frames The correction value calculation of these functions is performed in the above cycle, but the correction amount applied to each frame at the time of shooting is controlled so as to change step by step in one frame.
For example, taking brightness correction as an example, the target value of the amount to be corrected is calculated once every 12 frames. However, since the correction amount for 12 frames until the next calculation is calculated in one calculation, The amount changes step by step in 1 frame increments. The same applies to other functions. However, the embodiment is not limited to this, and the correction amount per frame can be appropriately adjusted.

Based on the above assumptions, the operation flow of the entire scene detection of FIG. 12 will be described.
First, when the shooting start button 112 is pressed to start shooting, the scene detection process of the image processing unit 108 is started.
Steps S1201 to S1205 are steps in which it is first determined whether each function is set to the automatic / fixed mode, and the scene detection process is changed according to the result. If it is determined in step S1201 that the color correction setting is automatic, the flow proceeds to a flow A for performing scene detection for all of the person detection, blue sky detection, natural green detection, and sunset scene detection, and is not automatic, that is, fixed. If it is determined, the process proceeds to step S1202. If it is determined in step S1202 that the brightness correction setting is automatic, the process proceeds to a flow B in which scene detection is performed focusing on person detection and sunset scene detection. If it is determined that the brightness correction setting is not automatic, the process proceeds to step S1203. If it is determined in step S1203 that the WB correction setting is automatic, the process proceeds to flow B. If it is determined not to be automatic, the process proceeds to step S1204. If it is determined in step S1204 that the exposure control setting is automatic, the flow proceeds to a flow C in which scene detection is focused on person detection. If it is determined that the exposure control is not automatic, the flow proceeds to step S1205. In step S1205, if it is determined that the focus control setting is automatic, the process proceeds to flow C. If it is determined that the focus control setting is not automatic, that is, if all five functions are in the fixed mode, no scene detection is necessary. Then, the scene detection process is terminated.

  A flow of flow A for performing scene detection for human detection, blue sky detection, natural green detection, and sunset scene detection will be described. In step S1206, 1 is set to the scene flag S. In step S1207, it is determined whether the number of frames is a multiple of 12. If the number of frames is a multiple of 12, the process proceeds to the next step. This process means that one scene is detected in a cycle once every 12 frames.

  In step S1208, the scene flag S is determined. If S = 1, the process proceeds to step S1209 to execute a person detection process. Since this person detection processing method has already been described, description thereof is omitted here.

  In step S1213, it is determined whether the scene flag S is six. Since this time is not 6, the process proceeds to step S1214, and 1 is added to the scene flag S. Next, in step S1215, it is determined whether or not shooting is finished. If shooting has not been completed, the process returns to step S1207, and waits until the number of frames becomes a multiple of twelve. In step S1208, the scene flag S is determined. Since the scene flag S is 2 this time, the process advances to step S1210 to detect a blue sky scene. Since this blue sky scene detection processing method has already been described, description thereof is omitted here.

Next, in step S1213, it is determined whether or not the scene flag S is 6. When the scene flag S becomes 6, the process returns to step S1206 and the scene flag is set to 1. As described above, the processes in steps S1206 to S1215 are repeated until it is determined in step S1215 that the photographing has been completed.
Since only the scene flag determination process in step S1208 is repeated, the process is different, so step S1208 will be described in detail.
In step S1208, a value obtained by adding 1 to the scene flag S every 12 frames is determined. The detection scene for each scene flag is shown below.

Scene flag S = 1 ... person detection (step S1209)
Scene flag S = 2... Blue sky detection (step S1210)
Scene flag S = 3 ... person detection (step S1209)
Scene flag S = 4... Natural green detection (step S1211)
Scene flag S = 5 ... person detection (step S1209)
Scene flag S = 6 ... evening scene detection (step S1212)
Note that the details of each scene detection method in steps S1209 to S1212 have already been described, and a description thereof will be omitted here.
According to this flow, the human scene is detected once every 24 frames, and the other scenes are detected once every 72 frames.

  Next, a flow of flow B in which scene detection is focused on human detection and sunset scene detection will be described. In step S1216, 1 is set to the scene flag S. Next, in step S1217, it is determined whether or not the number of frames is a multiple of 12. If the number of frames is a multiple of 12, the process proceeds to the next step.

  In step S1218, the scene flag S is determined. If S = 1, the process proceeds to step S1209 to execute a person detection process. Since this person detection processing method has already been described, description thereof is omitted here.

In step S1219, it is determined whether the scene flag is 2. Since it is 1 at this time, the process proceeds to step S1220. In step S1220, 1 is added to the scene flag S. That is, the scene flag S is 2. Next, in step S1221, it is determined whether the shooting end button 112 has been pressed and shooting has ended. If shooting has not been completed, the process returns to step S1217 and waits until the number of frames becomes a multiple of twelve. In step S1218, the scene flag S is determined again. Since the scene flag is 2 this time, the evening scene detection process in step S1212 is performed. Since this sunset scene detection processing method has already been described, description thereof is omitted here. Next, in step S1219, it is determined whether or not the scene flag is 2. Since this time it is the scene flag 2, the process returns to step S1216. In step S1216, the scene flag S becomes 1 again. The processing from step S1216 to step S1221 is repeatedly performed until it is determined in step S1221 that photographing has been completed.
That is, according to the processing from step S1216 to step S1221, when the previous color correction setting is fixed, the person detection process and the sunset scene detection process are each performed once every 24 frames.

Next, the flow of flow C for performing scene detection focusing on person detection will be described. In the flow C, it is determined whether or not the number of frames is a multiple of 12 in step S1222. If it is a multiple of 12, the process proceeds to the next step S1209. In step S1209, a person scene is detected, and then in step S1223, it is determined whether the shooting end button 112 has been pressed and shooting has ended. If shooting has not been completed, the process returns to step S1222 and waits until the number of frames becomes a multiple of twelve. If it is determined in step S1223 that shooting has ended, the scene detection process ends.
As described above, the functions of brightness correction, white balance, color correction, exposure, and focus are controlled as separate tasks in parallel with the flow of FIG. 12 described here. The latest scene detection result is reflected in each function as needed.
FIG. 3 shows a table summarizing detection scene cycles according to the flow shown in FIG. 12 in the first embodiment.
The detection scene cycle table in FIG. 3A corresponds to the processing of the flow A.
Further, the detection scene cycle table in FIG. 3B corresponds to the processing of the flow B.
Further, the detection scene cycle table in FIG. 3C corresponds to the process of the flow C.
As can be seen from FIG. 3, in the case of Flow A, priority is given to the number of detected scenes over the detection speed, and four scenes of people, blue sky, natural green, and sunset are detected in a cycle of 72 frames each. Yes. However, as shown in FIG. 5, a person scene is used for exposure and focus control that require a particularly high response speed, so that it is detected in one cycle in 24 frames shorter than other scenes. .
In the case of the flow B, priority is given to the detection speed, and only the person scene and the sunset scene are detected in a cycle of once every 24 frames. As shown in FIG. 5, since the blue sky and natural green scene detection results are used only in the color correction process, such a process is possible. It can be seen that the detection cycle of the sunset scene in flow B is shorter than that in flow A, once in 24 frames (the detection speed is faster).
In the case of the flow C, like the flow B, priority is given to the detection speed, and only the person scene is detected in a cycle of 12 frames. As shown in FIG. 5, since the scene detection result other than the human scene is not used for the exposure control and the focus control, such a process is possible. It can be seen that the detection cycle of the human scene in flow C is shorter than that in flows A and B (detection speed is increased) once every 12 frames.

As described above, in the present embodiment, by effectively performing the process that prioritizes the detection speed of the scene detection and the process that prioritizes the number of detected scenes according to the shooting conditions, while reducing the calculation load due to the scene detection, Correction processing and shooting control corresponding to more accurate scene detection are enabled.
In this embodiment, four scenes of a person, a blue sky, a natural green, and an evening scene are used as detection candidates. However, the present invention is not limited to this, and the number of scenes that are detection candidates may be more or less than that of this embodiment. . For example, a detection scene cycle table for each scene when a person scene is not detected as compared with the present embodiment is shown. As shown in FIG. 16, when the color correction process is automatically set, priority is given to the number of detected scenes, and each scene is detected in a cycle of 36 frames. On the other hand, when the color correction processing is fixed, priority is given to the detection speed, and only the sunset scene is detected in a cycle of once every 12 frames.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.
In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.
Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

DESCRIPTION OF SYMBOLS 100 Lens 101 Aperture 102 Shutter 103 Image pick-up element 104 A / D converter 105 Timing generation part 106 Memory control part 107 Memory 108 Image processing part 109 External storage device 110 Exposure control part 111 Distance control part 112 Shooting start button 113 System control part 114 Image processing apparatus 114 Operation unit

Claims (13)

Image acquisition means for acquiring image data of a plurality of frames;
A scene detection means for specifying a scene corresponding to the image data from a plurality of preset scenes;
Condition setting means for automatically setting a plurality of image acquisition conditions or a plurality of image processing conditions according to the result of scene specification by the scene detection means;
Specifying means for specifying image acquisition conditions or image processing conditions for automatic setting by the condition setting means,
The condition set by the condition setting means includes a first condition in which a unique setting is made in each of the first scene, the second scene, and the third scene among the plurality of preset scenes,
A unique setting different from the other scenes in the first scene, and a second condition in which the same setting is made in the second scene and the third scene,
The scene detection means determines, in a predetermined cycle, whether or not the first condition is designated when the first condition is designated by the designation means;
The first condition is not specified by said specifying means, and if the second condition is designated, shortens the cycle to determine whether or not corresponding to the first scene than the predetermined period An image processing apparatus. The scene detection means includes
When the first condition is designated by the designation means, it is determined whether the second scene and the third scene are satisfied,
When the first condition is not specified by the specifying means and the second condition is specified, it is not determined whether or not the second scene and the third scene are satisfied. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the plurality of image acquisition conditions include at least one of an exposure control condition and a focus control condition.   The plurality of image processing conditions include at least one of a brightness correction condition using a gain or contrast correction curve, a white balance process condition, and a color correction condition. The image processing apparatus described in one. The first condition is a condition for color correction,
5. The image processing apparatus according to claim 1, wherein the first scene is an evening scene and the second scene is a natural green scene or a blue sky scene. 6.   6. The condition for the color correction is to set the three-dimensional lookup table used in a color correction process in which color correction is performed on the image data using a three-dimensional lookup table. An image processing apparatus according to 1. The scene detection means divides the image data into blocks to determine whether it corresponds to the sunset scene , and occupies the ratio of the blocks in the first area in the color space to all the blocks of the image data. The image processing apparatus according to claim 5, wherein the determination is based on the determination. The scene detection means divides the image data into blocks to determine whether or not the natural green scene corresponds to all blocks of the image data of blocks in the second area on the color space. The image processing apparatus according to claim 5, wherein the determination is made based on an occupation ratio. The scene detection means determines whether or not the image corresponds to the blue sky scene by dividing the image data into blocks and occupying a ratio of blocks in the third area in the color space to all blocks of the image data. The image processing apparatus according to claim 5, wherein the determination is based on the determination.   The image processing apparatus according to claim 5, wherein the second condition is setting of brightness correction using a gain or contrast correction curve, or white balance processing. An image acquisition step of acquiring image data of a plurality of frames;
A scene detection step for identifying a scene corresponding to the image data from a plurality of preset scenes;
A condition setting step of automatically setting a plurality of image acquisition conditions or a plurality of image processing conditions in accordance with the scene identification result of the scene detection step;
A designation step for designating an image acquisition condition or an image processing condition in which the setting is automatically performed in the condition setting step,
The conditions set in the condition setting step include a first condition in which a unique setting is made in each of the first scene, the second scene, and the third scene among the plurality of preset scenes. ,
A unique setting different from the other scenes in the first scene, and a second condition in which the same setting is made in the second scene and the third scene,
In the scene detection step, when the first condition is designated in the designation step, it is determined at a predetermined cycle whether or not it corresponds to the first scene,
Wherein the designating step the first condition is not specified, and if the second condition is designated, shortens the cycle to determine whether or not corresponding to the first scene than the predetermined period And a control method for the image processing apparatus. Image acquisition means for acquiring video data composed of a plurality of frames;
Scene determination means for determining whether the moving image data corresponds to one scene selected in a predetermined order from a plurality of scenes set in advance at a predetermined frame period;
Image processing means for performing image processing on the moving image data according to the type of the corresponding scene;
For each of the plurality of scenes, comprising: designation means for designating a function of image processing that is automatically executed by the image processing means,
The image processing apparatus, wherein the scene determination unit varies the number of scene determination candidates according to the content specified by the specification unit. An acquisition step of acquiring video data composed of a plurality of frames;
A scene determination step for determining whether the moving image data corresponds to one scene selected in a predetermined order from a plurality of scenes set in advance at a predetermined frame period;
An image processing step for performing image processing on the moving image data in accordance with the type of the corresponding scene;
A designation step for designating a function of image processing that is automatically executed in the image processing step for each of the plurality of scenes,
In the scene determination step, the number of scene determination candidate scenes varies according to the content specified in the specification step.