JP5451918B2 - Imaging apparatus, control method thereof, and program - Google Patents

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a program.

  Conventionally, in an imaging apparatus such as a digital still camera, a scene discrimination evaluation value indicating a subject situation (scene) is calculated by judging various circumstances such as brightness, color, distance, and movement of sequentially photographed subjects. There is. Such an imaging apparatus performs imaging according to the scene specified by the calculated scene discrimination evaluation value and the face detection result. For example, in Patent Document 1, it is detected that a scene is a backlight scene based on a scene discrimination evaluation value indicating a difference in brightness between an area where a main subject is estimated to exist and an area other than the area where the main subject is present. However, a technique for performing exposure control that eliminates backlight is disclosed.

JP 2008-287091 A

  However, when an image sensor using a CCD or CMOS image sensor is used to determine various circumstances such as the brightness, color, distance, and movement of a subject, the narrow dynamic range becomes a problem. An image sensor used for taking a still image or a moving image does not have a wide dynamic range in the first place, and is not suitable for photometry of a scene having a strong contrast between light and dark. If the amount of exposure is inappropriate, overexposure in bright areas and blackout in dark areas will be noticeable. Even if you try to determine various circumstances such as the brightness, color, distance, and movement of subjects that have been sequentially captured, Cannot be detected. If scene discrimination is performed under such circumstances and imaging is performed according to the discrimination result, there is a risk that an inappropriate image will be obtained.

  The present invention has been made for the purpose of solving such problems of the prior art. An object of the present invention is to provide an imaging apparatus capable of appropriate scene discrimination even when the dynamic range of the imaging element is narrow, a control method thereof, and a program.

  The object is to periodically calculate a scene discrimination evaluation value used for scene discrimination of a subject from images taken in time series by the imaging means, and to calculate the scene discrimination evaluation value calculated by the calculation unit. And a scene discriminating unit that discriminates the scene of the subject and a control unit that controls the scene discriminating unit, and the control unit is imaged in time series by the scene discriminating unit. Among the images, the scene of the subject is discriminated using the scene discrimination evaluation value calculated from the image obtained when the luminance of the subject does not converge within a predetermined range with respect to the target luminance. First, the scene of the subject using the scene discrimination evaluation value calculated from the image obtained when the luminance of the subject converges within the predetermined range with respect to the target luminance. It is achieved by the image processing apparatus according to the present invention characterized by causing another.

  Further, the object is to periodically calculate a scene discrimination evaluation value used for scene discrimination of a subject from images captured in time series by the imaging unit, and the scene discrimination evaluation calculated by the calculation unit. A method of controlling an image processing apparatus having a scene determination means for determining a scene of the subject using a value, wherein the control means of the image processing apparatus images the scene determination means in the time series. Among the obtained images, the scene determination of the subject using the scene determination evaluation value calculated from the image obtained when the luminance of the subject does not converge within a predetermined range with respect to the target luminance is The subject using the scene discrimination evaluation value calculated from the image obtained when the luminance of the subject converges within the predetermined range with respect to the target luminance without performing Also achieved by a method of controlling an image processing apparatus according to the present invention characterized by causing the determination of a scene.

  According to the present invention, it is possible to appropriately determine a scene even if the dynamic range of the image sensor is narrow.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to an embodiment. (A) is a figure which shows the front external appearance of the imaging device which concerns on this embodiment, (b) is a figure which shows the back external appearance of the imaging device which concerns on this embodiment. It is a figure which illustrates the live image of a display part. It is a figure which illustrates the scene icon which abstracted each scene. It is a figure which shows an example of the imaging by an imaging device. 6 is a timing chart showing an example of timings of scene discrimination evaluation value generation, face detection, and scene discrimination in the imaging apparatus. 6 is a timing chart showing an example of timings of scene discrimination evaluation value generation, face detection, and scene discrimination in the imaging apparatus. 6 is a timing chart showing an example of timings of scene discrimination evaluation value generation, face detection, and scene discrimination in the imaging apparatus. 6 is a timing chart showing an example of timings of scene discrimination evaluation value generation, face detection, and scene discrimination in the imaging apparatus. 6 is a timing chart showing an example of timings of scene discrimination evaluation value generation, face detection, and scene discrimination in the imaging apparatus. It is a flowchart which shows the whole flow of the imaging device which concerns on this embodiment. It is a flowchart which shows a scene discrimination | determination process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. Further, the embodiment of the present invention shows the most preferable mode of the invention, and does not limit the scope of the invention.

  FIG. 1 is a block diagram illustrating an overall configuration of an imaging apparatus 100 according to the present embodiment. As shown in FIG. 1, in the imaging apparatus 100, external light is condensed on the imaging element 16 by the lens 10. In FIG. 1, the lens is expressed as one lens, but it is also possible to mount a lens unit composed of a plurality of lenses.

  In addition, the lens 10 can adjust the focal point and the angle of view by moving the lens position back and forth along the optical axis by the lens driving circuit 42. Further, based on the blur amount of the imaging apparatus 100 measured by the blur amount detection circuit 44, the lens is driven by the camera shake correction circuit 40, and the optical axis is changed in a direction in which the camera shake is canceled. It is also possible to perform camera shake correction. Note that the blur amount detection circuit 44 includes a gyro sensor that measures the blur amount of the imaging apparatus 100. In FIG. 1, camera shake correction is realized by driving the lens, but camera shake correction may be realized by driving the image sensor 16 in the same manner.

  The amount of light passing through the lens 10 is adjusted by the diaphragm 14. In the imaging apparatus 100, the control unit 60 can control the diaphragm 14 by transmitting the diaphragm control information to the diaphragm driving circuit 26.

  The diaphragm 14 includes an iris diaphragm composed of a plurality of wings and a round diaphragm in which holes of various diameters are punched in advance on a plate. The control unit 60 uses the diaphragm 14 and the diaphragm drive circuit 26 to control the diaphragm so as to reduce the amount of light when the subject brightness is high, and to capture a large amount of light by opening the diaphragm when the subject brightness is low. Control.

  In the imaging apparatus 100, the control unit 60 can control the mechanical shutter 12 by transmitting mechanical shutter control information to the mechanical shutter drive circuit 28. The exposure time at the time of still image capturing is determined by the opening / closing time of the mechanical shutter 12, and this exposure time is determined by the control unit 60 judging the time and giving an instruction to the mechanical shutter drive circuit 28.

  Light that has passed through the lens 10, the mechanical shutter 12, and the aperture 14 is received by the image sensor 16. The imaging device 100 can control the imaging device 16 by the control unit 60 transmitting an imaging device control signal to the TG 24 (Timing Generator).

  The TG 24 drives the image sensor 16 based on the image sensor control signal received from the control unit 60. The image sensor 16 is a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like, and photoelectrically converts a subject image received during exposure. Based on the drive signal from the TG 24, the image sensor 16 reads out image data that has undergone photoelectric conversion during exposure and exposure. Note that this exposure and readout of photoelectrically converted imaging data may be periodically performed based on a drive signal from the TG 24.

  Moreover, only the specific line or the specific area may be read out from the imaging data photoelectrically converted by the imaging element 16. This can be realized by changing the readout method according to the readout drive signal output from the TG 24. The control unit 60 determines an optimum reading method according to the situation, and instructs the TG 24. For example, since a high-resolution image data is required when capturing a still image, a method for reading out all data of the image sensor 16 is determined. In addition, since a high frame rate such as 30 fps or 60 fps is required at the time of electronic viewfinder (Electrical ViewFinder) or moving image shooting, a method of thinning and reading out a specific line of the image sensor 16 is determined.

  Further, the TG 24 can control the exposure time of the image sensor 16. This can be realized by outputting a drive signal to the image sensor 16 so that the image sensor 16 releases charges charged by photoelectric conversion at an arbitrary timing.

  The imaging data read from the imaging element 16 passes through the CDS circuit 18 (Correlated Double Sampler). The main role of the CDS circuit 18 is to remove noise components of the imaging data by the correlated double sampling method. Thereafter, the imaging data is attenuated / amplified by a PGA circuit 20 (Programmable Gain Amplifier). The controller 60 controls the attenuation / increase / decrease amount in the PGA circuit 20 by transmitting the amplification level to the PGA circuit 20.

  Usually, appropriate exposure of the image sensor 16 is realized by appropriately setting the exposure amount to the image sensor 16 with the aperture 14 and appropriately setting the exposure time in the mechanical shutter 12. The PGA circuit 20 can provide a function of changing exposure in a pseudo manner by performing attenuation / amplification of imaging data. That is, the PGA circuit 20 provides a function of switching sensitivity, which is one of the exposure conditions at the time of imaging along with the aperture and shutter speed.

  The imaging data attenuated / amplified by the PGA circuit 20 is converted from an analog signal to a digital signal by an A / D circuit 22 (Analog / Digital Converter). In FIG. 1, the CDS circuit, the PGA circuit, and the A / D circuit are represented as separate blocks, but a single IC (Integrated Circuit) package having these functions may be used.

  The imaging data converted into a digital signal by the A / D circuit 22 is input to the image processing circuit 50. The image processing circuit 50 includes a plurality of blocks for performing image processing, and performs various image processing. The image sensor 16 extracts a specific color component for each pixel through a color filter (not shown). Therefore, the image data output from the A / D circuit 22 has a data format corresponding to the arrangement of the pixels and the color filters of the image sensor 16. Therefore, the data format is not suitable for use in automatic exposure control (AE) in which exposure control is performed by evaluating only the luminance component from the A / D circuit 22. For this reason, the image processing circuit 50 has a function of excluding color information from image data and extracting only luminance information. Conversely, it also has a function of extracting color information, and can be used for white balance (AWB) processing that specifies the light source of the subject and appropriately adjusts the color.

  Further, the image processing circuit 50 has a function of extracting only the frequency component of the image data from the image sensor 16 and can be used for automatic focus control (AF: AutoFocus). A function of determining which region of the image data read from the image sensor 16 is to be extracted and how to divide the region is provided.

  Further, the image processing circuit 50 has a function of operating the increase / decrease of the level of the imaging data converted into a digital signal by the A / D circuit 22 and the color effect of the image, and also plays a role of adjusting the image quality of the captured image. Yes. Regarding the level of image data, gamma conversion that converts the signal level according to the magnitude of the original signal level, and the amplification factor according to the frequency component for each area in the screen Various adjustments, such as increasing or decreasing the level, are possible.

  The imaging data converted into a digital signal by the A / D circuit 22 may be input to the image processing circuit 50 and stored in the temporary storage memory 30 at the same time. The imaging data once stored in the temporary storage memory 30 can be read again, the imaging data can be referred to from the image processing circuit 50, or the read imaging data can be input to the image processing circuit 50. Further, the image data processed by the image processing circuit 50 can be written back to the temporary storage memory 30, or arbitrary data from the image processing circuit 50 can be written to the temporary storage memory 30.

  The imaging data processed by the image processing circuit 50 is input to the image recognition circuit 38. The image recognition circuit 38 is capable of recognizing a person's face, its facial expression, and character information when there is a character, in addition to recognizing the brightness state, focus state, and color state of the input image. It has become. It is possible to input a plurality of images to the image recognition circuit 38. For example, two images are input and the characteristics of the two images are compared to determine whether or not they are the same image. It is possible.

  In addition to the method of recognizing an image by the image recognition circuit 38, the control unit 60 can also perform image recognition processing. The control unit 60 can execute a program coded in advance on a CPU (Central Processing Unit) (not shown). By sequentially executing this program, the control unit 60 can read the imaging data stored in the temporary storage memory 30, analyze the read imaging data, and recognize the situation at the time of imaging.

  When image data is displayed and output on the display unit 108 such as an LCD (Liquid Crystal Display), the image data subjected to image processing by the image processing circuit 50 is developed in a VRAM 34 (Video Random Access Memory). The imaging data developed in the VRAM 34 is converted into analog data by the D / A circuit 36 and displayed on the display unit 108. The electronic viewfinder display is realized by sequentially displaying image data read out from the image sensor 16 on the display unit 108. Note that on the VRAM 34, the captured data is expanded so as to correspond to various display forms such as displaying one captured image on the display unit 108 so as to be the largest, or displaying a plurality of captured images on a multi-screen. It's okay.

  The display unit 108 may display not only an image but also arbitrary information alone or together with the image. Specifically, the state of the imaging device 100, the shutter speed selected by the user or determined by the imaging device 100, character information such as aperture value and sensitivity information, a graph such as the luminance distribution measured by the image processing circuit 50, face recognition Results include scene recognition results. In the display unit 108, the display position and display color of information can be arbitrarily selected according to the user's operation instruction received by the operation unit 70. In the imaging apparatus 100, the user interface is realized by displaying these various types of information on the display unit 108 in accordance with a user operation instruction received by the operation unit 70.

  The operation unit 70 includes a power switch 102, a shutter switch 104, a mode switch 110, parameter selection switches 151, 153, 157, 159, and the like. The power switch 102 receives a power ON / OFF instruction from the user. The shutter switch 104 receives an imaging preparation instruction and an imaging start instruction for preparing imaging from the user. The mode switch 110 receives an instruction to switch the imaging mode from the user. Parameter selection switches 151, 153, 157, and 159 accept parameter selection instructions from the user. Details of each of the switches described above will be described later.

  The display unit 108 can also read and display image data stored in the storage medium 82 connected to the imaging device 100 via the storage medium I / F 80. If the image data is compressed, the compression / decompression unit 32 decompresses the image data and expands the decompressed image data in the VRAM 34.

  The storage medium 82 is a non-volatile memory that can be connected to the imaging apparatus 100 via the storage medium I / F 80 and can read or store data under the control of the control unit 60. The storage medium 82 can store mainly captured image data. In addition, imaging information such as an aperture value, shutter speed, ISO sensitivity, and imaging time can be added to each image data. The added data may be stored in the storage medium 82 together with the image data.

  The external device I / F 84 is a communication interface for connecting to an external device 86 such as a PC (Personal Computer). The imaging apparatus 100 and the external device 86 are connected to each other via an external device I / F 84 so that they can communicate with each other, and can transmit and receive data under the control of the control unit 60. For example, transfer of image data captured in accordance with an instruction from the external device 86 is performed via the external device I / F 84.

  FIG. 2 shows an external view of the imaging apparatus 100. FIG. 2A is a diagram illustrating an appearance of the front surface of the imaging apparatus 100. FIG. 2B is a diagram illustrating a rear appearance of the imaging apparatus 100.

  As shown in FIG. 2A, a lens 10 is disposed on the front surface of the imaging apparatus 100, and the imaging apparatus 100 can capture a subject image. A strobe unit 90 whose light emission is controlled by the control unit 60 is disposed on the same surface (front surface) on which the lens 10 is provided. The imaging apparatus 100 can obtain a sufficient amount of light for imaging by causing the flash unit 90 to emit light when the main subject is dark, and can obtain a suitable captured image while maintaining a fast shutter speed even in the dark.

As illustrated in FIG. 2B, the display unit 108 is disposed on the back surface of the imaging device 100. The display unit 108 performs an electronic viewfinder display that continuously displays captured images sequentially captured by the image sensor 16 and is used as a finder for a user to capture a subject with reference to the continuous images (live images). May be. At this time, the photometry area information and the distance measurement area information in AE (automatic exposure adjustment) and AF (automatic focus adjustment) can be displayed superimposed on the live image. Furthermore, on the display unit 108, as subject recognition information, a frame is superimposed on the person's face according to the result of recognizing the person's face, and background scene conditions such as blue sky, sunset, and backlight are displayed. The recognized result may be displayed as an icon.
Further, a conventional optical viewfinder 106 may be provided on the back surface of the imaging apparatus 100.

  Further, a mode change switch 110 is provided on the back surface of the imaging apparatus 100. The mode switch 110 receives from the user an instruction to switch the operation mode of the imaging apparatus 100 such as a still image capturing mode, a moving image capturing mode, and a playback mode.

  In addition, parameter selection switches 151, 153, 155, 157, and 159 such as a determination switch and a cursor switch are provided on the back surface of the imaging apparatus 100. The parameter selection switches 151, 153, 155, 157, and 159 are used to select parameters such as selection of imaging conditions such as a distance measurement area and photometry mode, page feed during reproduction of captured images, and general operation settings. Can be selected and set. Further, ON / OFF of the electronic viewfinder display described above may be selected. The display unit 108 may display an image and accept a setting similar to that of the parameter selection switches 151, 153, 155, 157, and 159 from the user as a touch panel.

  As shown in FIGS. 2A and 2B, a power switch 102 and a shutter switch 104 are arranged on the upper portion of the imaging apparatus 100. The shutter switch 104 receives a two-step pressing operation from the user when the switch is pressed lightly (SW1 operation) and when pressed deeply (SW2 operation). The shutter switch 104 receives from the user an instruction to start imaging preparation for still image capturing in the SW1 operation and an instruction to start capturing still images in the SW2 operation. Therefore, when the user presses the shutter switch 104 shallowly, the image capturing apparatus 100 performs automatic exposure adjustment and automatic focus adjustment as preparation for image capturing. Next, when the user depresses the shutter switch 104 deeply, the image capturing apparatus 100 performs still image capturing, image recognition, and the like.

  In the automatic exposure adjustment, for example, the control unit 60 performs exposure control according to a control diagram corresponding to the imaging mode selected by the mode switch 110, and a suitable exposure is obtained in the imaging mode selected by the mode switch 110. Operate to get. The imaging mode includes a general-purpose imaging mode such as an auto mode. Further, there may be an imaging mode such as a shutter speed priority mode that prioritizes a shutter speed set in advance by the user and an aperture priority mode that prioritizes an aperture value set in advance by the user.

  In these imaging modes, the imaging sensitivity set by the PGA circuit 20 can be automatically selected and set appropriately, or the user can specify the imaging sensitivity in advance. When the user designates the imaging sensitivity in advance, the S / N of the captured image signal deteriorates as the imaging sensitivity is increased. Therefore, it is assumed that the user who wants to give priority to the image quality selects the low sensitivity.

  Here, how the imaging apparatus 100 identifies a scene and displays the identification result on the display unit 108 will be described with reference to FIG. FIG. 3 is a diagram illustrating a live image on the display unit 108, and shows a state of an image displayed on the display unit 108 when the person 391 is imaged by the imaging device 100.

  The subject image of the person 391 is imaged on the image sensor 16 through the lens 10, photoelectrically converted, and then read out as image data from the image sensor 16. Image data read from the image sensor 16 is stored in the temporary storage memory 30 or directly input to the image processing circuit 50. The image processing circuit 50 generates image data suitable for face recognition, and stores the generated image data in the temporary storage memory 30. Further, the image processing circuit 50 generates image data suitable for scene recognition, and stores the generated image data in the temporary storage memory 30. Further, the image processing circuit 50 generates image data suitable for display on the display unit 108, and stores the generated image data in the VRAM 34.

  Thereafter, image data generated for face recognition is input to the image recognition circuit 38, and the image recognition circuit 38 performs face recognition for detecting a human face. As a result of this face recognition, the number, position, and size of a person's face are acquired. Based on the result of face recognition by the image recognition circuit 38, as shown in FIG. 3, the display unit 108 detects the face based on the result of face recognition when displaying the display image data generated in advance in the VRAM 34. The frame 341 is superimposed and drawn.

  At the same time, image data suitable for scene recognition generated in the temporary storage memory 30 is read, and scene determination (situation determination) is performed in which the controller 60 determines the state of the subject. The scene determination is performed according to an algorithm realized by executing a preprogrammed code by the CPU. Therefore, the type of scene that can be discriminated and the scene discrimination performance depend on the program.

  The scene determined by the control unit 60 is superimposed and displayed on the screen of the display unit 108 together with the face recognition result by the image recognition circuit 38 as an icon 343 showing the state in an abstract manner. Specifically, as shown in FIG. 3, when display image data generated in advance in the VRAM 34 is displayed on the display unit 108, it is superimposed and displayed as an icon 343. In the example of FIG. 3, a form in which a person scene icon 311 is displayed at the upper left position of the display unit 108 is shown. However, the display form is not limited to this, and the display position, size, and icon design can be arbitrarily changed. It has become.

  FIG. 4 shows an example of an identifiable scene and an icon obtained by abstracting each scene. A scene icon 301 indicates a case where a human face is detected by the image recognition circuit 38 and is determined to be a backlight scene by the scene determination process. In the backlight scene, there is a bright subject such as the sun behind the main subject and the main subject becomes dark, but the scene recognition image is divided into multiple blocks and the distribution of luminance information for each block is checked, This is determined by determining whether the pattern is a backlight pattern.

  Similarly, the scene icons 303 and 305 indicate a backlight scene, but the main subject is not a person. Further, the scene icon 303 indicates that the in-focus position is a close distance. Such in-focus position information may be abstracted as an icon.

  Scene icons 311, 313, and 315 are not backlight scenes, respectively, and indicate that the main subject is a person, the main subject is not a person, the in-focus position is a close distance, and the main subject is not a person.

  Scene icons 321 and 325 indicate a case where a night scene is determined by the scene determination process. The night scene is discriminated by analyzing an image for scene recognition by the control unit 60 as in the case of the backlight scene. Specifically, image analysis such as determination of whether or not a dark sky area exists in the screen more than a predetermined amount, and determination of whether or not a point light source indicating illumination or the like exists is performed.

  In addition, the night scene may be determined using the blur amount of the image pickup apparatus 100 that is originally detected by the blur amount detection circuit 44 used for camera shake correction. In the case of a night view scene, the user is prevented from blurring by capturing the image capturing apparatus 100 on a tripod or placing it on an arbitrary place for an image capturing with a long shutter speed. Therefore, in the imaging apparatus 100, one of the conditions for shifting to the night scene is a case where it can be detected that the blur amount has become minute based on the detection result of the blur amount detection circuit 44.

  The scene icon 333 indicates a case where it is determined as an evening scene by the scene determination process. An evening scene is identified by its characteristic color. The scene recognition image generated in the temporary storage memory 30 includes color information, and the sunset scene is determined by analyzing the image by the control unit 60 in the same manner as the backlight scene. Specifically, it is determined that a predetermined area of a predetermined color temperature exists in a predetermined area of the scene recognition image and is determined to be an evening scene.

  The control unit 60 controls the lens driving circuit 42, the aperture driving circuit 26, the mechanical shutter driving circuit 28, the TG 24, the PGA circuit 20, and the image processing circuit 50 according to the scene determination result. For example, even if the average value of the luminance information obtained from the imaging data is the same, the control unit 60 determines whether the main subject is a person, whether it is backlit, Change the weighting of brightness information or change the combination of aperture and exposure time. Alternatively, the amplification level of the PGA circuit 20 is changed, or the gamma conversion setting of the image processing circuit 50 is changed. Further, depending on whether or not the main subject is a person, the depth of field of the lens 10 may be varied even if the frequency component values are equal. Further, depending on whether it is a sunset scene or a night scene, the setting of the color processing for the image data by the image processing circuit 50 may be changed.

  In the steady state before accepting the operation of the shutter switch 104, the above-described evaluation value (scene discrimination evaluation value) used for scene discrimination is periodically generated from the images imaged in time series by the image sensor 16. Then, when the scene discrimination evaluation value is obtained a predetermined number of times, the latest scene discrimination evaluation value is referred to, and if there is a change in the scene, the display of the scene icon is updated. The reason for judging whether or not there is a scene change after the scene discrimination evaluation value has been obtained a predetermined number of times is that if the scene discrimination result is updated too frequently, the scene discrimination result will change drastically depending on the situation of the subject. It is. In such a case, the user may be uncomfortable or the imaging apparatus may become unstable.

  However, in imaging with the imaging apparatus 100, there are many situations in which imaging is performed immediately after a scene suddenly changes, such as when the subject is suddenly changed or when the subject suddenly moves. When imaging is performed, imaging preparation and imaging are performed by the SW1 operation and SW2 operation of the shutter switch 104, respectively. At this time, if there is an abrupt scene change due to the movement of the subject or the like, the shorter the time required for imaging preparation by the SW1 operation is better. In addition, it is required that the image capturing apparatus 100 appropriately detect a scene change and perform appropriate exposure adjustment, focus adjustment, and scene icon display on the subject after the scene change.

  FIG. 5 shows an example of imaging by the imaging apparatus 100 described above. In the example of FIG. 5, it is assumed that the user intends to image the flower 395 but is still away from the flower 395, and various subjects are included in the imaging field angle of the imaging device 100. In addition, since the imaging field angle includes a person 391 sitting near the flower 395 with the sun 393 in the background, the scene icon display is a scene icon 301 indicating a backlight scene including the person.

  When the user who has the imaging device 100 approaches the flower 395 from the above-described state, the person 391 deviates from the imaging angle of view, and the flower 395, which is the imaging purpose, is located at the center of the angle of view. Here, when the person 391 deviates from the angle of view and immediately prepares for imaging by the SW1 operation, the flower 395 to be imaged is located at the center of the screen, and is close to the scene regardless of the scene discrimination cycle. It is desirable to display a scene icon 313 that is focused at a distance. In short, at the time of imaging preparation by the SW1 operation, it is desired that the processing time required for scene determination is short, and a new scene is also appropriately determined.

  Here, with reference to the timing chart of FIG. 6, the timing of scene discrimination evaluation value generation, face detection, and scene discrimination will be described. As shown in FIG. 6, in the imaging apparatus 100, exposure 403 is performed at each vertical synchronization (VD) with reference to the vertical synchronization signal 401 of the image sensor 16. Further, in the imaging apparatus 100, the image data exposed by each VD is read from the imaging element 16, and the scene discrimination evaluation value 405 is generated based on the read image data. That is, in the imaging apparatus 100, scene discrimination evaluation values relating to subject situation discrimination are sequentially calculated from images taken in time series. In the imaging apparatus 100, face detection 407 is performed based on image data read from the imaging element 16. In other words, in the imaging apparatus 100, a human face is sequentially detected from images captured in time series.

  The exposure 403 is performed for each VD, and the scene discrimination evaluation value 405 is output every VD from the next VD, whereas the face detection 407 is performed every 2 VD from the next VD. It has been broken. Specifically, the output of the scene discrimination evaluation value 405 based on the image data in the exposure period 411 is performed in the next VD period 413. The face detection 407 based on the image data in the exposure period 411 is the next VD, and is performed in the period 415 at the same timing as the start of the exposure period 417.

  The timing at which this face detection 407 is performed depends on the processing speed of the image recognition circuit 38. In the example of FIG. 6, the image recognition circuit 38 indicates that the process does not end within 1 VD in order to perform face detection from one input image. However, FIG. 6 is an example, and the face detection processing time varies depending on the performance of the image recognition circuit 38, the detection accuracy, the size of the input image, and the like. For example, the face detection processing time may be shorter or longer than 1 VD.

  When performing scene discrimination at time 421 at the timing illustrated in FIG. 6, if it is attempted to use the latest scene discrimination evaluation value 405 and face detection 407 at that time, one scene using image data of different exposure periods is used. A determination is made. Specifically, the latest scene discrimination evaluation value 405 at time 421 is output in the period 419 based on the image data in the exposure period 417, and the latest face detection 407 is based on the image data in the exposure period 411. In the period 415. Since the exposure period 411 and the exposure period 417 have a temporal shift of 2 VD, it is difficult to accurately perform scene discrimination using the latest scene discrimination evaluation value 405 and face detection 407 at time 421.

  FIG. 7 illustrates a timing chart for eliminating the difference between the scene discrimination evaluation value and the exposure period that is the basis of face detection. As shown in FIG. 7, in the scene discrimination at time 437, the latest scene discrimination evaluation value and face detection result are not used, but the scene discrimination evaluation value and face detection result based on image data of the same exposure period are used. Use.

  Specifically, when the latest face detection result is output in the period 435 based on the exposure period 431 at the time 437, the scene discrimination evaluation value output in the period 433 based on the same exposure period 431 is used. The scene is discriminated in the period 439. Thus, more accurate scene discrimination can be performed when scene discrimination is performed using the scene discrimination evaluation value and the face detection result generated from the image data for the same exposure period. At time 437, since the scene discrimination evaluation value of several VDs in the past is used, the temporary storage memory 30 has an area for storing the scene discrimination evaluation value over several VDs.

  In the example of FIG. 7, the face detection result is output after the scene discrimination evaluation value is output. Conversely, the output of the scene discrimination evaluation value is the output of the face detection result. It may be later. In this case, the scene discrimination is still performed using the scene discrimination evaluation value generated from the image data of the same exposure period and the face detection result. Since the face detection results of several VDs in the past are used, an area for storing the face detection results over several VDs is secured in the temporary storage memory 30.

  Further, FIG. 8 illustrates a timing chart from the steady state before the SW1 operation to the SW1 operation. In the example shown in FIG. 8, similarly to FIG. 7, scene discrimination is performed using a scene discrimination evaluation value and a face detection result generated from image data for the same exposure period. Prior to the SW1 operation at time 437a, the discriminator continues to discriminate sequentially at a certain period. During the SW1 operation, the latest face detection result and the scene discrimination evaluation value generated from the image data with the same exposure period as the face detection result are used. Scene discrimination is performed.

  However, what is assumed in FIG. 8 is that the subject does not change during the SW1 operation from the steady state before the SW1 operation at time 437a. In order to appropriately perform scene discrimination corresponding to a new scene when the SW1 operation is performed in a state where tracking of the subject is not completed, such as when imaging is performed immediately before the SW1 operation, It is necessary to discriminate the scene after finishing following the subject.

  FIG. 9 illustrates a timing chart in the case where the luminance tracking to the subject is not completed during the SW1 operation. As shown in FIG. 9, during the SW1 operation at time 441, the subject brightness is measured (photometric) in order to capture an image with the optimum exposure for the subject.

  In photometry using the image sensor 16 using a CCD or CMOS image sensor, the narrow dynamic range is an issue. The image sensor 16 is not suitable for photometry of a scene that has a wide dynamic range and has a strong contrast. This is because when the exposure amount is inappropriate, overexposure of bright portions and overexposure of dark portions become prominent, and accurate photometry cannot be performed. When photometry is performed with the image sensor 16 having a narrow dynamic range, it is necessary to adjust the exposure of the aperture, electronic shutter, gain, and the like in stages, and perform photometry by appropriately controlling the amount of light received by the image sensor 16. There is.

  Therefore, as shown in FIG. 9, when performing photometry during the SW1 operation in the exposure period 411, an AE convergence period 443 for making the light quantity appropriate is provided because the light quantity to the image sensor 16 is inappropriate. . The AE convergence period is a period of waiting until the luminance obtained from the imaging data converges within a predetermined range (for example, within ± 1/3 step) with respect to the target luminance. Then, scene determination is performed in a period 455 after the luminance obtained from the imaging data is within a predetermined range with respect to the target luminance. This is because if the scene discrimination is performed using the scene discrimination evaluation value obtained before the AE convergence period elapses, blackout or overexposure may occur in a different area from when waiting for the AE convergence period. This is because the nature is high. Under such circumstances, not only the luminance distribution of the imaged data but also the frequency components and color information cannot be detected properly, leading to erroneous scene discrimination results.

  Therefore, the scene discrimination evaluation value and the face detection result used for scene discrimination are the latest data (obtained earliest) when the brightness obtained from the captured data converges within a predetermined range with respect to the target brightness. Data). Specifically, the scene discrimination evaluation value used for scene discrimination in the period 455 is generated in the period 453 based on the image data of the exposure period 451 after AE convergence. Further, the face detection result used in the scene discrimination in the period 455 is generated in the period 449 based on the image data in the exposure period 445 before AE convergence.

  When scene detection requires AE convergence, a time lag corresponding to the AE convergence period occurs. When face detection is performed based on image data during the exposure period after AE convergence, a new time lag increases. . Therefore, in the example of FIG. 9, when convergence of AE is required, the time lag related to scene discrimination is shortened using the scene discrimination evaluation value and the face detection result generated based on the image data of different exposure periods. The sequence is focused on. In other words, after the AE has converged, scene discrimination is performed based on the scene discrimination evaluation value and the face detection result obtained earliest. In this case, the face detection result is generated based on the image data before the convergence of the AE, but if the image data is just before the convergence, there is a large difference compared with the image data obtained after the convergence of the AE. It is thought that it is hard to occur. Therefore, if it is a face detection result in the image data just before convergence, it is determined that a certain level of reliability is obtained, and when the scene discrimination evaluation value is obtained from the image data of the exposure period after AE convergence, these are obtained. Use to perform scene discrimination immediately.

  However, the above-described sequence may be changed according to the state of the user SW1 operation and SW2 operation. FIG. 10 uses the scene discrimination evaluation value and the face detection result generated from the image data for the same exposure period even when the luminance tracking to the subject is not completed during the SW1 operation and convergence of AE is required. Shows the sequence for scene discrimination.

  The imaging apparatus 100 performs exposure adjustment and focus adjustment as imaging preparation during the SW1 operation, and performs imaging during the SW2 operation. The user's operations in these two stages include taking a SW2 operation after holding the SW1 operation and confirming the exposure adjustment, focus adjustment, and composition, and taking a SW2 operation without a gap after the SW1 operation. There is a way. When the SW1 operation is temporarily held as in the former case, it is assumed that the time required for reducing the time lag related to scene discrimination is not so severe because there is a period until the image is captured by the SW2 operation.

  In such a case, as shown in FIG. 10, scene discrimination is performed using the scene discrimination evaluation value of the image data and the face detection result in the same exposure period after the AE convergence period 463 by the SW1 operation at time 461 has elapsed. Do. Specifically, the scene discrimination evaluation value and the face detection result are acquired in the periods 467 and 469 based on the image data of the exposure period 465 after AE convergence, and based on the acquired scene discrimination evaluation value and the face detection result. Scene discrimination is performed in period 471.

  In the scene discrimination using the scene discrimination evaluation value of the image data and the face detection result in the same exposure period after AE convergence, a time lag of about 2 VD occurs as is apparent from the comparison between FIG. 9 and FIG. Become. However, if there is no emphasis on shortening the time lag related to scene discrimination, scene discrimination is performed using the scene discrimination evaluation value of the image data and the face detection result in the same exposure period, so the scene discrimination accuracy is improved. Priority can be given to the benefits of enhancement.

  Next, how to use the above-described sequence properly in accordance with the AE convergence and the user's operation in the imaging apparatus 100 will be described with reference to FIG. FIG. 11 shows an overall flow from activation of the imaging apparatus 100 to imaging.

  As shown in FIG. 11, in the imaging apparatus 100, when processing is started by operating the power switch 102 or the like (S201), the EVF state in which a live image is displayed on the display unit 108 is controlled under the control of the control unit 60. . In the EVF state, the display unit 108 displays the result of appropriately adjusting the image quality of the display image on the display unit 108, performing face detection, and scene discrimination. Therefore, in the EVF state, under the control of the control unit 60, AE processing (S203), AWB processing (S205), AF processing (S207), face detection processing (S209), and scene discrimination processing (S211) are performed.

  In the scene discrimination process (S211) at this time, scene discrimination using the sequence illustrated in FIG. 8 is performed under the control of the control unit 60. That is, in the EVF state, scene discrimination using the scene discrimination evaluation value of the image data and the face detection result for the same exposure period continues to be performed.

  Next, the control unit 60 determines whether or not the SW1 operation is performed (ON / OFF) by the shutter switch 104 (S213), and continues the processing of S203 to S211 until the SW1 operation is confirmed. When the SW1 operation is confirmed, the control unit 60 performs AE processing for still image capturing (S215). In this AE process, the subject brightness is measured based on the image picked up by the image sensor 16, but the AE is converged when the subject brightness is not at an appropriate level or is not stable.

  Next, the control unit 60 performs a scene determination process for capturing a still image (S301). Here, the scene determination processing in S301 will be described with reference to FIG. As shown in FIG. 12, when the scene determination process is started, the control unit 60 determines whether or not the AE is in a stable state (AE has converged) (S303).

  When AE is stable (converged), it is not necessary to converge AE (already converged). Therefore, the control unit 60 performs scene discrimination processing in which the scene discrimination evaluation value and the exposure period of the image data used for face detection are synchronized, that is, a scene using the scene discrimination evaluation value of the image data and the face detection result in the same exposure period. A determination is made (S309). In other words, in S309, scene discrimination processing is performed based on the face detection result and the scene discrimination evaluation value for images taken at the same timing among images taken in time series.

  If AE is not stable (converged), it means that it is during the AE convergence period. Therefore, the control unit 60 waits for AE convergence (S305), and determines whether SW1 and SW2 are pressed at once after AE convergence (S307). In S307, it is determined whether or not the SW2 operation after the SW1 operation is performed within a predetermined time, and it is determined whether or not the SW2 operation is performed without taking time after the SW1 operation. Is done.

  When it is not pressed at once (when the SW1 operation is once held and the interval between the SW1 operation and the SW2 operation is longer than a predetermined time), it is possible to focus on improving the scene discrimination accuracy rather than reducing the time lag related to scene discrimination. . Therefore, as illustrated in FIG. 10, the control unit 60 performs scene discrimination processing in which the scene discrimination evaluation value and the exposure period of image data used for face detection are synchronized (S309).

  In the case of pressing at a stroke (when there is an SW2 operation without waiting for a time after the SW1 operation), imaging is performed according to the SW2 operation, and therefore it is important to shorten the time lag rather than improving the scene discrimination accuracy. Therefore, as illustrated in FIG. 9, the control unit 60 does not set the image data used for the scene discrimination evaluation value and the face detection to be the same exposure period, and the exposure period of the image data is asynchronous. A discrimination process is performed (S311), and the time lag of scene discrimination is shortened. In other words, in S311, a scene discrimination process based on the scene discrimination evaluation value and the face detection result obtained earliest after the SW2 operation is performed.

  As shown in FIG. 11, after the scene determination process, the control unit 60 performs an AF process for capturing a still image (S219) and adjusts the focus on the subject. By performing the above-described steps up to S219, the processing during the SW1 operation is completed.

  Next, the control unit 60 determines whether or not SW1 is turned off (S211). If SW1 is OFF, it is determined that the imaging preparation has been canceled, and the control unit 60 returns the process to the EVF steady state loop.

  When SW1 is ON and SW1 is held down, the control unit 60 determines whether or not SW2 is ON (S223), and performs an imaging operation in response to SW2 being ON, that is, a still image imaging process. (S225 to S229) is started. If SW2 remains OFF, the control unit 60 returns the process to S215.

  When the still image capturing process is started, the control unit 60 determines whether or not to perform strobe imaging with a strobe light on the basis of information set in advance regarding the presence or absence of strobe light emission and subject brightness (S225). In the case of performing strobe light emission, the control unit 60 performs light adjustment processing and determines the strobe light emission amount at the time of capturing a still image (S227). In the light control processing, the strobe is pre-flashed with a predetermined light emission amount, and the reflected light is measured by the photometric sensor or the image sensor 16, and the reflectance of the subject and the degree of arrival of the strobe light are confirmed based on the measurement result. To determine the flash output.

  Thus, the shooting conditions including exposure, scene, focus, strobe, etc. for still image shooting are determined, and still image shooting appropriate for the subject is performed (S229). That is, in S229, still image capturing is performed based on the subject situation determination result by the scene determination process. Note that after the still image is captured, the control unit 60 returns the processing to the EVF steady state loop.

  In this embodiment, the case where the face detection result acquisition takes more time than the calculation of the scene determination evaluation value is illustrated, but conversely, the calculation of the scene determination evaluation value takes longer than the acquisition of the face detection result. It may take. In short, in order to reduce the scene specific time lag when an instruction to capture a still image is given, the face detection result and the scene discrimination evaluation value that are obtained earliest after the imaging instruction is given may be used.

  Note that the description in the above-described embodiment shows an example, and the present invention is not limited to this. The configuration and operation in the embodiment described above can be changed as appropriate.

(Other embodiments)
The above-described embodiment can also be realized in software by a computer of a system or apparatus (or CPU, MPU, etc.). Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto. A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code. Then, the program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers. That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention.

  In addition, a storage medium in which the computer program for realizing the above-described embodiment is encrypted and distributed is distributed, and key information for decrypting is supplied to a user who satisfies a predetermined condition, and the user's computer Installation may be allowed. The key information can be supplied by being downloaded from a homepage via the Internet, for example. Further, the computer program for realizing the above-described embodiment may use an OS function already running on the computer. Further, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU provided in the expansion board. Good.

DESCRIPTION OF SYMBOLS 100 Imaging device 10 Lens 16 Image sensor 30 Temporary memory 34 VRAM
38 Image recognition circuit 50 Image processing circuit 60 Control unit 70 Operation unit 90 Strobe unit 104 Shutter switch 108 Display unit

Claims (5)

Calculation means for periodically calculating a scene discrimination evaluation value used for scene discrimination of a subject from an image taken in time series by the imaging means;
Scene discrimination means for discriminating the scene of the subject using the scene discrimination evaluation value calculated by the calculation means;
Control means for controlling the scene discrimination means,
The control means calculates from the images obtained when the luminance of the subject does not converge within a predetermined range with respect to the target brightness among the images taken in time series by the scene discrimination means. The scene of the subject is not discriminated using the determined scene discrimination evaluation value, and is calculated from an image obtained when the luminance of the subject converges within the predetermined range with respect to the target luminance. An image processing apparatus, comprising: discriminating a scene of the subject using the determined scene discrimination evaluation value. Face detection means for sequentially detecting a human face from images imaged in time series by the imaging means;
The image processing apparatus according to claim 1, wherein the control unit causes the scene determination unit to use a face detection result by the face detection unit and the scene determination evaluation value for determining a scene of the subject. .   The control means uses the face detection result and the scene discrimination evaluation value obtained for each of the images picked up in time series to the scene discrimination means for scene discrimination of the subject. The image processing apparatus according to claim 2, wherein: Using the calculation means for periodically calculating the scene discrimination evaluation value used for scene discrimination of the subject from the images taken in time series by the imaging means, and using the scene discrimination evaluation value calculated by the calculation means, A control method of an image processing apparatus having scene determination means for determining the scene of the subject,
Obtained when the control means of the image processing device has the luminance of the subject not converged within a predetermined range with respect to the target luminance among the images taken in time series by the scene discrimination means. Obtained when the subject scene is not discriminated using the scene discrimination evaluation value calculated from the obtained image and the luminance of the subject converges within the predetermined range with respect to the target luminance. A method for controlling an image processing apparatus, comprising: discriminating a scene of the subject using the scene discrimination evaluation value calculated from a captured image.   The program for functioning a computer as an image processing apparatus of any one of Claims 1 thru | or 3.

Images