JP4645413B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus that extracts a main subject from a captured electronic image.

  Electronic cameras that have a smaller imaging area compared to 35 mm cameras have become widespread. Since this type of camera has a short actual focal length, it is difficult to obtain a photographed image with a blurred background like a 35 mm version camera even if a photographing lens having the same F value (optical aperture value) as that of a 35 mm version camera is used. . Patent Document 1 discloses a technique for adding blur to an image by image processing.

JP 2003-125281 A

  According to Patent Document 1, it is necessary for the user to specify an area to which image blur is to be added (that is, a background area different from the main subject). For this reason, a technique is desired in which the imaging apparatus automatically extracts the main subject area (or background area).

The image pickup apparatus according to the first aspect of the invention is set with an image pickup device that picks up a subject image and outputs an image signal, and main subject region extraction means that extracts a main subject region from the picked-up image. A plurality of different extractions including a two-image reference method for extracting the main subject region using an image shot with the lighting device emitting light and an image shot without the lighting device emitting light according to the shooting scene mode Control means for controlling the main subject region extraction means so as to perform region extraction suitable for the shooting scene by a method selected from the methods, and the control means instructs the illumination device that illuminates the subject not to emit light. If shooting scene mode has been set, to and controls the main subject region extraction means so as to select another extraction method except for the two-image reference method .
According to a second aspect of the present invention, there is provided the imaging device according to the first aspect, wherein the plurality of different extraction methods extract the main subject region using images captured under different photographing conditions. A two-image reference method, a method of extracting the main subject region by analyzing a spatial frequency of the captured image, and a ridge line information reference method of extracting the main subject region using ridge line information detected from the captured image. It is characterized by.
According to a third aspect of the present invention, there is provided an imaging device that captures a subject image and outputs an image signal, a main subject region extracting unit that extracts a main subject region from the captured image, and a shooting environment. And selecting from a plurality of different extraction methods including a two-image reference method for extracting the main subject region using an image shot with the lighting device emitting light and an image shot without the lighting device emitting light. Control means for controlling the main subject region extraction means so as to perform region extraction suitable for the shooting scene by the method described above, and the control means excludes the two-image reference method when the photometric value is a predetermined value or more. The main subject region extracting means is controlled so as to select another extraction method.
According to a fourth aspect of the present invention, there is provided the imaging device according to the third aspect, wherein the plurality of different extraction methods extract the main subject region using images captured under different photographing conditions. A two-image reference method, a method of extracting the main subject region by analyzing a spatial frequency of the captured image, and a ridge line information reference method of extracting the main subject region using ridge line information detected from the captured image. It is characterized by.
An image pickup apparatus according to a fifth aspect of the invention is set with an image pickup device that picks up a subject image and outputs an image signal, and main subject region extraction means that extracts a main subject region from the picked-up image. A region suitable for a shooting scene by a method selected from a plurality of different extraction methods including a two-image reference method for extracting the main subject region using two images shot under different shooting conditions according to a shooting scene mode Control means for controlling the main subject region extraction means to perform extraction, the control means excluding the two-image reference method when a shooting scene mode for shooting a moving subject is set. The main subject region extracting means is controlled so as to select the extraction method.
According to a sixth aspect of the present invention, in the imaging device according to the fifth aspect, the plurality of different extraction methods extract the main subject region using images captured under different photographing conditions. A two-image reference method, a method of extracting the main subject region by analyzing a spatial frequency of the captured image, and a ridge line information reference method of extracting the main subject region using ridge line information detected from the captured image. It is characterized by .

  According to the present invention, it is possible to provide an imaging apparatus that appropriately extracts a main subject region.

The best mode for carrying out the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram illustrating an example of an imaging apparatus according to an embodiment of the present invention. In FIG. 1, an imaging device includes an imaging device 1, a photographing lens 2, a lens driving circuit 3, a control circuit 5, an operation member 7, an imaging device driving circuit 8, a signal processing circuit 9, and a data processing circuit. 10, a compression / expansion circuit 11, a monitor 13, a display control circuit 14, a photometry circuit 15, and an illumination device 20, and a detachable recording medium 12 is mounted. The recording medium 12 is composed of a memory card or the like.

  The taking lens 2 is composed of a plurality of lens groups constituting a taking optical system, and forms a subject image on the image pickup surface of the image pickup device 1. The photographic lens 2 includes a focus lens (not shown), and the lens driving circuit 3 drives the focus lens to move back and forth in the optical axis direction, thereby adjusting the focus of the photographic lens 2. The photographing lens 2 includes a zoom lens (not shown), and the lens driving circuit 3 drives the zoom lens to move back and forth in the optical axis direction, thereby adjusting the zoom of the photographing lens 2. The lens driving circuit 3 generates a lens driving signal in accordance with a lens driving command output from the control circuit 5, and moves each lens by driving a lens driving mechanism (not shown) with the generated lens driving signal.

  The image sensor 1 can capture a still image as well as continuous images of a still image and a moving image. The image sensor 1 is configured by, for example, a CCD image sensor or a CMOS image sensor.

  The image sensor drive circuit 8 generates a drive signal at a predetermined timing in response to a command output from the control circuit 5 and supplies the generated drive signal to the image sensor 1. The image sensor 1 controls charge accumulation (imaging) and reading of accumulated charge by the supplied drive signal. The control circuit 5 obtains brightness information using the photometric data of the subject from the photometric circuit 15 and determines the charge accumulation time of the image sensor 1 based on the brightness information.

  The image signal read from the image sensor 1 is input to the signal processing circuit 9. The signal processing circuit 9 performs signal processing such as amplification, direct current reproduction, A / D conversion, white balance, and gamma conversion on the input signal in accordance with a command from the control circuit 5, and the data after the signal processing is used as image data. Output to the data processing circuit 10.

  In response to a command from the control circuit 5, the data processing circuit 10 performs resolution conversion (pixel number conversion) processing necessary for displaying a reproduced image on the monitor 13 on the image data, and the image data after the resolution conversion processing is processed. The data is output to the compression / decompression circuit 11 and the display control circuit 14, respectively.

  In response to a command from the control circuit 5, the display control circuit 14 performs predetermined signal processing on the image data input from the data processing circuit 10 and outputs it to the monitor 13. The display control circuit 14 further performs processing for superimposing overlay image data such as a shooting menu and a cursor on the image data. Thereby, the subject image on which the overlay image is superimposed is displayed on the monitor 13.

  The compression / decompression circuit 11 performs a compression process on the image data input from the data processing circuit 10 in a predetermined format in response to a command from the control circuit 5, and records the compressed data on the recording medium 12. When image data is recorded on the recording medium 12 during shooting, a reproduced image corresponding to the image data to be recorded is displayed on the monitor 13.

  Note that it is also possible to display a reproduced image based on the image data recorded on the recording medium 12 on the monitor 13. In this case, the compression / decompression circuit 11 reads the image data recorded on the recording medium 12 in accordance with a command from the control circuit 5, performs decoding (decompression) processing on the read data, and then performs decoding. The subsequent data is sent to the data processing circuit 10. During reproduction, the data processing circuit 10 performs resolution conversion processing on the decoded data and outputs it to the display control circuit 14, whereby a reproduced image is displayed on the monitor 13.

  The illuminating device 20 emits light with a light amount instructed according to a light emission instruction from the control circuit 5 at the time of photographing, and illuminates the main subject. The imaging device is set to a photometric value (photometric value) when the setting for prohibiting the light emission of the illumination device 20 is performed, when the photographing scene mode is set to the museum mode, and when the light emission of the illumination device 20 is not prohibited. When the brightness information obtained from the photometric data by the circuit 15) is equal to or greater than a predetermined value, the illumination device 20 is not caused to emit light during photographing. If none of the above applies, the imaging device causes the illumination device 20 to emit light during shooting. Here, the museum mode is a shooting scene mode suitable for shooting in a room where light emission is prohibited. The control circuit 5 determines whether or not to instruct the lighting device 20 to emit light.

  The operation member 7 includes a release button, a shooting scene mode setting dial, and the like, and outputs an operation signal corresponding to the operation content to the control circuit 5. When a release operation signal based on the pressing operation of the release button is input from the operation member 7, the control circuit 5 sets a focus detection area set in advance in the imaging screen in the image signal read from the imaging device 1. A known contrast AF (autofocus) operation is performed using the corresponding signal.

  Specifically, the lens is set so that the integrated value (so-called focus evaluation value) of the high-frequency spatial frequency component for the data corresponding to the focus detection area among the image data processed by the signal processing circuit 9 is maximized. A drive command (focus adjustment signal) is sent to the lens drive circuit 3. The position of the focus lens that maximizes the focus evaluation value is a focus position that eliminates blurring of the edge of the subject image captured by the image sensor 1 and maximizes the contrast of the image (increases sharpness).

  In the present embodiment, image data (video signal) sequentially obtained while moving the focus lens in the imaging lens 2 is analyzed for each of a plurality of focus detection areas in the shooting screen, and the contrast value of the image corresponding to each area is analyzed. The (relative) shooting distance of the subject located in each area can be acquired based on the focus lens position that maximizes the angle.

  In addition to the contrast AF operation, a phase difference AF operation by a known pupil division method may be performed. Also in this case, the shooting distance of the subject located in each region can be obtained.

  The operation member 7 also includes a zoom operation member. When a zoom operation signal based on a zoom operation is input from the operation member 7, the control circuit 5 generates the lens drive command described above, and causes the lens drive circuit 3 to drive the zoom lens forward and backward. Thereby, the subject image formed on the imaging surface of the imaging device 1 is enlarged or reduced, and the zoom is optically adjusted.

  Further, when a zoom operation signal based on the zoom operation is input from the operation member 7, the control circuit 5 outputs a command to the data processing circuit 10 and uses the conversion ratio of the resolution conversion (pixel number conversion) processing for the image signal as the operation signal. Change accordingly. Thereby, the image reproduced and displayed on the monitor 13 is enlarged or reduced, and the zoom is electrically adjusted (electronic zoom). The resolution conversion ratio corresponds to the electronic zoom magnification. When the data processing circuit 10 changes the conversion ratio in the direction of increasing the electronic zoom magnification, a part of the reproduced image is enlarged and displayed on the monitor 13 (while the enlargement ratio is increased, the display range of the reproduced image is narrowed). On the contrary, when the data processing circuit 10 changes the conversion ratio in the direction of decreasing the electronic zoom magnification, the enlargement ratio of the reproduced image displayed on the monitor 13 is lowered, but the display range of the reproduced image is widened.

  A through image shooting operation and a still image book shooting operation performed by the imaging apparatus will be described. Through image shooting is preliminary shooting performed as a pre-stage of main shooting, and main shooting is shooting performed in response to a release full-press operation signal (shooting instruction).

<Through image shooting operation>
When the half-pressing operation signal based on the half-pressing operation of the release button is input from the operation member 7 to the control circuit 5, the imaging apparatus according to the present embodiment starts the AF operation and the through image photographing operation described above. The through image shooting operation is repeatedly performed at a predetermined frame rate while the input of the half-press operation signal is continued.

  When the control circuit 5 detects a half-press operation signal from the operation member 7, it sends an instruction to the image sensor drive circuit 8 to output a drive signal for executing a through image photographing operation. The imaging device 1 receives a drive signal for a through image shooting operation and continuously outputs accumulated charges at a high frame rate of, for example, 30 frames / second. The exposure conditions for through image shooting are determined based on the photometric data from the photometric circuit 15. The signal processing circuit 9 performs the above-described signal processing on the input signal and adds the signals of the same color pixels (in the case of a single-plate color image sensor) located in the vicinity on the image sensor 1, compared with the time of actual photographing. And output to the data processing circuit 10 as a low resolution video signal.

  The data processing circuit 10 outputs an image signal subjected to resolution conversion processing to the display control circuit 14, whereby a through image is displayed on the monitor 13. As a result, the photographer can observe the state of the object scene to be photographed from now on the screen of the monitor 13.

<Still image shooting>
When the release button is fully pressed following the through image shooting operation, a full press operation signal is input to the control circuit 5. The imaging device starts a still image book shooting operation in response to the full-press operation signal.

  When the control circuit 5 detects the full-press operation signal from the operation member 7, it sends an instruction to the image sensor drive circuit 8 to output a drive signal for executing the main photographing operation. The image sensor 1 receives a drive signal for the main photographing operation, accumulates the main photographing charge based on the exposure calculation result, and outputs the accumulated charge. The exposure condition for the main photographing is determined based on the brightness information obtained from the signal value of the latest live view image data. The signal processing circuit 9 performs the above-described signal processing on the input signal, and outputs it to the data processing circuit 10 as an image signal having a higher resolution than when a through image is captured.

  The data processing circuit 10 outputs an image signal subjected to resolution conversion (pixel number conversion) processing to the display control circuit 14, whereby the actual captured image is displayed on the monitor 13. The image data compressed by the compression / decompression circuit 11 is recorded on the recording medium 12.

<Blur addition processing>
An operation of adding electrical blur to an image (that is, adding blur to an image) will be described. The process of electrically blurring an image (hereinafter referred to as adding an image blur) is to process an image by image processing and generate a pseudo-focused image. In the imaging apparatus according to the present embodiment, the image data is subjected to low-pass filter (LPF) processing to degrade the high-frequency spatial frequency component of the image, thereby blurring the edge of the image and lowering the contrast of the image.

  The imaging apparatus is permitted to add image blur (and change the added content) when the shooting scene mode is set to either portrait mode or macro mode, and the shooting scene mode is set to the distant view mode. In such a case, addition of image blur (and change of the added content) is prohibited. Here, the portrait mode is a shooting mode in which a person is a main subject, the macro mode is a shooting mode in which close-up shooting is performed, and the distant view mode is a shooting mode in which a landscape is a subject. The control circuit 5 determines whether to permit or prohibit. When the portrait mode and the macro mode are set, a main subject region extraction process described later is performed, and an image blur is added to the background region. On the other hand, when the distant view mode is set, the extraction process of the main subject area is not performed.

  In addition, when the shooting scene mode is set to the night scene portrait mode, and there is an area having a brightness of a predetermined level or more in the background area of the image, the imaging apparatus adds image blur (and Change of additional contents) is permitted. On the other hand, when there is no area having brightness of a predetermined level or more in the background area of the image, addition of image blur (and change of the added content) is prohibited, and extraction processing of a main subject area described later is not performed. . Whether or not a brightness area of a predetermined level or more exists in the background area of the image is determined using a video signal for a through image input to the data processing circuit 10.

  The control circuit 5 may be configured to prohibit the above-described image blur addition (and change of the added content), at least at the time of shooting, and to allow at the time of reproduction.

  The present invention is characterized in the process of determining an area to which image blur is added by the imaging apparatus described above. A flow of photographing processing in which the control circuit 5 adds image blur will be described with reference to a flowchart of FIG. The imaging process according to FIG. 2 is started when the addition of image blur (and the change of the added content) is permitted.

  In step S101 in FIG. 2, the control circuit 5 determines whether or not the release button has been pressed halfway. When a half-press operation signal is input from the operation member 7 (release button), the control circuit 5 makes an affirmative determination in step S101 and proceeds to step S102. If no half-press operation signal is input, the control circuit 5 makes a negative determination in step S101. The determination process is repeated.

  In step S102, the control circuit 5 sets the exposure mode to the aperture priority auto mode, starts the above-described through image photographing operation, and proceeds to step S103. By setting the aperture priority auto mode, shooting is performed with the currently set aperture value.

  In step S103, the control circuit 5 performs a main subject extraction process from the video signal for the through image, and proceeds to step S104. Details of the process of extracting the main subject will be described later.

  In step S104, the control circuit 5 instructs the data processing circuit 10 to add image blur to the region other than the main subject extracted in step S103 (ie, the background region) in the through image video signal, and then proceeds to step S105. move on. The data processing circuit 10 outputs a signal subjected to image blur addition and resolution conversion processing to the display control circuit 14, whereby an image with image blur added is displayed on the monitor 13.

  The data processing circuit 10 adds image blur by performing LPF processing on the data corresponding to the background region of the input through image data. The characteristics of LPF processing (the number of filter stages, cutoff frequency, etc.) are changed according to the distance information acquired in the focus detection area. Specifically, the shooting distance acquired in the focus detection area in the main subject area is compared with the shooting distance acquired in the focus detection area in the background area, and the difference in shooting distance between the two is large. The cutoff frequency is lowered. As a result, a stronger blur effect is added to the background area having a larger difference from the shooting distance of the main subject.

  Instead of causing the data processing circuit 10 to perform LPF processing, convolution calculation processing using a point spread function may be performed.

  In step S105, the control circuit 5 determines whether or not there is a movement in the entire shooting screen. The control circuit 5 determines the magnitude and direction deviation (variation) of the motion vectors between frames calculated from the through image data of a plurality of frames acquired every 1/30 seconds by the through image shooting on the entire shooting screen. Affirmative determination is made in step S105 when each is within a predetermined range (in all blocks obtained by dividing the shooting screen into a predetermined number) and the average value of the motion vectors in the shooting screen is greater than a predetermined value; Proceed to step S106.

  On the other hand, when the average value of the motion vectors between frames calculated from the through image data of a plurality of frames is equal to or less than a predetermined value, the control circuit 5 or the size of the motion vector in the shooting screen If it is not determined that the deviation in the direction is within the predetermined range over all the blocks of the shooting screen, a negative determination is made in step S105, and the process proceeds to step S107. In step S107, the control circuit 5 determines whether or not some of the subjects have movement. The control circuit 5 determines the deviation of the direction and size of the motion vector between frames calculated from the through image data of a plurality of frames acquired every 1/30 seconds by through image shooting over all the blocks in the shooting screen. If it is determined that it is not within the range, an affirmative determination is made in step S107, and the process proceeds to step S108.

  When the process proceeds to step S106, it is determined that there has been a movement of the photographing range due to the movement of the imaging device. In step S106, the control circuit 5 moves the target region (background region) to which the image blur is to be added according to the direction and size of the motion vector, and proceeds to step S109. Here, a blur equivalent to the blur amount added at the screen edge of the frame before the moving of the shooting range is added to the area newly added to the shooting range as the shooting range moves (similarly). Of LPF treatment). The movement of the target area to which image blur is added is reflected in the display image of the monitor 13 in real time.

  The amount of blur added to the area newly added to the shooting range with the movement of the shooting range is calculated by extrapolation from the distribution of the blur amount in the shooting screen added before the shooting range moves, The calculated blur amount may be added. Also in this case, the movement of the target area to which the image blur is added is reflected on the display image of the monitor 13 in real time.

  On the other hand, the control circuit 5 does not detect any change in the motion vector between frames calculated from through-image data of a plurality of frames acquired every 1/30 seconds by through-image shooting in any of the predetermined number of divided blocks. In this case, a negative determination is made in step S107, and the process proceeds to step S109. In this case, the process proceeds to step S109 without changing the target area to which the image blur is added.

  When the process proceeds to step S108, it is determined that there is a movement for a part of the subject in the shooting screen. In step S108, the control circuit 5 moves the target area (area corresponding to a part of the subject) to which image blur is to be added according to the direction and magnitude of the motion vector in the divided block in which the motion is detected, and then proceeds to step S109. move on.

  FIG. 3 is a diagram illustrating through images acquired at times T0, T1, and T2. In the current frame screen at time T1, the amount of blur added to the partial subject area O1 in which the motion has been detected is the same as the partial subject area O0 in the frame screen (time T0) before the partial subject moves. The amount of blur to be added to the region P in the current frame screen (time T1) corresponding to the position of the region O0 is interpolated from the distribution of the amount of blur added to the periphery of the region P. Calculate by processing. The movement of the target area to which image blur is added is reflected in the display image of the monitor 13 in real time.

  You may determine as follows about the blur addition amount with respect to the area | region which existed before the one subject moved. That is, assuming that the current frame is time T2, the amount of blur added to the area O2 of the partial subject whose movement has been detected in the current frame screen is the same as that in the frame screen (time T1) before the partial subject moves. The blur amount added to the area Q in the current frame screen (time T2) corresponding to the position of the area O1 is set to be the same as the blur amount added to the area O1 of the partial subject. The blur amount added to the corresponding region R in the previous frame screen (time T0) is the same.

  In step S109, the control circuit 5 determines whether or not an operation for changing the image blur addition amount has been performed. The control circuit 5 makes an affirmative determination in step S109 when a blur amount change operation signal is input from the operation member 7, and proceeds to step S110. If a blur amount change operation signal is not input, the control circuit 5 makes a negative determination in step S109. Proceed to step S111. The blur amount changing operation is configured to be capable of both a changing operation performed by designating a specific area on the screen of the monitor 13 and a changing operation performed in common for the entire screen. Of course, the blur amount may be automatically changed according to the shooting distance or time of each subject.

  In step S110, the control circuit 5 instructs the data processing circuit 10 to change the characteristics of the LPF process according to the change operation signal, and proceeds to step S111. Thereby, the amount of blur added is changed. In addition, when performing a convolution operation process instead of the LPF process, the point spread function is changed according to the change operation signal. The image after the blur amount change is displayed on the monitor 13 in real time.

  In step S111, the control circuit 5 determines whether or not the release button is half-pressed. When the half-press operation signal is continuously input from the operation member 7, the control circuit 5 makes an affirmative decision in step S111 and proceeds to step S112. On the other hand, the control circuit 5 when the half-press operation signal is not input cancels the addition of image blur by making a negative determination in step S111 and returns to step S101.

  Regardless of whether the release button is half-pressed or not, in the case of an image pickup apparatus that is set to repeat the through image shooting operation when the main switch is turned on, a negative determination is made in step S111 and the process returns to step S101. Until the half-press operation signal is input, a through image without image blur is displayed on the monitor 13.

  In step S112, the control circuit 5 determines whether or not the release button has been fully pressed. The control circuit 5 makes an affirmative decision in step S112 when a full-press operation signal is input from the operation member 7, and proceeds to step S113. If a full-press operation signal is not input, the control circuit 5 makes a negative determination in step S112, and step S105. Return to.

  In step S113, the control circuit 5 starts the main photographing operation and proceeds to step S114. The control circuit 5 performs an automatic exposure (AE) operation for main photographing using the through image data displayed on the monitor 13 immediately before the full-press operation signal is input. As a result, when image blur is added to the latest through image data, the exposure for the main shooting is determined based on the signal value including the added image blur, and the image blur is added to the latest through image data. If not, the exposure for the main photographing is determined based on the signal value to which no image blur is added.

  In step S114, the control circuit 5 adds an image blur similar to the image blur added to the through image immediately before the full-press operation signal is input to the image signal obtained in the actual photographing ( The target area and the image blur addition ratio are made the same). Further, the control circuit 5 records the actual captured image data compressed after the image blur is added to the recording medium 12, returns the aperture priority auto mode to the most recent setting mode, and then ends the processing of FIG.

  When the control circuit 5 records the actual captured image data on the recording medium 12, the thumbnail image and the quick view image are recorded together with the actual captured image. The thumbnail image is an image for reducing the captured image on the monitor 13 and is obtained by reducing the number of pixels from the actual captured image for the thumbnail. The quick view image is an image used when the captured image is reproduced and displayed on the monitor 13 and is an image obtained by reducing the number of pixels from the actual captured image in accordance with the display pixels of the monitor 13. The thumbnail image and the quick view image are recorded on the recording medium 12 in association with the actual captured image.

<Display example of monitor>
A display mode when an image with added image blur is displayed on the monitor 13 will be described with reference to FIG. FIG. 4A is a diagram illustrating a monitor 13 that displays a through image without adding image blur. In FIG. 4A, the subject 301 exists behind the main subject 300, and the subject 302 exists further behind the subject 301. The imaging apparatus is focused by the AF operation, and is focused on the main subject 300 closest to the imaging apparatus.

  When the image blur is added to the through image illustrated in FIG. 4A, the image blur is added to a region of a subject located far from the focused main subject 300. The amount of blur to be added is larger for the subject 302 than for the subject 301 that has a larger difference (positioned farther) from the shooting distance of the main subject 300. The imaging apparatus of the present embodiment displays a through image with such image blur added on the monitor 13 as shown in FIG.

  According to FIG. 4 (b), the boundary portion of the region to which no image blur is added (in this case, the region of the main subject 300) so that the photographer can easily recognize the region to which the image blur has been added / the region to which the image blur has not been added. To highlight. In the present embodiment, the outline of the main subject 300 is displayed with a bold line to indicate emphasis. Further, the added amount of image blur may be increased as the subject is located far away, and the difference in emphasis can be expressed by changing the thickness of the outline of each main subject. In FIG. 4B, an example of a region where no image blur is added = a region of the main subject 300 is shown, but there may be a case where image blur is not added even if the subject is not in focus. That is, the area where no image blur is added is not necessarily the main subject in focus.

  Note that the above monitor display adds image blur when there is a movement of the entire imaging range (Yes in step S105) and when there is a movement of a part of a subject (Yes in step S107). The movement of the target area is reflected in real time.

  The example of displaying the through image with the image blur added to the monitor 13 has been described, but the same applies to the case where the image blur is added to the actual captured image, the thumbnail image, and the quick view image and each is displayed on the monitor 13.

  Thus, an image with image blur added includes an area to which no image blur is added (in this case, an area of the main subject 300), and an area to which image blur is added (in this case, an area including the main subjects 301 and 302). Is displayed on the monitor 13 as an entire region image including.

  The imaging apparatus according to the present embodiment extracts a main subject area different from the background area as follows in order to add image blur to the background area of the captured image.

The imaging apparatus extracts the main subject region from the captured image by analyzing the captured image using one extraction method selected from the following five extraction methods.
1. First Extraction Method In the first extraction method, a main subject region is extracted using a captured image acquired by causing the illumination device 20 to emit light during photographing and a captured image obtained without causing the illumination device 20 to emit light during photographing. (See JP-A-10-210340). The imaging apparatus extracts an area where the ratio of the image data is greater than or equal to a predetermined value for each pixel between the two images, and determines this area as the main subject area. The data processing circuit 10 determines an area where the ratio of image data is equal to or greater than a predetermined value under the control of the control circuit 5. Such an extraction method that refers to two images requires alignment for associating pixel data constituting the subject between the two images.

2. Second Extraction Method The second extraction method extracts a main subject region using a photographed image acquired by focusing on the main subject and a photographed image obtained by focusing out of the main subject (Japanese Patent Laid-Open No. 09). -327037). The imaging apparatus calculates a difference for each pixel between two images (that is, a signal difference between pixel data (image signals) of both images constituting a common subject), and compares the calculation result with a predetermined determination threshold value. To extract the boundary between the main subject and the background, and determine the region indicated by the boundary as the main subject. The data processing circuit 10 performs the difference calculation, the boundary extraction, and the region determination under the control of the control circuit 5. Thus, the extraction method for referring to two images requires alignment for matching pixel data constituting a subject between the two images.

3. Third Extraction Method The third extraction method extracts a main subject region using two captured images acquired at different depths of field (see Japanese Patent Application Laid-Open No. 2004-310504). The imaging device obtains relative blur information from the two images, and determines a main subject area according to the blur information. The blur information is obtained by comparing pixel data (image signals) of both images constituting a common subject. Acquisition of blur information and area determination are performed by the data processing circuit 10 under the control of the control circuit 5. Thus, the extraction method for referring to two images requires alignment for matching pixel data constituting a subject between the two images.

4). Fourth Extraction Method The fourth extraction method extracts a main subject region by analyzing a spatial frequency component of a captured image. The imaging device analyzes the spatial frequency component for each predetermined block on the entire screen of the through image, and selects the region where the high frequency component is extracted more than the predetermined determination threshold or the region where the high frequency component is extracted most as the main subject. Is determined. Analysis of the spatial frequency component is performed by a DCT (discrete cosine transform) calculation unit in the compression / decompression circuit 11, and region determination is performed by the data processing circuit 10 under the control of the control circuit 5.

5. Fifth Extraction Method The fifth extraction method extracts a main subject region by acquiring a ridge line of a captured image (see Japanese Patent Application Laid-Open No. 2000-132706). The imaging device determines each subject distance in the shooting screen based on the position on the lowermost screen of each subject image and the ridgeline, and determines the closest subject as the main subject. The detection of the ridge line is performed by the control circuit 5, and the area determination is performed by the data processing circuit 10 under the control of the control circuit 5.

  The flow of processing by which the control circuit 5 extracts the main subject will be described with reference to the flowcharts of FIGS. In the present embodiment, the processing shown in FIGS. 5 and 6 is configured as a subroutine of step S103.

  In step S11 of FIG. 5, the control circuit 5 determines whether or not the light emission prohibition setting is set. The control circuit 5 makes an affirmative determination in step S11 when the setting for prohibiting the light emission of the lighting device 20 is made, and proceeds to step S15. If the light emission prohibition setting has not been made, the control circuit 5 makes a negative determination in step S11. Proceed to S12. The light emission prohibition is set and canceled by the control circuit 5 in accordance with an operation signal from a switch (not shown) constituting the operation member 7.

  In step S12, the control circuit 5 determines whether or not the museum mode is set. The museum mode is a shooting mode for shooting an exhibit. When this mode is set, shooting is performed without causing the lighting device 20 to emit light in order to protect the exhibit. If the shooting scene mode is set to the museum mode, the control circuit 5 makes an affirmative decision in step S12 and proceeds to step S15. If not set to the museum mode, the control circuit 5 makes a negative decision in step S12 and proceeds to step S14. move on.

  In step S14, the control circuit 5 starts AF processing and proceeds to step S16. In the AF operation, the focus evaluation value is maximized for each of the plurality of focus detection areas in the moving process while the focus lens is moved at a predetermined speed from the position corresponding to infinity to the position corresponding to the closest end. A focus lens position (focus position) is detected, and shooting distance information corresponding to each focus lens position is acquired. Thereby, distance information to a subject existing in each of the plurality of focus detection areas can be obtained.

  The control circuit 5 sequentially displays the through image on the monitor 13 at the above frame rate during the AF operation, and superimposes and displays the AF marks indicating a plurality of focus detection areas on the through image.

  The process proceeds to step S15 when the illumination device 20 is not caused to emit light. In step S15, the control circuit 5 excludes the first extraction method and proceeds to step S14. Thereby, the control circuit 5 selects an extraction method from other extraction methods other than the first extraction method in step S27 described later.

  In step S16, the control circuit 5 determines whether or not an in-focus ridge line is detected. When the ridge line is detected from the through image, the control circuit 5 makes a positive determination in step S16 and proceeds to step S18. When the ridge line is not detected, the control circuit 5 makes a negative determination in step S16 and proceeds to step S17.

  In step S17, the control circuit 5 determines whether or not the AF operation is finished. When the AF operation is finished, the control circuit 5 makes an affirmative decision in step S17 and proceeds to step S18. If not finished, the control circuit 5 makes a negative decision in step S17 and returns to step S16. When returning to step S16, the AF operation is continued.

  In step S18, the control circuit 5 stores the ridge line information in an internal memory (not shown) and proceeds to step S17. The ridge line information includes the position (area) of the ridge line on the photographing screen and distance information to the subject existing in this area.

  In step S19, the control circuit 5 determines whether or not the photometric value is greater than or equal to a predetermined value. The control circuit 5 makes an affirmative decision in step S19 when the brightness information obtained from the photometric data by the photometry circuit 15 is greater than or equal to a predetermined value and proceeds to step S21. If the brightness information is not greater than or equal to the predetermined value, the control circuit 5 proceeds to step S19. Is negatively determined, and the process proceeds to step S20.

  The process proceeds to step S21 when the illumination device 20 is not caused to emit light. In step S21, the control circuit 5 excludes the first extraction method and proceeds to step S20. Thereby, the control circuit 5 selects one extraction method from the other extraction methods excluding the first extraction method.

  In step S20, the control circuit 5 determines whether or not it is a sports shooting mode. When the shooting scene mode is set to the sports shooting mode, the control circuit 5 makes a positive determination in step S20 and proceeds to step S23. When the shooting scene mode is not set to the sports shooting mode, the control circuit 5 makes a negative determination in step S20. Proceed to S22. Here, the sports shooting mode is a mode for shooting a subject having movement such as a sports scene.

  In step S22, the control circuit 5 determines whether or not the variation in motion of the through image is equal to or greater than a predetermined value. The control circuit 5 has a small change in the motion vector between frames calculated from the through image data of a plurality of frames acquired every 1/30 seconds by the through image shooting (all directions of the motion vector and the size of the shooting screen). Step S22 is negative when it is determined that the amount of change between frames in the block is less than a predetermined value), or that the deviation of the direction and magnitude of the motion vector is within a predetermined range over all blocks on the shooting screen Then, the process proceeds to step S24 in FIG. In this case, it can be considered that the alignment of the through image between the frames is easy.

  On the other hand, if the control circuit 5 does not determine that the motion vector orientation and magnitude deviation between frames calculated from the through image data of a plurality of frames is within a predetermined range across all the above-mentioned divided blocks. Affirmative determination is made in S22, and the process proceeds to step S23.

  When the process proceeds to step S23, it may be difficult to align the through image between frames. In step S23, the control circuit 5 excludes the first to third extraction methods and proceeds to step S24 in FIG. Thereby, the control circuit 5 selects the extraction method from the fourth and fifth extraction methods for extracting the main subject region from one captured image.

  In step S24 of FIG. 6, the control circuit 5 determines whether or not the ridge line information is stored. If the ridge line information stored in step S18 exists, the control circuit 5 makes a positive determination in step S24 and proceeds to step S25. If the ridge line information does not exist, the control circuit 5 makes a negative determination in step S24 and proceeds to step S26.

  When the process proceeds to step S26, the use of the fifth extraction method is not suitable because there is no ridge line information. In step S26, the control circuit 5 excludes the fifth extraction method and proceeds to step S25. As a result, the control circuit 5 selects one extraction method from other extraction methods other than the fifth extraction method.

  In step S <b> 25, the control circuit 5 determines whether or not a significant high-frequency spatial frequency component difference is detected by the region in the frequency analysis for the entire screen region of the through image. The control circuit 5 makes an affirmative decision in step S25 when a difference is found in the spatial frequency component between the predetermined blocks used in the compression process, and proceeds to step S27. If no difference is found in the spatial frequency component, the control circuit 5 A negative determination is made in S25, and the process proceeds to step S28.

  When the process proceeds to step S28, the use of the fourth extraction method is not suitable because the high frequency spatial frequency component is not extracted. In step S28, the control circuit 5 excludes the fourth extraction method and proceeds to step S27. Thereby, the control circuit 5 selects one extraction method from the other extraction methods excluding the fourth extraction method.

  In step S27, the control circuit 5 includes a technique for extracting a known specific color (skin color, etc.) area, a technique for extracting an area having a contour similar to the registered specific outline information, and first to fifth. The main subject region is extracted from the through image using one of the extraction methods that are not excluded. For example, a skin color area is extracted from a through image, and the extracted skin color area is selected from extraction methods that are not excluded (priorities selected from a plurality of extraction methods are 1, 2, 3, 4, 5 To determine the main subject area. Here, a plurality of methods that are not excluded from the first to fifth extraction methods may be selected.

  When the first extraction method is selected, the control circuit 5 performs, for example, the first frame through image shooting without causing the illumination device 20 to emit light in the through image shooting for extracting the main subject region. Each unit is controlled so that the illumination device 20 emits light and a through image is taken. Note that the position of the focus lens has been moved to a position that focuses on the closest subject detected by the AF operation.

  When the second extraction method is selected, the control circuit 5 performs the through image shooting for extracting the main subject region. For example, the control circuit 5 focuses on the main subject closest to the imaging apparatus and captures the first frame through image. The focus lens is moved by a predetermined amount in the second frame (out of focus from the main subject), and each unit is controlled to perform through image shooting. Note that the moved focus lens is returned to a position where the closest subject detected by the AF operation is focused. The shooting order of the first frame and the second frame may be reversed.

  When the third extraction method is selected, the control circuit 5 performs the first frame through image shooting with the control value based on aperture priority auto in the through image shooting for extracting the main subject region, for example. Each part is controlled so that through image shooting is performed by changing the aperture value to a predetermined number of steps (changing the depth of field). The changed aperture value is returned to the aperture value before the change. In addition, the position of the focus lens has been moved to a position that focuses on the closest subject detected by the AF operation.

  When the fourth extraction method is selected and when the fifth extraction method is selected, the control circuit 5 is in a state in which the focus lens is moved to a position where the closest subject detected by the AF operation is in focus Thus, it is sufficient to perform at least one frame shooting for extracting the main subject region.

  The control circuit 5 drives the focus lens to a position where the extracted main subject is focused. When there are a plurality of extracted subjects, the subject closest to the imaging device is extracted using the distance information to the plurality of subjects acquired by the AF operation.

  When the control circuit 5 extracts the area of the main subject, the process in FIG. 6 is terminated, and the process returns to step S104 in FIG.

According to the embodiment described above, the following effects can be obtained.
(1) One of the methods for extracting a known specific color (skin color, etc.) region and the first to fifth subject region extraction methods that are not excluded when determining the main subject region The main subject area is determined from the through image by combining one or more. Compared with the case where the main subject region is determined using only the method of extracting the specific color (skin color or the like) region, the determination accuracy of the main subject region can be improved. Thereby, the background area to which the image blur is added can be accurately obtained.

(2) Since the first extraction method is excluded from the selection range so as not to be selected in a state where the lighting device 20 is not emitting light (step S15), the first extraction method performed by causing the lighting device 20 to emit light and The contradiction between the setting state of the imaging device that causes the illumination device 20 to emit no light can be eliminated. For example, the configuration described above is also applied to a case where the imaging device is driven by a battery, the control circuit 5 determines that the remaining capacity of the battery is less than a predetermined value, and the light emission of the lighting device 20 is prohibited. Can be applied.

(3) Since the first extraction method is excluded from the selection range so that the photometric value is not selected in a state of a predetermined value or more (step S21), the brightness between images when the lighting device 20 emits light and when it does not emit light. It can be excluded that the first extraction method that requires a difference in thickness is applied in a very bright photographing environment.

(4) With regard to the first to third extraction methods, there is a variation in the direction and size of the motion vector between frames in a sport shooting mode in which the movement of the subject is expected (Yes in step S20) or in a through image. If it is detected above a predetermined value (step S22 is affirmative), it is excluded from the selection range so that it is not selected (step S23), so that the position of the pixel data constituting the subject is difficult to align between frames. Since it is not necessary to perform the motion compensation calculation process for performing the alignment, the burden on the control circuit 5 can be reduced.

(5) The fifth extraction method is excluded from the selection range so as not to be selected in a state where a focused ridge line is not detected from the through image (step S26). Therefore, the fifth extraction method that requires ridge line information is used. Application to a through image not including a ridge line can be eliminated.

(6) Since the fourth extraction method is excluded from the selection range so as not to be selected in a state where there is no difference in spatial frequency components between the predetermined blocks used for the compression process (step S28), the space between the predetermined blocks It is possible to eliminate applying the fourth extraction method that requires a difference in frequency components to a through image in which a difference in spatial frequency components between blocks does not occur. An image having no difference in spatial frequency components between blocks is, for example, an image when a monochrome flat plate is photographed.

(7) Since the selection range of the extraction method is changed flexibly as described in (2) to (6) above, the shooting scene mode set in the imaging device, the acquired shot image (in this case) An appropriate main subject region extraction method can be selected according to the characteristics (motion, ridgeline state, spatial frequency) of the image indicated by the “through image”, the shooting environment (brightness), and the remaining battery capacity.

(Modification 1)
In the above description, the selection priority in the case where one method is selected from the extraction methods that are not excluded among the plurality of methods for extracting the main subject region in the image is 1, 2, 3, 4, 5. However, the selection order may be changed as appropriate according to the photographer's intention.

(Modification 2)
Although the example in which the lighting device 20 is built in the imaging device has been described, the present invention can also be applied to the case where an external lighting device is connected to the imaging device.

(Modification 3)
In step S22 described above, the motion variation determination may be performed step by step. For example, the determination is made while moving the focal plane, or the adjustment is made while adjusting the aperture value. The control circuit 5 in the case of performing motion variation determination while moving the focal plane moves the focal plane to change the position of the focus lens (in a different focus adjustment state) in through-image shooting of consecutive frames. And) Control each unit in the imaging apparatus so as to perform through image shooting. The control circuit 5 makes an affirmative decision in step S22 and performs the second extraction when a change in the motion vector between frames is detected in any of the predetermined number of divided blocks at or above the determination threshold at the time of focal plane movement. Exclude methods.

  In addition, the control circuit 5 in the case of performing motion variation determination while changing the aperture value varies the aperture value in the case of through-image shooting of consecutive frames (the depth of field is set so as to be in different optical blur states). Control each unit in the imaging apparatus so as to perform through image shooting. The control circuit 5 makes an affirmative decision in step S22 when the change in the motion vector between the frames is detected in any of the predetermined number of divided blocks above the determination threshold value at the time of changing the depth of field. Exclude the extraction method. Note that, in the case of the modified example 3 as well, in the sports shooting mode in which the movement of the subject is expected (Yes in step S20), the first extraction method is excluded as in step S20.

(Modification 4)
Further, in step S22 described above, it may be configured to perform motion variation determination using a detection signal from the motion sensor. The imaging device in this case includes an angular velocity sensor or an acceleration sensor. The control circuit 5 compares the magnitude and direction of the motion vector indicated by the detection signal from the motion sensor with the magnitude and orientation of the motion vector obtained from a predetermined block in the through image, and the difference between the two is determined in advance. If the value is equal to or smaller than the value, it is regarded as parallel movement of the photographed image due to camera shake, and a negative determination is made in step S22. When it is regarded as parallel movement, the first to third extraction methods are not excluded.

  The photometric value may be obtained from the signal level of the image signal from the image sensor 1 in addition to the photometric value obtained from the photometric circuit 15.

  The main subject area acquired by the extraction method determined as described above is used to determine an area to add image blur, to be used as a focus area at the time of focus adjustment, to be used as a photometry area at the time of photometry, or to zoom up ( It may be used as a central area when zooming down.

  The above description is merely an example, and the interpretation of the invention is not limited to the correspondence between the components of the above-described embodiment and the components of the present invention.

It is a block diagram which shows an example of the imaging device by this invention. It is a flowchart explaining the flow of the imaging | photography process in which a control circuit adds image blur. It is a figure which illustrates the through image acquired at different time. (a) It is a figure which illustrates a through image, without adding image blur. (b) It is a figure which illustrates the through image to which the image blur was added. It is a flowchart explaining the flow of the process in which a control circuit extracts a main subject. It is a flowchart explaining the flow of the process in which a control circuit extracts a main subject.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Imaging device 2 ... Shooting lens 3 ... Lens drive circuit 5 ... Control circuit 7 ... Operation member 9 ... Signal processing circuit 10 ... Data processing circuit 11 ... Compression / decompression circuit 12 ... Recording medium 13 ... Monitor 14 ... Display control circuit 15 ... Photometry circuit 20 ... Lighting device

Claims (6)

An image sensor that captures a subject image and outputs an image signal;
Main subject region extraction means for extracting a main subject region from the photographed image;
Including a two-image reference method for extracting the main subject region using an image shot with a lighting device emitting light and an image shot without the lighting device emitting light according to a set shooting scene mode Control means for controlling the main subject area extraction means so as to perform area extraction suitable for the shooting scene by a method selected from a plurality of different extraction methods;
The control means is configured to cause the main subject region extraction means to select an extraction method other than the two-image reference method when a shooting scene mode for instructing non-light emission is set for an illumination device that illuminates the subject. An imaging device characterized by controlling . The imaging device according to claim 1,
The plurality of different extraction methods include a two-image reference method for extracting the main subject region using images taken under different shooting conditions, and extracting the main subject region by analyzing the spatial frequency of the shot image. An imaging apparatus comprising: a method; and a ridge line information reference method for extracting the main subject region using ridge line information detected from a captured image. An image sensor that captures a subject image and outputs an image signal;
Main subject region extraction means for extracting a main subject region from the photographed image;
A plurality of different extraction methods including a two-image reference method for extracting the main subject region using an image shot with a lighting device emitting light and an image shot without the lighting device emitting light according to the shooting environment Control means for controlling the main subject area extraction means so as to perform area extraction suitable for the shooting scene by a method selected from
The image pickup apparatus according to claim 1, wherein the control unit controls the main subject region extraction unit to select an extraction method other than the two-image reference method when a photometric value is a predetermined value or more. The imaging device according to claim 3.
The plurality of different extraction methods include a two-image reference method for extracting the main subject region using images taken under different shooting conditions, and extracting the main subject region by analyzing the spatial frequency of the shot image. An imaging apparatus comprising: a method; and a ridge line information reference method for extracting the main subject region using ridge line information detected from a captured image. An image sensor that captures a subject image and outputs an image signal;
Main subject region extraction means for extracting a main subject region from the photographed image;
A shooting scene selected from a plurality of different extraction methods including a two-image reference method for extracting the main subject region using two images shot under different shooting conditions according to a set shooting scene mode. Control means for controlling the main subject region extraction means so as to perform region extraction suitable for
The control means controls the main subject region extraction means to select an extraction method other than the two-image reference method when a photographing scene mode for photographing a moving subject is set. An imaging device. The imaging apparatus according to claim 5,
The plurality of different extraction methods include a two-image reference method for extracting the main subject region using images taken under different shooting conditions, and extracting the main subject region by analyzing the spatial frequency of the shot image. An imaging apparatus comprising: a method; and a ridge line information reference method for extracting the main subject region using ridge line information detected from a captured image .

Images