JP2019135839A - Image processing apparatus, method of controlling image processing apparatus, imaging apparatus, and program - Google Patents

Abstract

To provide an image processing apparatus capable of performing image recording having high convenience while minimizing a recorded amount.SOLUTION: An image processing apparatus acquires a plurality of image data (moving image data or the like) and records the data in a recording medium. The image processing apparatus acquires depth distribution information of an object corresponding to an image. A recording mode control unit 104 performs record control by dynamically determining whether to record in a first mode or a second mode with respect to the image data. The recording mode control unit 104 controls such that, when recording is performed in the first mode, the image data and the depth distribution information corresponding to the image data are recorded in the recording medium by a recording unit 109 while, when recording is performed in the second mode, the image data is recorded in the recording medium by the recording unit 109.SELECTED DRAWING: Figure 1

Description

  The present invention relates to image and distance information recording control technology.

  There is a technique in which a user selects a favorite image from continuously acquired moving images and performs various image processing afterward to acquire a single still image that matches the user's preference. Patent Document 1 discloses a technique for determining whether or not a subject is in a predetermined state with respect to continuously shot images, and relatively increasing the recording image quality of the image in the predetermined state. Patent Document 2 discloses a technique for acquiring a high-quality image by not performing frame compression when the shooting mode is changed to the super-resolution mode. Further, as disclosed in Patent Literature 3, a technique for adjusting blur of a captured image by image processing by generating distance distribution information as additional information in addition to the captured image is disclosed.

JP 2011-166391 A JP 2007-189665 A JP-A-2015-119416

  However, various types of image processing include processing that requires subject outline extraction with high accuracy. There are a background blur process for electronically blurring the background, a gradation process for each area that obtains a high-quality dynamic range compressed image by separately performing a gradation correction process on the main subject and other areas. In order to realize these techniques, not only the frame image data is recorded, but also a distance map indicating the distance distribution within the angle of view (at least information indicating the relative relationship of the distances) is acquired together. It is necessary. In the gradation processing for each area, it is necessary to record a RAW image before image processing.

On the other hand, in the form in which the distance map and the raw image data are recorded for performing the image processing on all the frames of the moving image, there is a problem that the recording capacity becomes enormous and the burden on the user increases. Therefore, it is very important to minimize the data amount of the frame image when acquiring post-processing information.
An object of the present invention is to provide an image processing apparatus capable of acquiring a captured image and additional information, which can perform highly convenient image recording while suppressing a recording capacity.

  An apparatus according to an embodiment of the present invention is an image processing apparatus including a recording unit that acquires a plurality of image data and records the acquired image data on a recording medium, and acquires an object depth distribution information corresponding to the image data. A first mode for recording the image data and the depth distribution information corresponding to the image data on the recording medium by the recording means, and the image data without recording the depth distribution information by the recording means. An image processing apparatus comprising: control means for performing a recording process of the plurality of image data by switching a second mode for recording on the recording medium.

  According to the present invention, it is possible to perform highly convenient image recording while suppressing the recording capacity.

1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. FIG. 3 is a block diagram of a recording mode control unit in the first embodiment. It is a flowchart which shows the process of the recording mode control part in 1st Embodiment. It is explanatory drawing which shows an example of the main subject area | region detection result. It is a figure which illustrates the distance histogram of a main subject area and a background area. It is a figure which shows an example of the optical model which shows an object distance and an image distance. It is a figure which shows the relationship between a defocus amount and an object surface distance with a graph. It is a block diagram of an image processing part in a 1st embodiment. It is a flowchart of the background image blurring process in 1st Embodiment. It is a figure which shows the example of a screen which notifies that the blurring process of a background image is possible. It is a figure explaining the process of a distance map shaping part. It is a figure explaining the process of a focus subject extraction part. It is a figure explaining a blurring process. It is a block diagram of the recording mode control part in 2nd Embodiment of this invention. It is a flowchart which shows the process of the recording mode control part in 2nd Embodiment. It is a figure explaining the process of a threshold value calculation part. It is a block diagram of the image processing part in 2nd Embodiment. It is a flowchart of the gradation correction process classified by area in 2nd Embodiment. It is a figure which shows the example of a screen which notifies that the gradation correction process according to area | region is possible. It is a figure explaining the gradation characteristic calculation method in 2nd Embodiment. It is a figure explaining a synthetic | combination process. It is a block diagram which shows the structure of the imaging device which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the process of the recording information adjustment part in 3rd Embodiment. It is a figure explaining a main subject score calculation process. It is a figure explaining the calculation process of the main subject score threshold value. It is a block diagram which shows the structure of the imaging device which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process of the exposure condition control part in 4th Embodiment. It is a figure explaining the recording format of an image and a distance map.

  Hereinafter, each embodiment of the present invention will be described with reference to the drawings. In each embodiment, an image processing apparatus that performs image processing with high-performance region extraction processing later according to a user instruction will be described.

[First Embodiment]
In the first embodiment of the present invention, an explanation is given on the assumption that a background image blurring process is performed on an arbitrary frame selected by a user later, with respect to a recorded continuous image of a plurality of frames in an image processing apparatus. To do. Therefore, the present embodiment provides an image processing apparatus that can be controlled by switching between a mode in which depth distribution information as additional information is recorded and a mode in which recording is not performed for a plurality of captured images. FIG. 1 is a block diagram illustrating a configuration applicable to an imaging apparatus as an example of an image processing apparatus of this embodiment.
The imaging optical system 101 includes a lens group such as a zoom lens and a focus lens, a diaphragm adjusting device, and a shutter device. The imaging optical system 101 adjusts the magnification, focus position, and light amount of the subject image that reaches the imaging unit 102. The imaging unit 102 includes a CCD (charge coupled device) image sensor, a CMOS (complementary metal oxide semiconductor) image sensor, and the like, and converts a light beam from a subject that has passed through the imaging optical system 101 into an electrical signal by photoelectric conversion. The A (analog) / D (digital) conversion unit 103 converts the input analog electric signal into a digital image signal.

  The recording mode control unit 104 has a plurality of recording modes, and controls the amount of information of an image recorded in the recording unit 109. The image processing unit 105 performs various processes on the image signal read from the recording unit 109 in addition to the image signal output from the A / D conversion unit 103. For example, correction processing such as distortion and noise caused by the imaging optical system, demosaicing processing, white balance adjustment, color conversion processing, and gamma correction are executed. The image processing unit 105 performs background blur processing assumed in the present embodiment in addition to predetermined image processing. In the present embodiment, among the image processing performed by the image processing unit 105, an image that has not been subjected to at least a part of image processing other than distortion and noise correction processing caused by the imaging optical system is defined as a RAW image. In this embodiment, an example in which the image processing unit 105 performs the reproduction process will be described. However, a reproduction processing unit may be provided separately from the image processing unit 105.

  The system control unit 106 is a control central unit that controls the overall operation control of the imaging apparatus, and includes a CPU (Central Processing Unit), a memory, and the like. The system control unit 106 performs drive control of the imaging optical system 101 and the imaging unit 102, predetermined image processing in the image processing unit 105, and the like in accordance with an operation instruction from the operation unit 107 by the user.

  The display unit 108 is configured by a liquid crystal display, an organic EL (Electro Luminescence) display, or the like, and displays an image according to an image signal generated by the imaging unit 102 or an image signal read from the recording unit 109. The recording unit 109 performs processing for recording an image signal on a recording medium. Recording processing is performed in an encoding format for still images (for example, JPEG) and an encoding format for moving images (for example, H.264, H.265, etc.) according to the recording settings for each of still images and moving images. In addition, when the settings are set so that still images and moving images are recorded as raw images before image processing, low compression, such as non-compression or lossless compression, with less image quality deterioration than when image data is recorded after image processing. Compression and recording are performed at a rate. The recording medium is an information recording medium using, for example, a package containing a rotating recording body such as a memory card on which a semiconductor memory is mounted or a magneto-optical disk. The recording medium is detachable from the imaging device.

The bus 110 is used for transmitting and receiving signals among the recording mode control unit 104, the image processing unit 105, the system control unit 106, the display unit 108, and the recording unit 109.
Hereinafter, the flow of processing in the present embodiment will be described with a focus on the recording mode control unit 104 and the image processing unit 105. The recording mode control unit 104 is a characteristic processing block in this embodiment. The image processing unit 105 is a processing block that performs a background image blurring process according to an operation instruction after shooting.

The operation of the recording mode control unit 104 will be described with reference to FIGS.
FIG. 2 is a block diagram of the recording mode control unit 104. The main subject area detection unit 201 acquires input image data and detects a main subject area. The main subject area is an image area of a subject selected from a plurality of subjects, for example. The distance map calculation unit 202 obtains input image data and calculates a distance map (depth distribution information). The main subject distance calculation unit 203 acquires information on the main subject area detected by the main subject area detection unit 201 and the distance map calculated by the distance map calculation unit 202 to calculate the main subject distance. The main subject distance is a distance from the imaging device to the main subject. The background distance calculation unit 204 calculates the background distance from the background area information and the distance map. The background area is an area other than the main subject, and the background distance is the distance from the imaging device to the background. The threshold calculation unit 205 acquires the main subject distance and calculates a threshold for the difference in distance between the main subject and the background. The recording mode determination unit 206 acquires the main subject distance, the background distance, and the threshold value, and compares the difference between the main subject distance and the background distance with the threshold value to perform a recording mode determination process. The recording mode includes at least a first mode for recording image data and distance information, and a second mode for recording image data. In the present embodiment, control for switching between the first and second modes at the time of recording each frame for a moving image having image data acquired by imaging as each frame will be described, but the control is not limited thereto. For example, in still image capturing, an embodiment in which the first and second modes are switched in each image capturing is also included in the present invention. That is, it is only necessary to control the switching between the first mode and the second mode for a plurality of acquired image data. Details of the processing of each unit will be described later.

  FIG. 3 is a flowchart for explaining the processing of the recording mode control unit 104. In this flowchart, it is assumed that moving images are being acquired, and frame images are continuously input at a predetermined frame rate from the time point when the user starts shooting until the time point when the shooting operation ends. To do.

  First, in S301, the main subject area detection unit 201 detects a main subject area. A specific example will be described with reference to FIG. FIG. 4 shows an example of the detected main subject area, and the main subject area is detected using the face detection frame. In addition, when the main subject is not a person, the subject region is detected by general object detection, and is not limited to a specific detection method. Further, as shown in FIG. 4, an area that is not the main subject area in the entire image area is defined as a background area.

In step S302, the distance map calculation unit 202 calculates a distance map. As a distance map calculation method, for example, as described in Patent Document 3, the light from the subject is divided into pupils to generate a plurality of viewpoint images (parallax images), the amount of parallax is calculated, and the depth of the subject is calculated. There is a method for acquiring distribution information. The depth distribution information of the subject is data representing the distance from the camera as the imaging means to the subject (subject distance) as an absolute value, or data indicating a relative distance relationship (image depth) in the image data ( Parallax amount distribution, defocus amount distribution, etc.). In the present embodiment, the following description will be given as data represented by a distance value. However, when the disparity amount or defocus amount distribution is used as the depth distribution information, the distance value is replaced with the disparity amount and the defocus amount, respectively. Shall be made.
Regarding the depth distribution information, there are various embodiments as information corresponding to the depth in the depth direction (depth direction) of each subject in the captured image. That is, the information indicated by the data corresponding to the depth of the subject directly represents the subject distance from the imaging device to the subject in the image, or the relative relationship between the distance (subject distance) and the depth of the subject in the image. May be any information that represents For example, control for changing the in-focus position is performed on the imaging unit 102, and a plurality of captured image data captured is acquired. The depth distribution information can be acquired from the focus area of each captured image data and the focus position information of the captured image data. In addition, when the imaging element of the imaging unit 102 has a pupil division type pixel configuration, it is possible to acquire depth distribution information for each pixel from a phase difference between a pair of image signals. Specifically, the imaging device converts a pair of light beams that pass through different pupil partial regions of the imaging optical system into optical signals, and converts the paired image data into a plurality of photoelectric conversion units. Output from. The image shift amount of each region is calculated by the correlation calculation between the paired image data, and an image shift map representing the distribution of the image shift amount is calculated. Alternatively, the image shift amount is further converted into a defocus amount, and a defocus map representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image) is generated. When this defocus amount is converted into the subject distance based on the conditions of the imaging optical system and the imaging element, distance map data representing the distribution of the subject distance is obtained. Image shift map data, defocus map data, or distance map data of the subject distance converted from the defocus amount can be acquired.
In addition, the subject distance from the imaging device to the subject in the image is directly measured using the TOF (Time Of Flight) method that measures the delay time from the light projection to the subject and receiving the reflected light to measure the distance to the subject. May be acquired automatically. In the TOF method, pulse light is projected onto a subject (object) by a light projecting means, the reflected light is received by the imaging unit 102, and the flight time (delay time) of the pulse light is measured to measure the subject distance ( Measure the distance to the object) and obtain the depth distribution information.
In the present embodiment, the following description will be given as data represented by a distance value. However, when the disparity amount or defocus amount distribution is used as the depth distribution information, the distance value is replaced with the disparity amount and the defocus amount, respectively. Shall be made.

  In step S303, the main subject distance calculation unit 203 calculates a main subject distance (a parallax amount and a defocus amount corresponding to the main subject) using the main subject region information and the distance map. In S304, the background distance calculation unit 204 calculates the background distance using the background area information and the distance map. A specific example of the processing of S303 and S304 will be described with reference to FIG. FIG. 5 is a diagram illustrating a method of calculating the main subject distance and the background distance. The horizontal axis represents the distance in the optical axis direction of the imaging apparatus, and the vertical axis represents the frequency (frequency) of the distance histogram.

  First, the main subject distance calculation unit 203 acquires a distance histogram in the main subject area shown in FIG. 5A, and sets the distance corresponding to the peak value with the maximum frequency as the main subject distance. FIG. 5A shows an example in which two peak values exist. The reason why there is a peak value other than the main subject distance is that a part of the background image has entered the main subject region because the boundary of the main subject region is defined by a rectangular frame.

  Next, the background distance calculation unit 204 acquires a distance histogram in the background region shown in FIG. 5B, and sets the distance corresponding to the peak value having the maximum frequency as the background distance. FIG. 5B shows an example in which three peak values exist.

In S305, the threshold value calculation unit 205 calculates a threshold value for the difference in distance between the main subject and the background. The threshold is determined using an optical model from the value of the main subject distance. This will be described with reference to FIG. The optical model in FIG. 6 shows the object distance and the image distance when the imaging optical system 101 is approximated as one lens. The side where the object exists (left side of the lens in FIG. 6) is the object surface side with respect to the lens of the focal length (denoted as f), and the side on which the light from the object forms an image through the lens (the lens (Right side) is defined as the image plane side. Further, assuming that the main subject is in focus, the following distance is defined.
・ Main subject distance: Dist_Obj
・ Background distance: Dist_Back
・ Main subject image distance: Img_Obj
The main subject image distance is a distance at which light from the main subject forms an image on the image plane side. Further, a position on the image plane side where light from the main subject forms an image is defined as a focus plane, and a displacement amount from the focus plane to a position where background light is imaged is defined as a defocus amount def. Here, regarding the sign of the defocus amount def, the direction away from the object side with respect to the lens is positive. Therefore, the defocus amount at a position where the background light at a position farther from the main subject on the object image forms a negative value. Qualitatively, the greater the absolute value of the defocus amount, the greater the degree of scattering of the background image on the focal plane, and the greater the background blur.

In FIG. 6, the following formulas (1) and (2) are established from the lens formula.
Here, when the main subject image distance Img_obj is deleted from the expressions (1) and (2) and transformed into an expression for the background distance Dist_Back, the following expression (3) is obtained.
From the expression (3), the background distance Dist_Back can be regarded as a function of the defocus amount def when the main subject distance Dist_Obj and the focal distance f are uniquely determined. Therefore, in the present embodiment, an example in which the subject distance is mainly calculated as the distance map is shown, but the object is also achieved by the defocus amount and the parallax amount.

FIG. 7 is a graph illustrating the relationship between the defocus amount (horizontal axis) and the object surface distance (vertical axis) created based on Expression (3). Two examples are shown when the focal length is 50 mm and the main subject distance Dist_obj is 3 m (3000 mm) and 5 m (5000 mm). Since it is assumed that the main subject is in focus, the defocus amount is 0 when the object distance is equal to Dist_obj. If the allowable defocus amount that can be considered that the background is sufficiently blurred is −0.2 mm, the corresponding object distance is about 3911 mm when Dist_obj = 3 m. Therefore, the threshold for the distance difference between the subject and the background is | 3911-3000 | = 911 mm, and is calculated as approximately 90 cm.
The above is description of the process which the threshold value calculation part 205 performs in S305.

  In step S <b> 306, the recording mode determination unit 206 compares the difference between the main subject distance and the background distance with the threshold acquired from the threshold calculation unit 205. If the difference between the main subject distance and the background distance is within the threshold value, the recording mode determination unit 206 determines that there is a possibility that the background image is not sufficiently blurred and an electronic blurring process may be required later (S307). The recording process is performed in the high quality mode, which is the first mode. In the high quality mode, a process of recording a distance map in addition to the data of the normal compressed frame is executed. The process of blurring the background using information recorded in the high quality mode will be described later. On the other hand, if the difference between the main subject distance and the background distance is larger than the threshold, the recording mode determination unit 206 considers that the background image is sufficiently blurred. That is, the recording mode determination unit 206 determines that electronic blurring processing is unnecessary, and the process proceeds to S308, where recording processing in the normal mode that is the second mode is performed. In the normal mode, processing for recording only data of a normal compressed frame is executed. Note that the predetermined signal processing accompanying the frame compression processing and the development processing is performed in the image processing unit 105. The processing by the recording mode control unit 104 is performed at the time of shooting. Further, the image recorded in the high quality mode is not limited to the normal compressed frame, but may be a RAW image before image processing. Here, the RAW image may be compressed by a reversible compression method at the time of recording.

  Next, a background image blurring process performed in accordance with a user instruction after recording will be described. This processing is mainly performed by the image processing unit 105, but the operation unit 107 and the system control unit 106 that receive user instructions are also involved. Since this process is performed after recording, it is hereinafter referred to as “post-blurring process”. The post-blurring process will be described with reference to FIGS.

  FIG. 8 is a processing block diagram of the image processing unit 105 regarding the post-blurring process. The distance map shaping unit 801 performs distance map shaping processing. The distance map shaping process will be described later. A focus subject extraction unit 802 extracts an image of the focus subject. The focus subject is the main subject in focus. The blur processing unit 803 performs a blur process on the background image, and outputs the processed frame image data. Details of the processing of each unit will be described later.

FIG. 9 is a flowchart of the post-blurring process.
First, in step S <b> 901, the imaging apparatus receives an instruction for a background image blurring process from a user. Since only frames recorded in the high quality mode can be subjected to post-blurring processing, it is necessary to notify the user whether or not the frames are processable. FIG. 10 shows a display example when the display unit 108 notifies (notifies) the user that post-blurring processing is possible. Display and input processing are performed for an instruction as to whether or not to perform the background image blurring process. If the user selects to perform background image blurring during moving image playback, the system control unit 106 receives an operation instruction from the operation unit 107 and instructs the image processing unit 105 to perform the processing from S902 onward.

  In step S902, the distance map shaping unit 801 acquires frame image and distance map data, and performs distance map shaping processing. FIG. 11 is a diagram for explaining the distance map shaping process. FIG. 11A illustrates an input frame image, FIG. 11B illustrates an input distance map before the shaping process, and FIG. 11C illustrates an output distance map after the shaping process.

  The subject outline of the input distance map shown in FIG. 11B may be less accurate than the subject outline in the input frame image shown in FIG. The reason is that the distance map is calculated by reducing the resolution of the frame image due to the influence of the perspective conflict at the subject boundary part when calculating the distance. Background blurring processing requires high-precision contour extraction. For this reason, it is necessary to perform processing for matching the subject contour of the distance map with the subject contour of the frame image, and this processing is called shaping processing. Of course, the distance map after the shaping process is highly versatile and can be used in processes that require a distance map other than the blurring process of the background image.

The shaping process is performed by a bilateral filter process. In the bilateral filter processing, the shaping result is used as a frame image (see FIG. 11A), and the filter result (denoted as Jp) at the target pixel position p is expressed by the following equation (4).
Jp = (1 / Kp) ΣI1q · f (| p−q |) · g (| I2p−I2q |) (4)
The meaning of each symbol in Formula (4) is as follows.
q: peripheral pixel position Ω: integration target area centered on the target pixel position p Σ: integration in the q∈Ω range I1q: distance map signal value at the peripheral pixel position q f (| p−q |): target pixel position p Gaussian function centered at I2p: pixel value of the shaping image at the target pixel position p I2q: pixel value of the shaping image at the peripheral pixel position q g (| I2p−I2q |): pixel value I2p of the shaping image Gaussian function centered on Kp: normalization coefficient, integrated value of f · g weights.
In the equation (4), the f value increases as the target pixel position p and the peripheral pixel position q are closer. The smaller the difference between I2p at the target pixel position p and I2q at the peripheral pixel position q, that is, the closer the pixel value of the target pixel and the peripheral pixel in the shaping image is, the g weight (smoothing weight) of the peripheral pixel is. growing. The output obtained by weighting and adding the signal value I1q of the input distance map with the f · g weight becomes the signal value Jp of the output distance map after shaping.

  FIG. 11D shows a profile 1100pf at the position x of the frame image in FIG. The acquisition position of the profile 1100 pf is indicated by a line 1100 in FIG. The profile 1100pf has a step shape that changes at the position xa. FIG. 11E shows a profile at a position x in the distance maps of FIGS. 11B and 11C. The acquisition position of the profile 1101pf is indicated by a line 1101 in FIG. 11B, and the acquisition position of the profile 1102pf is indicated by a line 1102 in FIG. The position of the line 1101 and the position of the line 1102 are the same.

  In the input distance map shown in FIG. 11B, the signal value indicating the distance of the subject protrudes outside the contour of the subject image. The change in the profile 1101pf indicated by the broken line in FIG. 11E is shifted from the position xa where the profile 1100pf in FIG. 11D changes. A shaping process by the bilateral filter is executed, and a profile 1102 pf corresponding to the shaped distance map of FIG. 11C is obtained. The position at which the profile 1102pf changes corresponds to the position xa at which the profile 1100pf in FIG. 11D changes, and has a shape that matches the contour of the subject image. That is, the profile 1102pf has a step shape that changes greatly at the position xa.

  In step S903 of FIG. 9, the focused subject extracting unit 802 extracts a focused subject. This will be specifically described with reference to FIG. FIG. 12A illustrates an input frame image, and FIG. 12B illustrates a distance map after shaping. FIG. 12C illustrates the extraction characteristics, and FIG. 12D illustrates the extraction result of the focus subject.

  The extraction characteristics shown in FIG. 12C are applied to the shaped distance map shown in FIG. The horizontal axis of FIG. 12C represents the distance of the subject, and the vertical axis represents the output value of the extraction result. The extraction characteristic is a characteristic that outputs a maximum value only for a distance within a predetermined focus surface distance range, and outputs zero or a minimum value for other distances. An extraction result output by applying the extraction characteristic is shown in FIG. Only the main subject in focus is extracted.

  In S904 of FIG. 9, the blurring processing unit 803 performs a blurring process of the background image. The blurring process will be described with reference to FIG. FIG. 13A shows an example of the shape of the kernel of the blur filter, and FIG. 13B exemplifies a focus surface extraction image. FIG. 13C illustrates a blur filter for the position of interest.

  In order to simulate the round blur caused by the lens, the shape of the kernel of the blur filter is substantially circular as shown in FIG. 13A, and the filter weight (weighting coefficient value) is constant. In this embodiment, the filter size is 5 × 5. The blur filter is applied to each image of the input frame image. At that time, the blurring processing unit 803 refers to the focus surface extraction image and performs control so that only the background portion is subjected to filter processing. FIG. 13B illustrates a case where the target position P is the right position of the subject. In this case, the blur filter performs a filter operation on the position indicated by the symbol “◯” in FIG. The position indicated by the symbol x is a position belonging to the image area of the focused subject. If these positions are included in the target of the filter operation, the outline portion is mixed and the image quality may be deteriorated. For this reason, control is performed so that only the background portion is filtered. By the above processing, an image obtained by performing blurring processing only on the background region other than the focused subject is obtained.

  In S905 of FIG. 9, post-adjustment is executed. This process is a process for adjusting the blur intensity when the blur intensity does not match the user's preference as a result of checking whether the blur intensity matches the user's preference. As a result, it is possible to obtain a blurred image of the degree desired by the user. The blur intensity can be adjusted by changing the number of TAPs in the filter kernel in FIG. The post-blurring process executed by the image processing unit 105 is performed according to a user instruction after shooting. Finally, the image data generated by the image processing unit 105 is recorded on the recording medium by the recording unit 109.

  In the present embodiment, whether to acquire additional information for region extraction at the same time is dynamically switched according to the distance difference between the main subject and the background. For this reason, it is possible to reduce a burden on the user due to an enormous amount of recording, and to obtain information for post-blurring processing in advance. By recording high-precision area extraction information only for frames that are likely to require post-processing, it is possible to perform image recording with high user merit while suppressing the recording capacity. . Here, in the present embodiment, when the distance difference between the main subject and the background is large, only the captured image is recorded as the normal mode. When the distance difference between the main subject and the background is larger than the threshold, the background blur processing is performed to further emphasize the distance difference, and recording is performed in the high quality mode.

  In this embodiment, based on the distance difference between the main subject and the background, the presence / absence of acquisition of additional information is determined from the comparison result between the distance difference and the threshold value. Without being limited thereto, for example, depth information may be acquired from the F value, and the depth information may be used as an element of determination. In this embodiment, as a distance map calculation method, the distance is calculated from the parallax of the pupil divided image. Without being limited thereto, the distance may be acquired using, for example, a contrast AF (autofocus) evaluation value or the like. These matters are the same in the embodiments described later.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the first embodiment, it is assumed that post-blurring processing is performed on a recorded image. In the second embodiment, for example, it is assumed that the gradation correction processing for each region is performed later on a frame image in which the main subject is dark due to backlight. When a main subject that is dark due to backlight and a background area other than that are processed with the same tone conversion characteristics, the dark portion of the background becomes extremely bright, resulting in an unnatural image. The significance of performing tone correction for each region is to suppress unnaturalness by correcting the main subject region and the background region in the image with different tone characteristics. On the other hand, this processing requires advanced region extraction processing.

  The processing in this embodiment differs from the first embodiment in the operation of the recording mode control unit 104 and the operation of the image processing unit 105 related to post-processing. Hereinafter, description will be made centering on portions where processing is different from that of the first embodiment, and the same configurations as in the case of the first embodiment will be omitted by using the same reference numerals already used. Such omission of description is the same in embodiments described later.

  The operation of the recording mode control unit 104 will be described with reference to FIGS. FIG. 14 is a processing block diagram of the recording mode control unit 104 in the second embodiment. Differences from the configuration shown in FIG. 2 are as follows.

  The main subject Bv value calculation unit 1402 acquires information on the main subject region from the main subject region detection unit 201 and calculates the Bv value of the main subject. The Bv value is an exposure value that represents a luminance difference with respect to a target luminance value of the region of interest. The background Bv value calculation unit 1403 acquires background area information and calculates a background Bv value. The exposure level difference calculating unit 1404 acquires the Bv values of the main subject and the background, and calculates the exposure level difference between the main subject and the background. A threshold calculation unit 1405 calculates a threshold for the exposure step from the Bv value of the main subject. The recording mode determination unit 1406 acquires the exposure level difference between the main subject and the background calculated by the exposure level calculation unit 1404 and the threshold value calculated by the threshold value calculation unit 1405. The recording mode determination unit 1406 compares the exposure step with the threshold value to determine the recording mode. Details of the processing of each unit will be described later.

FIG. 15 is a flowchart for explaining the processing of the recording mode control unit 104.
First, in step S1501, the main subject area detection unit 201 detects a main subject area in an image. In next step S1502, the main subject Bv value calculation unit 1402 calculates the Bv value of the main subject. When the Bv value of the main subject is expressed as Bv_obj, this is calculated by the following equation (5).
Bv_obj = log2 (Y_obj / Y_obj_target) (5)
In the equation (5), log2 is a logarithmic function with 2 as the base. Y_obj is a representative luminance value of the main subject, and is calculated as an average value of the luminance values of the main subject region. Y_obj_target is a target luminance value of the main subject area. The target luminance value is a luminance value regarded as appropriate exposure, and is a predetermined value. The target luminance value may be changed according to whether or not the main subject is a person. From equation (5), the darker the main subject, the smaller the Bv value. For example, when the representative luminance value of the main subject is ½ of the target luminance value, the Bv value is −1. When the representative luminance value of the main subject is twice the target luminance value, the Bv value is +1.

In step S1503, the background Bv value calculation unit 1403 calculates a background Bv value. When the background Bv value is expressed as Bv_back, this is expressed by the following equation (6).
Bv_back = log2 (Y_back / Y_back_target) (6)
Y_back in the equation (6) is a representative luminance value of the background, and is calculated by an average value of luminance values of the background region. Y_back_target is the target luminance value of the background area.

In the next step S1504, the exposure level difference calculation unit 1404 calculates an exposure level difference (denoted as delta_Bv) between the main subject area and the background area by the following equation (7).
delta_Bv = | Bv_obj-Bv_back | (7)
As can be seen from the equations (5) to (7), the exposure level difference increases as the background is bright and the main subject is dark, or vice versa, that is, as the D range (dynamic range) when focusing on the main subject is increased. The value of delta_Bv increases. Moreover, the value of delta_Bv becomes smaller as the D range becomes narrower. In this embodiment, an exposure step is used as an evaluation value indicating the D range of the frame.

  In step S <b> 1505, the threshold calculation unit 1405 calculates a threshold for the exposure step from the main subject Bv value. An example of threshold calculation shown in FIG. 16 will be described. In the graph shown in FIG. 16, the horizontal axis represents the main subject Bv value, and the vertical axis represents the exposure step threshold. In the example of FIG. 16, the threshold corresponding to Bv_min is TH_min, and the threshold when the main subject Bv value is greater than or equal to zero is TH_max. Although an example in which linear interpolation is performed with a linear expression in a section from Bv_min to zero is shown, higher-order interpolation processing of second or higher order may be performed.

  As shown in FIG. 16, qualitatively, when the main subject Bv value is a negative value, that is, when the main subject is darker than the appropriate exposure, it is more necessary to perform tone correction processing later. Therefore, the threshold for the exposure step is set small. On the other hand, when the main subject Bv value is near zero, the main subject is close to proper exposure. Therefore, since the necessity for gradation correction processing is reduced, the threshold value is set large. Further, when the main subject Bv value is a positive value, the main subject is too bright. In this case, the threshold value is set high (TH_max) in order not to be subjected to the tone correction main process.

  In step S1506, the recording mode determination unit 1406 compares the threshold calculated by the threshold calculation unit 1405 with the exposure level difference to determine the recording mode. If the exposure step is greater than or equal to the threshold, the recording mode determination unit 1406 determines that there is a high possibility that gradation correction will be necessary later, and proceeds to S1507 to perform recording processing in the high quality mode. In the high quality mode, recording processing of frame image, distance map, and RAW image data is performed. The reason for recording the RAW image before image processing is to perform gradation correction before nonlinear processing such as gamma conversion. On the other hand, if the exposure step is less than the threshold value, the recording mode determination unit 1406 determines that it is not necessary to perform tone correction later, and proceeds to S1508 to perform recording processing in the normal mode. The processing of the recording mode control unit 104 is performed at the time of shooting.

  Next, a description will be given of the tone correction processing for each area performed according to a user instruction after recording. This processing is mainly performed by the image processing unit 105, but the operation unit 107 and the system control unit 106 that receive user instructions are also involved. Hereinafter, the area-specific gradation correction processing performed after recording is referred to as “post-correction processing”. The post correction process will be described with reference to FIGS. 17 and 18.

FIG. 17 is a processing block diagram of the image processing unit 105 regarding post-correction processing. Differences from the configuration shown in FIG. 8 are as follows.
The first gradation characteristic calculation unit 1703 acquires frame image data and calculates the first gradation characteristic. The first gradation characteristic is the gradation characteristic of the main subject area. The second gradation characteristic calculation unit 1704 acquires frame image data and calculates the second gradation characteristic. The second gradation characteristic is the gradation characteristic of the background area. The first gradation correction unit 1705 performs gradation correction processing on the input frame image using the first gradation characteristic. The second tone correction unit 1706 performs tone correction processing on the input frame image using the second tone characteristics. The synthesizing unit 1707 acquires information on the focused subject from the focused subject extracting unit 802, and the first tone corrected by the first tone correcting unit 1705 and the second tone correcting unit 1706 perform tone correction. The second image that has been obtained is acquired and the composition process is performed. Details of the processing of each unit will be described later.

FIG. 18 is a flowchart for explaining post-correction processing.
First, in step S <b> 1801, the imaging apparatus receives an instruction for post-correction processing from the user. FIG. 19 shows a display example, and the display unit 108 indicates to the user that post-correction processing is possible for a high-quality recording frame that can be subjected to post-correction processing. In addition, display and input processing are performed to ask the user whether or not to perform post-correction processing. When the user selects to perform tone correction processing after shooting during moving image reproduction, the system control unit 106 receives an instruction from the operation unit 107 and instructs the image processing unit 105 to perform the processing from S1802 onward. .

  In step S1802, the distance map shaping unit 801 performs distance map shaping processing, and in step S1803, the focus subject extraction unit 802 extracts a focus subject. In step S1804, the first gradation characteristic calculation unit 1703 calculates a first gradation characteristic. In step S1805, the second gradation characteristic calculation unit 1704 calculates the second gradation characteristic. With reference to FIG. 20, the calculation process of the first gradation characteristic and the second gradation characteristic will be described. FIG. 20A illustrates an input frame image and shows a main subject region and a background region in the image. That is, the main subject region detection unit 201 detects the main subject region and the background region in the image separately from the input frame image. Separate tone correction processing is performed on the main subject area and the background area. First, uniform gain processing is performed on the main subject area. The reason for the uniform gain processing is that, when this function is applied, there is a high possibility that the main subject is uniformly dark due to backlight or shade. On the other hand, the gain processing for each luminance is performed on the background area. This is because there are generally subjects having various luminances in the background area, and the D range is wide. Gradation compression is performed so as not to impair the gradation of each region. Note that the gain characteristic is not limited to the above concept, and can have a characteristic of an arbitrary shape.

FIG. 20B illustrates characteristics of gain (vertical axis) with respect to input luminance (horizontal axis) in the main subject region. The gain takes a constant value (denoted Gain_obj). In order to perform correction so that the average luminance value Y_obj of the main subject region becomes the target luminance value Y_obj_target that is set to an appropriate exposure, the gain Gain_obj is calculated by the following equation (8).
Gain_obj = Y_obj_target / Y_obj (8)

  FIG. 20C shows a solid line graph of input / output luminance characteristics when processing is performed using the gain characteristics shown in FIG. That is, this characteristic is the first gradation characteristic. A dotted line indicates a case where the ratio between the input luminance value and the output luminance value is 1: 1. The slope of the graph line of the first gradation characteristic is larger than the slope of the graph line indicated by the dotted line.

FIG. 20D illustrates the representative luminance value in the luminance histogram of the background area. The horizontal axis represents input luminance, and the vertical axis represents frequency (frequency). FIG. 20E illustrates gain characteristics with respect to input luminance in the background region. The gain changes according to the input luminance. The second tone characteristic calculation unit 1704 calculates the representative luminance values of the dark part side and the bright part side of the input luminance before calculating the gain characteristic. In the calculation process, a luminance histogram of the background area is acquired as shown in FIG. The representative luminance value Y_back_low is a dark portion representative luminance value calculated when a predetermined number of pixels are counted from the minimum luminance. The representative luminance value Y_back_high is a representative luminance value of a bright part calculated when a predetermined number of pixels are counted from the maximum luminance. The maximum gain (denoted as Gain_back) for the background region is calculated by the following equation (9) so that the average luminance value Y_back of the background region becomes a target luminance value Y_back_target that is set to appropriate exposure.
Gain_back = Y_back_target / Y_back (9)

  In the gain characteristic shown in FIG. 20E, Gain_back is a constant value in a section where the input luminance value is Y_back_low or less, and the minimum gain amount is 1 in a section where the input luminance value is Y_back_high or more. In the section between T_back_low and Y_back_high, the characteristic monotonously decreases according to the input luminance value. FIG. 20F illustrates a solid line graph of input / output luminance characteristics when processing is performed using the gain characteristics illustrated in FIG. This characteristic is the second gradation characteristic. The solid line graph representing the second gradation characteristic has a polygonal line shape that protrudes upward at Y_back_low, and is a dotted line (in the ratio of the input luminance value to the output luminance value of 1: 1 in the section above Y_back_high). The shape matches the case).

  In S1806 of FIG. 18, the first gradation correction unit 1705 performs gradation correction processing of the input frame image using the first gradation characteristic. In step S1807, the second tone correction unit 1706 performs tone correction processing of the input frame image using the second tone characteristics. These processes are processes for converting the luminance value of the input frame image with the gradation conversion characteristics illustrated in FIGS. 20C and 20F, respectively. In next step S1808, the synthesizing unit 1707 synthesizes the two images subjected to the gradation correction processing. With reference to FIG. 21, the process of the synthesis unit 1707 will be described.

  FIG. 21A illustrates an input frame image, and FIG. 21B illustrates a focus plane image extracted by the focus subject extraction unit 802. In the synthesis process, an image that has been subjected to tone correction with the first tone characteristics is output to the focus plane region in the image indicated by the white region in FIG. For the non-focus surface area (background area) in the image indicated by the black area in FIG. 21B, an image that has been subjected to gradation correction with the second gradation characteristics is output. When this processing is performed, the image after the synthesis processing is the image shown in FIG. 21C with respect to the input frame image of FIG. FIG. 21C shows an image in which gradation correction processing is performed with different gradation conversion characteristics for the main subject area that is the focus plane and the background area that is the non-focus plane area. Finally, the image data on which the gradation correction for each region has been performed is recorded on the recording medium by the recording unit 109.

  In the present embodiment, whether to acquire additional information for region extraction at the same time is dynamically switched according to the exposure step (brightness difference) between the main subject region and the background region. For this reason, it is possible to reduce a burden on the user due to an enormous amount of recording, and to acquire information for post-correction processing (gradation correction). In the present embodiment, the image and the depth distribution information are recorded in the high quality mode when the exposure level difference between the main subject area and the background area is larger than the threshold value. However, the present invention is not limited to this. For example, even when recording is performed in the high quality mode, it is assumed that the subject is shot with appropriate brightness compared to when the exposure step between the main subject area and the background area is smaller than the threshold value. Good. At this time, when the exposure step is larger than the threshold value, the image is recorded without recording the depth distribution information in the normal mode.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. The present embodiment is premised on the post-blurring process described in the first embodiment, and aims to further reduce the recording capacity, particularly when recording information. In the first embodiment, post-processing information is acquired based on the difference in distance between the main subject and the background, for example, for a frame for which it is determined that the background image is not sufficiently blurred. However, depending on the shooting scene, the distance between the main subject and the background may always be short. In such a case, post-processing information is acquired over almost the entire frame. As a result, the recording capacity of the captured image may be enormous. Thus, in the present embodiment, a process for further narrowing down the frames to be recorded in the high quality mode in order to keep the recording capacity appropriate will be described. That is, in this embodiment, the determination of whether or not it is necessary to reduce the recording capacity, which will be described later, and the determination of the recording mode by subject score determination are performed in combination with part or all of them in the first and second embodiments. It can be done.

  FIG. 22 is a block diagram illustrating a configuration applicable to the imaging apparatus of the present embodiment. The difference from the configuration shown in FIG. 1 is that a recording information adjustment unit 2210 is added. FIG. 23 is a flowchart showing the processing of the recording information adjustment unit 2210. The recording information adjustment unit 2210 will be described with reference to FIG. The following processing is executed after the recording unit 109 records a series of moving image data in the recording mode controlled by the recording mode control unit 104.

In step S2301, the recording information adjustment unit 2210 acquires the image recording capacity (denoted as MEM), and calculates the maximum recording capacity (denoted as MEM_MAX) in step S2302. The maximum recording capacity MEM_MAX is calculated by the following formula (10).
MEM_MAX = MEM_FRAME x NUM_FRAME x k1 x k2 (10)
In equation (10), MEM_FRAME is the recording amount per frame, and in this case, inter-frame compression or intra-frame compression is not performed. NUM_FRAME is the total number of frames of the moving image to be processed. k1 is a predetermined compression rate and takes a value of 1 or less. The actual compression rate varies depending on the shooting scene, but here it is a predetermined compression rate. k2 is an allowable value of the capacity increase rate due to the recording of the additional information, and takes a value of 1 or more.

  In next S2303, the recording information adjustment unit 2210 compares the recording capacity MEM with the maximum recording capacity MEM_MAX. If MEM is less than or equal to MEM_MAX, the processing ends without adjusting the recording information. If MEM is larger than MEM_MAX, the recording information adjustment unit 2210 determines that it is necessary to adjust the recording information, and the process proceeds to S2304.

  In the present embodiment, when it is determined that the recording capacity needs to be reduced, processing for further narrowing down the frames to be recorded in the high quality mode is performed based on the size information and position information of the main subject. The reason is that the frame that the user wants to record as a still image with a blurred background after shooting is a frame in which the main subject is in good condition. Specifically, the frame acquired by the narrowing-down process is a frame in which the main subject image area is present near the center of the captured image and the main subject size is large.

  In S2304 of FIG. 23, the recording information adjustment unit 2210 acquires the size information and position information of the main subject, and calculates the main subject score based on the size information and position information of the main subject in the next S2305. Note that the processing after S2304 is performed on all the recorded frames. The processes of S2304 and S2305 will be described with reference to FIG. FIG. 24A illustrates an input frame image. The coordinates of the center of gravity of the main subject area are expressed as (X, Y), the height of the main subject area is expressed as Height, and the width of the main subject area is expressed as Width.

In S2304, a process of extracting the main subject region in a rectangular shape from the input frame image shown in FIG. 24A and acquiring the width Width and height Height is executed. Further, the center-of-gravity position coordinates (X, Y) of the extracted main subject area are acquired. From the acquired information, a normalized size (denoted as Size) corresponding to the area is calculated by the following equation (11).
Size = Width x Height / Size_all (11)

Size_all in the equation (11) is a normalization coefficient for normalization so as not to depend on the image size. For example, Size_all is the area of the entire image. In this case, Size indicates the ratio of the area of the main subject area to the entire image area. When the coordinates of the image center position are expressed as (Xc, Yc), the normalized distance (denoted as Dist) from (Xc, Yc) to the center of gravity position coordinate (X, Y) of the main subject region is ( 12) Calculated by the equation.

  R in the equation (12) is a normalization coefficient corresponding to the distance from the image center position to the image edge. That is, the normalized distance Dist indicates the ratio of the distance difference between the coordinates (Xc, Yc) and (X, Y) with respect to the distance from the image center position to the image edge.

  In step S2305, a main subject score is calculated. This will be described with reference to FIGS. FIG. 24B illustrates distance score calculation characteristics. The horizontal axis represents the normalized distance Dist, and the vertical axis represents the distance score (denoted as Score_Dist). FIG. 24C illustrates size score calculation characteristics. The horizontal axis represents the normalized size Size, and the vertical axis represents the size score (denoted as Score_Size).

  In the present embodiment, first, a distance score Score_Dist and a size score Score_Size are respectively calculated from the calculated normalized distance information and normalized size information according to the characteristics shown in FIGS. With respect to the characteristics of the distance score Score_Dist shown in FIG. 24B, the score increases as the subject image is closer to the center of the image. FIG. 24B illustrates characteristics obtained by linearly interpolating between two points with a linear expression. In a range where the Dist value is smaller than the first threshold D1 with respect to the normalized distance Dist, the distance score Score_Dist is constant. In the range where the Dist value is larger than the second threshold D2 with respect to the normalized distance Dist, the distance score Score_Dist is constant. When the Dist value is greater than or equal to the first threshold and less than or equal to the second threshold, the value of the distance score Score_Dist decreases linearly as the Dist value increases.

With respect to the characteristics of the size score Score_Size shown in FIG. 24C, since the score increases as the image size of the subject increases, the characteristic increases monotonously with respect to the Size. FIG. 24C illustrates characteristics obtained by linearly interpolating between two points with a linear expression. In a range where the Size value is smaller than the first threshold value S1 for the normalized size Size, the size score Score_Size is constant. Further, the size score Score_Size is constant in a range where the Size value is larger than the second threshold value S2 with respect to the normalized size Size. When the Size value is greater than or equal to the first threshold and less than or equal to the second threshold, the value of the size score Score_Size increases linearly as the Size value increases.
The characteristics shown in FIGS. 24B and 24C are examples, and interpolation processing may be performed by setting three or more points.

Next, from the calculated distance score Score_Dist and size score Score_Size, a main subject score (denoted as Score) is calculated by the following equation (13).
Score = w_d × Score_Dist + w_s × Score_Size (13)
In Expression (13), w_d and w_s are arbitrary weighting coefficients.

  In step S2306 in FIG. 23, the recording information adjustment unit 2210 performs high-quality recording frame narrowing processing. In S2305, the main subject score Score is calculated over all frames. The recording information adjustment unit 2210 compares the main subject score Score with a predetermined threshold (denoted TH_Score). The recording process in the high quality mode is executed for the frame in which the value of the main subject score Score exceeds the threshold value TH_Score. A frame in which the value of the main subject score Score is equal to or less than the threshold value is in the normal mode, and additional information is deleted and is not recorded. This will be specifically described with reference to FIG. The horizontal axis represents the threshold value TH_Score, and the vertical axis represents the recording capacity MEM. As the threshold TH_Score increases, the number of high-quality frame images (the number of frames) decreases, so the recording capacity MEM decreases. The maximum threshold value that the recording capacity MEM falls below the upper limit value MEM_MAX calculated in S2302 is defined as TH_Score_min. By narrowing down high-quality frames using the threshold TH_Score_min, a moving image can be acquired while the moving image recording capacity is suppressed to MEM_MAX or less.

In the present embodiment, an increase in the recording capacity of moving images can be suppressed by the processing of the recording information adjustment unit 2210. In the present embodiment, position information and size information of the main subject area are used as an index for narrowing down high quality frames.
In another embodiment, the scene change degree may be used as another index for switching the recording mode. The scene change degree is an index representing the magnitude of a scene change when an image changes between different frames (for example, the current frame and the previous frame). The scene change degree is calculated by aligning a plurality of time-series images, for example, a current frame and a previous frame, and calculating a difference between the images. When there is almost no change in the image between two temporally continuous frames, a process of thinning out the additional information in the high quality mode is executed. When there is a large change in the image between two temporally continuous frames, processing for recording additional information relating to both frames is executed.
The determination based on the subject score and the determination based on the degree of scene change are performed when it is determined that the recording capacity needs to be reduced in the present embodiment, but the present invention is not limited to this. For example, the recording mode may be switched using a subject score or a scene change degree as a high-quality frame determination method without detecting or determining the recording capacity.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described. The present embodiment is based on the post-blurring process described in the first embodiment, and aims to reduce motion blur of each frame. A control for changing the shutter speed and the frame rate when recording in the high quality mode in accordance with the movement of the main subject will be described.

  With respect to a frame that is a target for generating a high-quality still image by post-processing, it is preferable that there is no motion blur of the subject and the subject is stopped. In exposure control when capturing a moving image, the shutter speed is controlled so that the motion of the subject image does not appear discontinuous when the frame images are continuously viewed. That is, control is performed so that the shutter speed does not become too fast. On the other hand, when viewing one frame of a moving image frame imaged as such as a still image, motion blur may occur depending on the moving speed of the subject. In view of this, the present embodiment has an object to suppress motion blur of a subject with respect to exposure control of a frame image acquired in a high quality mode, in particular, shutter speed.

  FIG. 26 is a block diagram illustrating a configuration applicable to the imaging apparatus of the present embodiment. The difference from the configuration shown in FIG. 1 is that an exposure condition control unit 2606 is provided. FIG. 27 is a flowchart showing the processing of the exposure condition control unit 2606. The exposure condition control unit 2606 will be described with reference to FIG. The following processing is performed after the recording mode control unit 104, and is performed every frame or at a predetermined frame interval.

  First, in S2701, a determination process is performed as to whether or not the recording mode is a high quality mode. If the recording mode is the normal mode, the process advances to step S2705, and the shooting operation is performed with the moving image exposure in the normal mode for the next frame that is one frame time after the current frame. On the other hand, if the recording mode is the high quality mode, the process proceeds to S2702, and exposure control is performed in accordance with the movement of the main subject. In S2702, the exposure condition control unit 2606 calculates a motion vector of the main subject. The motion vector is calculated by a known pattern matching process.

In next step S2703, the exposure condition control unit 2606 calculates the shutter speed of the next frame. The shutter speed of the next frame is expressed as Tv, and the magnitude of the motion vector calculated in S2702 is expressed as v (unit: pixel). The frame interval is expressed as T_frame (1/60 second when the frame rate is 60 fps). The shutter speed Tv is calculated from the following equation (14) from v and T_frame.
Tv = T_frame / v (14)
The equation (14) means that Tv is calculated as the shutter speed at which the moving amount of the main subject image is 1 pixel within the photographing time. In other words, Tv is a shutter speed at which motion blur is contained in one pixel. For example, T_frame is 1/60 seconds and v is 6 pixels. In this case, the shutter speed corresponding to the movement of one pixel of the main subject image is Tv = 1/360 seconds. Therefore, in accordance with the object of this embodiment, a value smaller than Tv calculated by equation (14) may be used as the shutter speed.

  In S2704, the exposure condition control unit 2606 determines other exposure conditions and a frame rate. The other exposure conditions are specifically sensitivity and aperture value. Changing the aperture value changes the depth of field and loses continuity with the previous and next frames. For this reason, in the present embodiment, for the amount of change in the shutter speed Tv, control is performed to keep the exposure constant by changing the sensitivity. Further, control is performed to change the frame rate as the shutter speed Tv increases. For example, when the value of Tv is 1/120 seconds or less, processing for changing the frame rate from 60 fps to 120 fps is executed. This process provides an effect that it is difficult to miss a critical moment of a fast-moving subject. Finally, the exposure condition control unit 2606 performs feedback control of the exposure condition determined in S2704 or S2705 in the control of the imaging optical system 101 and the imaging unit 102, and performs control to perform the imaging process for the next frame. .

  In the present embodiment, the processing of the exposure condition control unit 2606 enables shooting at an optimal shutter speed that matches the speed of the subject (moving object). In addition, when viewing as a moving image, the subject is electronically controlled according to the amount of motion of the subject in order to avoid discontinuous movement of the moving object due to the large shutter speed Tv associated with the image frame. A process of adding blur to the image is performed.

In the first, second, third, and fourth embodiments described above, the recording mode is basically determined and switched for each frame. However, the present invention is not limited to this. For the purpose of reducing the amount of recording data, control may be performed by periodically switching the recording mode, such as recording in a high quality mode every predetermined frame manually or automatically.
When manual setting is performed, for example, when the predetermined number of frames is set to 5 by a user operation via the operation unit 107, the depth distribution information is recorded together with the image as 1 frame in 5 frames and the high quality mode. Alternatively, the period of recording in the high quality mode may be determined according to the imaging frame rate set by the user.
When the setting is automatically performed, at least one of determinations such as a difference in distance between the main subject and the background, an exposure step, a subject score, and a scene change degree in each embodiment described above is periodically performed. Then, a cycle for recording in the high quality mode may be determined, and control may be performed so that recording in the high quality mode is performed in that cycle until the next determination.

<Pattern of recording format in each embodiment>
In each of the above-described embodiments, any of the following may be used as a format for recording a plurality of captured images (frames) and depth distribution information corresponding to some of the frames.
FIG. 28 shows a diagram in which each pattern is imaged with respect to a format for recording a plurality of captured images (frames) and depth distribution information corresponding to some of the frames. That is, as a recording format, as shown in FIG. 28A, a plurality of images 1, 2, and 3 sequentially captured and acquired, and distance maps 1 and 3 respectively corresponding to images 1 and 3 are all separate files. It is recorded as. In this case, information that associates the image with the distance map (or information that there is no corresponding distance map) is recorded in the header of each image file, and the corresponding image information is also recorded on the distance map side.

  Further, as shown in FIG. 28B, each image is associated as a continuous image (may be encoded) as one moving image file, and a plurality of distances are synchronized with the moving image file. Each map may be recorded individually as a distance map file. The distance map records the time code of the corresponding moving image, and can be read out and used as necessary in association with the image when editing the moving image including still image clipping.

  Also, as shown in FIG. 28C, each image is associated as a continuous image (may be encoded) as one moving image file, and a plurality of distances are synchronized with the moving image file. The map may be recorded as a separate file. In this case, the time code synchronized with the time code of each frame of the moving image may be recorded in the corresponding distance map and recorded as one file. Compared to the form of FIG. 28 (B), the distance map can be handled as a pair with a moving image file by making it a single file, and between distance maps can be encoded using a known encoding technique as necessary. As a result, a reduction in data volume can be expected.

  Further, as shown in FIG. 28D, a format in which an image and a distance map corresponding to the image before or after the image are connected and recorded, and one moving image file is formed as a whole may be recorded. In this format, a distance Map that is depth distribution information corresponding to an image on which recording processing has been performed is recorded before or after the image data as metadata of the image data. In this format, there can be advantages such that it can be handled by one file, and the distance map corresponding to the image is adjacent, so that access is easy.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As described above, according to the present invention, it is possible to perform highly convenient image recording while suppressing the recording capacity.

DESCRIPTION OF SYMBOLS 101 ... Imaging optical system 102 ... Imaging part 104 ... Recording mode control part 105 ... Image processing part 106 ... System control part 109 ... Recording part 2210 ... Recording information adjustment part 2606 ... Exposure condition control unit

Claims (28)

An image processing apparatus including a recording unit that acquires a plurality of image data and records the acquired image data on a recording medium,
Acquisition means for acquiring subject depth distribution information corresponding to image data;
A first mode for recording the image data and the depth distribution information corresponding to the image data on the recording medium by the recording unit; and recording the image data by the recording unit without recording the depth distribution information. An image processing apparatus comprising: control means for performing a recording process of the plurality of image data by switching a second mode for recording on a medium.   The control means performs recording processing by switching between the first mode and the second mode during recording of the plurality of image data, and uses the recording means as the moving image to record the recording medium. The image processing apparatus according to claim 1, wherein the image processing apparatus is recorded.   3. The recording unit according to claim 2, wherein the recording unit records depth distribution information corresponding to the plurality of image data and image data subjected to recording processing in the first mode in one moving image file. The image processing apparatus described.   The recording means records depth distribution information corresponding to the image data recorded in the first mode in the moving image file before or after the image data as metadata of the image data. The image processing apparatus according to claim 3.   The control means acquires the depth distribution information including depth information of a main subject and a background among a plurality of subjects, and a difference between depth information of the main subject and background depth information included in the depth distribution information is within a threshold value The image processing apparatus according to claim 1, wherein recording processing is performed in the first mode.   The image processing apparatus according to claim 5, further comprising a threshold value calculation unit that calculates the threshold value from a focal length of an imaging optical system, depth information of a main subject included in the depth distribution information, and an allowable defocus amount. .   6. The control unit according to claim 5, wherein the control unit determines whether or not to perform recording processing in the first mode based on a comparison result between the difference and the threshold value and an F value of the imaging optical system. 6. The image processing apparatus according to 6.   The control means determines whether or not to perform recording processing in the first mode by calculating an evaluation value indicating a dynamic range of a frame and comparing the evaluation value with a threshold value. Item 8. The image processing apparatus according to Item 1.   The control means calculates, as the evaluation value, an exposure step between a main subject region and a background region related to the main subject among a plurality of subjects, and when the exposure step is greater than or equal to the threshold value, The image processing apparatus according to claim 8, wherein a recording process is performed, and the recording process is performed in the second mode when the exposure level difference is smaller than the threshold value. Extraction means for extracting information on a main subject area related to a main subject among a plurality of subjects from image data and depth distribution information recorded in the first mode;
The image processing apparatus according to claim 1, further comprising: an image processing unit that acquires information on a main subject area extracted by the extraction unit and performs image processing on the image data.   The image processing apparatus according to claim 10, wherein the image processing unit determines a background area in the image from information on the main subject area and performs a blurring process on the background image. Extraction means for extracting information on a main subject area related to a main subject among a plurality of subjects from image data and depth distribution information recorded in the first mode;
Image processing means for acquiring information on the main subject region extracted by the extraction means and performing image processing on the image data;
The image processing means determines a main subject region and a background region in an image from information on the main subject region, and performs different gradation correction processing on the main subject region and the background region, respectively. The image processing apparatus according to 1. An adjustment unit that acquires a recording capacity of the moving image and adjusts a recording amount by narrowing down a frame to be recorded in the first mode when the recording capacity is equal to or greater than a threshold;
The image processing apparatus according to claim 1, wherein the adjustment unit performs control to record the narrowed-down image data of the frame and the depth distribution information.   The control means compares a score calculated from at least one of position information and size information of a main subject among a plurality of subjects with a threshold value, and in the first mode in a frame where the score is larger than the threshold value. The image processing apparatus according to claim 1, wherein a recording process is performed, and the recording process is performed in the second mode when the score is equal to or less than the threshold value.   The image processing according to claim 1, wherein the control unit detects a change in a scene from a difference between images, and performs a recording process in the first mode when the change in the scene is detected. apparatus.   The image processing apparatus according to claim 1, wherein the control unit performs a recording process in the first mode for each predetermined frame in a plurality of time-series images acquired by the acquisition unit.   2. The control unit according to claim 1, wherein when the moving image recorded in the first mode is reproduced, the control unit notifies that the image is recorded in the first mode. The image processing apparatus according to any one of 16.   The depth distribution information includes an image shift map based on a parallax amount of a plurality of viewpoint images, a defocus map based on a defocus amount for each region, a distance map indicating a relative distance relationship between each subject in image data, and a TOF method. 18. The image processing apparatus according to claim 1, wherein the image processing apparatus is one of distance information indicating a distance relationship from the acquired imaging apparatus to each subject.   The acquisition unit calculates and acquires an image shift map that is the depth distribution information corresponding to the image data, based on a parallax amount of a pair of parallax images corresponding to the acquired image data. The image processing apparatus according to any one of claims 1 to 18.   The acquisition means calculates and acquires a defocus map, which is the depth distribution information corresponding to the image data, based on a defocus amount for each area of the acquired image data. The image processing apparatus according to any one of 1 to 18.   The acquisition means calculates a relative distance relationship between the subjects as the depth distribution information corresponding to the image data based on a defocus amount for each area of the acquired image data and an imaging optical system or an imaging element. The image processing apparatus according to claim 1, wherein the image processing apparatus calculates and acquires the image processing apparatus.   The acquisition means uses the TOF method that measures the delay time from the light projection to the subject to the reception of the reflected light to measure the distance to the subject, from the imaging device that is the depth distribution information corresponding to the image data The image processing apparatus according to claim 1, wherein a subject distance to each subject is acquired.   23. The image processing apparatus according to claim 1, wherein the control unit performs control to record RAW image data before image processing in the first mode.   The image processing apparatus according to claim 23, wherein the RAW image is an image that has not been subjected to image processing including demosaicing processing, white balance adjustment, color conversion processing, or gamma correction. The image processing apparatus according to any one of claims 1 to 24;
An imaging apparatus comprising: imaging means for imaging a subject. An exposure condition control means for controlling an exposure condition when the subject is imaged by the imaging means and image data is generated;
The exposure condition control means acquires the amount of movement of the subject in the image of the frame in the first mode, and determines the shutter speed and the frame rate related to imaging the next frame of the frame from the amount of movement and the frame interval. 26. The imaging apparatus according to claim 25, wherein one or more of them are determined. A method for controlling an image processing apparatus including a recording unit that acquires and records a plurality of image data on a recording medium,
An acquisition step of acquiring depth distribution information of the subject corresponding to the image data;
A first mode for recording the image data and the depth distribution information corresponding to the image data on the recording medium by the recording unit; and recording the image data by the recording unit without recording the depth distribution information. And a control step of performing recording processing of the plurality of image data by switching to a second mode for recording on a medium. A program causing a computer of an image processing apparatus to execute each step according to claim 27.