JP2016058814A - Image recording apparatus, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image recording apparatus that obtains seamlessly a still image during photographing a moving image while suppressing to the minimum influences of shading caused by pupil division and a change in a depth of field and an influence of SN degradation caused by time division photographing.SOLUTION: In a digital camera, when a still image acquisition instruction is given during photographing a moving image, a plurality of A images photographed on the basis of a predetermined photographing condition is acquired, and a still image is generated by adding and synthesizing a number of images required for generating one still image out of the plurality of A images. From A images and B images obtained by adding and synthesizing a number of images required for generating a moving image frame, a moving image frame is generated.SELECTED DRAWING: Figure 10

Description

  The present invention relates to an image recording apparatus and method, and a program, and more particularly to a control technique for an image recording apparatus that enables still image shooting without interrupting moving image shooting.

  Conventionally, a technique for acquiring a still image (hereinafter referred to as “moving still image”) at an arbitrary timing during moving image shooting has been proposed. With this technology, in a digital camera with a single imaging optical system, when the still image shooting mode is entered by a user instruction, the movie recording is interrupted, and when viewing the movie later, the movie is captured only when the still image is acquired. There is a problem that is stopped and played. Furthermore, when both a moving image and a still image are acquired at the same time, there is a problem that a desirable shutter speed differs between the moving image and the still image. For example, it occurs when a user wants to acquire a still image with a shutter speed longer than 1/60 seconds at the same time as shooting a moving image at 60 fps.

  In order to solve the above problems, techniques shown in Patent Document 1 and Patent Document 2 have been proposed. In the technique disclosed in Patent Document 1, incident light is divided into two light beams by a pupil division optical system, and each is incident on an image pickup device for still image shooting and an image pickup device for moving image shooting. To get.

  In the technique disclosed in Patent Document 2, when acquiring a still image in a moving image, a plurality of continuously shot images are acquired by dividing the exposure time of one frame of a moving image into a plurality of images, and the images are synthesized without alignment. Acquire a moving image as a still image and a synthesized image while performing positioning.

JP 2013-160782 A JP2013-005136A

  However, the technique described in Patent Document 1 has a problem that shading occurs due to pupil division, and luminance unevenness due to shading or noise unevenness due to shading correction occurs in the obtained moving image and still image. In addition, there is a problem that the depth of field of moving images and still images acquired by pupil division becomes deeper than a desired setting value. In patent document 1, there is no description for solving the said subject. Details of shading and depth of field changes due to pupil division will be described later.

  Furthermore, the technique described in Patent Document 2 has a problem that the SN of the composite image is significantly deteriorated because each of the plurality of images taken in time division has a small exposure amount. In addition, since still images are acquired with an image size equivalent to a moving image, the size of the still image obtained is generally smaller than desired, and even if it is enlarged and converted to a size equivalent to a still image, the enlargement process is performed. Cause degradation of resolution.

  The present invention has been made in view of the above problems, and during moving image shooting while minimizing the effects of shading and depth of field changes due to pupil division and SN degradation due to time division shooting. The purpose is to obtain still images seamlessly.

  In order to achieve the above object, an image recording apparatus according to the present invention includes a plurality of pixels having one microlens and a pair of photoelectric conversion units, and each of the plurality of pixels passes through an exit pupil of an optical system. In an image recording apparatus including an imaging device that receives a light beam by each of the pair of photoelectric conversion units by the microlens and outputs a signal, the first image obtained from one of the pair of photoelectric conversion units and the other A moving image frame generating means for generating a moving image frame by adding and synthesizing the second images obtained from the above, a determining means for determining shooting conditions by the image sensor for obtaining a still image, and a moving image by the image sensor When a still image acquisition instruction is given during shooting, a plurality of first images shot based on the shooting conditions determined by the determination unit are acquired, and the plurality of first images are acquired. A still image generating unit that generates a still image by adding and synthesizing a required number of images to generate one still image of the image, and the moving image frame generating unit is necessary for generating a moving image frame. A moving image frame is generated from the first image obtained by adding and combining the number of images and the second image.

  According to the present invention, still images can be obtained seamlessly during moving image shooting while minimizing the effects of shading and depth changes due to pupil division and the effects of SN degradation due to time-division shooting.

It is a block diagram which shows the function structure of the digital camera which is an example of the image recording device which concerns on the 1st Embodiment of this invention. (A) The figure which shows the pixel arrangement | sequence of the imaging part in FIG. 1, (b) It is the figure which expanded the pixel 200 shown to Fig.2 (a). It is a figure which shows the whole imaging surface of an imaging part. FIG. 2B is a cross-sectional view of the S-S cross section of the pixel shown in FIG. 2B viewed from the + y side and an exit pupil plane of the optical system. It is a figure which shows the light reception characteristic in the x-axis cross section of the pixel shown in FIG. (A) The figure which showed the projection image of the pupil division pixel which the pixel arrange | positioned in a peripheral image height part has, (b) It is a figure for demonstrating the pupil intensity distribution of Fig.6 (a). It is a figure which shows the change of the brightness (luminance) of the pupil division pixel A and the pupil division pixel B. FIG. It is a flowchart which shows the operation | movement process of the image process part at the time of the moving image priority mode in 1st Embodiment. It is a figure which shows an example of the determination method of the imaging condition of a time division still image. It is a figure which shows the specific example of the operation | movement process in step S811 of FIG. It is a figure which shows the specific example of the moving image frame production | generation for a display in step S809 of FIG. It is a flowchart which shows the operation | movement process of the image process part at the time of the still image priority mode in 2nd Embodiment. It is a figure which shows the specific example of the operation | movement process in step S1211 of FIG. It is a figure which shows the specific example of the moving image frame for a display in step S1209 of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First embodiment]
FIG. 1 is a block diagram showing a functional configuration of a digital camera which is an example of an image recording apparatus according to the first embodiment of the present invention.

  The digital camera 100 includes a system control unit 101, a ROM 102, a RAM 103, an optical system 104, an imaging unit 105, an A / D conversion unit 106, an image processing unit 107, a recording medium 108, and a system bus 109.

  The system control unit 101 is a CPU, for example, and controls operations of a plurality of functional blocks included in the digital camera 100 by reading a control program from the ROM 102 and developing the program in the RAM 103 and executing it. The ROM 102 is a rewritable nonvolatile memory, and stores parameters necessary for the operation of various functional blocks in addition to the control program. The exit pupil distance is also stored in the ROM 102 as lens information necessary for focus detection and the like.

  A RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for output data in the operation of various functional blocks provided in the digital camera 100.

  The optical system 104 is a lens group that forms a subject image on the imaging unit 105. The optical system 104 also includes a stop, and the amount of light at the time of photographing is adjusted by adjusting the stop. The imaging unit 105 is an imaging element such as a CCD or CMOS sensor, for example, photoelectrically converts an optical image formed on the imaging element by the optical system 104, and outputs the obtained analog image signal to the A / D conversion unit 106. To do. The A / D conversion unit 106 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data to the RAM 103 for storage.

  The image processing unit 107 mainly reads image data from the RAM 103 and performs predetermined image processing. The recording medium 108 is a detachable memory card or the like, and the image data image-processed by the image processing unit 107, the image data A / D converted by the A / D conversion unit 106, and the like are stored as a recorded image. The system bus 109 is used for exchanging signals between various functional blocks.

  FIG. 2A is a diagram illustrating a pixel array of the imaging unit 105 in FIG. FIG. 2B is an enlarged view of the pixel 200 shown in FIG.

  In the imaging unit 105, as shown in FIG. 2A, the pixels 200 are regularly arranged two-dimensionally.

  As shown in FIG. 2B, the pixel 200 includes a microlens 201 and a pair of photoelectric conversion units 203A and 204B (hereinafter referred to as “pupil divided pixels 203A and 204B”).

  FIG. 3 is a diagram illustrating the entire imaging surface of the imaging unit 105, and the pixel 200 is a pixel disposed near the center (reference numeral 300) of the imaging surface. In addition, any pixel of the imaging unit 105 can output a signal obtained by each of the pupil division pixels 203A and 204B receiving light independently. Note that reference numeral 301 in FIG. 3 will be described later.

  When the signal output from the pupil division pixel 203A on the right side of the pixel 200 is A (x, y) and the signal output from the pupil division pixel 204B on the left side is B (x, y), the full aperture signal g (X, y) is expressed by the following (formula 1).

g (x, y) = A (x, y) + B (x, y) (Formula 1)
In the following description, an image composed of signals from the pupil division pixel 203A or the pupil division pixel 204B is referred to as a pupil division image, an image obtained from the pupil division pixel 203A is an A image, and an image obtained from the pupil division pixel 204B is a B image. Call. An image made up of the full aperture signal g is called a full aperture image.

  Next, the concept of pupil division in the pixel 200 will be described with reference to FIG.

  4 is a cross-sectional view of the S-S cross section of the pixel 200 shown in FIG. 2B viewed from the + y side, and an exit pupil plane of the optical system 104. FIG. In FIG. 4, in order to correspond to the coordinate axis of the exit pupil plane, the x-axis and y-axis of the cross-sectional view are inverted with respect to FIG. In FIG. 4, the same parts as those in FIG. 2 are denoted by the same reference numerals.

  The imaging unit 105 is disposed in the vicinity of the imaging plane of the optical system 104. The light flux from the subject passes through the exit pupil 400 of the optical system 104 and enters each pixel. The size of the exit pupil 400 varies depending on the size of the stop and the size of the lens frame that holds the lens.

  The pupil partial areas 401 </ b> A and 402 </ b> B are generally in a conjugate relationship by the light receiving surfaces of the pupil division pixels 203 </ b> A and 204 </ b> B divided into 2 × 1 and the microlens 201. Therefore, the light flux that has passed through each pupil partial region is received by each pupil division pixel that is in a conjugate relationship.

  The exit pupil 400 of the optical system 104 is Np-divided into different pupil partial areas, with the number of pupil divisions being Np = M × N. When the aperture value of the optical system 104 is F, the effective aperture value of the pupil partial region is approximately √ (M × N) F. The pupil region 403 is a pupil region (entire aperture region) that can receive light in the entire pixel 200 when all the M × N divided photoelectric conversion units are combined.

  In this embodiment, since the pupil division number M × N is 2 × 1, the effective aperture value of the pupil partial region is √2F. That is, each of the A image and the B image is a dark image having a depth of field that is one step deeper than the full aperture image.

  FIG. 5 is a diagram showing the light receiving characteristics in the x-axis cross section of the pixel 200 shown in FIG.

  In FIG. 5, the horizontal axis represents the horizontal coordinate on the exit pupil plane of the optical system 104, and the vertical axis represents the light receiving efficiency of the pupil-divided pixel. Further, since this vertical axis is also the transmittance distribution of each aperture stop, it can be regarded as the light beam receiving efficiency of the pupil division pixel. This distribution characteristic of the light beam receiving efficiency is called a pupil intensity distribution. Further, the dotted line in FIG. 5 is a projection of the width of the exit pupil 400.

  4 and 5, the light amounts received by the pupil division pixels 203 </ b> A and 204 </ b> B in the pixels near the center of the imaging surface are equal.

  Paying attention to the pupil intensity distribution 501A of the pupil division pixel 203A, the + x side has a gentle curve due to diffraction blur due to insufficient pupil division performance of the sensor. On the other hand, the -x side has a sharp curve due to the vignetting of the lens frame. Therefore, the pupil intensity distribution 501A is asymmetric with respect to the intensity peak. On the other hand, regarding the pupil intensity distribution 502B of the pupil division pixel 204B, the same can be said because the pupil intensity distribution 501A has a horizontally inverted shape.

  Next, pixels arranged in the peripheral image height portion indicated by 301 of the imaging portion 105 shown in FIG. 3 will be described.

  FIG. 6A is a diagram showing a projected image of pupil-divided pixels included in pixels arranged in the peripheral image height portion. FIG. 6B is a diagram for explaining the pupil intensity distribution of FIG.

  In FIG. 6A, the configuration of the pupil partial regions 601A and 602B and the pupil region 603 is the same as that in FIG. On the other hand, the shape of the exit pupil 600 changes due to vignetting. As a result, shading occurs. Therefore, the center of gravity of the projected image of the exit pupil is closer to the center of the pupil partial region 601A, and the pupil partial region 601A has higher light receiving efficiency and brighter than the pupil partial region 602B, as shown in FIG. It becomes an image. On the other hand, at the opposite peripheral image height, the reverse of this phenomenon occurs, and the pupil partial region 602B has higher light receiving efficiency than the pupil partial region 601A, resulting in a bright image. Here, the relationship between the brightness of the pupil division pixel A and the pupil division pixel B in the peripheral image height portion is shown in FIG.

  FIG. 7 is a diagram illustrating changes in brightness (luminance) of the pupil division pixel A and the pupil division pixel B. FIG.

  In FIG. 7, the solid line represents the pupil division pixel A, and the dotted line represents the pupil division pixel B. In FIG. 7, the pupil division pixel A is brighter than the pupil division pixel B on the −x side. Therefore, the A image has a shallower depth of field than the B image, and the index represents the noise amount of the image. This improves the S / N ratio. On the other hand, the opposite is true on the + x side.

  The above is the concept of pixel arrangement and pupil division in the imaging unit 105, and the description of the problem of shading and occurrence of a depth of field difference associated with pupil division.

  Next, a process for acquiring a still image in moving image executed by the image processing unit 107 in the digital camera 100 will be described.

  In the present embodiment, the processing is more focused on the image quality of the moving image than the acquired still image, and the processing focused on the image quality of the still image will be described in the second embodiment. . Hereinafter, the former process is referred to as a moving image priority mode, and the latter process is referred to as a still image priority mode.

  FIG. 8 is a flowchart showing an operation process of the image processing unit 107 in the moving image priority mode in the first embodiment.

  As a premise, it is assumed that the digital camera 100 is shooting a moving image at the start of this process. In addition, the imaging mode of the imaging unit 105 includes a moving image mode and a still image mode, and these two modes have different imaging conditions such as the number of pixels of a read image and a shutter speed. In the following, since the operation differs between the A image and the B image, the description will be given separately.

  Regarding the processing of the A image, first, in step S801, the image processing unit 107 determines whether there is a still image acquisition instruction. The still image acquisition instruction includes an external operation such as half-pressing a release button (not shown) of the digital camera 100, for example.

  If it is determined in step S801 that there is no still image acquisition instruction, the process proceeds to step S802, and shooting in the moving image mode is performed. On the other hand, if it is determined that there is a still image acquisition instruction, the process advances to step S803 to perform shooting in the still image mode.

  In step S803, the image processing unit 107 determines shooting conditions for the still image mode. The shooting conditions in the still image mode are the shooting conditions when shooting a single still image. Specifically, the shooting sensitivity, aperture value, and shutter speed, and the optimal exposure based on the brightness of the shooting angle of view. Automatically determined to be

  In step S <b> 804, the image processing unit 107 performs time-division still image shooting condition determination processing. Here, there are mainly two reasons for acquiring a still image in time division. One is that the shutter speed in the moving image mode may be different from the shutter speed in the still image mode. The other is to generate a moving image frame with a full-aperture image in this embodiment in order to solve image quality problems (shading and depth of field differences) due to pupil division. This is because it is necessary to generate a moving image frame paired with the B image.

  A difference between the shutter speed in the moving image mode and the shutter speed in the still image mode will be described.

  For example, in the case of a moving image of 60 fps, the shutter speed is 1/60 second or less due to the frame rate, but it is preferable that the shutter speed of a still image when shooting a dark scene is longer than that. Further, when a still image is shot with underexposure rather than appropriate brightness in dynamic range expansion processing or the like, the shutter speed of the still image is preferably shorter than that of a moving image. For the above reasons, the shutter speeds in the movie mode and still image mode may differ, so the shooting speed can be shortened so that both the movie and still images can be handled by changing the number of images to be combined in the subsequent combining process. Shoot time-divisionally. Accordingly, the number of still images acquired in time division is required in addition to the exposure conditions as the time division still image shooting conditions.

  The method for determining the shooting conditions for the time-division still image in step S804 will be described in detail with reference to FIG.

  FIG. 9 is a diagram illustrating an example of a method for determining a shooting condition for a time-division still image.

  FIG. 9 shows a case where the moving image shutter speed Tv_movie is 1/100 seconds and the still image shutter speed Tv_still is 1/60 seconds.

  First, Tv_movie and Tv_still are expressed by a simple integer ratio. That is, in the example of FIG. 9, Tv_still: Tv_movie = 5: 3. Here, the integer ratio is the number of added sheets N_still, N_movie necessary for generating a moving image and a still image. Furthermore, a value obtained by dividing the desired shutter speeds for moving images and still images by the respective added number is the shutter speed Tv_div for time-division still images. In the example of FIG. 9, Tv_div = 1/300 seconds.

  In this embodiment, it is necessary to acquire five time-division still images in order to generate one still image, but three time-division still images are required to generate one frame of moving image. is there. Therefore, in order to generate moving image frames continuously, 6 time-division still images are acquired, 5 of them are combined to generate a still image, and 3 still images are combined to generate a 2 frame moving image. Generate a frame. Therefore, the number of time-division still image acquisition is six. As described above, the shooting conditions, the number of sheets required to generate one still image and the number of sheets required to generate a moving image frame are determined based on the signal level of the signal output from the image sensor.

  Returning to FIG. 8, in step S805, the image processing unit 107 acquires time-division still images according to the shooting conditions determined by the above-described method, and in step S806, determines whether a predetermined number of time-division still images have been acquired. To do. If it is determined that the image has not been acquired, the image processing unit 107 returns to step S805 and continues acquiring the still image. On the other hand, if it is determined that the image has been acquired, the image processing unit 107 turns off (cancels) the still image acquisition instruction (step S807), and transitions to the moving image shooting mode.

  On the other hand, in the B image, moving image shooting is continuously performed in step S808 regardless of whether there is a still image acquisition instruction.

  In the display moving image frame generation processing in step S809, when there is a still image acquisition instruction, only the B image moving image frame is input, and when there is no still image acquisition instruction, the A image and B image moving image frames are input. Is done. Note that the moving image frames and time-division still images acquired for the A and B images are stored in the ROM 102. Details of the display moving image frame generation processing in step S809 will be described later.

  Next, in step S810, the image processing unit 107 determines whether there is a moving image recording stop instruction. The moving image recording stop instruction includes, for example, an external operation such as pressing a moving image shooting button (not shown) of the digital camera 100.

  If it is determined in step S810 that there is no moving image recording stop instruction, the flow returns to step S801 to continue moving image shooting. On the other hand, if it is determined that there is a moving image recording instruction, the main recording moving image and still image creation processing in step S811 is executed.

  In step S811, the image processing unit 107 generates a moving image and a still image for main recording from the moving image frame and the time-division still image recorded in the ROM 102. A specific example of the operation process in step S811 is shown in FIG.

  FIG. 10 shows a process of generating a moving image and a still image for main recording when an arbitrary seven frames of a moving image are extracted and a still image acquisition instruction is issued during moving image shooting.

  In FIG. 10, the horizontal axis indicates time t, and the vertical axis indicates the resolution (pixel) in the vertical direction of the acquired image. Also, the moving image frames obtained from the A and B images are A1, A2,... A7, B1, B2,... B7, respectively, and acquisition of time-division still images is started at the timing of the third frame.

  The time-division still image is acquired as an A image, and the shooting conditions are the same as in the example of FIG. That is, the shutter speeds of moving images, still images, and time-division still images are 1/100 seconds, 1/60 seconds, and 1/300 seconds, respectively, and the number of acquired time-division still images is six. Note that the acquired time-division still images are AS1, AS2,. Furthermore, the resolution (pixels) in the vertical direction of the obtained moving image frame and still image is 2000 pixels and 8000 pixels, respectively.

  First, as described above, six time-division still images are shot, and two moving image frames are generated. Therefore, the period during which the still image acquisition instruction is issued is the third frame and the fourth frame. Therefore, the other 1, 2, 5, 6, and 7 frames acquire moving image frames for both the A image and the B image, and the recorded moving image is a full aperture image obtained by adding the A and B images of each frame.

  Next, 3 and 4 frames that are in the still image shooting mode will be described.

  First, the recorded still image is obtained by combining five still images AS1 to AS5 among six time-division still images that are continuously shot. Note that the composition processing method does not matter, but here, it is assumed that five time-division still images are added and combined to generate one still image.

  Next, generation of the recorded moving image in the still image shooting mode will be described.

  In the B image, moving image frames B3 and B4 are obtained, whereas in the A image, a time-division still image is taken. Therefore, it is necessary to obtain a corresponding moving image frame by addition synthesis processing. Further, since the A image is taken at a resolution (pixel) equivalent to a still image, it is necessary to reduce the number of pixels to an equivalent of a moving image frame. Here, AS1 to AS6 are reduced by bilinear reduction processing. The reduced A image still image is referred to as reduced AS1 to reduced AS6, and the A image frames corresponding to the moving image frames B3 and B4 are referred to as A3 'and A4'. In this case, as shown in FIG. 10, A3 'is generated by addition synthesis of reduced AS1 to reduced AS3, and A4' is generated by addition combination of reduced AS4 to reduced AS6. Then, A3 'and B3, and A4' and B4 frames are added to obtain a moving image frame of a full aperture image.

  The above is the operation in step S811, and the moving image and the still image to be recorded are generated in step S811. The generated main recording moving image and still image are recorded in the ROM 102, and the processing ends.

  Next, details of the display moving image frame generation processing in step S809 in FIG. 8 will be described with reference to FIG.

  Since the generation of this recorded moving image involves a synthesis process that acquires and synthesizes a plurality of time-division still images, it is considered difficult to display live on the display unit of the digital camera from the viewpoint of real-time processing. . Therefore, although it is different from the moving image to be recorded, it is necessary to easily generate a moving image for display for confirming the moving image taken by the user in real time.

  FIG. 11 is a diagram showing a specific example of the display moving image frame generation processing in step S809 in FIG.

  As shown in FIG. 11, for the frames 1, 2, 5, 6, and 7 in which still image shooting is not performed, the full aperture moving image obtained by adding the A image and the B image is used as the moving image for display as in the case of the main recording moving image. For the frames 3 and 4 for taking still images, only the moving image frames B3 and B4 acquired as B images are used.

  However, since the moving image frames B3 and B4 are images on one side of the pupil division, there are differences in shading and depth of field due to the above-described pupil division.

  Therefore, as shown in FIG. 11, first, gain is applied to B3 and B4 to correct shading due to pupil division. Specifically, it is multiplied by a surface gain having a characteristic that cancels the influence of shading shown in FIG. Next, the defocus amount of the image is calculated from the pair of the already acquired moving image frames of the A image and the B image, and the blur for compensating for the difference in depth of field according to the defocus amount is calculated by B3 after gain multiplication. , B4.

  DefMAP in FIG. 11 is a defocus map representing the distribution of the defocus amount. As a specific method of the defocus map process, for example, a technique disclosed in Japanese Patent Laid-Open No. 2008-015754 can be cited.

  In the example of FIG. 11, the pair of moving image frames is A1 and B1 two frames before, but the number of frames before the defocus map is generated is actually determined by the processing time required for the blur addition process. Since the angle of view changes every moment due to moving image shooting, it is desirable to periodically update the defocus map.

  The above is the description of the processing in step S809. In step S809, a moving image frame for display is generated and displayed live on the display unit of the digital camera.

  According to the first embodiment, a still image and a moving image are acquired with the A image and the B image, which are pupil-divided images, respectively. It is possible to acquire a still image without interrupting shooting. In particular, in the present embodiment, moving image shooting is continuously performed with one of the pupil-divided images, and a paired frame is formed from the still image of the other image that has been time-divided and added. As a result, a moving image corresponding to an all-aperture moving image can be acquired, and in the moving image, adverse effects on image quality due to pupil division (shading, depth of field difference) can be suppressed.

[Second Embodiment]
In the first embodiment, the method for seamlessly acquiring a still image in a moving image giving priority to the image quality of the moving image has been described. In the second embodiment, a method (still image priority mode) for prioritizing the image quality of still images while seamlessly acquiring still images in moving images will be described. Hereinafter, the same parts as those of the first embodiment are denoted by the same reference numerals and the description thereof is omitted.

  In the second embodiment, the processing content of the image processing unit 107 is different from that in the first embodiment.

  FIG. 12 is a flowchart showing an operation process of the image processing unit 107 in the still image priority mode in the second embodiment.

  First, if it is determined from the determination result in step S1201 that there is no still image acquisition instruction, the image processing unit 107 acquires images in the moving image mode for both the A image and the B image, as in the first embodiment. The process proceeds to step S1207) and step S1209.

  On the other hand, if it is determined in step S1201 that there is a still image acquisition instruction, the image processing unit 107 acquires a still image as the A image (step S1202), and acquires a time-division still image as the B image (step S1203). Step S1206). Here, the still image of the A image is not a time-division shooting but a single shooting at a predetermined shutter speed. Further, the method for determining the time-division still image shooting conditions in steps S1203 and S1204 is the same as that in steps S803 and S804 of FIG. 8 described above.

  Next, the image processing unit 107 acquires time-division still images until the number of time-division still images acquired reaches a predetermined number in steps S1205 and S1206. When the predetermined number is reached, the moving image and still image for main recording are generated through steps S1208 to S1210 (step S1211). The acquired still image is recorded in the ROM 102. Details of the display moving image frame generation processing in step S1209 will be described later.

  In step S <b> 1211, the image processing unit 107 generates a moving image and a still image for main recording from the still image and the time-division still image recorded in the ROM 102. A specific example of the operation process in step S1211 is shown in FIG. Note that the moving image and still image shooting conditions and the number of pixels in the vertical direction of the moving image and still image in FIG. 13 are the same as those shown in FIG. 10 in the first embodiment.

  In FIG. 13, first, as in the first embodiment, in the 1, 2, 5, 6, and 7 frames where there is no still image acquisition instruction, both the A image and the B image are shot in the moving image mode. A full aperture moving image obtained by adding the A image and the B image is recorded as the main recording moving image.

  Next, in 3 and 4 frames with a still image acquisition instruction, a still image is captured without performing time division for the A image, and a time division still image is captured for the B image. Here, the still image acquired with the A image is referred to as AS, and the time-division still images acquired with the B image are referred to as BS1 to BS6.

  First, for the recorded moving image, time-division still images BS1 to BS6 continuously shot are reduced (hereinafter referred to as “reduced BS1 to reduced BS6”), and reduced BS1 to reduced BS3 are added and combined. Then, a B3 ′ frame corresponding to the third frame of the moving image is generated, and the reduced BS4 to the reduced BS6 are added and combined to generate a B4 ′ frame corresponding to the fourth frame of the moving image.

  The still image recorded is obtained as a full-opening still image by adding the still image AS obtained from the A image and the still image of the B image corresponding to the AS. Here, the still image of the B image is obtained by adding and synthesizing the time-division still images BS1 to BS5 obtained by the time-division processing.

  The above is the operation in step S1211. In step S1211, the moving image and the still image to be recorded are generated. The generated main recording moving image and still image are recorded in the ROM 102, and the processing is completed.

  Next, details of the display moving image frame generation processing in step S1209 will be described.

  FIG. 14 is a diagram showing a specific example of the display moving image frame generation processing in step S1209 of FIG.

  As shown in FIG. 14, for frames 1, 2, 5, 6, and 7 where still image shooting is not performed, a full aperture moving image obtained by adding the A and B images is used as a moving image for display, as in the case of the recorded moving image. For frames 3 and 4 for taking a still image, only BS1 and BS4 among the time-division still images acquired as the B image are used.

  However, since the time-division still images BS1 and BS4 are images on one side of the pupil division, there are differences in shading and depth of field due to the above-described pupil division. Furthermore, BS1 and BS4 are time-divided, and the shutter speed is short, so the image is darker than the appropriate exposure (in this embodiment, 1/3 the brightness of the appropriate exposure).

  Therefore, as shown in FIG. 14, first, a gain is applied to BS1 and BS4 to make the brightness equivalent to an appropriate exposure, and shading correction by pupil division is performed. Next, the defocus amount of the image is calculated from a pair of the already obtained moving image frames of the A image and the B image, and a blur for compensating for the difference in depth of field according to the defocus amount is obtained by BS1 after gain multiplication. , BS4. Note that the blur addition process is the same as that described with reference to FIG. 11 in the first embodiment.

  According to the second embodiment, a still image and a time-division still image are respectively acquired from the A-image and the B-image that are pupil-divided images, and moving image shooting is performed particularly by appropriately combining the time-division still images. A still image can be acquired without interruption. In particular, in this embodiment, still image shooting is performed without time-sharing on one of the pupil-divided images, and a paired still image is created from the still images of the other time-divided image and added. As a result, a still image corresponding to an all-aperture still image can be acquired, and in the still image, adverse effects on image quality (shading, depth of field difference) due to pupil division can be suppressed.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

101 System control unit 102 ROM
103 RAM
104 Optical system 105 Imaging unit 106 A / D conversion unit 107 Image processing unit 108 Recording medium 109 System bus

Claims (12)

Consists of a plurality of pixels having one microlens and a pair of photoelectric conversion units, and each of the plurality of pixels receives light beams that have passed through the exit pupil of the optical system by the microlens to each of the pair of photoelectric conversion units. In an image recording apparatus including an image pickup device that outputs a signal,
A moving image frame generating means for generating a moving image frame by adding and synthesizing a first image obtained from one of the pair of photoelectric conversion units and a second image obtained from the other;
Determining means for determining shooting conditions of shooting by the imaging device for acquiring a still image;
When a still image acquisition instruction is given during moving image shooting by the image sensor, a plurality of first images shot based on the shooting conditions determined by the determination unit are acquired, and the plurality of first images are acquired. A still image generating means for generating a still image by adding and synthesizing a number necessary to generate one still image,
The image recording apparatus, wherein the moving image frame generating unit generates a moving image frame from the first image and the second image obtained by adding and synthesizing a number necessary for generating the moving image frame.   The determination means determines the number of sheets necessary for generating the still image and the moving image frame as the shooting condition based on a simple integer ratio of the moving image shutter speed and the still image shutter speed. The image recording apparatus according to claim 1.   The determination unit divides the shutter speed for acquiring the plurality of still images by dividing the shutter speed of the moving image by a necessary number for generating the moving image frame, or the shutter speed of the still image. The image recording apparatus according to claim 2, wherein the image recording apparatus is calculated by dividing by a number necessary for generating the still image.   When the still image acquisition instruction is issued, the second image is reduced to the number of pixels equivalent to a moving image frame, and after applying a predetermined gain, a predetermined blur addition process is performed to display a moving image for display. 4. The image recording apparatus according to claim 1, further comprising display moving image frame generation means for outputting as a frame. Consists of a plurality of pixels having one microlens and a pair of photoelectric conversion units, and each of the plurality of pixels receives light beams that have passed through the exit pupil of the optical system by the microlens to each of the pair of photoelectric conversion units. In an image recording apparatus including an image pickup device that outputs a signal,
A moving image frame generating means for generating a moving image frame by adding and synthesizing a first image obtained from one of the pair of photoelectric conversion units and a second image obtained from the other;
Determining means for determining shooting conditions of shooting by the imaging device for acquiring a still image;
When a still image acquisition instruction is given during moving image shooting by the imaging element, a plurality of second images shot based on the shooting conditions determined by the determination unit are acquired, and the plurality of second images are acquired. A still image generating means for generating a still image by adding and synthesizing a still image obtained by adding and synthesizing a number necessary for generating one still image and the first image;
The moving image frame generating means generates the moving image frame from a second image obtained by adding and combining the necessary number of frames for generating a moving image frame.   The determination means determines the number of sheets necessary for generating the still image and the moving image frame as the shooting condition based on a simple integer ratio of the moving image shutter speed and the still image shutter speed. The image recording apparatus according to claim 5.   The determination unit divides the shutter speed for acquiring the plurality of still images by dividing the shutter speed of the moving image by a necessary number for generating the moving image frame, or the shutter speed of the still image. The image recording apparatus according to claim 6, wherein the image recording apparatus is calculated by dividing by a number necessary for generating the still image.   When a still image acquisition instruction is issued, a display moving image frame generation is performed in which a predetermined gain is applied to the second image, and then an image subjected to a predetermined blur addition process is output as a display moving image frame. The image recording apparatus according to claim 5, further comprising a unit.   The shooting conditions, the number of sheets required to generate the one still image and the number of sheets required to generate the moving image frame are determined based on a signal level of a signal output from the image sensor. The image recording apparatus according to claim 1, wherein: Consists of a plurality of pixels having one microlens and a pair of photoelectric conversion units, and each of the plurality of pixels receives light beams that have passed through the exit pupil of the optical system by the microlens to each of the pair of photoelectric conversion units. In an image recording method of an image recording apparatus including an image pickup device that outputs a signal,
A moving image frame generating step of generating a moving image frame by adding and combining a first image obtained from one of the pair of photoelectric conversion units and a second image obtained from the other;
A determination step of determining shooting conditions of shooting by the imaging device for acquiring a still image;
When a still image acquisition instruction is given during moving image shooting by the imaging element, a plurality of first images shot based on the shooting conditions determined in the determination step are acquired, and the plurality of first images is acquired. A still image generating step of generating a still image by adding and synthesizing a required number of images to generate one still image,
The moving image frame generating step generates a moving image frame from the first image and the second image obtained by adding and synthesizing a number necessary for generating a moving image frame. Consists of a plurality of pixels having one microlens and a pair of photoelectric conversion units, and each of the plurality of pixels receives light beams that have passed through the exit pupil of the optical system by the microlens to each of the pair of photoelectric conversion units. In an image recording method of an image recording apparatus including an image pickup device that outputs a signal,
A moving image frame generating step of generating a moving image frame by adding and combining a first image obtained from one of the pair of photoelectric conversion units and a second image obtained from the other;
A determination step of determining shooting conditions of shooting by the imaging device for acquiring a plurality of still images;
When a still image acquisition instruction is given during moving image shooting by the image pickup device, a plurality of second images shot based on the shooting conditions determined in the determination step are acquired, and the plurality of second images is acquired. A still image generating step of generating a still image by adding and synthesizing a still image obtained by adding and synthesizing a required number of images to generate one still image and the first image.
The moving image frame generating step generates the moving image frame from a second image obtained by adding and synthesizing a necessary number of frames for generating a moving image frame.   A computer-readable program for causing a computer to execute the image recording method according to claim 10 or 11.

Images