JP2016100868A - Image processing apparatus, image processing method, and program, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem in which, when converting a frame rate of a recorded moving picture, using the same frame for a plurality of times gives an unnatural feeling in the change of a focal plane.SOLUTION: An image processing apparatus includes: refocusing means for generating a refocused image at least on one virtual focal plane from light field data of a reference frame by making one frame of moving picture data composed of a plurality of frames of light field data containing ray route information and light quantity information to be a reference frame; interpolation means for interpolating the refocused image to positions neighboring the reference frame in time series; and virtual focal plane determination means for determining a virtual focal plane of the refocused image on the basis of the change in focal plane of the plurality of frames containing the reference frame.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus that can generate an image of an arbitrary focal plane by reconstructing shooting data, and more particularly to an image processing apparatus that performs processing such as frame rate conversion using the reconstructed image.

  For example, Patent Document 1 discloses a technique for converting the frame rate of a moving image shot with a digital camera or the like. In the technique according to this document, an interpolation frame image is generated from temporally adjacent frame images, and the frame rate is converted by adding the interpolation frame image to the original moving image. This enables smooth display in the moving image after the frame rate conversion.

  On the other hand, light beams that have passed through different regions of the exit pupil of the imaging optical system are acquired by an imaging device, and the obtained signals are combined to generate image data of an image at an arbitrary focal plane in one shooting. A technique is disclosed (Patent Document 2). A camera having such a function is generally known as a light field camera.

JP 2011-244330 A JP 2007-004471 A

  In the above-mentioned Patent Document 1, interpolation is performed even with respect to a change in focal plane by interpolating the movement of a subject between frames, but this is not always appropriate as focal plane interpolation. When the interpolated frame is generated by addition averaging, it appears that the change in the focal plane is smooth, but actually there are a plurality of focal planes.

  Further, in Patent Document 2, an image of an arbitrary focal plane can be generated from a single shooting data, but there is no mention regarding interpolation of time-series focal plane changes in moving images.

  In view of the above problems, an object of the present invention is to provide an image processing apparatus that makes it possible to smoothly change a focal plane of a moving image after the addition of an interpolation frame in the moving image.

  The image processing apparatus uses, as a reference frame, one frame of moving image data composed of a plurality of frames of light field data including light path information and light amount information, and reconstructs at least one virtual focal plane from the light field data of the reference frame. Refocusing means for generating a focus image; interpolation means for interpolating the refocused image as an interpolation frame of moving image data at a position adjacent to the reference frame in time series; and focal planes of a plurality of frames including the reference frame Virtual focal plane determining means for determining a virtual focal plane of the refocused image based on the change.

  ADVANTAGE OF THE INVENTION According to this invention, in the addition of the interpolation frame in a moving image, the image processing apparatus which makes it possible to smooth the change of the focal plane of the moving image after the addition of an interpolation frame can be provided.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is a figure for demonstrating the structure of the image pick-up element in a light field camera. It is a figure which shows typically the relationship between a division pixel and incident light. It is a figure which shows typically the refocus process of light field data. It is a figure which shows the flowchart of the frame rate conversion processing operation in the image processing apparatus concerning 1st Example of this invention. It is a figure which shows typically the determination of the focal plane and frame interpolation in the frame rate conversion concerning 1st Example of this invention. It is a block diagram which shows the structure of the imaging device concerning the 2nd Example of this invention. It is a figure which shows the flowchart of the frame rate conversion processing operation in the imaging device concerning 2nd Example of this invention. It is a figure for demonstrating the interpolation frame of a defect | deletion frame in the still image imaging | photography process in the moving image imaging | photography concerning 3rd Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  An image processing apparatus according to the first embodiment of the present invention will be described with reference to FIGS. The present invention uses a reconstructed image (refocused image) in a virtual focal plane obtained from light field data for a predetermined process including interpolation of a frame of the moving image data with a frame image generated from the moving image data. It is. In the first embodiment, frame rate conversion processing is used as an example of predetermined processing, and a reconstructed image generated from moving image data (light field data) captured by an imaging device that is a light field camera is used as an interpolation frame. . This makes it possible to smooth the change in the focal plane of the moving image after the frame rate conversion.

  FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus 100 according to the present embodiment. In FIG. 1, the image processing apparatus 100 includes a setting unit 101, an input unit 102, a frame rate conversion unit 103, and an output unit 107.

  The setting unit 101 is an operation unit such as various switches or a graphical user interface, and accepts a frame rate setting after conversion. The frame rate set by the setting unit 101 is output to the virtual focal plane determination unit 104. The input unit 102 reads the designated moving image data (light field data) and outputs it to each unit of the frame rate conversion unit 103.

  The frame rate conversion unit 103 includes a virtual focal plane determination unit 104, a refocus image generation unit 105, and a conversion data generation unit 106, and converts moving image data recorded by moving image shooting from a low frame rate to a high frame rate. . The virtual focal plane determination unit 104 determines a virtual focal plane of an interpolation frame to be interpolated by frame rate conversion based on focal plane information given to input moving image data and a set frame rate, and generates a refocus image Output to the unit 105.

  Based on the input moving image data and the virtual focal plane determined by the virtual focal plane determination unit 104, the refocus image generation unit 105 generates a refocus image on the virtual focal plane by shift addition calculation. The refocus image generated by the refocus image generation unit 105 is output to the conversion data generation unit 106.

  Here, the configuration of the image sensor in the light field camera and the principle of refocus will be described with reference to FIGS.

  FIG. 2A is an enlarged view of a part of the image sensor in the light field camera, and is a diagram illustrating a correspondence relationship between the microlens array and the pixels. A circle 201 in the figure indicates a microlens. A rectangular area corresponding to the microlens 201 is a pixel. The imaging device has a two-dimensional array of microlenses 201 and pixels shown in FIG.

  Next, the configuration of the pixel will be described. FIG. 2B is a diagram in which one pixel on the image sensor is observed from above. FIG. 2C is a cross-sectional view of the pixel along the line P-P ′ shown in FIG. In FIG. 2, the same parts are denoted by the same reference numerals.

  The micro lens 201 condenses the light beam output from the imaging optical system. The microlens 201 is disposed in the vicinity of the imaging plane of the imaging optical system. Reference numerals 202 to 205 denote photoelectric conversion units. Hereinafter, these photoelectric conversion units are referred to as divided pixels. As shown in FIG. 3, the divided pixel 202 and the divided pixel 203 respectively receive light beams that have passed through specific areas 302 and 303 on the exit pupil 301 of the imaging optical system shown in FIG. 3.

  FIG. 4 is a diagram for explaining the operation of the refocus processing for the light field data acquired by the light field camera. FIG. 4A shows a relationship between incident light and a focal plane in a region where an image sensor is present. The light flux that has passed through the exit pupil regions 302 and 303 described in FIG. 3 is pupil-divided by the microlens and received by the divided pixels. The divided pixels 401, 403, 405, 407, and 409 receive light beams that have passed through the exit pupil region 302, and the divided pixels 402, 404, 406, 408, and 410 receive light beams that have passed through the exit pupil region 303, respectively.

  FIG. 4B is a diagram schematically showing a light beam received by each divided pixel. By adding the divided pixels 401 and 402, 403 and 404, 405 and 406, 407 and 408, 409 and 410, respectively, under the same lens, it is possible to obtain an image in which light beams represented by the same type of line are imaged. .

  FIG. 4D shows an example in which received signal groups are shifted and added. Since each divided pixel has the light path information and light quantity information shown in FIG. 4A, the signal obtained by the shift addition is obtained by imaging a light beam represented by a different kind of line. It is done. This can be handled by approximating a signal that can be acquired by the virtual focal plane 411 as shown in FIG.

  FIG. 4F is an example in which addition is performed by shifting in the reverse direction, and can be handled as a signal approximate to the signal obtained on the virtual focal plane 412 as shown in FIG.

  Similarly, by performing shift addition in the vertical direction and performing processing for all the pixels, a refocus image in which the focal plane is changed to an arbitrary position can be obtained.

  By using shift addition in this way, it is possible to generate a refocus image focused on an arbitrary subject area.

  Here, the refocus processing has been described using an example in which there are four divided pixels with respect to the microlens, but the scope of the present invention is not limited to this. It is only necessary to obtain a refocus image corresponding to the synthesis virtual focal plane by performing the recombination processing in accordance with the pixel structure such as the number of divided pixels and the configuration of the optical system. Details are well-known techniques, and are therefore omitted.

  Further, for the sake of explanation, an example of the refocus processing by a simple shift addition operation has been shown, but the present invention is not limited to this, and a refocus image at each virtual focal plane may be obtained using another method. . For example, as in Patent Document 2, processing by weighted shift addition may be performed.

  Returning to FIG. 1, the conversion data generation unit 106 generates data obtained by converting the frame rate from the refocus image generated by the refocus image generation unit 105 and the input moving image data. The generated conversion data is output to the output unit 107. The detailed operation of each part of the frame rate conversion unit 103 will be described later.

  The output unit 107 performs codec processing such as data compression by encoding on the conversion data generated by the frame rate conversion unit 106, and records the generated moving image data in a storage unit such as a memory.

  The configuration of the image processing apparatus 100 in the embodiment has been described above. Next, the frame rate conversion operation in the present embodiment will be described with reference to FIGS.

  FIG. 5 is a diagram showing a flowchart of the conversion processing operation when moving image data is converted from a low frame rate to a high frame rate. FIG. 6 is a diagram schematically showing a process in the case of converting the frame rate from 30 fps to 120 fps. FIG. 6A shows an example of change of the focal plane (focus position) with respect to the shooting time when the input moving image data is shot. Here, the focal plane is a value calculated from the optical system information of the lens and the focus lens position, and basically corresponds to the distance to the main subject. FIG. 6B shows an example of a recording frame when the moving image recording frame rate of the input moving image data is 30 fps. FIG. 6C shows an example of a frame when the frame rate conversion unit 103 converts the input moving image data to a frame rate of 120 fps.

  Next, processing of each step in the flowchart of FIG. 5 will be described with reference to FIGS. 6 (a), (b), and (c). The processing operation according to the flowchart of FIG. 5 is realized by a CPU of a control unit (not shown) of the image processing apparatus 100 executing a control program stored in a memory included in the control unit.

  In step S501, the virtual focal plane determination unit 104 reads the setting value of the conversion frame rate input by the setting unit 101. When the conversion frame rate is set, the process proceeds to step S502.

  In step S502, the virtual focal plane determination unit 104 determines the number of interpolation frames to be interpolated from the moving image recording frame rate of the input data read by the input unit 102 and the converted frame rate set in step S501. As shown in FIGS. 6B and 6C, when the recording frame rate is 30 fps and this is converted to 120 fps, the number of interpolation frames to be interpolated is 611 to 613, 615 to 617, and 619 to 621. Thus, it becomes 3 frames. Here, the frames 607, 608, and 609 before the frame rate conversion correspond to the converted frames 610, 614, and 618, respectively, and are the same frames. When the number of interpolation frames to be interpolated is determined by the virtual focal plane determination unit 104 in step S502, the process proceeds to step S503.

  In step S503, the conversion data generation unit 106 determines whether the frame generated as an output frame is in the order of moving image recording frames that do not require interpolation processing (hereinafter referred to as reference frames) or the order of interpolation frames to be interpolated. Determine. If an interpolation frame to be interpolated as an output frame is generated, the process proceeds to step S504. If a moving image recording frame is used as an output frame as it is, the process proceeds to step S506.

  In step S504, the virtual focal plane determination unit 104 determines virtual focal planes of a plurality of interpolation frames to be interpolated. As shown in FIGS. 6A and 6B, when the focus lens is driven during moving image shooting and the focal plane 601 changes, the continuity of the change is maintained and the converted moving image The virtual focal plane of the interpolation frame to be interpolated is determined so that the change of the focal plane becomes smooth. The virtual focal plane of the interpolation frame to be interpolated is obtained by linearly interpolating the focal planes in two frames adjacent in the time series before conversion. Here, an example will be described in which a virtual focal plane of an interpolation frame to be interpolated between moving image recording frames 607 and 608 is determined. The focal plane change between the frames is linearly interpolated from the focal planes 602 and 603 of the recording frames 607 and 608, and virtual focal planes 604 to 606 of interpolation frames to be interpolated are determined according to the number of interpolation frames determined in step S502. . When the virtual focal plane is determined in step S504, the process proceeds to step S505.

  In step S505, the refocus image generation unit 105 generates a refocus image based on the virtual focal plane determined in step S504, and generates a plurality of interpolation frames to be interpolated. Here, the moving image recording frame 607 of the input moving image data is used as a reference frame, and an interpolated frame interpolated between this frame and the next frame 608 is refocused with respect to the reference frame 607 based on the virtual focal planes 604 to 606. The interpolation frames 611 to 613 are generated.

  In step S506, the conversion data generation unit 106 generates frame rate conversion data from the interpolation frame and the moving image recording frame generated by the refocus processing based on the virtual focal plane, and outputs the frame rate conversion data to the output unit 107.

  In step S507, the output unit 107 performs codec processing such as data compression by encoding on the frame rate conversion data generated in step S506, and records the data in a storage medium (not illustrated) of the image processing apparatus 100. When the conversion data recording process in step S507 ends, the process proceeds to step S508.

  In step S508, it is determined whether unconverted data remains in the input unit 102 or not. If unconverted data remains, the process returns to step S503, and if the conversion of all data is completed, the frame rate conversion process ends.

  As described above, according to the present embodiment, the change of the focal plane is smooth in the conversion from the low frame rate to the high frame rate by interpolating the frame generated by interpolating the focal plane change between the frames. It is possible to generate a simple moving image.

  In the present embodiment, when the virtual focal plane of the interpolation frame to be interpolated is determined, an example in which linear interpolation is performed between two frames adjacent in time series has been described. However, the present invention is not limited to this, and another interpolation calculation is performed. May be obtained by interpolating the focal plane between frames. For example, processing by Lagrange interpolation may be performed using focal plane information in two or more frames.

  Further, the interpolation frame is based on the moving image recording frame, and after that, the frame generated by refocusing the reference frame by the number of interpolation frames is interpolated. However, the present invention is not limited to this. An interpolation frame obtained by refocusing the reference frame by the number of interpolation frames before, before, after, or after the reference frame may be interpolated.

  Further, although the present embodiment has been described with reference to the focal plane interpolation, an interpolation frame to be interpolated may be generated in consideration of the interpolation with respect to the motion of the subject. For example, interpolation is performed by shifting each divided pixel so that the focal planes of two adjacent reference frames coincide with each other and then performing addition-average synthesis, and then performing refocus processing so as to interpolate the focal plane. Generate a frame. By generating the interpolation frame in such a procedure, it is possible to interpolate the movement of the subject and also to interpolate the change of the focal plane without having a plurality of focal planes.

  In addition, although an example in which the refocus processing when generating an interpolation frame to be interpolated is performed on the entire frame has been shown, the present invention is not limited to this, and processing may be performed for each region such as a subject and a background.

  If the focal plane has not changed between moving image recording frames to be converted, or if the reference frame is used as an output frame as it is, the processing of virtual focal plane determination S504 and refocus image generation S505 is not performed. . As a result, the processing load and power consumption can be reduced.

  In the present embodiment, the hardware configuration is described as an example. However, the present invention is not limited to this, and the operation flow that characterizes the present invention may be realized as software.

  A second embodiment of the present invention will be described with reference to FIGS. In this embodiment, an example in which the frame rate conversion unit 103 of the image processing apparatus 100 described in the first embodiment is applied to an imaging apparatus will be described. By applying the image processing apparatus of the present invention to an imaging apparatus, it is possible to perform frame rate conversion processing with smooth focal plane change simultaneously with moving image shooting, and processing such as reading of recorded data is reduced, and power consumption is also reduced. Be expected.

  FIG. 7 is a block diagram showing a configuration of the image pickup apparatus 700 according to the present embodiment, and FIG. 8 is a flowchart showing a moving image shooting operation in the image pickup apparatus according to the present embodiment. Note that portions already described in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted here.

  In FIG. 7, reference numeral 701 denotes an imaging optical system including a focus lens. The imaging optical system 701 receives light from a subject and guides the light to the imaging element 702 through a plurality of lens groups and a diaphragm. The focus lens included in the imaging optical system 701 is driven based on a drive command from a lens drive control unit (not shown) or a rotation operation of the focus ring, and enables adjustment of the focus position. The light beam that has passed through the imaging optical system 701 forms an optical image of the subject on the imaging element 702.

  As described in the first embodiment, the imaging element 702 includes a plurality of microlenses arranged in a lattice pattern, and a plurality of divided pixels are provided below each microlens. The image sensor 702 converts the received light flux into an electrical signal and outputs it as an analog signal to the A / D converter 703.

  An A / D converter 703 is a converter for converting an analog signal from the image sensor 702 into a digital signal. The A / D converter 703 outputs the converted digital signal to the image processing unit 704.

  The image processing unit 704 is a group of programs executed by a circuit having an image processing function including a focus detection unit 705 described later and the frame rate conversion unit 103 described in the first embodiment, and a system control unit 706. The image processing unit 704 performs image processing on the digital signal group acquired by the A / D conversion unit 703 based on a control command from the system control unit 706 to generate image data for recording and display. Image processing performed by the image processing unit 704 is development processing such as magnification chromatic aberration correction, distortion correction, noise reduction processing, and gamma processing.

  The focus detection unit 705 performs correlation calculation based on the digital signal group obtained via the A / D converter 703 and calculates an evaluation value used for lens position adjustment. The evaluation value calculated by the focus detection unit 705 is converted into a lens driving command via the system control unit 706 and used for driving the imaging optical system 701.

  Here, the detailed operation of the focus detection unit 705 will be described. The focus detection unit 705 first generates a pupil divided image in the autofocus area. As described in the above description of the image sensor 702, each of the divided pixels receives a light beam that has passed through a different region on the exit pupil. Therefore, a pair of pupil-divided signals can be obtained by collecting the outputs of the divided pixels having a symmetrical positional relationship with respect to the center of the microlens. Next, correlation calculation is performed on the obtained pair of pupil-divided images to calculate a signal difference, that is, a deviation amount. In the correlation calculation, the evaluation value is acquired by integrating the absolute value of the difference value for each overlapping pixel while shifting the pair of pupil divided images in the horizontal direction. Next, an image shift amount to a position where the evaluation value is the lowest, that is, a position where the degree of coincidence of the pupil divided images is the highest is calculated (SAD calculation). The imaging optical system 701 is driven based on the shift amount to adjust the focal plane.

  Although an example in which the absolute value of the difference of each pixel is used as the evaluation value has been described here, the evaluation value may be acquired by a sum of squares of the difference value (SSD calculation) or other methods.

  Returning to FIG. 7, the system control unit 706 includes a CPU, and controls the operation of the entire system according to a control program stored in a memory (not shown). The system control unit 706 also performs input / output of signals between blocks such as data obtained by image processing and acquisition of lens position information of the imaging optical system 701.

  The operation unit 707 is an operation member such as various switches and dials provided in the imaging apparatus 700, and the photographer can control shooting parameter settings, shooting operations, and the like using the operation unit 707. An operation input by the operation unit 707 is output to the system control unit 706.

  A display unit 708 is a display such as an LCD, and displays an image at the time of shooting transferred from the system control unit 706, image data stored in the storage unit 709, various setting screens, and the like. The storage unit 709 is a recording medium such as an SD card or a CF card, and performs recording of image data transferred via the system control unit 706 and reading of the recording data.

  The above is the configuration of the imaging apparatus 700 according to the present embodiment. Next, the frame rate conversion operation in the imaging apparatus according to the present embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating a flowchart of a shooting operation in a case where a moving image with a shooting recording frame rate and a moving image converted into a high frame rate are simultaneously generated during shooting of a moving image in the imaging apparatus 700. In the imaging apparatus 700, when a moving image shooting start operation is executed by the operation unit 707, the system control unit 706 executes a control program to realize the following processing operations.

  In step S <b> 801, the virtual focal plane determination unit 104 reads the setting values of the recording / recording frame rate and the conversion frame rate input by the operation unit 707 via the system control unit 706. When shooting / recording and conversion frame rate are set, the process proceeds to step S502.

  In step S502, as described in the first embodiment, the virtual focal plane determination unit 104 determines the number of interpolation frames to be interpolated from the frame rate set in step S801. When the number of interpolation frames to be interpolated is determined by the virtual focal plane determination unit 104 in step S502, the process proceeds to step S802.

  In step S <b> 802, the system control unit 706 determines whether the operation unit 707 has stopped moving image shooting. If it is determined in step S802 that the moving image shooting stop operation has been performed, the moving image shooting operation is terminated. On the other hand, if the stop operation has not been performed in step S802, the process proceeds to step S803.

  In step S803, a preliminary shooting operation such as determination of an autofocus, an aperture value by an automatic exposure function, and a shutter speed is performed. When the preliminary shooting operation is completed in step S803, the process proceeds to step S804.

  In step S804, exposure is performed at a shutter speed corresponding to the moving image shooting frame rate, and a signal of the shooting target area is output from the image sensor 702. When the exposure operation is completed in step S804, the process proceeds to step S805.

  In step S <b> 805, the image processing unit 704 performs image processing such as development processing on the digital signals in the imaging target area acquired by the image sensor 702 and the A / D converter 703, and records image data for recording and display. Generate. When the image processing is completed in step S805, the process proceeds to step S806.

  In step S806, the frame rate conversion unit 103 determines the virtual focal plane of the interpolation frame to be interpolated by the processing in steps S504 to S506 described in the first embodiment, performs digital signal refocus processing, and generates conversion data. The frame rate conversion process is performed. The focal plane information of each frame is acquired by the focus detection unit 705.

  The focus detection unit 705 can acquire the focal plane information of the current frame from the current position of the focus lens, and can acquire the focal plane information of the next frame based on the calculated shift amount. The frame rate conversion unit 103 determines a virtual focal plane of an interpolation frame to be interpolated by linear interpolation using the focal plane information of the current frame and the next frame acquired by the focus detection unit 705.

  In step S807, the recording data generated in step S805 and the frame rate conversion data generated in S806 are stored in a storage medium slot (not shown) of the image processing apparatus 700 via the system control unit 706. Part 709 is recorded. The display data is displayed on the display unit 708 via the system control unit 706.

  As described above, when configured as an imaging device, frame rate conversion processing can be performed simultaneously with moving image shooting, and processing such as reading of recorded data is reduced, so that power consumption can also be reduced.

  With reference to FIG. 9, an image processing apparatus according to a third embodiment of the present invention will be described. In the first and second embodiments, the process of converting moving image data from a low frame rate to a high frame rate is taken as an example, and the configuration for smoothing the focal plane change of the present invention is shown. In the present embodiment, the predetermined processing including frame interpolation is exemplified by processing for generating moving image recording data and display data by interpolating frames that are lost when still image shooting is performed during moving image shooting. A configuration for smoothing the change of the focal plane related to the above will be described. It is obvious that the configuration according to the present embodiment can be applied to the imaging apparatus according to the second embodiment or the imaging apparatus to which the image processing apparatus according to the first embodiment is not applied.

  FIG. 9 is a diagram schematically showing a moving image recording frame that is lost when still image shooting is performed during moving image shooting. FIG. 9A shows a moving image recording frame shot at a certain time, and frames 905 to 908 are lost by the still image shooting processing 913 performed during moving image shooting shown in FIG. 9B. It shows the frame that will be.

  In the imaging apparatus 700 described in the second embodiment, when the shutter button included in the operation unit 707 is pressed halfway during the moving image shooting process, a preliminary operation such as autofocus is performed, and when the shutter button is fully pressed, the image capturing apparatus 700 is stationary. Image capture processing is performed. In still image shooting during moving image shooting, mode switching occurs, and therefore moving image shooting cannot be continued at the shooting / recording frame rate while still image shooting is being performed. Accordingly, there is a problem in that the frames of the moving image recording data and the display data are not updated until the still image shooting process is completed and the moving image shooting mode is restored.

  For the above problem, the frame rate conversion processing described in the first and second embodiments is applied with the frame immediately before the start of still image shooting as the reference frame. Accordingly, it is possible to record and display a frame that smoothly expresses a change in focal plane, instead of recording and displaying the same frame during the still image shooting processing period. Note that the number of interpolation frames can be determined from the frame rate of moving image shooting and the time required for still image shooting.

  As described above, if the present invention is applied to moving image recording and display data during a period of still image shooting during moving image shooting according to the present embodiment, the focal plane changes smoothly in the captured moving image. It becomes possible to express it. As a result, it is possible to provide a moving image display with less discomfort to the photographer.

[Other embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

Claims (12)

One frame of moving image data including a plurality of frames of light field data including light path information and light amount information is used as a reference frame, and a refocus image is generated on at least one virtual focal plane from the light field data of the reference frame. Refocusing means,
Interpolating means for interpolating the refocus image as an interpolation frame of the moving image data at a position adjacent to the reference frame in time series;
Virtual focal plane determination means for determining a virtual focal plane of the refocused image based on a change in focal plane of a plurality of frames including the reference frame;
An image processing apparatus comprising:   The image processing according to claim 1, wherein the virtual focal plane determination unit determines the virtual focal plane by performing an interpolation operation on a focal plane of each of a plurality of frames including the reference frame. apparatus.   When there is no change in the focal planes of a plurality of frames including the reference frame, the virtual focal plane determining unit does not determine the virtual focal plane, and the refocusing unit does not generate the refocus image. The image processing apparatus according to claim 1, wherein:   The refocusing unit generates the refocused image using a frame generated by performing addition average synthesis after matching the focal plane of the frame of the moving image data adjacent to the reference frame and the reference frame as a reference frame. The image processing apparatus according to claim 3. Processing means for performing predetermined processing on the moving image data, including interpolation of the refocus image by the interpolation means;
The image processing apparatus according to claim 1, wherein the refocusing unit determines the number of the refocused images generated from the reference frame according to the predetermined processing.   The predetermined process is a frame rate conversion process for converting a frame rate of the moving image data, the processing means includes a setting means for setting a frame rate to be converted, and the virtual focal plane determination means includes the setting The number of interpolation frames to be interpolated by the interpolation unit is determined based on the determined frame rate and the frame rate of the moving image data, and the frame rate of the moving image data is lower than the set frame rate. The image processing apparatus according to claim 5.   A reference frame is each frame of the moving image data, and the processing means interpolates the determined number of interpolation frames before, before, after, or after the reference frame, and a moving image whose frame rate is converted The image processing apparatus according to claim 6, wherein data is generated.   The predetermined process is an interpolation process for a frame missing from moving image data shot for still image shooting during moving image shooting, and the reference frame is a frame immediately before the still image shooting in the moving image data. The image processing apparatus according to claim 5. An imaging optical system;
A plurality of frames of light field data having a two-dimensional array of pixels including a plurality of divided pixels that receive a light beam that has passed through a specific exit pupil region of the imaging optical system and including path information and light amount information of the light beam Imaging means for generating moving image data comprising:
Refocus means for generating a refocus image in at least one virtual focal plane from the light field data of the reference frame, with one frame of the moving image data composed of the plurality of frames as a reference frame;
Interpolating means for interpolating the refocus image as an interpolation frame of the moving image data at a position adjacent to the reference frame in time series;
Virtual focal plane determination means for determining a virtual focal plane of the refocused image based on a change in focal plane of a plurality of frames including the reference frame;
Processing means for performing predetermined image processing including interpolation of the refocused image by the interpolation means on the moving image data;
An imaging apparatus comprising: One frame of moving image data including a plurality of frames of light field data including light path information and light amount information is used as a reference frame, and a refocus image is generated on at least one virtual focal plane from the light field data of the reference frame. Refocusing process,
An interpolation step of interpolating the refocused image as an interpolation frame of the moving image data at a position adjacent to the reference frame in time series;
A virtual focal plane determination step for determining a virtual focal plane of the refocused image based on a change in focal plane of a plurality of frames including the reference frame;
An image processing method comprising: A program for controlling an image processing apparatus,
Computer
One frame of moving image data including a plurality of frames of light field data including light path information and light amount information is used as a reference frame, and a refocus image is generated on at least one virtual focal plane from the light field data of the reference frame. Refocusing means,
Interpolating means for interpolating the refocus image as an interpolation frame of the moving image data at a position adjacent to the reference frame in time series,
Virtual focal plane determination means for determining a virtual focal plane of the refocused image based on a change in focal plane of a plurality of frames including the reference frame;
Program to function as.   A computer-readable storage medium storing the program according to claim 11.

Images