JP2015023552A - Image reproduction device, image reproduction method, program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately determine a focal position in each frame of motion picture data based on write-field image data.SOLUTION: An image reproduction device comprises a capturing part and an arithmetic operation part. The capturing part captures motion picture data which are imaged by an imaging apparatus capable of capturing write-field image data, for which a focal distance of a moving image for each imaging frame and for each subject is measured by the imaging apparatus and of which the information of the focal distance is recorded in association with the imaging frame. In the case where the information of the focal distance for each subject recorded in association with the imaging frame is present in the moving image data, the arithmetic operation part performs refocus arithmetic operation on the basis of the information of the focal distance. In the case where the information of the focal distance for each subject recorded in association with the imaging frame is not present in the moving image data, the arithmetic operation part performs refocus arithmetic operation while using the information of the focal distance or information of a stop value which is interpolated on the basis of information of focal distance for each subject recorded in association with another imaging frame.

Description

  The present invention relates to a refocus technique for reproducing a moving image.

  Conventionally, there has been a demand for an effective reproduction effect not only at the time of recording but also when viewing a recorded image. For example, Patent Document 1 proposes a still image capturing apparatus that records a face position and a focus position in association with image data, and shifts the entire reproduced image in parallel to the face position center or the focus position center in enlarged reproduction. Yes.

  On the other hand, in recent years, an imaging apparatus having a configuration in which a microlens array arranged at a ratio of one for a plurality of pixels is arranged on the front surface of a solid-state imaging device has been proposed in Non-Patent Document 1 and the like. With such a configuration, it is possible to obtain information on the direction of light rays incident on the image sensor. When this imaging device is used, in addition to generating a normal captured image based on the output signal from each pixel, subjecting the captured image data to a subject at an arbitrary distance can be achieved by performing predetermined image processing. It is possible to reconstruct a focused image.

JP 2007-195099 A

Ren. Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, since the imaging apparatus of Patent Document 1 uses only a still image as a reproduction target, there is no description about a technique for reproducing a plurality of still images that are temporally continuous as a moving image.

  The present invention has been made in view of the above-described problems, and an object thereof is to appropriately determine a focal position in each frame of moving image data based on light field image data.

  The image reproduction device according to the present invention is picked up by an image pickup device capable of acquiring light field image data, and the image pickup device measures a focus distance for each image pickup frame and subject of a moving image, and Acquisition means for acquiring moving image data in which focal distance information is recorded in association with the imaging frame; and in-focus distance information for each subject recorded in the moving image data in association with the imaging frame. If the focus distance information for each subject recorded in association with the imaging frame does not exist in the moving image data, a refocusing operation is performed based on the focusing distance information. Calculating the refocus using the information on the focus distance or the aperture value interpolated based on the information on the focus distance for each subject recorded in association with Characterized in that it comprises a and.

  According to the present invention, it is possible to appropriately determine a focal position in each frame of moving image data based on light field image data.

1 is a block diagram illustrating a schematic configuration of an imaging unit, a recording unit, and a reproduction unit of a digital camera that is an imaging apparatus according to a first embodiment of the present invention. The figure which shows the example of the imaged moving image data. The figure which shows the metadata row | line | column of the focusing distance information according to to-be-photographed object of the 1st-4th embodiment. The figure which shows the example of the to-be-photographed object distance in 1st Embodiment. The flowchart which shows operation | movement of 1st Embodiment. The figure which shows the example of the to-be-photographed object distance of 2nd Embodiment. The flowchart which shows operation | movement of 2nd Embodiment. The figure which shows the example of the to-be-photographed object distance of 3rd Embodiment. The flowchart which shows operation | movement of 3rd Embodiment. The figure which shows the example of the to-be-photographed object distance of 4th Embodiment. The flowchart which shows operation | movement of 4th Embodiment. The figure which shows the metadata row | line of the focusing distance information according to to-be-photographed object of 5th Embodiment. The figure which shows the metadata row | line of the focusing distance information according to to-be-photographed object of 6th Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
In the first embodiment, when the distance value used for the refocus calculation is missing in a part of the moving image frame (moving image data), the refocus parameter obtained by interpolating the distance value of the current frame with the distance value of the previous frame is used. An example of performing a refocus calculation using the above will be described. Hereinafter, an imaging apparatus according to a first embodiment of the present invention will be described with reference to FIG.

  FIG. 1 is a block diagram showing a schematic configuration of an imaging unit, a recording unit, and a reproduction unit of a digital camera that is an imaging apparatus (image reproduction apparatus) according to the first embodiment of the present invention. In FIG. 1, 101 is an objective lens for condensing light rays from a subject, 103 is a microlens array arranged adjacent to an image sensor, and 105 is an image that outputs an electrical signal obtained by photoelectrically converting an optical image of the subject. It is an element. Reference numeral 107 denotes an analog front end and an AD conversion circuit for performing an amplification process on an image signal output from the image sensor 105. Reference numeral 111 denotes a subject distance map generation circuit.

  Here, the operation of the subject distance map generation circuit 111 will be briefly described. First, a temporary distance value is given to the light field image data, and an image as a result of performing a refocus calculation is derived for a plurality of temporary distance values. Then, by comparing the images of the plurality of refocus calculation results with respect to the plurality of temporary distance values and selecting the image with the maximum contrast, the focus corresponding to the subject whose temporary distance value given here has the maximum contrast is selected. Distance. This selection calculation is performed for each subject on the screen. Then, correspondence data between the focus distance value corresponding to the subject area and the screen coordinate value is generated for each screen. The above is the operation of the subject distance map generation circuit 111.

  Returning to FIG. 1, reference numeral 121 denotes a refocusing arithmetic circuit, 131 denotes an image data display device such as a liquid crystal monitor, and 141 denotes a recording medium such as a memory card for storing imaging data. Reference numeral 151 denotes a memory composed of a RAM that temporarily stores image signals, 161 denotes a CPU that controls various arithmetic controls, and 171 denotes a user interface such as a touch panel.

  First, the recording operation will be described. Note that the operation of each circuit block, including the imaging operation, recording operation, and reproduction operation described later, is controlled by the CPU 161 executing a program recorded in its internal memory.

  The optical image of the subject is collected by the objective lens 101, passes through the microlens array 103, is projected onto the image sensor 105, and is subjected to photoelectric conversion. The image data output from the image sensor 105 is amplified by the analog front end and the AD conversion circuit 107 and then AD converted and stored in the RAM 151 as image data. This image data is input to the subject distance map generation circuit 111, and the focusing distance for each subject is calculated. The in-focus distance for each subject is stored in the recording medium 141 as metadata of each image data as a set with the image data.

  Here, FIG. 2 is an example of captured moving image data, and FIG. 3 is an example of in-focus distance information metadata corresponding to FIG. 2, that is, an example of a 12-frame metadata string. That is, FIG. 3 illustrates the distance information from the image sensor of each subject of the objects A, B, and C in FIG. 2, that is, the focus distance information generated by the subject distance map generation circuit 111 in the order of frame numbers. is there.

  Next, a schematic reproduction operation of recorded image data will be described. The image data and metadata bundled in a set in the recording medium 141 are once transferred to the RAM 151 for each frame. The CPU 161 inputs the image data to the refocus calculation circuit 121, analyzes distance information in the metadata as will be described later, and inputs the distance value to the refocus calculation circuit 121 as a refocus parameter. The refocus calculation circuit 121 performs refocus calculation on the image data using the given distance parameter. As a result, the image data focused at one distance value is transferred to the RAM for each frame, and this image data is drawn as a continuous frame on the data display device 131.

  Next, the control operation of the in-focus distance value which is a refocus parameter in the refocus calculation of the reproduction image data is shown in the block diagram of FIG. 1, the example of the metadata string of FIG. 3, the example of the subject distance of FIG. 4, and the flowchart of FIG. Will be described. The operation of the flowchart of FIG. 5 is controlled by the CPU 161 executing a program recorded in its internal memory.

  First, the CPU 161 outputs a message prompting the user to select a main subject to the display device 131, and records the main subject to be focused in the CPU 161 in accordance with a user response from the user interface 171 (step S203). Here, it is assumed that the object A in FIG. 2 is selected.

  Next, the CPU 161 checks the presence / absence of a value corresponding to the main subject instructed in step S203 in the distance information of the plurality of subjects in the metadata bundled with the image data of the current display frame ( Step S205). In FIG. 3, since the distance information of the object A exists between the frames 1 and 5, the process proceeds to step S211, and the current frame, that is, the distance value 2.4 m of the frame 1 is used as the refocusing parameter to the image data of the frame 1. Refocus calculation is performed.

  Next, the operation of outputting the calculation result to the display device 131 (step S212), proceeding to the next frame (step S213), and returning to the inspection of the presence of the distance value corresponding to the main subject (step S205) is performed in frame 1. Repeat until frame 5.

  And since the distance value of the object A is not preserve | saved in the flame | frame 6 of FIG. 3, it transfers to step S221 of the flowchart of FIG. In this step, the subject distance value in the metadata is inspected as to whether or not the framed subject is to be framed again during the same moving picture cut. Further, when re-frame-in is performed, the number of frames after the current frame number is checked for re-frame-in, and the control is switched depending on the number of frames up to the re-frame-in. For example, in this embodiment, the threshold frame number N is assumed to be 9 frames. On the other hand, in this embodiment, the value of the object A is recorded again from 5 frames after the frame 6 (FIG. 3). Therefore, in this example, the frame 11 to be framed in again is less than N (below the threshold), and the process proceeds to step S223.

  Since the distance value is not recorded in the frame 6 (FIG. 3), the distance value 2.1m of the immediately preceding frame 5 is given as an interpolation refocus parameter, and the refocus calculation is performed on the image data of the frame 6 (currently reproduced frame). (Step S223). Then, the calculation result is output to the display device 131 (step S224), and the process proceeds to step S225. On the other hand, if the subject out of the frame does not re-frame in step S221 or if the number of frames until re-frame-in is 10 or more, the CPU sends a message prompting the user to change the main subject to the display device 131. Output (step S231). Next, the CPU checks the user response from the user interface 171. If the main subject has been updated, the process proceeds to step S203. If the main subject has not been updated, the process proceeds to step S223, and interpolation is performed using the distance value of the previous frame. Use refocus parameters.

  As described above, in this embodiment, the in-focus distance value, which is a refocus parameter in the refocus calculation of reproduced image data, is controlled as indicated by the curve objA in FIG. In FIG. 4, the solid line portion of the objA curve is the metadata recorded value itself without interpolation, and the broken line portion of the objA curve is the interpolated value from the metadata recorded value.

  As described above, in this embodiment, when the main subject is out of frame, a method of interpolating the distance value of the current frame with the distance value of the frame immediately before the frame out is used. This avoids the problem that if the distance value used for the refocus calculation is missing at a part of the moving image frame, the in-focus position frequently moves and flickers and becomes an image that is difficult to view.

  In addition, by prompting the user to update the main subject when the main subject is out of the frame, the in-focus position becomes unstable, and the problem of flickering and difficult to view images is avoided.

(Second Embodiment)
In the second embodiment, an example in which refocus calculation is performed using a refocus parameter considering an interpolation period of distance values used for refocus calculation will be described. Basic imaging operation, recording operation, and reproduction operation are the same as those in the first embodiment. Therefore, the block diagram of the apparatus is the same as that shown in FIG. The difference between the first embodiment and the second embodiment is the control operation of the in-focus distance value that is the refocus parameter in the refocus calculation of the reproduction image data.

  A control operation of a focus distance value, which is a refocus parameter in refocus calculation of reproduced image data, will be described with reference to the block diagram of FIG. 1, the metadata string example of FIG. 3, the subject distance example of FIG. 6, and the flowchart of FIG. To do. Note that the operation of the flowchart of FIG. 7 is controlled by the CPU 161 executing a program recorded in its internal memory.

  First, the CPU 161 outputs a message prompting the user to select a main subject to the display device 131, and records the main subject to be focused in the CPU in accordance with a user response from the user interface 171 (step S203). Here, it is assumed that the object A in FIG. 2 is selected. Next, the CPU checks whether or not there is a value corresponding to the main subject instructed in step S203 in the distance information of the plurality of subjects in the metadata bundled with the image data of the current display frame ( Step S205). In FIG. 3, since the distance information of the object A exists between the frames 1 and 5, the process proceeds to step S211, and the current frame, that is, the distance value 2.4 m of the frame 1 is used as the refocusing parameter to the image data of the frame 1. Refocus calculation is performed. Next, the operation of outputting the calculation result to the display device 131 (step S212), proceeding to the next frame (step S213), and returning to the inspection of the presence of the distance value corresponding to the main subject (step S205) is performed in frame 1. Repeat until frame 5.

  And since the distance value of the object A is not preserve | saved in the flame | frame 6 of FIG. 3, it transfers to step S221 of the flowchart of FIG. In this step, it is checked whether or not the framed-out subject is framed again during the same moving image cut. Further, when re-frame-in is performed, the number of frames after the current frame number is checked for re-frame-in, and the control is switched depending on the number of frames up to the re-frame-in. For example, in this embodiment, the threshold frame number N is assumed to be 9 frames. On the other hand, in this embodiment, the value of the object A is recorded again from 5 frames after the frame 6 (FIG. 3). Accordingly, in this example, the frame 11 to be framed in again is less than N (below the threshold), and the process proceeds to step S241.

  In step S241, the difference between the frame number at which the subject out of the frame enters the frame again (future) and the current frame number is inspected. If the time is less than L frames from the start frame of the second frame-in, the main subject distance value at the second frame-in is used for the refocus calculation. For example, in step S241 of this embodiment, this frame time L is assumed to be two frames, and the process proceeds to step S251 from frame 6 to frame 9 two frames before the re-in (frame 11).

  In step S251, the duration of the immediately preceding frame replacement treatment using the subject distance immediately before the frame out is inspected. Then, if the number of continuing frames of the immediately preceding frame replacement procedure has reached a time of K frames or more (a predetermined number of frames or more), a reframe-in-frame replacement procedure that uses the subject distance in the next frame-in from the previous frame replacement procedure. Switch to. For example, in step S251 of this embodiment, this K frame time is assumed to be 5 frames. In this case, in the frame 6 to the frame 9, the process proceeds to step S223, and the refocus calculation is performed on the image data using the distance value of the frame immediately before the frame out as the refocus parameter. Next, the calculation result is output to the display device 131 (step S224), the process proceeds to the next frame (step S225), and the process returns to the inspection of the existence of the distance value corresponding to the main subject (step S205).

  On the other hand, if the number of continuing frames has reached K frames or more in step S251, the main subject distance value in the second frame-in, that is, the distance value replaced with 3.0 m of frame 11 is used as the refocus parameter to convert the image data. Focus calculation is performed (step S243). Then, the calculation result is output to the display device 131 (step S244), and the process proceeds to the next frame (step S245).

  On the other hand, in step S241, in the case of the frame 10 one frame before the second frame-in, the following operation is performed because the frame time L is assumed to be 2 frames. That is, the refocusing operation is performed on the image data using the main subject distance value in the re-frame-in, that is, the distance value replaced with 3.0 m of the frame 11 as the refocus parameter (step S243). Then, the calculation result is output to the display device 131 (step S244), and the process proceeds to the next frame (step S245).

  On the other hand, if the subject out of the frame does not re-frame in step S221 or if the number of frames until re-frame-in is 10 or more, the CPU sends a message prompting the user to change the main subject to the display device 131. Output (step S231). Next, the CPU checks the user response from the user interface 171. If the main subject has been updated, the process proceeds to step S203. If the main subject has not been updated, the process proceeds to step S223. Use focus parameters.

  As described above, in this embodiment, the in-focus distance value, which is the refocus parameter in the refocus calculation of the reproduced image data, is controlled as indicated by the curve objA in FIG. In FIG. 6, the solid line portion of the objA curve is the metadata recorded value itself without interpolation, and the broken line portion of the objA curve is the interpolated value from the metadata recorded value.

  As described above, in this embodiment, the duration of the focus distance interpolation of the refocus parameter is inspected, and when it is long, the focus distance is switched, so that an unnatural long time after the main subject is framed out. It is avoided that the focusing position is fixed.

  Further, the period until the main subject is re-framed is inspected, and if it is short, it is pre-focused to the distance at which the main subject will be re-framed in the future. By controlling the in-focus distance in this way, the main subject can be put into frame-in standby while the in-focus state around the subject is stable, and an effective reproduction effect can be achieved.

(Third embodiment)
In the third embodiment, an example will be described in which interpolation is performed on the assumption that the subject distance at the time of frame-out of the subject and the subject distance at the time of frame-in again transition gradually in proportion to time. The basic imaging operation, recording operation, and reproduction operation are the same as in the first embodiment. The block configuration of the imaging apparatus is also the same as that in FIG. The difference between the first embodiment and the third embodiment is the control operation of the in-focus distance value that is a refocus parameter in the refocus calculation of reproduced image data.

  A control operation of a focus distance value, which is a refocus parameter in refocus calculation of reproduced image data, will be described with reference to the block diagram of FIG. 1, the example of the metadata string of FIG. 3, the example of the subject distance of FIG. 8, and the flowchart of FIG. To do. The operation of the flowchart of FIG. 9 is controlled by the CPU 161 executing a program recorded in its internal memory.

  First, the CPU 161 outputs a message prompting the user to select a main subject to the display device 131, and records the main subject to be focused in the CPU in accordance with a user response from the user interface 171 (step S203). Here, it is assumed that the object A in FIG. 2 is selected. Next, the CPU checks the presence / absence of a distance value corresponding to the main subject instructed in step S203 in the distance information of the plurality of subjects in the metadata bundled with the image data of the current display frame. (Step S205). In FIG. 3, since the distance information of the object A exists between the frames 1 and 5, the process proceeds to step S211, and the current frame, that is, the distance value 2.4 m of the frame 1 is used as the refocusing parameter to the image data of the frame 1. Refocus calculation is performed.

  Next, the operation of outputting the calculation result to the display device 131 (step S212), proceeding to the next frame (step S213), and returning to the inspection of the presence of the distance value corresponding to the main subject (step S205) is performed in frame 1. Repeat until frame 5. And since the distance value of the object A is not preserve | saved in the flame | frame 6 of FIG. 3, it transfers to step S221 of the flowchart of FIG.

  In this step S221, it is checked whether or not the frame-out subject is framed in again during the same moving image cut. Further, when the subject is framed in again, it is checked how many frames later the current frame number is to be reframed, and the control is switched depending on the number of frames until reframed. For example, in this embodiment, the threshold frame number N is assumed to be 9 frames. On the other hand, in this example, the value of the object A is recorded again from 5 frames after the frame 6 (FIG. 3). Therefore, the reframe-in-frame 11 is less than N (below the threshold), and the process proceeds to step S261.

  Since the frame 6 has no distance value recorded (FIG. 3), the distance value is between the distance value 2.1 m of the frame 5 where the main subject is out of frame and the distance value 3.0 m of the reframe-in frame 11. The distance value of each frame is calculated on the assumption that the transition is in proportion to time, and is used as an interpolation value. The interpolated value is given as a refocus parameter, refocus calculation is performed on the image data of frame 6 (step S261), and the calculation result is output to the display device 131 (step S262). Then, the process proceeds to step S263, and the process returns to the inspection for the existence of the distance value corresponding to the main subject (step S205).

  On the other hand, if the subject out of the frame does not re-frame in step S221 or if the number of frames until re-frame-in is 10 or more, the CPU sends a message prompting the user to change the main subject to the display device 131. Output (step S231). Next, the CPU checks the user response from the user interface 171. If the main subject has been updated, the process proceeds to step S203. If the main subject has not been updated, the process proceeds to step S223, and interpolation is performed using the distance value of the previous frame. Use refocus parameters. Next, the calculation result is output to the display device 131 (step S224), the process proceeds to the next frame (step S225), and the process returns to the inspection of the existence of the distance value corresponding to the main subject (step S205).

  As described above, in this embodiment, the in-focus distance value, which is a refocus parameter in the refocus calculation of the reproduction image data, is controlled as indicated by the curve objA in FIG. In FIG. 6, the solid line part of the curve of objA is the metadata recorded value itself without interpolation, and the broken line part of the curve of objA is the interpolated value from the metadata recorded value.

  As described above, in the present embodiment, interpolation is performed on the assumption that the in-focus distance from the subject distance at the time of frame-out to the subject distance at the time of frame-in again changes in proportion to the time. Therefore, it is possible to produce an effective reproduction effect such as a smooth zoom transition during the period from frame out to frame in again.

(Fourth embodiment)
In the fourth embodiment, interpolation is performed with an aperture value and a focus distance value corresponding to the depth of field including the distances when the subject is out of frame and when the subject is re-framed. The basic imaging operation, recording operation, and reproduction operation are the same as in the first embodiment. The block configuration of the imaging apparatus is also the same as that in FIG. The difference between the first embodiment and the fourth embodiment is the control operation of the distance value and the aperture value which are refocus parameters in the refocus calculation of the reproduction image data.

  The control operation of the distance value and the aperture value, which are refocus parameters in the refocus calculation of the reproduction image data, will be described with reference to the block diagram in FIG. 1, the metadata string example in FIG. 3, the subject distance example in FIG. 10, and the flowchart in FIG. explain. The operation of the flowchart of FIG. 11 is controlled by the CPU 161 executing a program recorded in its internal memory.

  First, the CPU 161 outputs a message prompting the user to select a main subject to the display device 131, and records the main subject to be focused in the CPU in accordance with a user response from the user interface 171 (step S203). Here, it is assumed that the object A in FIG. 2 is selected. Next, the CPU checks the presence / absence of a distance value corresponding to the main subject instructed in step S203 in the distance information of the plurality of subjects in the metadata bundled with the image data of the current display frame. (Step S205). In FIG. 3, since the distance information of the object A exists between the frames 1 and 5, the process proceeds to step S211, and the current frame, that is, the distance value 2.4 m of the frame 1 is used as the refocusing parameter to the image data of the frame 1. Refocus calculation is performed. Next, the operation of outputting the calculation result to the display device 131 (step S212), proceeding to the next frame (step S213), and returning to the inspection of the presence of the distance value corresponding to the main subject (step S205) is performed in frame 1. Repeat until frame 5.

  And since the distance value of the object A is not preserve | saved in the flame | frame 6 of FIG. 3, it transfers to step S221 of the flowchart of FIG. In this step, it is checked whether or not the framed-out subject is framed again during the same moving image cut. Further, when re-frame-in is performed, the number of frames after the current frame number is checked for re-frame-in, and the control is switched depending on the number of frames up to the re-frame-in. For example, in this embodiment, the threshold frame number N is assumed to be 9 frames. On the other hand, in this example, the value of the object A is recorded again from 5 frames after the frame 6 (FIG. 3). Therefore, the reframe-in frame 11 is less than N (below the threshold), and the process proceeds to step S271.

  Since the distance value is not recorded in the frame 6 (FIG. 3), the distance value 2.1 m of the frame-out frame 5 and the distance value 3.0 m of the re-frame-in frame 11 are respectively set to the depth-of-field front end distance, Set the trailing edge distance. That is, the aperture value and the focus distance value corresponding to the depth of field including the distance at the time of frame-out and the time of re-frame-in in the focus range are calculated as interpolation values (step S271). Then, this interpolation value is given as a refocus parameter, refocus calculation is performed on the image data of frame 6 (step S273), and the calculation result is output to the display device 131 (step S274). Then, the process proceeds to step S275, and the process returns to the inspection of the existence of the distance value corresponding to the main subject (step S205).

  On the other hand, if the subject out of the frame does not re-frame in step S221 or if the number of frames until re-frame-in is 10 or more, the CPU sends a message prompting the user to change the main subject to the display device 131. Output (step S231). Next, the CPU checks the user response from the user interface 171. If the main subject has been updated, the process proceeds to step S203. If the main subject has not been updated, the process proceeds to step S223. The refocus parameter interpolated with is used. Next, the calculation result is output to the display device 131 (step S224), the process proceeds to the next frame (step S225), and the process returns to the inspection of the existence of the distance value corresponding to the main subject (step S205).

  As described above, in this embodiment, the in-focus distance value, which is the refocus parameter in the refocus calculation of the reproduced image data, is controlled as indicated by the curve objA in FIG. In FIG. 10, the solid line portion of the curve of objA is the metadata recorded value itself without interpolation, and the shaded area of the curve of objA is the interpolation value from the metadata recorded value.

  As described above, in the present embodiment, interpolation is performed with the aperture value and the focus distance value corresponding to the depth of field including the distances at both the frame-out time and the re-frame-in time in the focus range. Accordingly, since the in-focus distance and the depth of field are constant in the period from the frame-out to the frame-in where the main subject is absent in the frame, a stable reproduction image can be output.

(Fifth embodiment)
In the fifth embodiment, when the imaging apparatus measures the focusing distance for each imaging frame and each imaging subject and records the focusing distance for each subject in association with the recording imaging frame, the number of recording subjects M is limited. To do.

  This is because the CPU 161 limits the number M of recorded subjects in accordance with an instruction from the user interface 171 when storing the in-focus distance value for each subject calculated by the subject distance map generation circuit 111 as metadata in the recording medium 141. To record.

  FIG. 12 is an example of in-focus distance information metadata corresponding to FIG. 2, that is, a 12-frame metadata string. FIG. 12A is an example in which the number M of recorded subjects is limited to 3, and distance information from the image sensor of each subject of object A, object B, and object C, that is, in-focus distance information, in the order of frame numbers. It is illustrated. FIG. 11B is an example in which the number M of recorded subjects is limited to 2, and distance information from the image sensor of each subject of object A and object C, that is, in-focus distance information is illustrated in the order of frame numbers. .

  As described above, in this embodiment, the number M of recording subjects is limited when recording the in-focus distance for each subject in association with the recording imaging frame. Therefore, since the metadata capacity can be controlled, the recording medium can be effectively used by adjusting the number of user requests for the subject corresponding to the focus control during reproduction.

(Sixth embodiment)
In the sixth embodiment, when the imaging apparatus measures the focusing distance for each imaging frame and each imaging subject and records the focusing distance for each subject in association with the recording imaging frame, only when the first subject is detected. An example of recording the distance value will be described.

  This is because when the focus distance value for each subject calculated by the subject distance map generation circuit 111 is stored as metadata in the recording medium 141, first, the distance value acquired by the CPU 161 is the first acquisition frame for each subject. It is determined whether it is a continuous acquisition frame. Then, the CPU 161 selects only the in-focus distance value for each subject in the first acquisition frame and stores it as the metadata in the recording medium 141.

  FIG. 13 is an example of in-focus distance information metadata corresponding to FIG. 12, that is, a metadata frame for 12 frames. FIG. 13A shows distance information from the image sensor of each subject of the object A, object B, and object C, that is, in-focus distance information in the order of frame numbers for all the frames detected and identified by the subject. It is. FIG. 13B illustrates distance information from the image sensor of each subject of the objects A, B, and C, that is, in-focus distance information, in the order of frame numbers only for the first frame detected and identified. Is.

  First, when arranging the light field image data of one frame, if the aperture value and the distance value are given as the refocus parameters, the refocus calculation can be performed on the reproduced image data. First, regarding this aperture value, if a user-specified value, for example, F4 is given, a refocus calculation can be performed on the reproduced image data. On the other hand, in order to calculate this distance value, that is, the subject in-focus distance value, a temporary distance value is given to the light field image data and a refocus calculation is performed to obtain a result image for a plurality of temporary distance values. . Then, it is possible to determine the in-focus distance of the subject by a method of comparing the images of the plurality of refocus calculation results for a plurality of temporary distance values and selecting an image with the maximum contrast.

  When the range of the temporary distance value given by this calculation is infinite, the number of in-focus distance calculations is also infinite, and the amount of calculation becomes very large. However, there is some clue about the assumed temporary distance value range, and the amount of calculation can be reduced if the range of the temporary distance value can be limited. On the other hand, since the change in the subject focus distance between adjacent frames is generally small, the amount of computation can be reduced by limiting to a temporary distance value obtained by adding or subtracting a minute value before or after the distance of the immediately preceding frame, and the contrast during playback processing The in-focus distance can be calculated by comparison. In other words, if only the distance information of the first frame in which the subject is detected is recorded for one subject, the temporary distance between adjacent frames can be limited, so the focus distance calculation by contrast comparison during playback processing Is possible.

  As described above, in this embodiment, when the in-focus distance for each subject is recorded in association with the imaging frame, the number of frames for recording the distance value can be greatly reduced. Accordingly, the metadata capacity can be controlled, and the recording medium can be effectively used.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  In addition, the present invention is not limited to devices such as digital cameras, but includes built-in or external connection of imaging devices such as mobile phones, personal computers (laptop type, desktop type, tablet type, etc.), game machines, etc. It can be applied to any device. Therefore, the “imaging device” in this specification is intended to include any electronic device having an imaging function.

Claims (8)

The image is picked up by an imaging device capable of acquiring light field image data, and the imaging device measures the focusing distance for each imaging frame of the moving image and for each subject. Acquisition means for acquiring moving image data recorded in association;
When the moving image data includes in-focus distance information for each subject recorded in association with the imaging frame, refocus calculation is performed based on the in-focus distance information, and the moving image data When there is no information on the focus distance for each subject recorded in association with the imaging frame, the information on the focus distance interpolated based on the information on the focus distance for each subject recorded in association with another imaging frame Computing means for performing refocusing computation using information or aperture value information;
An image reproducing apparatus comprising:   The calculation means eliminates the information on the in-focus distance when the in-focus distance information for each subject recorded in association with the imaging frame does not exist for a predetermined number of frames from the currently reproduced imaging frame. Using the information on the in-focus distance for each subject recorded in association with the immediately preceding imaging frame, the information on the in-focus distance of the currently reproduced imaging frame is interpolated, and the information on the interpolated in-focus distance is used. The image reproducing apparatus according to claim 1, wherein refocus calculation is performed.   When the information on the focal distance for each subject recorded in association with the imaging frame does not exist for a predetermined number of frames from the currently reproduced imaging frame, the calculation means The focus distance information of the currently reproduced imaging frame is interpolated using the focus distance information for each subject recorded in association, and the refocus calculation is performed using the interpolated focus distance information. The image reproducing apparatus according to claim 1, wherein:   The calculation means eliminates the information on the in-focus distance when the in-focus distance information for each subject recorded in association with the imaging frame does not exist for a predetermined number of frames from the currently reproduced imaging frame. Focusing gradually changes from information on the focal distance for each subject recorded in association with the immediately preceding imaging frame to information on the focal distance for each subject recorded in association with the imaging frame to be reproduced in the future. The image according to claim 1, wherein information on a focus distance of an imaging frame currently reproduced is interpolated with distance information, and refocus calculation is performed using the information on the focus distance thus interpolated. Playback device.   The calculation means eliminates the information on the in-focus distance when the in-focus distance information for each subject recorded in association with the imaging frame does not exist for a predetermined number of frames from the currently reproduced imaging frame. The refocusing calculation of the currently reproduced imaging frame is performed using an aperture value having a depth of field deeper than the aperture value used in the refocusing calculation of the immediately preceding imaging frame. Image playback device. The image is picked up by an imaging device capable of acquiring light field image data, and the imaging device measures the focusing distance for each imaging frame of the moving image and for each subject. An acquisition step of acquiring moving image data recorded in association;
When the moving image data includes in-focus distance information for each subject recorded in association with the imaging frame, refocus calculation is performed based on the in-focus distance information, and the moving image data When there is no information on the focus distance for each subject recorded in association with the imaging frame, the information on the focus distance interpolated based on the information on the focus distance for each subject recorded in association with another imaging frame A calculation process for performing a refocus calculation using information or aperture value information;
An image reproduction method comprising:   A computer-executable program in which the procedure of the image reproduction method according to claim 6 is described.   A computer-readable storage medium storing a program for causing a computer to execute each step of the image reproduction method according to claim 6.

Images