JP2015081984A - Imaging device, control method and program of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which in light field cameras having a refocus function, due to backlashes of lenses, the refocus has been difficult depending upon moving subjects.SOLUTION: An imaging device comprises: an imaging optical system that including a focus lens; imaging means for photographing an optical image of a subject formed by the imaging optical system, and generating refocusable pixel data; distance metering means for acquiring a distance to the subject; and lens drive means for driving the focus lens in accordance with a lens location determined on the basis of the acquired subject distance. The imaging device comprises, on the basis of movement information on the subject based on the subject distance and drive characteristic information on the lens drive means, control means for determining a target lens location having an amount of deviation falling within a refocusable range with respect to the lens location, and controlling the lens drive means to drive the focus lens in accordance with the target lens location.

Description

  The present invention relates to an imaging apparatus capable of acquiring a refocusable image, and more particularly to lens drive control during shooting by an imaging apparatus capable of acquiring a refocusable image.

  Currently, many cameras have a so-called moving body tracking function that controls the driving of the taking lens so that a focused image can be taken in accordance with a change in the subject distance of a moving subject. In the following description, each part of the photographing optical system is defined as shown in FIG. 9 in order to explain lens drive control. Further, when “focusing” is described with respect to the photographing optical system, the following lens driving is assumed. That is, the image distance is adjusted by driving the lens so that the focus position of the subject at an arbitrary subject position (subject distance) matches the imaging plane with respect to the imaging plane whose position is fixed in a general camera. That is.

  In a general photographing optical system, the focusing is performed by mechanically driving a focus lens. When driving a mechanical mechanism that reciprocates without limiting to the lens, as shown in FIG. 10, a phenomenon called so-called backlash occurs in which the mechanism cannot immediately respond to the reversal of the drive signal (in FIG. 10, from forward feed to reverse feed). To do. Backlash is caused by play that cannot be avoided due to the structure of the mechanism. Due to this backlash, a delay time occurs in the mechanical drive.

  The adverse effect of the mechanical drive delay time due to the backlash can be considered in the subject tracking operation when the subject moving in the depth direction of the shooting angle of view suddenly reverses the moving direction. For example, when the subject moving in the close direction continues to be focused by moving object tracking operation, if the subject suddenly reverses the moving direction to the infinity side, the lens cannot be driven to reverse immediately due to backlash. As a result, the focus may be lost.

  By the way, in recent years, there has been proposed an imaging apparatus capable of acquiring multi-viewpoint parallax images and controlling the focus position of an image after shooting by image reconstruction processing (image synthesis processing). The focus position control by this reconfiguration process is called refocus process, and an imaging apparatus having such a configuration is called a light field camera. In this light field camera, a reconstructed image in focus can be acquired by refocus processing after shooting even for an image shot at a lens position out of focus.

  For example, in a light field camera as disclosed in Patent Document 1, when a moving object is tracked by driving a lens by a conventional method, the focus position does not coincide with the imaging surface, and there is a slight shift between them. When shooting at a certain lens position, a blurred image is obtained. However, with a light field camera, an in-focus image can be acquired by performing a refocus process after shooting. That is, by adopting the configuration of the light field camera, the ratio of the out-of-focus image acquisition during moving object tracking is reduced.

International Publication Number WO08 / 050904

  However, since there is a limit to the focus depth at which refocusing can be performed for a light field camera, if a certain amount of deviation occurs between the focus position and the imaging surface, the refocus processing will focus. It will not be possible to obtain images that match. Therefore, the ratio of acquiring an out-of-focus image due to backlash during the moving body following operation is not necessarily reduced even in a light field camera.

  Therefore, an object of the present invention is to reduce the acquisition ratio of an out-of-focus image in a light field camera having a refocus function.

  In order to achieve the above object, according to the present invention, an imaging optical system including a focus lens and an imaging unit that captures an optical image of a subject formed by the imaging optical system and generates refocusable pixel data. An imaging device comprising: a distance measuring unit that acquires a distance of the subject; and a lens driving unit that drives the focus lens according to a lens position determined based on the acquired subject distance. Based on the movement information of the subject and the driving characteristic information of the lens driving means, a target lens position having a deviation amount within a refocusable range with respect to the lens position is determined, and the lens driving means is controlled to follow the target lens position. An imaging apparatus comprising control means for driving a focus lens.

  According to the present invention, in a light field camera having a refocus function, it is possible to reduce the acquisition ratio of an out-of-focus image.

1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. The figure for demonstrating the control method of a focus position in the Example of this invention. The figure for explaining the refocus possible range in the light field camera The figure for demonstrating the lens drive for the moving body tracking in the Example of this invention The figure which shows subject distance dependence of the amount of focus position shifts possible in refocusing The figure which shows the flowchart of the control operation of the lens drive of the imaging device which concerns on the Example of this invention. Diagram for explaining the relationship between lens aperture and refocusable range in a light field camera The figure for demonstrating the reconstructed image acquired by the imaging device which concerns on the Example of this invention. The figure which shows the name of each part of photographing optical system Illustration for explaining backlash of focus lens

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  The configuration of the imaging apparatus according to the first embodiment of the present invention will be described below with reference to FIGS. In the present embodiment, the focus lens is moved by shifting the lens drive position by a distance within the refocusable range to shift the focus position from the imaging surface, and this shift corresponds to a moving object tracking delay due to backlash. Specifically, by driving the focus lens in advance by shifting the focus lens drive position by an image distance within a refocusable range in a direction in which the change in the image distance is delayed before the lens drive reversal operation is required. The structure that absorbs the moving object follow-up delay due to backlash is adopted.

  FIG. 1 is a block diagram illustrating an electrical configuration of a digital camera and a lens that are imaging apparatuses according to the present embodiment. A camera system including the camera 1 and the lens 2 has an imaging system, an image processing system, a recording / reproducing system, and a control system. The imaging system includes a photographing optical system 3 and an imaging element 6. In this embodiment, the combination of the photographing optical system 3, the image sensor 6, and a microlens on the image sensor 6 described later constitutes a parallax image acquisition unit that acquires a parallax image of the subject space. The photographing optical system 3 includes a focus lens 4, a shake correction lens 16, and a lens diaphragm 17. The image processing system includes an image processing unit 7. The image processing unit 7 is an image generation unit that generates a composite image in which the focus position is controlled from the image acquired by the parallax image acquisition unit. The recording / reproducing system includes a memory unit 8 and a display unit 9. The control system includes a camera system control unit 5, an operation detection unit 10, a lens system control unit 12, a focus lens driving unit 13 that is a lens driving unit, a blur correction lens driving unit 14, and a lens diaphragm driving unit 15.

  In this embodiment, since the camera system control unit 5 controls the lens by sending a control signal to the lens system control unit 12, the combination of the camera system control unit 5 and the lens system control unit 12 is the lens. The control unit is configured. The camera system control unit 5, the image sensor 6, and the image processing unit 7 constitute a distance measuring unit. The lens system control unit also functions as a storage unit that stores the backlash amount of the focus lens driving unit 13. The camera system control unit 5 is a subject movement direction calculation unit that calculates the movement direction of the subject based on the distance measurement result over time, and a subject movement speed calculation unit that calculates the movement speed of the subject based on the distance measurement result over time. It has the function of. Further, it also has functions of a focus control determination unit that determines whether or not the subject can be refocused at the current lens position, and an area determination unit that extracts an area within the angle of view based on the distance measurement result. The focus lens drive unit 13 drives the focus lens 4, the shake correction lens drive unit 14 drives the shake correction lens 16, and the lens aperture drive unit 15 drives the lens aperture 17.

  The imaging system generates a photoelectric conversion signal (pixel signal) by forming an optical image from an object on the imaging surface of the imaging element 6 by the imaging optical system 3. Microlenses (hereinafter referred to as ML) are arranged in a lattice pattern on the surface of the imaging element 6 to form a so-called microlens array (hereinafter referred to as MLA). The MLA constitutes pupil dividing means in this embodiment. Light rays from the subject are incident on different pixels of the image sensor 3 through the photographing optical system 6 and the MLA, depending on the position of the light rays on the subject surface according to the angle. Thereby, light field information (pixel data) is acquired.

  The image processing unit 7 includes an A / D converter, a white balance unit, a gamma correction unit, an interpolation calculation unit, and the like, and can generate image data for recording. In addition, refocus processing is performed to reconstruct an image focused on an arbitrary subject by shifting the pixel data so that a specific subject overlaps and synthesizing a single viewpoint image.

  The memory unit 8 includes a processing unit necessary for recording in addition to a recording unit that actually stores data. The memory unit 8 compresses images, moving images, sounds, and the like by this processing circuit using a predetermined method.

  The camera system control unit 5 generates and outputs a timing signal at the time of imaging, and controls an imaging system, an image processing system, and a recording / reproducing system in response to an external operation. For example, the operation detection unit 10 detects that a shutter release button (not shown) is pressed, and controls driving of the image sensor 6, operation of the image processing unit 7, compression processing of the memory unit 8, and the like. Furthermore, the camera system control unit 5 also controls the state of each segment of the information display device for displaying information on a liquid crystal monitor or the like by the display unit 9.

  The adjustment operation of the optical system by the control system will be described. An image processing unit 7 is connected to the camera system control unit 5, and a lens position and an aperture value suitable for shooting conditions are obtained based on a signal from the image sensor 6. When determining an appropriate lens position, distance measurement is performed from a different pupil image by a so-called phase difference method, and the lens position is determined based on the distance measurement result. Alternatively, distance measurement may be performed by comparing and evaluating the contrast of the composite image that is refocused at different focus positions generated by the image processing unit 7. Further, the distance may be measured by a phase difference detection method using an image sensor (not shown) provided separately from the image sensor 6.

  The camera system control unit 5 transmits a command to the lens system control unit 12 via the electrical contact 11, and the lens system control unit 12 appropriately controls the lens driving unit 13 according to the command.

  The image distance reversal in the photographing optical system 3 when the subject reverses its moving direction will be described with reference to FIG. 2A and 2B, the horizontal axis indicates the subject distance, and the vertical axis indicates the image distance. A curve L indicates the relationship between the subject distance and the image distance when the subject is in focus.

  FIG. 2A shows a case where the subject moves in the close direction indicated by the arrow D1 and reverses in the infinite direction indicated by the arrow D2 at the time of the subject distance s. In order to focus on the subject, the lens is driven to change the image distance in the direction indicated by the arrow D3, and then at the time of the image distance s ′ corresponding to the subject distance s where the subject is reversed, the arrow D4 is used. It must be reversed so that the image distance changes in the direction shown. At this time, as described above, the lens cannot be driven to reverse immediately due to backlash, and the reversal of the image distance is delayed accordingly. If the focus position and the imaging surface at this time do not match and the amount of shift between the two is large, the period becomes a time region in which focus cannot be achieved even after refocusing after shooting.

Here, the refocusable range in the refocus processing will be described with reference to FIG. Consider the depth of focus corresponding to the depth of field of the composite image. The refocusable range is determined by the sampling pitch of the angular component of the light beam, the size of the allowable circle of confusion, the size of one pixel in the composite image, and the like. For example, when the size of the permissible circle of confusion of the depth of focus is ε and the sampling pitch of the angle component of the luminous flux is Δu, the refocus coefficient α _± can be given by the following equation (1).

The _± coefficient of refocusing alpha is a coefficient for obtaining the refocus range, α ₊ · s'be represented using any of a refocus coefficient and the image distance _{(s) ~α - · s'(} s) is the image side This is the refocusable range. The refocusable range on the image side and the range conjugate to the photographing optical system are the refocusable range on the object side. In this range, a focused image (refocused image) can be reconstructed from pixel data by post-processing while shooting at a lens position where the focus position and the imaging surface do not match.

  In this embodiment, the refocusing image reconstruction is used to control the driving position of the focus lens so as to absorb the moving object tracking delay due to backlash. Specifically, the drive position of the focus lens is shifted in advance by the image distance within the refocusable range in a direction in which the change in the image distance is delayed before the reversing operation is necessary. By doing this, the shot image is out of focus by the amount of shift of the drive position, but the actual lens position at the time of reverse drive becomes the position that has been driven by the image distance already shifted from the reverse position, and only by the amount of shift The moving object tracking delay can be absorbed. In addition, the out-of-focus blur of the captured image can be substantially corrected by reconstructing and displaying the refocused image at the time of capturing.

  In the case shown in FIG. 2A, since the lens drive in the reverse direction D4 is delayed, the image distance is shifted to the image distance s ′ (s) −ds′1 in the direction indicated by the arrow D4. By doing so, since the drive position of the focus lens is shifted in the direction in which the drive is delayed, it is possible to reduce the drive delay due to the backlash that occurs during the reversing operation of the subject. In addition, since the image-side refocusable range shown in FIG. 2A includes the image distance s ′ (s) in which the subject is in focus, an in-focus image can be acquired in the post-shooting process. Here, the lens position at which the focused image distance s ′ (s) is obtained without performing the refocus processing is set as the first lens position, and the image distance s ′ (s) shifted in the direction opposite to the moving direction D1 of the subject. The lens position where −ds′1 is set as the target lens position. By shifting the image distance in this way, it is possible to reduce the ratio of acquiring an out-of-focus image due to backlash during moving object tracking.

  FIG. 2B shows a case where the moving / reversing direction of the subject is opposite to that in FIG. The subject moves in the infinite direction as indicated by the arrow D5, and reverses in the closest direction as indicated by the arrow D6 at the time of the subject distance s. In order to focus on the subject, the lens is driven to change the image distance in the direction of the arrow D7, and then at the time of the image distance s ′ corresponding to the subject distance s where the subject is inverted, the direction of the arrow D8 The lens must be driven in reverse so that the image distance changes in reverse. In order to reduce the backlash drive delay at this time, the direction in which the image distance is shifted is the direction of the arrow D8. Therefore, the lens position corresponding to the image distance s ′ (s) + ds′2 in FIG. This is the lens position.

  An example of moving object tracking when the lens position described above is set as the target lens position will be described with reference to FIG.

  In FIG. 4A, the horizontal axis represents time, the left first axis represents the subject distance, and the right second axis represents the lens position. A curve 41a is a curve indicating the subject distance with the left first axis as an axis, and is reversed from the movement toward the close side to the movement toward the infinity side. A curve 41b indicates a lens position for focusing on a subject at a subject distance indicated by the curve 41a with the second axis on the right as an axis. The lens needs to be reversed to drive to the imaging surface side after being driven to the subject side. A short broken curve 42b has the lens position of the right second axis as an axis, and shows an actual lens position when the lens is driven according to the curve 41b. It is a curve which shows the lens position which shifted | deviated from the curve 42b by the backlash when a to-be-photographed object determined the moving direction, and drives a lens according to the conventional following curve. A long broken curve 43b is centered on the lens position of the right second axis, and shows a tracking curve for performing moving object tracking according to this embodiment.

  The following curve 42b will be described. The tracking curve 42b coincides with the lens position curve 41b before the time t1 when the subject is reversed. A time region 44 from time t1 to time t2 indicates a time region in which the lens position does not change due to a delay due to backlash. At time t2, the backlash disappears and the lens starts to move. Since the tracking curve 42b follows the lens position curve 41b so as to coincide with the lens position curve 41b, the lens position is changed at the maximum lens driving speed to try to catch up with the lens position 41b. At time t3, the lens position catches up with the lens position curve 41b, and the subsequent movement position coincides with the lens position curve 41b.

  The tracking curve 43b gives the lens position shifted in the opposite direction to the traveling direction of the subject so that the refocusable range shown in FIG. 3 includes the lens position curve 41b. Since the subject has moved from infinity to the closest point in time before time t1 in FIG. 4A, the lens position is shifted to a lens position corresponding to the subject position on the more infinite side. Then, after the time t2 after the inversion, since the subject has moved from close to infinity, the lens position is shifted to a lens position corresponding to the closer subject position.

  With reference to FIG. 4B, the amount of deviation between the focus position and the imaging surface when the lens movement follows the tracking curve 42b or 43b will be described. The horizontal axis is time as in FIG. 4A, the left first axis indicates the lens position, and the right second axis indicates the difference between the focus position and the imaging surface position.

  A curve 41b shows a lens position curve similar to that shown in FIG. 4A with the left first axis as an axis. A curve 42c indicates the amount of deviation between the focus position and the imaging surface when the tracking is performed on the tracking curve 42b with the right second axis as an axis. A curve 43c indicates the amount of deviation between the focus position and the imaging surface when following the curve 43b with the second right axis as the axis. A straight line 46 indicates a threshold value within a refocusable range with the second axis on the right as an axis. If the amount of deviation between the focus position and the imaging surface is equal to or less than this value, the focus is achieved by refocus processing after shooting. An image can be obtained.

  First, the curve 42c of the amount of deviation between the focus position and the imaging surface by the tracking curve 42b will be described along the time axis. The tracking curve 42b has almost no deviation until time t1 when the subject is inverted and affected by backlash. A shift amount occurs after time t1, and as a result of integration, the shift amount exceeds the refocusable range at time t4. Since the delay due to backlash disappears at time t2 and the lens resumes movement, the amount of deviation between the focus position and the imaging surface again falls below the threshold value that is within the refocusable range at time t5. At time t3, the tracking curve 42b catches up with the lens position curve 41b, so that there is almost no deviation thereafter. The time region 45 is a time region in which an in-focus image cannot be acquired even after refocusing after shooting, and this time region 45 is generated due to a delay due to backlash in the lens movement indicated by the tracking curve 42b.

Next, the curve 43c of the amount of deviation between the focus position and the imaging surface (difference between 41b and 43b) by the tracking curve 43b will be described along the time axis. In the deviation amount curve 43c, the deviation amount is a constant value ds ′ _adj until time t1 when the subject is inverted and affected by backlash. (Ds ′ _adj corresponds to ds′1 or ds2 described above.) Since backlash occurs at time t1, the amount of deviation changes, but the lens position is shifted to the reverse side in advance. The amount of deviation due to the delay is suppressed as compared with the tracking curve 42b. In the case shown in FIG. 4B because 41b and 43b intersect, the shift amount is not more than the threshold value within the refocusable range even at time t2 when the lens position can be driven, and also by the refocus processing after shooting. There is no time domain in which an in-focus image cannot be acquired. Moreover, focus is achieved at the point where the curves 43b and 41b intersect.

  As described above, by disposing the lens position so that the focus position corresponding to the position of the subject is deviated from the imaging surface, it is possible to reduce the rate of out-of-focus image acquisition during tracking of a moving body affected by backlash. .

  Next, the derivation of the lens position shift amount to compensate for the delay due to backlash will be described. The shift amount may be determined based on how much the subject moves (subject movement information) while the lens runs idle by a backlash amount (for example, a time period). For example, the time required for the backlash removal is calculated from the backlash amount of the lens and the driveable speed of the lens driving unit, and the moving amount ds′3 of the focus position is calculated from the moving speed of the subject on the image side at that time. Good. At this time, (backlash amount) ÷ (drive speed) ÷ (image-side subject moving speed) = (ds′3). This amount ds′3 is an amount to shift the focus position. Note that the amount of backlash and the lens drivable speed used for the calculation are memory values as driving characteristic information unique to the lens, and the time required for rattling can be similarly used as the memory value.

FIG. 5 is a graph showing the subject distance dependency characteristic of the amount ds′4 for shifting the focus position by the refocusing process. Here, the amount ds′4 that shifts the focus position is not a so-called refocusable range, but indicates a shift amount that allows the subject to be focused from the shifted focus position by refocusing. The amount by which the focus position is shifted can be expressed as | s ′ _{ref ±} (s) −s ′ (s) |. s ′ _{ref ±} (s) indicates the image distance when the image distance s ′ (s) corresponding to the subject is shifted in the directions of the farthest closest side / infinite side including the refocusable range, The refocus coefficient α _± may be used to give the following equation (2).

  The amount by which the focus position is shifted by refocusing does not asymptotically approach zero even in the infinite region, has a certain width, and increases in the close region. Note that the refocusable range on the subject side is narrower on the closest side and wider on the infinite side due to the relationship between the subject distance and the image distance shown in FIG.

  The shift amount of the focus position in the present invention is set according to the magnitude relationship between the shift amount ds′3 and the shift amount ds′4. Hereinafter, the determination of the target focus position including the setting of the shift amount will be described with reference to the process flowchart of FIG.

  FIG. 6 is a flowchart for determining the target focus position of the lens of the present invention. The process shown in FIG. 6 is performed by the camera system control unit 5.

  In step S <b> 61, the camera system control unit 5 acquires the backlash amount of the focus lens 4. This backlash amount may have a value in the memory unit 8 in the camera 1, or the lens system control unit 12 may have a value.

  In step S62, the camera system control unit 5 acquires the movement direction and the movement speed as the movement information of the subject. These values are calculated by the subject movement direction calculation unit and the subject movement speed calculation unit from the subject distance with time acquired during moving body tracking.

  In step S63, the camera system control unit 5 obtains the focus position shift amount ds′3 that compensates for the delay due to the backlash as described above from the lens backlash amount and the subject moving speed. In addition, a shift direction is set in the direction opposite to the subject movement direction acquired in step 2.

  In step S64, the camera system control unit 5 acquires the subject distance.

  Subsequently, in step S65, the camera system control unit 5 acquires an amount ds′4 for shifting the focus position by refocus using the subject distance. The amount by which the focus position is shifted by refocusing is also determined by the subject distance and the configuration of the imaging optical system 3, and similarly, the camera system control unit 5 acquires a value from the reference table based on information acquired from the lens system control unit 12. May be. Moreover, you may obtain | require by calculation at any time using the above-mentioned Formula (2).

  In step S66, a conditional branch is made based on the magnitude relationship between the two quantities.

  If the focus position shift amount ds′3 that compensates for the delay due to backlash is equal to or less than the amount ds′4 obtained by multiplying the focus position by refocus and the safety factor, the process proceeds to step S67.

  In step S67, the focus position that compensates for the delay due to backlash is set as the target focus position. That is, the shift amount is ds′3, and the shift direction is opposite to the subject moving direction. Here, the safety factor avoids the original focus position corresponding to the subject at the end of the refocusable range by refocusing, and is a value greater than 0 and less than 1. Further, the safety factor may be determined in consideration of a specific configuration of the photographing optical system 3 and driving characteristics of the lens driving unit. At the target focus position set in step 7, a delay due to backlash can be compensated.

  If it is determined in step S66 that the focus position shift amount ds′3 that compensates for the backlash delay is larger than the amount ds′4 obtained by refocusing and the amount ds′4 multiplied by the safety factor, the process proceeds to step S68.

  In step S68, the target focus position is shifted and set to a position that compensates for the backlash delay within a range obtained by multiplying the amount ds'4 that shifts the focus position by the safety factor. Even at the target focus position set in step 8, the backlash delay can be compensated as compared with the conventional case.

According to the present embodiment, the ratio of the out-of-focus image acquisition due to the delay due to the backlash can be reduced by setting the target focus position of the lens by the above procedure.
Next, a modification of the above-described embodiment will be described with reference to FIGS. This modification compensates for insufficient focus adjustment due to refocusing of the refocused image because the subject is likely to be out of the refocusable range due to the moving object follow-up delay due to backfashing and the speed of movement of the subject. Provide a configuration for.

  FIG. 7 is a diagram for explaining the relationship between the stop and the sampling pitch of the angle component. This figure shows an example of the relationship between the aperture and the sampling pitch of the angle component when the pupil area to be divided is 2 × 2, as shown on the left side of the figure, and how the pupil is divided on the exit pupil plane It shows.

Consider a case where the lens aperture, which is the aperture of the exit pupil, is stopped from the state 70 to 71. At this time, the sampling pitch of the angle component that is the interval between the divided pupils is reduced from Δu ₇₀ to Δu ₇₁ . At this time, according to the relationship shown in the equation (1), the refocusable range is expanded as the sampling pitch Δu of the angle component is reduced. That is, in the above-described pupil division configuration, the refocusable range can be expanded by narrowing the lens aperture. In the present invention, in the situation where the subject moves out of the refocusable range shown in FIG. 4A, if the refocusable range can be expanded, the out-of-focus image acquisition rate at the time of moving object tracking It can be reduced more. Therefore, the refocusable range may be expanded by narrowing the lens aperture just before the subject is out of the refocusable range. Therefore, the lens target position determination operation in this modification is the same as that in the first embodiment. The operation of this modification is performed based on a determination as to whether or not the subject is out of the refocusable range when shooting is performed by driving the lens according to the determined target position of the lens.

  Whether the subject is out of the refocusable range can be determined from the distance measurement result by the distance measuring unit and the subject-side refocusable range at the current lens position. For example, the determination can be made based on whether the distance measurement result is included in the subject-side refocusable range obtained by multiplying the subject-side refocusable range at the current lens position by a safety factor of 1 or less. These processes are performed in the camera system control unit 5 which is the focus control determination unit described above. Therefore, the configuration of the imaging apparatus is the same as that of the first embodiment.

  FIG. 8 is a diagram for explaining an image acquired when the lens aperture is narrowed just before the subject is out of the refocusable range. Reference numerals 80a, 80b, 80c, and 80d denote images reconstructed so as to focus on the main subject 81 in the refocus reconstruction process after photographing with respect to the images photographed along the time axis. . Reference numerals 81a, 81b, 81c, and 81d denote main subjects 81 that are reversed to the infinite side while moving to the closest side. Reference numerals 82a, 82b, 82c, and 83d denote backgrounds 82 in the respective images. The oblique lines shown in the figure schematically indicate the degree to which the image is out of focus and are more blurred as the distance between the oblique lines is narrower.

  Further, the reconstructed image 80c shows an image in which the lens aperture is narrowed down because the lens drive does not catch up due to backlash and is likely to fall out of the refocusable range with respect to the main subject 81 that is reversed from the nearest to infinity. . That is, the lens aperture is narrowed down from the aperture state set by the camera system control unit 5 or the aperture state set by the user. The reconstructed image 80d shows a state in which the lens aperture is returned because the backlash disappears and the driving of the lens is resumed and the main subject 81 returns to the refocusable range.

  Attention is now paid to the blurred state of the main subject 81 and the background 82 along the time axis. In the reconstructed images 80a and 80b, the main subject 41 is focused by lens driving and refocus processing. In the reconstructed image 80c, the lens drive cannot catch up and the refocus processing cannot be performed, but the refocusable range is expanded because the lens aperture is narrowed down. As a result, refocus processing can be performed on the main subject 41, and a reconstructed image focused on the main subject 81 can be generated. Since the lens drive catches up with the main subject 81 in the reconstructed image 80d, the main subject 81 returns to the refocusable range at the original aperture setting value again. For this reason, the lens aperture is returned to the opened state as originally set, and in this state, an image focused on the main subject 81 can be generated by the refocus processing. As described above, when the reconstructed image 80 is viewed along the time axis, the focus state of the main subject 81 is always in focus and does not change.

  Next, focus on the background 82. The focus state of the background 82 is blurred in the reconstructed images 80a and 80b. When the lens aperture is narrowed down in the reconstructed image 80c, the blur is weaker than in the reconstructed images 80a and 80b. Then, in the reconstructed image 80d, the lens aperture is returned to the aperture value in the reconstructed images 80a and 80b, so the blurred state of the background 82 also returns to the state of the reconstructed images 80a and 80b. When the focus state of the background 82 is viewed along the time axis, the blur is weakened only in the reconstructed image 80c. When viewing continuously acquired images as a moving image, the discontinuity of the blurred state of the background 82 is not a concern if the time for narrowing the lens aperture is sufficiently short. On the other hand, even in the case of viewing as a moving image, the discontinuity in the blurred state of the background 82 gives a sense of incongruity and cannot be ignored if the time for narrowing down the lens aperture becomes long. Further, in the case where the continuously acquired still images are simultaneously output and viewed, the discomfort between the still images is similarly given and cannot be ignored. Therefore, in such a case, it is desirable to reduce the discontinuity by performing image processing on the composite image 80c. For example, in the camera system control unit 5 serving as the region determination unit, the region of the background 82 is extracted based on the distance measurement information s (x, y) for each in-view angle position acquired by the distance measurement unit. A method of adding blur to the background 82 by image processing may be applied.

  Also according to the above-described modification, in the light field camera having the refocus function, it is possible to reduce the ratio of shooting an out-of-focus image at the time of shooting.

  As described above, in the present invention, unlike the conventional moving object tracking method, the moving object tracking is performed by driving the lens in a lens state in which the focus position of the main subject 81 is always deviated from the imaging surface. Then, when generating a reconstructed image from the acquired image in real time, refocus processing is performed to focus on the main subject 81.

  In addition, it is considered that the processing capability of the image processing unit 7 is not sufficient, the processing takes time, the processing is not in time, and the focused image cannot be output to the main subject 81 on the display unit 9 in real time. It is done. As a countermeasure in such a case, a reconstructed image may be generated using only the central single viewpoint of the divided pupil without performing the refocus processing in real time. Generation of the reconstructed image from only the central single viewpoint has a light processing load, and the generated image has a deep depth of field, so that it is difficult to recognize the out-of-focus of the main subject 81. Thereby, the uncomfortable feeling of the display part 9 can be reduced.

  As described above, according to the present invention, in a light field camera having a refocus function, it is possible to reduce the ratio of shooting an out-of-focus image during shooting.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, it goes without saying that the object of the present invention can also be achieved when the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the functions of the above-described embodiment can also be realized when an OS (basic system or operating system) operating on the computer performs part or all of the actual processing based on the instruction of the program code read by the computer. Realized. Needless to say, this case is also included in the present invention.

  Furthermore, after the program code read from the storage medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the processing based on the instruction of the program code is also performed. Included in the invention. That is, it goes without saying that the present invention also includes the case where the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code to realize the functions of the above-described embodiment. Yes.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (13)

A taking optical system including a focus lens;
An imaging unit that captures an optical image of a subject formed by the imaging optical system and generates refocusable pixel data;
Ranging means for obtaining the distance of the subject;
Lens driving means for driving the focus lens according to a lens position determined based on the acquired subject distance;
In an imaging apparatus comprising:
Based on the movement information of the subject based on the subject distance and the driving characteristic information of the lens driving means, a target lens position having a deviation amount within a refocusable range with respect to the lens position is determined, and the lens driving An imaging apparatus comprising: control means for controlling the means to drive the focus lens according to the target lens position.   The movement information of the subject is information on the movement speed and movement direction of the subject, the drive characteristic information of the lens driving means is the driving speed and backlash amount of the focus lens, and the control means is the lens The imaging apparatus according to claim 1, wherein the target lens position having an amount of deviation from a position is determined in a direction opposite to the moving direction of the focus lens corresponding to the moving direction of the subject.   Storage means for storing the driving speed and backlash amount of the focus lens, and the control means determines subject movement speed and direction based on the subject distance over time acquired by the distance measuring means. A speed calculation unit, and the amount of deviation of the target lens position is determined based on the backlash amount, the driving speed of the lens driving unit, the moving speed of the subject, and the size of the refocusable range The imaging apparatus according to claim 2, wherein the imaging device is determined according to a result of comparison with a threshold based on the length.   The photographing optical system includes a lens diaphragm, the imaging device includes a diaphragm driving unit that drives the lens diaphragm, and the control unit is capable of refocusing on a distance measurement result of the distance measuring unit and a lens position. And a focus control determination unit that determines whether an image distance corresponding to the subject distance is out of the refocusable range, and controls the aperture driving unit according to the determination result of the focus control determination unit The imaging apparatus according to any one of claims 1 to 3, wherein the imaging apparatus includes:   When it is determined that the image distance corresponding to the subject distance is out of a refocusable range, the control unit controls the diaphragm driving unit to stop the lens diaphragm, and then corresponds to the subject distance. The imaging apparatus according to claim 4, wherein when it is determined that the image distance returns to a refocusable range, the lens diaphragm is returned to an original state.   The control means includes area determination means for determining a main subject and a background area based on the object distance, and when the lens diaphragm is stopped according to a determination result of the focus control determination means, the image generation means The image pickup apparatus according to claim 5, wherein different image processing is performed on the main subject and the background by controlling the main subject and the background.   Image generation means for reconstructing a refocus image from pixel data generated by the imaging means, and display means for displaying the refocus image generated by the image generation means, wherein the control means includes the image generation means The imaging apparatus according to claim 1, further comprising: a display unit configured to control the display unit to perform reconstruction and display of the refocused image when the subject is captured.   The control unit controls the image generation unit and the display unit, and only the central single-viewpoint pixel data among the pixel data generated by the imaging unit instead of the reconstruction of the refocused image at the time of photographing the subject. The image pickup apparatus according to claim 7, wherein an image is generated from the image and displayed. A taking optical system including a focus lens;
An imaging unit that captures an optical image of a subject formed by the imaging optical system and generates refocusable pixel data;
Ranging means for obtaining the distance of the subject;
In a method for controlling an imaging apparatus comprising:
A lens driving step of driving the focus lens according to a lens position determined based on the acquired subject distance measurement;
The target movement information and the driving characteristic information of the lens driving step based on the subject distance are acquired, and the target having a deviation amount within a refocusable range with respect to the lens position based on the acquired information. A control method comprising: a control step of determining a lens position, controlling the lens driving step, and driving the focus lens according to the target lens position.   A program for causing a computer to execute the control method according to claim 9.   A computer-readable storage medium storing the program according to claim 10.   A program that causes a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 8.   A storage medium storing a program that causes a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 8.

Images