JP2017139576A - Imaging device, and control method and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of improving the accuracy of refocus processing, by reducing the impact of rolling shutter distortion.SOLUTION: An imaging device includes: an image pick-up device having multiple pixels arranged two-dimensionally, and capable of acquiring an image signal containing angle information of an incident light beam; focus adjustment means for focusing an imaging optical system on a subject; reading means capable of reading a signal of a pixel of the image pick-up device, at a different timing depending on the position of the pixel; and control means for controlling the focus adjustment means so as to focus the imaging optical system on a moving body, when the reading means reads a signal of a pixel at a different timing depending on the position of the pixel, and when the moving body exists in the subject, i.e., the shooting object.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a program.

  There has been proposed a light field camera (also referred to as an LF camera) that can acquire the image signal while maintaining the angle information of the light beam from the subject by dividing and recording the light beam that has passed through the microlens array ( Non-patent document 1). By using the image signal recorded by the LF camera, it is possible to reconstruct an image in which the viewpoint is changed after shooting or an image in which the focus state is changed (also referred to as a refocus process).

  The LF camera requires an optical component such as a microlens array, and distortion may occur depending on the assembly accuracy, so adjustment thereof is necessary. On the other hand, Patent Document 1 is an image pickup apparatus including a photographing optical system and a microlens array, and detects distortion of components constituting the apparatus, and a detection result of the distortion and a predetermined configuration. A device that corrects the distortion of a captured image based on the distortion characteristics of the member is disclosed.

Ren. Ng, 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

JP 2011-19088 A

  By the way, even when the assembly accuracy of the optical components is improved, when the image sensor used in the LF camera is, for example, a CMOS image sensor, distortion inherent to the image sensor may occur in the captured image. For example, rolling shutter distortion is caused by an image sensor reading method in which pixel readout timing on the image sensor varies depending on the position. When an image sensor that employs such a readout method is used, distortion occurs in a subject (also referred to as a moving object) that moves at high speed. In particular, when there is a moving object that is blurred when shooting while focusing on a stationary main subject, the moving object is further distorted due to the spread due to the blur. Therefore, if the refocusing process for focusing on this moving object is performed after shooting with the LF camera, the distortion of the moving object appears more greatly.

  The present invention has been made in view of the above-described problems of the prior art, and is capable of reducing the influence of rolling shutter distortion and improving the accuracy of refocus processing, and a control method and program therefor The purpose is to provide.

  In order to solve this problem, for example, an imaging apparatus of the present invention has the following configuration. That is, an imaging device having a plurality of pixels arranged two-dimensionally and capable of acquiring an image signal including angle information of incident light rays, and a focus adjusting unit for focusing the photographing optical system on a subject, This is a case where the readout means that can read out the pixel signal of the image sensor at different timings according to the position of the pixel, and the readout means read out the pixel signal at different timings according to the position of the pixel. And a control means for controlling the focus adjustment means so that the focus of the photographing optical system is adjusted to the moving object when the moving object is present in the subject to be imaged.

  According to the present invention, it is possible to reduce the influence of rolling shutter distortion and improve the accuracy of refocus processing.

The figure explaining the influence of rolling shutter distortion considered in the embodiment of the present invention 1 is a block diagram illustrating a functional configuration example of a digital camera as an example of an imaging apparatus according to Embodiment 1. FIG. 6 is a flowchart showing a series of operations of imaging processing according to the first embodiment. FIG. 5 is a diagram for explaining the principle of imaging processing according to the first embodiment. FIG. 6 is a diagram schematically illustrating refocus processing according to the first embodiment. FIG. 6 is a diagram schematically illustrating an effect of shooting processing according to the first embodiment. The figure which illustrates typically the case where the imaging | photography process which concerns on Embodiment 1 is not applied. 8 is a flowchart showing a series of operations of shooting processing according to the second embodiment.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, an example in which the present invention is applied to an arbitrary digital camera including a microlens array and a focusing lens will be described as an example of an imaging apparatus. However, the present invention is not limited to a digital camera, and can be applied to any device including a microlens array and a focusing lens. These devices may include, for example, mobile phones, game machines, tablet terminals, personal computers, clock-type or glasses-type information terminals, medical devices, and the like.

(Outline of light field camera)
First, after explaining the focus state of a general digital camera that is not a light field camera (also simply referred to as an LF camera), the principle of the LF camera will be described.

  In the focus state of a general digital camera, the position of a focusing lens is adjusted manually by a photographer or automatically by an automatic focus adjustment (autofocus) function at the time of shooting. Although the captured image signal can change the sharpness of the subject by adjusting the sharpness by digital image processing, for example, information that has already been optically blurred cannot be recovered. This is because the optical information is three-dimensional information including the depth, but when this optical information forms an image on the imaging surface of the solid-state image sensor of the digital camera, the information is converted into two-dimensional information. This is because it is converted. In other words, in a general digital camera, the incident light beam is recorded from where on the object plane facing the imaging surface of the solid-state image sensor, and what angle distribution of the incident light has? Not recorded. In other words, a general digital camera records a light beam generated from a certain point of an object after integrating it in an angle range corresponding to the aperture diameter of the photographing optical system. Specifically, when the two-dimensional coordinates of the diaphragm surface of the photographing optical system are (u, v), the information Eb (s, t) recorded on the two-dimensional coordinates (s, t) of the recording surface of the solid-state imaging device or the like. Is

It is the quantity represented by. Here, b represents the imaging distance from the rear principal point of the photographic optical system to the imaging surface, and Lb (s, t, u, v) represents the distance from the subject under the imaging distance b of the photographic optical system. Of the light rays, the light rays that pass through the coordinates (u, v) of the diaphragm surface and reach the coordinates (s, t) of the imaging surface are represented. The imaging distance b is uniquely determined by driving the focusing lens of the photographing optical system, and rays reaching the coordinates (s, t) of the imaging surface under the imaging distance b are also uniquely determined by an imaging formula described later. It is determined. For this reason, Lb (s, t, u, v) represents a light beam from the subject that passes through the coordinates (u, v) of the aperture plane under the condition of driving the focusing lens of the photographing optical system.

  On the other hand, the LF camera can record the light beam while maintaining the angle information of the light beam from the subject by a technique called light field photography. The LF camera, for example, forms a subject image mainly on a microlens array in which microlenses are arranged at a predetermined pitch, and receives light rays incident by a solid-state imaging device with a finer pitch arranged behind the microlens array. Record. Note that an image signal including such light angle information is also referred to as light space information.

  The light beam recorded in each pixel of the solid-state imaging device having a finer pitch than the pitch of the microlens array is light that has passed through a partial area of the stop of the photographing optical system in the image formation state on the microlens array. Here, the light beam that passes through a partial area of the stop is a light beam emitted from a certain point of the subject and emitted at a specific angle. Therefore, by providing pixels with a finer pitch than the microlens array, it is possible to divide and record a large number of light beams by an angle corresponding to the finer pitch. Since a plurality of light beams can be divided and recorded, an image with a changed viewpoint can be acquired, and refocus processing for changing the focus state after shooting can be performed.

(Principles of rolling shutter distortion)
The principle of generation of rolling shutter distortion is known as distortion generated in the structure of a CMOS image sensor. In the CMOS image sensor, two steps are executed: first, these pixels are exposed to pixels arranged in two dimensions horizontally and vertically, and secondly, the charges of these pixels are sequentially read out. To do. When a mechanical shutter can be used together with exposure, light can be shielded during sequential readout of pixels, so that rolling shutter distortion can be avoided. On the other hand, when the mechanical shutter cannot be used in combination, sequential readout of the pixel charges (ie, the exposure is completed when the accumulation is completed) and sequential readout of the pixel charges in the previous frame (ie, the start of the exposure when the accumulation starts) Is done). The sequential readout of the charge of the pixel includes an XY address type and a vertical sequential scanning type. In either method, the pixel existing at a certain coordinate on the solid-state image sensor is the same as the pixel existing in another vertical row. Have different accumulation start and accumulation end times. Although the time difference between the accumulation start and accumulation end (that is, the accumulation time) is kept constant for all pixels, the subject moving at a high speed on the screen or the subject moving at a relatively low speed due to the time difference of reading. Even so, distortion occurs in a subject having a large proportion of the angle of view. For example, when the subject existing in the horizontal coordinates x1 to x2 moves at the horizontal speed vh when the first row is read, the distortion moves to the horizontal coordinates x1 + 100 vh to x2 + 100 vh when the 100th row is read. It occurs because of being.

(Rolling shutter distortion in LF camera)
Furthermore, a case where such a CMOS image sensor in which rolling shutter distortion occurs is applied to an LF camera will be described with reference to FIG. In FIG. 1, a photographing optical system 1 includes a stop, and its focal length is f. The microlens array 2 has microlenses arranged two-dimensionally and is disposed between the solid-state imaging device 3 and the photographing optical system 1. In addition, the subject A1 and the subject A2 that are the photographing targets exist at different subject distances. In the example of FIG. 1, a subject A1 at a subject distance a1 focused by the photographing optical system 1 is imaged at the apex of the microlens array 2. That is, according to the imaging formula of the photographing optical system 1, the distance b1 from the rear principal point of the photographing optical system 1 to the apex of the microlens array 2 is
b1 = (a1−f) / f · a1 (Formula 2)
It is represented by

The subject A2 is a subject whose subject distance a2 satisfies the relationship of a2 <a1. Since the subject A1 at the subject distance a1 is in focus, the imaging distance b2 of the subject A2 photographed at the same time is
b2 = (a2−f) / f · a2 (Formula 3)
(Where b2> b1). That is, the light beam from the subject A2 forms an image on the side farther from the rear principal point of the photographing optical system 1 than the light beam from the subject A1. Since the solid-state imaging device 3 exists between b1 and b2, the light rays (in the angular range limited by the stop of the photographing optical system 1) all intersect at an imaging point that satisfies Equation 3 before Recording is performed on the solid-state imaging device 3 in a state where the projection area is widened. For this reason, the greater the blur of the optical image projected onto the solid-state imaging device 3, the greater the distance between the coordinates on which the subject A2 is projected, and the greater the difference in readout time. That is, when the optical image spreads, it is greatly affected by the rolling shutter distortion of the solid-state imaging device 3.

  In the general digital camera described above, the subject A2 blurred at the time of shooting is displayed as blurred, so it is difficult to recognize the effect of rolling shutter distortion on the blurred subject. However, since the LF camera can generate an image focused on an arbitrary subject after shooting by refocus processing, the greater the influence of rolling shutter distortion due to blur, the more easily the distortion of the subject when refocusing is recognized. Appears in state.

  On the contrary, an increase in rolling shutter distortion due to blur can be suppressed by focusing on a moving object that generates rolling shutter distortion.

(Configuration of digital camera 100)
FIG. 2 is a block diagram illustrating a functional configuration example of the digital camera 100 as an example of the imaging apparatus of the present embodiment. Note that one or more of the functional blocks shown in FIG. 2 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or by executing software by a programmable processor such as a CPU or MPU. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The photographing optical system 1 includes a lens group such as a focusing lens for adjusting a focus and a zoom lens for changing a field angle, and a diaphragm. The microlens array 2 is composed of a plurality of microlenses arranged two-dimensionally at a predetermined pitch horizontally and vertically. A subject image focused by the focusing lens of the photographing optical system 1 is formed on the apex of the microlens.

  The solid-state imaging device 3 has a plurality of pixels arranged in a two-dimensional form, and has a photoelectric conversion function for photoelectrically converting a subject image formed by the photographing optical system 1 and a photoelectrically converted electric signal as an image signal. And a signal transfer function for transfer. The solid-state imaging device 3 is configured such that a plurality of pixels correspond to one microlens of the microlens array 2. When the solid-state imaging device 3 is a CMOS image sensor, signals from each pixel are sequentially transferred for each row by so-called rolling scanning, for example. The A / D conversion unit 4 includes a circuit or a module that converts the sampled analog image signal into a digital image signal. The digitized image signal is stored in the image memory 8.

  The signal processing unit 7 includes a circuit or a module that performs predetermined image processing such as white balance correction and gamma correction on the image signal stored in the image memory 8. The signal processing unit 7 performs refocus processing described later on the image signal. The image signal output from the signal processing unit 7 is recorded on a nonvolatile recording medium 10 such as a memory card. The recording control unit 9 includes an interface circuit with the recording medium 10 and controls writing of an image signal to the recording medium 10 and reading of an image signal from the recording medium 10.

  The timing generator 5 drives an imaging system such as the solid-state imaging device 3. Further, the A / D converter 4 is driven and controlled in synchronization with the drive of the imaging system and, in turn, the output signal of the solid-state imaging device 3. The focusing lens driving unit 6 drives a focusing lens included in the photographing optical system 1. The focusing lens may be the same as that used for focusing on a subject in a generally known digital camera that does not rely on light field photography. The focusing lens driving unit 6 receives a control signal from the system control unit 13 by an automatic focus detection (AF) function or the like, and determines the imaging distance of the photographing optical system 1.

  The display control unit 11 includes a circuit or a module that controls an image displayed on the display device 12. The display device 12 includes a display device such as a liquid crystal display, for example, and displays an image signal subjected to various signal processing via the display control unit 11. The display device 12 realizes a live view display that continuously displays a screen to be imaged live, a playback display of a recorded moving image, a menu display for operation, and the like.

  The system control unit 13 includes, for example, a CPU or MPU, and controls each unit of the digital camera 100 by developing and executing a program stored in a ROM 15 that is a nonvolatile memory in a RAM 14 that is a volatile memory.

  The moving object detection unit 16 detects a moving object (moving object) particularly from moving image frames arranged in time series, and detects the speed of the detected object. The subject detection unit 17 detects a main subject (also simply referred to as a main subject) in the image. The main subject can be determined based on an evaluation value obtained by weighting, for example, that the person is recognized as a person, or that the recognized person is located near the center of the image.

  The refocus distance setting unit 18 sets a refocus distance that is a parameter for executing refocus processing in the LF camera. The refocus distance will be described later.

(A series of operations related to the shooting process of the present embodiment)
Next, a series of operations related to the photographing process will be described with reference to FIG. In this photographing process, a moving object is detected, and the moving object is preferentially focused and photographed, and a refocus process for focusing on a separately detected main subject is performed. Note that this series of operations is started when, for example, an operation from a user on an operation unit (not shown) is detected. Further, this series of operations is realized by the system control unit 13 expanding and executing the program stored in the ROM 15 in the work area of the RAM 14.

  In step S <b> 301, the system control unit 13 is activated in response to the main power being turned on by an operation switch (not illustrated in FIG. 2), and initializes each unit of the digital camera 100 and predetermined operations necessary for the operation of the digital camera 100. Are read from the ROM 15.

  In step S <b> 302, the system control unit 13 performs drive settings for the solid-state image sensor 3. For example, the system control unit 13 sets a drive mode of the solid-state imaging device 3 by transmitting a drive setting signal for setting a live view mode in which live images are continuously displayed on the display device 12.

  In S303, the system control unit 13 performs a so-called automatic exposure adjustment operation (AE). For example, the signal processing unit 7 performs photometry using image signals continuously acquired from the solid-state imaging device 3 in accordance with instructions from the system control unit 13, and outputs the photometry results to the system control unit 13. The system control unit 13 controls the exposure used for photographing the next frame according to the photometric result. More specifically, for example, the signal processing unit 7 performs weighting calculation on luminance information extracted from the image signal for each area of the screen, and calculates how much brighter it should be (to make it darker). . When the solid-state imaging device 3 has an electronic shutter function, the shutter speed of the electronic shutter may be changed at the same time.

  In S304, the system control unit 13 performs so-called autofocus (AF). AF is automatic focus adjustment that changes the focus position by a focusing lens so that a predetermined subject is focused. For example, first, the system control unit 13 drives the focusing lens of the photographing optical system 1 through the focusing lens driving unit 6 in a plurality of steps to obtain a plurality of images. A so-called contrast detection process is performed on a plurality of images to determine the position of the focusing lens that maximizes the evaluation value. Note that the image used in this step to S308 is an image using only the output of one pixel (for example, the center pixel of 5 × 5 pixels) among a plurality of pixels corresponding to one microlens. Use it. In addition to the method using contrast detection, a method using a dedicated distance measuring sensor may be applied to the AF method in this step.

  In step S <b> 305, the system control unit 13 uses the moving object detection unit 16 to detect a moving object in the image. As a method of detecting a moving object, a known method such as a method of calculating a motion vector between consecutive frames can be used. In this embodiment, any moving object in an image and a moving speed of the moving object in the frame can be detected. It may be a method. The system control unit 13 temporarily stores the number of moving objects detected in the frame and the moving speed of each moving object in the RAM 14.

  In step S306, the system control unit 13 determines whether the detection of the fastest moving object in the image is completed. For example, if the system control unit 13 sequentially detects moving objects for each region indicating a predetermined feature of the motion vector detected in the image, the system control unit 13 completes the detection of the moving object for all the regions. It is determined that the detection of the moving object has been completed, and the process proceeds to S307. On the other hand, if the moving object is not detected for all the regions, the process returns to S305 again. The system control unit 13 detects the fastest moving body among the speeds of the moving bodies stored in S305 as the fastest moving body.

  In step S <b> 307, the system control unit 13 drives the focusing lens of the photographing optical system 1 via the focusing lens driving unit 6. For example, the system control unit 13 collates the above-described contrast detection result with the moving object detection result, and drives the focusing lens so as to focus on the fastest moving object. By controlling in this way, an image signal photographed in a state where the moving object is focused, that is, an image signal photographed in a state where the moving object is focused on the apex of the microlens array 2 is obtained. This image signal is an image signal in which the influence of rolling shutter distortion is reduced because the projected area of the moving object onto the solid-state imaging device 3 is small.

  In S308, the system control unit 13 detects the main subject in the image using the subject detection unit 17 (main subject detection processing). The subject detection unit 17 uses, for example, a center-weighted method that weights evaluation values on the assumption that a main subject exists near the center of the image, or a subject recognition method that places importance on the evaluation value of a subject that is determined to be a person. Can do.

  Note that since the focusing lens is driven with priority given to the moving object by the processing of S307, the position of the focusing lens is not necessarily the position at which the main subject is focused. Since the subject that the user is interested in is assumed to be the main subject rather than the moving object, it is possible to refocus the image focused on the main subject based on the image signal obtained by focusing on the moving object. There is a need.

  Therefore, the system control unit 13 determines whether refocus processing for focusing on the main subject is possible. The imaging distance range in which the refocus processing is possible is limited to the range represented by the actual position ± dmax of the focusing lens. Therefore, it is necessary to control the focusing lens at a position where refocus processing for focusing on the main subject is possible and the blur of the moving object is minimized. Dmax is given by Equation 4.

ここで、Δθは光線の角度分解能、Ｎはマイクロレンズアレイ２の１つのマイクロレンズに対応する固体撮像素子３の画素分割数、Δｘは固体撮像素子３の画素ピッチである。

Here, Δθ is the angular resolution of the light beam, N is the number of pixel divisions of the solid-state imaging device 3 corresponding to one microlens of the microlens array 2, and Δx is the pixel pitch of the solid-state imaging device 3.

  For example, the system control unit 13 acquires distance information to the main subject detected in S308 using a dedicated distance measurement sensor (not shown). Then, the system control unit 13 determines whether or not the acquired imaging distance is within the range of the focusing lens driving position ± dmax determined in S307 (that is, whether or not refocus processing for focusing on the main subject is possible). Determine.

  If the system control unit 13 determines that the imaging distance is out of the above range, the process returns to S307 to provide feedback to the focusing lens drive, and the focusing lens is moved again so that the main subject is within the refocus range. To drive. For example, the process of determining whether or not the refocus process for moving the focusing lens by a predetermined distance from the current position and refocusing the main subject is possible is repeated. On the other hand, if the system control unit 13 determines that the imaging distance is within the range, the process proceeds to S310.

  In S310, the system control unit 13 controls the signal processing unit 7 and the refocus distance setting unit 18, and performs refocus processing using the image signal obtained at the position of the focusing lens set in S307. Details of the refocus processing will be described later. In step S311, the system control unit 13 records the image generated by the refocus process, and ends a series of operations related to the process.

(Details of refocus processing)
The refocusing process is a process for reconstructing and rearranging pixel information in consideration of the angle information of light rays. First, FIG. 4 shows a state in which the subject A2 which is a moving object is imaged on the apex of the microlens array 2 by the above-described processing. Unlike the example shown in FIG. 1, the main subject A1 is not focused at this time. A light beam emitted from one point of the subject A1 is divided into the number of divisions (four emission angle ranges for convenience) of the aperture regions p1 to p4 of the photographing optical system 1 and is photoelectrically converted by the solid-state imaging device 3. That is, the light beam emitted from the subject A1 that is not in focus is received in a wide area denoted by reference signs p11 to p14 on the pixels of the solid-state imaging device 3.

  The process of refocusing to the image information at the position of the imaging distance b1 using the signal received at the position of the imaging distance b2 is the result of averaging the pixel values of the signs p11 to p14 in the above-described equation 1. Reconstructing the pixel value at the image distance b1. That is, it corresponds to reproducing the intersection of light rays on the imaging distance b1 as one point from the information of the four light rays received at the imaging distance b2. In other words, this corresponds to removing the defocus of the subject A1.

  On the other hand, a light beam emitted from one point of the subject A2 formed to be focused because it is a moving body is divided into four emission angle ranges like the subject A1, and the pixels p21 to p24 of the solid-state image sensor 3 are divided. Is photoelectrically converted. The light beam emitted from the subject A2 is imaged at the apex of a microlens with the microlens array 2, and is received in a narrow region existing directly below the microlens. For this reason, the image of the subject A2 on the pixels of the solid-state imaging device 3 can be scanned with a small rolling shutter distortion as compared with a case where the image is taken out of focus.

  With respect to the subject A2, using the signal received at the imaging distance b2, the refocusing to the image information at the imaging distance b1 is performed by reversing the pixel order of reference numerals p21 to p24. It is to be. That is, this corresponds to reproducing a plane in which the light beams corresponding to the symbols p21 to p24 have passed the imaging distance b1. In other words, this corresponds to creating a defocused image of the subject A2. It should be understood that the solid-state imaging device 3 having a finer pitch than the pitch of the microlens array 2 is necessary to obtain a defocused image.

  In the case of performing refocusing from the imaging distance b2 to the virtual imaging distance b1 = αb2, a general expression relating to how much these pixels are reconstructed is given by Equation 5. That is, if the two-dimensional coordinates of the diaphragm surface of the photographing optical system 1 are (u, v), the information Eαb2 (s, t) recorded in the two-dimensional coordinates (s, t) of the virtual imaging surface αb2 is ,

と規定される。なお、αをリフォーカス係数と呼ぶ場合がある。

It is prescribed. Α may be referred to as a refocus coefficient.

  Equation 5 can be described with reference to FIG. FIG. 5 shows where in the imaging distance b2 the light ray that constitutes the coordinate s on the plane including the virtual refocus distance b1 among the light rays that have passed through the coordinate u of the aperture region. According to the similarity principle of the triangle, the corresponding coordinate on the plane including the imaging distance b2 is u + (su) / α, so s in Equation 1 is replaced with this value, and t is also v + (t− v) It may be replaced with / α. Therefore, Formula 5 is explained by the conversion formula of Formula 1.

(Effect of shooting processing according to this embodiment)
With reference to FIG. 6 and FIG. 7, the effect of the above-described photographing process will be described. FIG. 6A schematically shows a photographed image obtained at the position of the focusing lens at the time of photographing (that is, before refocus processing), and FIG. 6B schematically shows a generated image after refocus processing. . Since the subject A2 that is a moving object is focused before the refocus processing, it is possible to suppress rolling shutter distortion due to blurring on the subject A2 that is a moving object.

  On the other hand, since the subject A1, which is the main subject, is focused after the refocus processing, the moving subject A2 is in a defocused state, but the rolling shutter distortion before the refocus processing is suppressed. Can be played.

  FIG. 7 schematically shows an example in which the photographing process according to the present embodiment described above is not applied. FIG. 7A schematically shows a photographed image obtained by focusing on the main subject A1 described above with reference to FIG. 1, and FIG. 7B schematically shows a generated image when the subject is refocused on a moving subject A2. Is shown.

  The refocusing process assumed in Equation 5 is to reconstruct pixels from an isotropic range such as a circle or regular polygon reflecting the aperture shape. For this reason, for example, the range of pixels used for the reconstruction of the point X (s0, t0) after the refocus is the area of Xcir (s0, t0) before the refocus processing. Therefore, the contour extending in the horizontal direction of the subject A2 included in Xcir (s0, t0) is only affected by optical defocusing, and is thus reconstructed as a horizontal line after the refocus processing.

  On the other hand, the original vertical line included in Xcir (s0, t0) is recorded with an inclination in the moving direction of the subject obtained by adding rolling shutter distortion to optical defocusing. For this reason, a part of the pixels constituting the original vertical line departs from Xcir (s0, t0) and is reconstructed as an unclear contour. That is, if the influence of rolling shutter distortion increases, it seems as if the effect of eliminating blur differs between the vertical direction and the horizontal direction.

  Thus, when it is desired to obtain an image focused on the subject A2, if the captured image of FIG. 6 (a) is used, a moving object with reduced outline blur both vertically and horizontally compared to FIG. 7 (b). You can see that

  In the above-described embodiment, the series of operations illustrated in FIG. 3 has been described as the entire photographing operation of the LF camera. However, the above series of operations may exist as one mode of the LF camera. For example, even in a system in which a mechanical shutter can be used in combination, it is difficult to use all frames in a moving image mode having a large number of frames per hour due to restrictions on the operation speed of the mechanical shutter. For this reason, the rolling shutter distortion peculiar to a CMOS image sensor cannot be avoided in principle. Therefore, if the processing according to the present embodiment is applied to a moving image that does not use a mechanical shutter as the above-described one mode in the LF camera, a moving image in which a moving object is suppressed and the refocusing to the main subject is always reproduced, and the user feels uncomfortable. It is possible to present an image with little.

  In addition, an image signal recorded with priority given to a moving object can be recorded with a flag indicating that the image has been taken by driving a focusing lens (taken in a state where the moving object is in focus). In this way, even when shooting with a system in which refocus processing in the imaging device is difficult due to restrictions on processing time or the like, the external device that has read the captured image signal will focus on the main subject. It can be understood that the focus processing is necessary. That is, even when an external device reproduces a captured image signal, it is possible to reproduce a refocused image focused on the main subject as a default and present an image without a sense of incongruity.

  In the present embodiment, the example in which the focusing lens is controlled according to the fastest moving body has been described. However, in addition to the speed of the moving object, taking into account other situations where the rolling shutter distortion of the moving object increases (for example, when the moving object occupies a large area in the shooting angle of view), the focusing lens is controlled to a moving object that satisfies the other conditions. May be. In addition, when there are moving objects having the same moving speed, it is possible to determine a mode in which the magnitude of distortion changes depending on the moving direction vector of the moving object, and to control the focusing lens to a moving object that is more affected by rolling shutter distortion. Good. In this case, for example, when obtaining the speed of the moving body in S306, a method of giving importance to the movement vector parallel to the sub-scanning direction of the solid-state image sensor 3 can be considered.

  In the above-described embodiment, the configuration in which the main subject is detected in S308 has been described as an example. However, for example, when the user specifies a subject via an operation unit such as a touch panel (not shown), the system control unit 13 may perform the processing from S309 on the designated main subject. In this way, an image focused on the user's desired subject can be generated more easily.

  As described above, in the present embodiment, in the LF camera that can acquire the angle information of the incident light beam, the moving object is detected, and the focusing lens is controlled so as to match the focus with the detected moving object. In addition, refocus processing is performed using an image signal captured with a moving object in focus, and an image focused on a separately detected main subject is output. By doing in this way, the rolling shutter distortion which generate | occur | produces in a moving body can be reduced, and the distortion at the time of performing the refocus process which focuses on the said moving body can be reduced. In other words, it is possible to reduce the influence of rolling shutter distortion and improve the accuracy of refocus processing.

(Embodiment 2)
Next, Embodiment 2 will be described. In the first embodiment, when the fastest moving object is detected, the focusing lens is driven to focus on the moving object. In the second embodiment, the focusing lens is preferentially driven to a moving object that is assumed to have significant rolling shutter distortion, considering the speed of the moving object and the size of the detected moving object in the angle of view. Different. The digital camera 100 of the present embodiment has substantially the same configuration as that of the first embodiment, and only the points described above in the shooting process are different. For this reason, the same configurations and steps are denoted by the same reference numerals, and redundant description will be omitted, and differences will be mainly described.

(A series of operations related to the imaging processing of the present embodiment)
With reference to FIG. 8, a series of operations related to the imaging process of the present embodiment will be described. Note that this series of operations is started, for example, when a user operation on an operation unit (not shown) is detected, as in the first embodiment. Further, this is realized by the system control unit 13 expanding and executing the program stored in the ROM 15 in the work area of the RAM 14.

  First, the system control unit 13 performs the processes of S301 to S306 as in the first embodiment.

  In step S <b> 801, the system control unit 13 determines whether the size of the fastest moving object detected in step S <b> 305 occupies the angle of view of the moving object is a size in which rolling shutter distortion appears significantly. For example, when the size of the moving object to be processed exceeds a predetermined threshold (for example, ¼ of the screen vertical size), the system control unit 13 is a size at which rolling shutter distortion appears remarkably. And the process proceeds to S307. On the other hand, when the size of the moving object to be processed is equal to or smaller than the threshold value, it is determined that the rolling shutter distortion does not appear remarkably, and the process proceeds to S802.

  In step S802, the system control unit 13 determines whether there is a high-speed moving object next to the moving object determined in step S801. For example, the system control unit 13 refers to the moving speed of each moving object stored in the RAM 14 in S305, and if there is the next fastest moving object, the process returns to S801 again and the size of the next fastest moving object is determined. Determine. In other words, the processing is repeated so as to identify a moving object having a size that is a high-speed moving object and the rolling shutter distortion becomes remarkable. On the other hand, if there is no next high-speed moving object, the process proceeds to S803. If there is no moving object that meets the conditions, the process of S801 and S802 may be repeated by lowering the threshold value of the size described above. In step S <b> 803, the system control unit 13 sets a flag (e.g., binary) indicating that there is no moving object exceeding the above-described size threshold.

  In step S <b> 804, the system control unit 13 drives the focusing lens of the photographing optical system 1 via the focusing lens driving unit 6. For example, when the flag is not set in S803 (that is, there is a moving object exceeding the size threshold), the system control unit 13 sets the focusing lens so as to focus on a subject with a size that causes significant rolling shutter distortion. To drive. By controlling in this way, it is possible to obtain an image signal photographed while focusing on a moving object that is greatly affected by rolling shutter distortion. That is, it is possible to obtain an image signal in which the influence of rolling shutter distortion is reduced. On the other hand, when the flag is set (there is no moving object exceeding the size threshold), the focusing lens is driven so as to focus on the moving object that is the fastest but small in size.

  Further, the system control unit 13 performs the processing of S308 to S311 as in the first embodiment, and ends a series of operations related to this processing.

  As described above, in the present embodiment, focusing is preferentially applied to a moving object in which rolling shutter distortion is assumed to be significant by considering the speed of the moving object and the size of the detected moving object in the angle of view. The lens was driven. Further, refocus processing is performed using an image signal photographed while focusing on this moving object, and an image focused on a separately detected main subject is output. By doing so, it is possible to reduce the distortion when the refocus processing is preferentially performed from the moving object that is greatly affected by the rolling shutter distortion.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 1 ... Imaging optical system, 2 ... Micro lens array, 3 ... Solid-state image sensor, 6 ... Focusing lens drive part, 13 ... System control part,

Claims (12)

An imaging device having a plurality of pixels arranged in a two-dimensional shape and capable of acquiring an image signal including angle information of incident light rays;
Focus adjusting means for focusing the photographing optical system on the subject;
Reading means capable of reading out signals of pixels of the image sensor at different timings depending on the positions of the pixels;
When the readout means reads out the signal of the pixel at a different timing depending on the position of the pixel and there is a moving object in the subject to be photographed, the focus of the photographing optical system is Control means for controlling the focus adjusting means so as to match the moving body;
An imaging apparatus comprising: Refocusing means for generating a refocus image in which a subject different from the moving object is focused based on the image signal photographed with the focus of the photographing optical system aligned with the moving object;
The imaging apparatus according to claim 1. A moving object detecting means for detecting the moving object based on an image signal obtained from the image sensor;
The imaging apparatus according to claim 2. The moving object detection means detects the moving object in preference to a fast moving subject.
The imaging apparatus according to claim 3. The moving object detection means detects the moving object based further on whether the size of the moving subject in the angle of view exceeds a predetermined threshold;
The imaging apparatus according to claim 4. A main subject detection means for detecting a main subject that is a main subject that is different from the moving object among the subjects;
The refocusing unit generates the refocused image so as to focus on the detected main subject;
The image pickup apparatus according to claim 2, wherein the image pickup apparatus is an image pickup apparatus. A determination means for determining whether a refocus image focused on a subject different from the moving object can be generated when the photographing optical system is focused on the moving object;
The focus adjusting unit changes the focus of the photographing optical system from a position aligned with the moving body when the determining unit determines that a refocus image focused on a subject different from the moving body cannot be generated.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. The focus adjusting unit changes a focus of the photographing optical system so that blur of the moving object is reduced within a range in which a refocus image focused on a subject different from the moving object can be generated;
The imaging apparatus according to claim 7. The image sensor is a CMOS image sensor.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. The imaging device includes a microlens array, and each microlens is configured to correspond to the plurality of pixels.
The imaging apparatus according to any one of claims 1 to 9, wherein A control method of an imaging apparatus having a plurality of pixels arranged two-dimensionally and having an imaging element capable of acquiring an image signal including angle information of incident light rays,
A focus adjustment process for focusing the photographing optical system on the subject;
A readout step capable of reading out signals of pixels of the image sensor at different timings according to the positions of the pixels;
In the readout step, when the pixel signal is read out at different timings depending on the position of the pixel, and there is a moving object in the subject to be imaged, the focus of the imaging optical system is A control step of controlling the focus adjustment means to match the moving body;
An image pickup apparatus control method comprising:   The program for making a computer perform each process of the control method of the imaging device of Claim 11.

Images