JP6031366B2 - Imaging vibration control device - Google Patents

Description

  The present invention relates to an imaging damping device that suppresses blurring of a subject image during shooting.

  Conventionally, as a technique for correcting the influence of vibration such as camera shake during shooting, a technique for correcting blur of a subject image captured in an imaged (captured image) by image processing, or a technique for suppressing vibration mechanically during shooting Etc. In particular, the technology that suppresses vibration mechanically suppresses vibration before processing from when light is accumulated on the image sensor until the image is read out. It is effective and not only for digital cameras but also for silver halide cameras.

As an example of the technique for suppressing vibration mechanically, there are the following techniques (a) to (d), for example.
(A) Technology using an additional device.
This technology is a technology for controlling the vibration of the entire camera using gyro rigidity by rotating an object having a rotational moment.
(B) Technology using a gantry (see, for example, Patent Document 1 and Patent Document 2).
This technique is a technique for preventing fine vibrations of a support (such as a photographer) from being transmitted to the camera by non-rigidly connecting the support (such as a photographer) and the camera.
(C) A technique for moving or rotating a part of the optical system (lens, prism, etc.).
This technique is a technique for canceling vibration by moving or rotating a part of an optical system (such as a lens or a prism) according to vibration.
(D) Technology for moving the image sensor (see, for example, Patent Document 3).
This technique is a technique for canceling vibration by moving an image sensor in response to vibration. In this technique, for example, when the horizontal component value and the vertical component value of a motion vector of a subject image between two frames having different imaging times in a moving image are set to “uc” and “vc”, respectively, The control amount is set to (uc, vc), and the image pickup device is moved in the horizontal direction and the vertical direction along the image pickup surface of the image pickup device by the displacement control amount (uc, vc). Thereby, this technique changes the optical position of the image sensor with respect to the object scene. Hereinafter, the operation of changing the optical position of the image sensor with respect to the scene is referred to as “displacement”. Here, an apparatus related to a technique for moving the image sensor will be described as a “conventional image damping device”.

Japanese Patent No. 5009177 JP 2011-171933 A JP2012-234196A (FIG. 2)

  A conventional imaging vibration control device aims to minimize blurring of a subject image within an exposure time within a single frame and to suppress shaking of the subject image between a plurality of frames. Note that “minimizing blurring of the subject image within the exposure time within a single frame” means, for example, canceling the motion of the subject image within 1/60 seconds when using an electronic shutter operating at 60 Hz. This means that the blurring of the subject image is minimized. Further, “suppressing the sway of the subject image between a plurality of frames” means that the subject image is not continuously moved between predetermined frames (for example, between five frames).

  Such a conventional imaging vibration damping device can reduce the amount of movement of the subject image between the frames in the moving image, compared to the case where the effect of vibration is not corrected (when the vibration is not suppressed).

  However, the conventional imaging damping device cancels the movement of the subject image within the opening time (for example, 1/60 seconds) of the electronic shutter and minimizes the blurring of the subject image. When it is large, motion correction exceeding 0.5 pixels may be performed in the vertical direction or the horizontal direction.

  To supplement this explanation, the conventional image damping device is not designed to prevent the position displacement of the image sensor from protruding from the range of the unit cell region of the pixel, and simply forms an image. Is configured to be stationary on the image sensor. Therefore, the conventional imaging vibration control device can perform correction that deviates from the unit cell area. As a result, when this correction is applied to a moving image, a conventional imaging vibration suppression device has an unnatural image (for example, an image in which a subject that should have been moving is still stationary, or the subject has an allowable correction amount. In such a case, a video that moves greatly in a direction different from the original motion) may be acquired at the moment of exceeding. Here, the “pixel unit cell region” means a region where the center point of each unit cell (pixel) forming the image sensor can move.

  When the subject image moves in units of decimal pixels (that is, when the image formation moves in units of pixels not between pixels), the camera performs non-integer in motion compensation prediction processing when performing video compression coding The influence of the interpolation filter for performing the pixel motion compensation is generated, and an extra motion compensation residual is generated by that amount, so that the coding efficiency (rate distortion characteristic) is lowered.

As a result, the conventional imaging vibration control device has a problem that, for example, problems such as deterioration of image quality and increase of the bit rate may occur.
In addition, the conventional imaging vibration damping device has a problem that it may excessively cancel even the shake of the subject image intended for production reasons.

  The present invention has been made in order to solve the above-described problems, and improves the compression encoding efficiency of the video to improve the image quality, suppress the increase in the bit rate, etc. It is a main object to provide an image pickup vibration damping device that prevents excessive cancellation of even a subject image shake intended by the above.

  In order to solve the above-mentioned problem, the present invention is an image pickup vibration damping device, which is an image pickup device that takes an image, and a displacement that generates (executes) a displacement of an optical position of the image pickup device with respect to an object scene. And a control unit that controls the displacement unit, and the control unit includes a displacement amount estimation unit and a displacement control unit. In addition, the said displacement part is equipped with the structure which moves the position of an image pick-up element physically with respect to an optical system, the structure which bends an optical axis with respect to an image pick-up element, etc. as a structure for generating the said displacement, for example. It is good to have a configuration.

  With this configuration, the imaging vibration control device generates a displacement so that the positional displacement of the imaging element with respect to the object field does not protrude from the range of the unit cell region of the pixel.

  For this purpose, the displacement amount estimation unit of the imaging vibration control device uses an estimated motion vector estimation value that is generated when no displacement is generated as an estimated motion vector, and is applied to an imaging element between two frames at different imaging times. The estimated motion vector is calculated with an accuracy finer than the size of the unit lattice area of the pixel by matching the moving amounts of the image formation. In addition, the displacement control unit of the imaging vibration control device uses the motion vector of the imaging obtained as a result of generating the displacement as a displacement result vector, and calculates the value of the displacement result vector from the estimated motion vector calculated by the displacement estimation unit. Calculates a displacement control amount that matches a vector value or zero vector value in a vector value set obtained by connecting the center points of two adjacent pixels, and sets the displacement control amount as a displacement unit. Is configured to output.

  Here, “displacement control amount in which the value of the displacement result vector becomes a value that coincides with any vector value or zero vector value in the vector value set obtained by connecting the center points of two adjacent pixels” Means a displacement control amount in which the value of the displacement result vector becomes the nearest integer value.

  In the imaging vibration damping device, the displacement unit generates a displacement based on the displacement control amount. At this time, since the displacement result vector is set to the above-described value, the value of the displacement result vector becomes the nearest integer value. As a result, the image after the displacement is generated (hereinafter referred to as “image after displacement”). Is set to a position within the range of the unit cell region of the pixel.

  Accordingly, the image after displacement always exists at a position within the range of the unit cell region of the pixel, and does not protrude from the range of the unit cell region of the pixel. That is, the total amount of movement of the subject image between the frames at the time of shooting (for example, the average movement of the predetermined area in the image sensor, etc.) is made an integer in the size of the unit grid area of the pixel.

  For this reason, this imaging vibration control device can avoid the occurrence of an extra motion compensation residual due to the influence of the interpolation filter in the motion compensation prediction process when compressing and encoding video. That is, this imaging vibration control device can suppress a prediction process residual after the motion compensation prediction process when performing video compression encoding. As a result, this imaging vibration control device improves the compression encoding efficiency of the video, improves the image quality, suppresses the increase in the bit rate, etc., and also shakes the subject image intended for production reasons. Can be prevented from excessively canceling.

  Preferably, the displacement control unit of the imaging vibration control device is a unit of the pixel for each of the horizontal component value and the vertical component value of the estimated motion vector from the horizontal component value and the vertical component value of the estimated motion vector. A configuration in which the displacement control amount is calculated by subtracting a horizontal component calculation value and a vertical component calculation value obtained by performing a carry-up operation or a carry-down operation of a digit toward a nearest integer value in a decimal part in the size of the lattice area. Also good.

  According to such a configuration, the imaging vibration control device detects the displacement control amount at which the value of the displacement result vector becomes the nearest integer value (that is, the position of the imaging after the displacement result vector is within the range of the unit cell region of the pixel). It is possible to calculate an appropriate displacement control amount that is a value set at the position.

  In addition, the imaging vibration suppression apparatus preferably further includes another imaging element and a branching member that branches an optical path into the direction of the imaging element and the direction of the other imaging element, and the displacement amount estimation unit When the estimated motion vector of the imaging on the image sensor is “Vc” and the motion vector of the image on the other image sensor is “Vs”, the image is captured by the another image sensor. The motion vector Vs may be calculated on the basis of the video between the two frames, and the estimated motion vector Vc may be calculated on the basis of the calculated motion vector Vs.

  According to this configuration, the imaging vibration suppression device calculates the motion vector Vs based on the video between two frames captured by another imaging device, and further estimates based on the calculated motion vector Vs. A motion vector Vc can be calculated.

The displacement estimation unit of the imaging vibration control device preferably sets the imaging frame time interval, the horizontal pixel pitch, and the vertical pixel pitch of the imaging element to “Tc”, “Pc”, and “Qc”, respectively. , When the imaging frame time interval, the horizontal pixel pitch, and the vertical pixel pitch of the other imaging device are “Ts”, “Ps”, and “Qs”, respectively, the estimated motion based on the equation (1) The vector Vc may be calculated.

  According to such a configuration, the imaging vibration damping device can prepare Equation (1) in a memory or the like in advance and calculate the estimated motion vector Vc based on the motion vector Vs.

Further, the displacement amount estimation unit of the imaging vibration control device is preferably configured such that the imaging on the imaging element between a current frame used as an ornamental image by the imaging element and a frame imaged immediately before the current frame is used. The estimated motion vector of “Vc” is set to “Vc”, and the image formation on the image sensor between the two frames imaged by the image sensor at the times ta and tb (ta <tb) past the current frame. When the motion vector is set to “Vd”, the displacement control amount at time ta is set to “Da”, and the displacement control amount at time tb is set to “Db”, the estimated motion vector Vc is calculated based on Expression (2). May be configured to calculate.

  According to such a configuration, the imaging vibration control device calculates the estimated motion vector Vc of the imaging on the imaging element between the current frame used as an ornamental image by the imaging element and the frame imaged immediately before the current frame. Can do.

  Further, the displacement unit of the imaging vibration damping device is preferably a means for physically moving one of the imaging element and the optical system, or is disposed on an optical path, and the imaging An optical path changing element that optically changes the position of the image formation on the element may be used.

  The imaging vibration control device according to the present invention can suppress a prediction process residual after the motion compensation prediction process when performing video compression encoding. As a result, the imaging vibration damping device can improve the compression encoding efficiency of the video, and can realize, for example, improvement in image quality, suppression of increase in bit rate, and the like.

  Further, in this imaging vibration damping device, since the displacement unit suppresses the vibration only by the amount of displacement control described above, it does not excessively cancel the shake of the subject image that is a multiple of the size of the unit cell region of the pixel. . As a result, this imaging vibration control device can prevent excessive cancellation of even the sway of the subject image intended for production reasons.

It is a figure which shows the structure and operation example of the imaging damping device which concerns on Embodiment 1. FIG. (A), (b) is a figure which shows an example of the operation timing of the imaging damping device which concerns on Embodiment 1, (a) is the "accumulation" timing and "read" timing of the image in a 1st image sensor. (B) shows the “accumulation” timing and “read” timing of the video in the second image sensor. (A), (b) is a figure which shows an example of the operation timing of the imaging damping device which concerns on Embodiment 1 when a frame time interval differs between a 1st image sensor and a 2nd image sensor, (a) Shows the operation timing of the first image sensor, and (b) shows the operation timing of the second image sensor. It is a figure which shows an example of the unit lattice area | region of a pixel, and is an example of the unit lattice area | region of a pixel when the image pick-up element is comprised as a square grid of a monochrome single board, and the aperture ratio is 100% Show. (A), (b), (c) is a figure which shows an example of the unit lattice area | region of a pixel, (a) has shown the example of a structure of the image pick-up element comprised as a square lattice for colors, (B) shows an example of a unit grid area of green (G) pixels, and (c) shows an example of a unit grid area of red (R) pixels. It is a figure which shows an example of the unit cell area | region of a pixel, and shows an example of the unit cell area | region of a pixel in case an imaging element is comprised as a honeycomb lattice. It is a figure which shows the difference in the position of the image formation after a displacement with the case where this invention is not applied, and the case where this invention is applied. It is a figure which shows the structure and operation example of the imaging damping device which concerns on Embodiment 2. FIG. FIG. 10 is a diagram illustrating an example of operation timing of the imaging vibration damping device according to the second embodiment. It is a figure which shows the structure and operation example of the imaging damping device which concerns on Embodiment 3. FIG. It is a figure which shows the operation example of the displacement part (optical path changing element) used in Embodiment 3. It is a figure which shows the structure and operation example of the imaging damping device which concerns on Embodiment 4. FIG.

  Hereinafter, an embodiment of the present invention (hereinafter referred to as “the present embodiment”) will be described in detail with reference to the drawings. Each figure is only schematically shown so that the present invention can be fully understood. Therefore, the present invention is not limited to the illustrated example. Moreover, in each figure, the same code | symbol is attached | subjected about the common component and the same component, and those overlapping description is abbreviate | omitted.

  Here, the operation of changing the optical position of the image sensor with respect to the object scene is referred to as “displacement”. Further, an image formed as a result of generating (executing) displacement is referred to as “image after displacement”. An area in which the center point of each unit cell (pixel) forming the image sensor can move is referred to as a “pixel unit grid area”.

  The imaging damping device according to the present invention generates (executes) “displacement” so that “image after displacement” does not protrude from the “range of the unit cell region of the pixel” (see FIG. 4B). This improves video compression and coding efficiency, improves image quality and suppresses the increase in bit rate, etc., and overshoots the subject image shake that was intended for production reasons. To prevent it.

[Embodiment 1]
<Configuration of imaging damping device>
Hereinafter, the configuration of the imaging vibration damping device 1 according to the first embodiment will be described with reference to FIG.
As shown in FIG. 1, the imaging vibration damping device 1 according to the first embodiment includes a control unit 2, a lens 20, two imaging elements 10 and 60, displacement units 40 a to 40 d, and a half mirror 50. Yes. Hereinafter, when the two image sensors 10 and 60 are distinguished, the image sensor 10 is referred to as a “first image sensor 10”, and the image sensor 60 is referred to as a “second image sensor 60”.

  The control unit 2 is a functional unit that controls the overall operation of the imaging vibration damping device 1. The control unit 2 includes a CPU, and includes a “displacement amount estimation unit 70” described later and a “displacement control unit 80” described later.

  The lens 20 is an optical system that collects light on the first image sensor 10 and forms an image for viewing on the first image sensor 10. Here, the lens 20 has a function as a convex lens, and may be a single lens or a configuration in which a plurality of lenses are combined. The lens 20 is configured to be connected to a housing (not shown).

The first image sensor 10 is an element that captures an image of a subject or the like captured from the lens 20.
The second image sensor 60 is an element that captures an image used for calculation of a motion vector measurement value of the entire image formation or a part thereof. Here, the second image sensor 60 is used as a sensor for calculating “a measured value of the motion vector of the entire image or a part thereof”.

  Hereinafter, the “measured value of the motion vector of the entire image or a part thereof” is referred to as “measured motion vector Vs”. In addition, an image used for calculation of “measured value of motion vector of whole or part of image formation (measured motion vector Vs)” is referred to as “measurement image”.

  The first image sensor 10 and the second image sensor 60 may be the same type of element or different types of elements. The spatial resolution (view angle per pixel interval) of the second image sensor 60 is more preferably finer than that of the first image sensor 10 (however, this condition is not essential).

  The displacement units 40a to 40d are means for generating (executing) the above-described “displacement (that is, an operation of changing the optical position of the first image sensor 10 with respect to the object scene)”. The displacement units 40a to 40d can be configured as means for physically moving either the first image sensor 10 or the optical system (lens 20 or the like) in units of decimal pixels.

  In the first embodiment, the case where the displacement units 40a to 40d are configured by an actuator that physically changes the position of the first image sensor 10 with respect to the optical axis 30 of the lens 20 in units of decimal pixels will be described.

Specifically, a configuration in which the displacement units 40 a to 40 d move the first imaging element 10 in the horizontal direction and the vertical direction along the imaging surface of the first imaging element 10 (that is, perpendicular to the optical axis 30 of the lens 20). The first imaging element 10 is two-dimensionally translated in a plane that intersects with (1).
Such displacement portions 40a to 40d can be constituted by, for example, a piezoelectric element, a servo motor, a linear motor, or the like.

In the example illustrated in FIG. 1, the imaging vibration damping device 1 is provided with four displacement portions 40 a to 40 d. However, the number of the displacement portions may not be four as long as the first imaging device 10 can be translated two-dimensionally within a plane perpendicular to the optical axis 30 of the lens 20.
In the first embodiment, the imaging vibration damping device 1 is configured to move the first imaging element 10 using the displacement units 40a to 40d, but may be configured to move the optical system such as the lens 20 or the like. Is possible.

The half mirror 50 is a branching member that branches the optical path into the direction of the first image sensor 10 and the direction of the second image sensor 60. The imaging vibration control device 1 includes a half mirror 50 provided between the lens 20 and the first imaging element 10 on the optical axis 30 from the lens 20, and the optical path is branched by the half mirror 50. An image is formed on the first image sensor 10 and the second image sensor 60. That is, the half mirror 50 branches the optical path into the direction of the first image sensor 10 and the direction of the second image sensor 60, and guides the video of the object to be viewed to the first image sensor 10. In addition to forming an image on the top, the above-described “measurement image (that is, an image used for calculating“ measurement motion vector Vs ”)” is guided to the second image sensor 60 and imaged on the second image sensor 60. It is configured.
The imaging vibration control device 1 may be configured to use a half prism (not shown) as a branch member instead of the half mirror 50.

  A measurement image formed by the second image sensor 60 is stored in a storage unit (not shown) and then read out by the control unit 2 to control the displacement units 40a to 40d (hereinafter referred to as control amounts). , Referred to as “displacement control amount”).

Here, the control unit 2 includes a “displacement amount estimation unit 70” and a “displacement control unit 80” described below.
The “displacement amount estimation unit 70” and the “displacement control unit 80” are described later, for example, when the imaging elements 10 and 60 are configured as a monochromatic square lattice and the aperture ratio is 100%. As described in detail in the section “Calculation Method of Displacement Control Amount”, the “displacement control amount (that is, the control amount for operating the displacement units 40a to 40d)” described later is “(uc-round (uc) )), (Vc-round (vc)) ", and the" displacement result vector "to be described later is set to" (round (uc)), (round (vc)) ", and the video compression code The motion compensation prediction process when performing the conversion is performed in units of integer pixels.

  The displacement amount estimation unit 70 estimates an image motion vector generated when the above-described “displacement (that is, an operation of changing the optical position of the first image sensor 10 with respect to the object scene)” is not generated. (Hereinafter, referred to as “estimated motion vector Vc”) is a functional means for calculating with decimal pixel accuracy.

  In the first embodiment, the displacement amount estimation unit 70 first calculates the “measurement motion vector Vs” based on the “video for measurement” captured by the second image sensor 60, and then Based on the measured motion vector Vs, the aforementioned “estimated motion vector Vc” is calculated with decimal pixel accuracy. A method for calculating “measured motion vector Vs” and a method for calculating “estimated motion vector Vc” will be described later.

  Here, “decimal pixel accuracy” means an accuracy finer than the size of the “pixel unit cell region”. A specific example of the “pixel unit cell region” will be described later with reference to FIGS. 3A to 3C.

  The displacement control unit 80 is a functional unit that operates the displacement units 40 a to 40 d to generate the above-described “displacement (that is, an operation for changing the optical position of the first image sensor 10 with respect to the object scene)”. .

  Based on the estimated motion vector Vc calculated by the displacement amount estimation unit 70, the displacement control unit 80 calculates the “displacement control amount (that is, the control amount for operating the displacement units 40a to 40d)”.

  At this time, the displacement control unit 80 generates the above-described “displacement” by the “displacement control amount”, thereby moving the overall movement amount of the subject image between the frames at the time of shooting (for example, the first image sensor). The average movement of the predetermined area in FIG. 10 is an integer value in the size of the unit cell area of the pixel.

  That is, at this time, the displacement control unit 80 obtains “all or part of the image formed on the first image sensor 10 (for example, near the center) obtained as a result of the displacements 40a to 40d generating the“ displacement ”described above. Of the "preferred predetermined representative value" when measuring a plurality of frames at different positions or image capturing times with the size accuracy (hereinafter referred to as "pixel accuracy") of the unit cell region of pixels. The “displacement control amount” is calculated so as to minimize the absolute value, and the above-described “displacement” is generated by the “displacement control amount”.

  As a configuration for this, in the first embodiment, the displacement control unit 80 selects any one of the vector value sets obtained by connecting the center points of two adjacent pixels with the value of a “displacement result vector” described later. The displacement control amount is a value that coincides with the vector value or the zero vector value.

  Here, “displacement control amount in which the value of the displacement result vector becomes a value that coincides with any vector value or zero vector value in the vector value set obtained by connecting the center points of two adjacent pixels” Means a displacement control amount in which the value of the displacement result vector becomes the nearest integer value.

  In the first embodiment, “displacement in which the value of the displacement result vector becomes a value that coincides with any vector value or zero vector value in the vector value set obtained by connecting the center points of two adjacent pixels. The “control amount” is calculated by the displacement control unit 80 executing a method described in the section “Calculation method of displacement control amount” described later.

  The “preferred predetermined representative value” means, for example, a suitable value arbitrarily determined in advance according to the operation from among the mode value, the average value, the median value, the statistical value, and the like. ing. As the “preferred predetermined representative value”, the mode value is particularly preferable. A specific calculation method of the “displacement control amount” will be described later.

  When the displacement control unit 80 calculates the “displacement control amount”, the displacement control unit 80 outputs the “displacement control amount” (strictly, a control signal for operating the displacement units 40a to 40d by the “displacement control amount”) to the displacement units 40a to 40a. Output to 40d. In response to this, the displacement units 40a to 40d move the first imaging device 10 in the horizontal direction and the vertical direction along the imaging surface of the first imaging device 10 by the amount of “displacement control amount”, and the above-described operation is performed. Generate “displacement”.

  Hereinafter, a motion vector from the position before displacement of the image after displacement obtained by the first image sensor 10 as a result of generating “displacement” is referred to as “displacement result vector”. In other words, the “displacement result vector” means a movement amount measured in units of pixels of a part or the whole of the image captured by the first image sensor 10 before and after the occurrence of the “displacement”. Yes.

<Operation of imaging vibration control device>
Hereinafter, the operation of the imaging vibration damping device 1 will be described with reference to FIG. 1 and FIG. 2A. The imaging vibration control device 1 operates based on the time measured by a timer (not shown). The operation of the imaging vibration control device 1 is defined by a control program stored in advance in a readable manner in a storage unit (not shown), and is executed by the control unit 2. In addition, each data is temporarily stored in a storage unit (not shown) so as to be readable, and then output to a required component for performing subsequent processing. Hereinafter, since these points are conventional means in information processing, detailed description thereof will be omitted.

  As shown in FIG. 1, the imaging vibration damping device 1 collects light on the first image sensor 10 by the lens 20 and forms an ornamental image on the first image sensor 10 as described above. At that time, the imaging vibration control device 1 branches the optical path into the direction of the first imaging element 10 and the direction of the second imaging element 60 by the half mirror 50, and guides an ornamental image to the first imaging element 10. At the same time, the “measurement video (that is, video used to calculate“ measurement motion vector Vs ”)” is guided to the second image sensor 60 and imaged on the second image sensor 60.

  Each of the first image sensor 10 and the second image sensor 60 accumulates electric charges corresponding to light constituting the image formed on the image sensors 10 and 60 at the “accumulation” timing shown in FIG. 2A. Then, at the “reading” timing shown in FIG. 2A, the control unit 2 converts the electric charge accumulated in the imaging elements 10 and 60 into a voltage signal and stores it as a video signal in a storage unit (not shown). A video signal is read from a storage unit (not shown), and the video signal is used for a predetermined calculation.

  As the “predetermined calculation”, for example, the above-described calculation of the “measured motion vector Vs” and the above-mentioned “estimated motion vector Vc” by the displacement amount estimation unit 70 of the control unit 2 and the displacement control of the control unit 2 are performed. The above-described “displacement control amount” is calculated by the unit 80. Examples of the “predetermined calculation” include a video compression encoding process by a function unit (not shown) of the control unit 2 and a motion compensation prediction process when the video compression encoding process is performed. Here, the calculation of the “measured motion vector Vs” and the calculation of the “estimated motion vector Vc” described above by the displacement amount estimation unit 70 and the calculation of the “displacement control amount” by the displacement control unit 80 are emphasized. Explained.

  In the example shown in FIG. 2A, the frame time interval of the first image sensor 10 is “Tc”, and the frame time interval of the second image sensor 60 is Ts. Here, the “frame time interval” means an interval for periodically performing “accumulation” and “reading” of video.

  In the example shown in FIG. 2A, the frame time interval Tc of the first image sensor 10 and the frame time interval Ts of the second image sensor 60 are set to substantially the same value. However, as shown in FIG. 2B, the frame time interval Tc of the first image sensor 10 and the frame time interval Ts of the second image sensor 60 can be set to different values.

  In the example shown in FIG. 2A, the frame time intervals Ts and Tc are intervals from the start of the “accumulation” period of each cycle to the start of the “accumulation” period of the next cycle. However, the frame time intervals Ts and Tc may be intervals from the end of the “accumulation” period of each cycle to the end of the “accumulation” period of the next cycle. Alternatively, the frame time intervals Ts and Tc are the intervals from the start of the “read” period of each cycle to the start of the “read” period of the next cycle, or the end of the “read” period of each cycle, respectively. To the end of the “read” period of the next cycle.

  Note that the “accumulation” period of the second image sensor 60 is shorter than the “accumulation” period of the first image sensor 10. This is because the video imaged by the second image sensor 60 is the above-mentioned “video for measurement (that is, video used for calculation of“ measurement motion vector Vs ”)”, and is not an ornamental video. This is because the image quality may be lower than that of the image captured by the image sensor 10.

  In the example shown in FIG. 2A, the imaging vibration control device 1 performs “accumulation after displacement” as an ornamental image by the first imaging element 10 at the “accumulation” timing of the frames F1, F2, F3,. The "image" is being photographed.

  In order to realize such an operation, the imaging vibration control device 1 is used for measuring a plurality of frames having different imaging times by the second imaging element 60 at a timing preceding the “accumulation” timing in the first imaging element 10. Take a picture. For example, the imaging vibration damping device 1 captures a video for measurement by the second image sensor 60 at the “accumulation” timing of the frames fa and fb prior to the “accumulation” timing of the frame F1.

  The measurement images of the frames fa and fb photographed by the second image sensor 60 generate the above-mentioned “displacement” that is performed in advance when the first image sensor 10 captures the ornamental image of the frame F1. Used for displacement control. That is, the frames fa and fb are a frame pair used for estimating the motion vector for the frame F1. The details will be described below.

  The imaging vibration control device 1 performs “accumulation” of the video for viewing the frame F1 on the first image sensor 60 after the “accumulation” period of the video for measurement of the frame fb on the second image sensor 60 ends. Until the period starts, the following processing is performed.

  For example, in the imaging vibration damping device 1, when the video for measurement of the frames fa and fb is accumulated in the second imaging element 60 and stored in a storage unit (not shown), the control unit 2 performs the reading at the timing of t1. The video for each measurement of the frames fa and fb is read out.

  Next, at the timing of the calculation time t2, the displacement amount estimation unit 70 of the control unit 2 first calculates the above-described “measurement motion vector Vs” based on each measurement video of the frames fa and fb. In addition, based on the measured motion vector Vs, the above-described “estimated motion vector Vc” is calculated, and the displacement control unit 80 of the control unit 2 calculates the “displacement control amount”. A method for calculating “measured motion vector Vs”, a method for calculating “estimated motion vector Vc”, and a method for calculating “displacement control amount” will be described later.

  Next, at the timing of the displacement control time t3, the displacement control unit 80 operates the displacement units 40a to 40d by “displacement control amount” (strictly, “displacement control amount”) calculated at the calculation time t2. Control signal) is output to the displacement portions 40a to 40d.

  Accordingly, the displacement control unit 80 causes the displacement units 40a to 40d to move the first image sensor 10 in the horizontal direction and the vertical direction along the imaging surface of the first image sensor 10 by the “displacement control amount”. Then, displacement control for generating the above-described “displacement” is performed.

  As a result, the first image sensor 10 captures “image formation after displacement” as an ornamental image at the “accumulation” timing of the frame F1.

  Thereafter, the imaging vibration damping device 1 repeatedly performs the same operation in accordance with the “accumulation” timing of the frames F2, F3,. Thus, the first image sensor 10 captures “image formation after displacement” as an image for viewing at the “accumulation” timing of each frame F2, F3,.

<Calculation method of measurement motion vector Vs>
Hereinafter, with reference to FIG. 1, the calculation method of the “measured motion vector Vs” will be described.
As shown in FIG. 1, the displacement estimation unit 70 determines the “measurement motion” from a plurality of measurement images captured by the second image sensor 60 or measurement images between a plurality of frames at different imaging times. Vector Vs "is calculated.

  Here, the displacement amount estimation unit 70 performs block matching on the whole or a part of the video in the measurement video between a plurality of frames taken by the second imaging device 60 at different imaging times, thereby obtaining the “measurement motion vector”. It is assumed that “Vs” is calculated.

  Here, “a plurality of frames taken by the second image sensor 60 at different imaging times” is, for example, photographed by the second image sensor 60 prior to the current frame to be photographed by the first image sensor 10. It means two frames, such as frame-1 and frame-2.

  Note that the calculation accuracy of the “measurement motion vector Vs” is obtained by block matching of a measurement image between two frames photographed by the second image sensor 60, so that the pixel pitch of the second image sensor 60 is the first. When the pixel pitch is sufficiently finer than the pixel pitch of one image sensor 10 (for example, 0.5 times or less), it may be integer pixel accuracy or decimal pixel accuracy.

  However, the calculation accuracy of the “measured motion vector Vs” is when the pixel pitch of the second image sensor 60 is not sufficiently finer than the pixel pitch of the first image sensor 10 (for example, larger than 0.5 times). The decimal pixel accuracy is required.

  As a technique for performing block matching with sub-pixel accuracy, for example, an enlarged image is generated by interpolating interpolation pixels in each of two frames of video, and a motion vector is obtained by comparing the generated videos. There is a method of obtaining by dividing by a magnification.

  Alternatively, as a technique for performing block matching with decimal pixel precision, for example, there is a technique called “parabolic fitting”. This method called “parabolic fitting” correlates images between two frames, obtains a motion vector having a maximum correlation value (minimum value of error), and centers on the motion vector. A quadratic surface of a correlation value sequence (or error value sequence) is calculated for a total of nine pixels within ± 1 pixel in the horizontal direction and within ± 1 pixel in the vertical direction, and the maximum value (or minimum) on the quadric surface This is a method in which a point giving a value is obtained and this point is regarded as a motion vector with decimal pixel precision.

  The displacement amount estimation unit 70 calculates the “measurement motion vector Vs” by performing such block matching. Note that the “measurement motion vector Vs” is obtained by block matching of a measurement video between two frames, and thus is affected by the size of a unit cell region of video pixels used for block matching.

<Calculation method of estimated motion vector Vc>
Hereinafter, a method for calculating the “estimated motion vector Vc” will be described with reference to FIG.
As shown in FIG. 1, when the displacement amount estimation unit 70 calculates the “measured motion vector Vs”, the “estimated motion vector Vc” is decimally calculated based on the “measured motion vector Vs” as follows. Calculate with pixel accuracy.

The displacement amount estimation unit 70 sets the imaging frame time interval of the first image sensor 10 to “Tc”, sets the horizontal pixel pitch of the first image sensor 10 to “Pc”, and sets the vertical pixel pitch of the first image sensor 10 to “Qc”. Furthermore, the imaging frame time interval of the second image sensor 60 is “Ts”, the horizontal pixel pitch of the second image sensor 60 is “Ps”, and the vertical pixel pitch of the second image sensor 60 is “Qs”. Then, the “estimated motion vector Vc” with decimal pixel precision can be calculated based on the following equation (1).

In addition, Formula (1) is a thing on the assumption that the conditions of the following (a)-(c) are satisfy | filled.
(A) The first image sensor 10 and the second image sensor 60 are arranged so that the horizontal axis and vertical axis of the first image sensor 10 and the horizontal axis and vertical axis of the second image sensor 60 are parallel to each other. The condition of being.
(B) The condition that it has been compensated that the image on the second image sensor 60 becomes a mirror image by the half mirror 50.
(C) A condition that the first image sensor 10 and the second image sensor 60 are arranged at the same distance from the second optical principal point of the lens 20.

<Calculation method of displacement control amount>
Hereinafter, the calculation method of the “displacement control amount” will be described with reference to FIG.
As shown in FIG. 1, when the displacement estimation unit 70 calculates the “estimated motion vector Vc”, the displacement control unit 80 calculates the displacement control amount (ΔX based on the “estimated motion vector Vc” as follows. , ΔY).

  In the first embodiment, the displacement control unit 80 calculates the horizontal component value and the vertical component value of the “estimated motion vector Vc” from the horizontal component value and the vertical component value of the “estimated motion vector Vc (estimation result)”. By subtracting the horizontal component calculation value and the vertical component calculation value obtained by performing the carry-up calculation or the carry-down calculation with the decimal part in the size of the unit cell region of the pixel being directed to the nearest integer value, the displacement control amount (ΔX, ΔY) is calculated.

Specifically, the displacement control unit 80 sets the estimated motion vector when the horizontal component value of the “estimated motion vector Vc” calculated by the displacement amount estimation unit 70 is “uc” and the vertical component value is “vc”. Based on Vc, a displacement control amount (ΔX, ΔY) is calculated based on the following equation (2).

  Here, the round function represents a rounding operation. "Rounding operation" means rounding off to the first decimal place or two types of nearest rounding specified in IEEE 757-2008, rounding down to the one that adds 0.5, rounding up to the one that subtracts 0.5 Is an operation to perform.

  The horizontal component calculation value round (uc) and the vertical component calculation value round (vc) by such rounding calculation may be larger than the horizontal component value uc and the vertical component value vc, respectively, or the horizontal component value uc and In some cases, the vertical component value vc may be smaller.

  Then, the displacement control amounts (ΔX, ΔY) are calculated from the horizontal component value uc and the vertical component value vc of the motion vector Vc (estimated result) by the rounding calculation, the horizontal component calculated value round (uc) and the vertical component calculated value round (vc). Is subtracted from the pixel pitch of the first image sensor 10 in the horizontal direction and the vertical direction (specifically, the first image sensor 10 is assumed to be a monochromatic single plate square lattice). When the aperture ratio is 100%, the value is within a radius of 0.5 pixels in the Chebyshev distance.

  The displacement units 40a to 40d are arranged in the horizontal direction by (uc-round (uc)) and in the vertical direction by (vc-round (vc)) according to the displacement control amount (ΔX, ΔY). 1 The image sensor 10 is displaced.

  As a result, the “displacement result vector (that is, the motion vector from the position before displacement of the image after displacement)” has a horizontal component value of (uc−ΔX) = (uc− (uc−round (uc)). )) = Round (uc), and the vertical component value is (vc−ΔY) = (vc− (vc−round (vc))) = round (vc).

  In other words, the “displacement result vector” has a horizontal component value and a vertical component value corresponding to the horizontal component value and the vertical component value of the “estimated motion vector Vc”, respectively, and the nearest part of the size of the unit cell region of the pixel A value obtained by performing a carry operation or a carry operation for a digit toward an integer value. As described above, as a result of the displacement control by the control unit 2, the motion compensation prediction process when the video is compressed and encoded can be performed in units of integer pixels.

  As described above, the “displacement result vector” is a movement amount measured in units of pixels of a part or the whole of the image captured by the first image sensor 10 before and after the occurrence of the “displacement”. Means. Since the “displacement result vector” is converted into an integer value between the frames, the value of the decimal part is the minimum value.

  The imaging vibration control device 1 generates “displacement” so that the image after displacement always exists at a position within the range of the unit cell region of the pixel and does not protrude from the range of the unit cell region of the pixel. Let As a result, the imaging vibration control device 1 determines the overall movement amount of the subject image between frames at the time of shooting (for example, the average movement of a predetermined area in the image sensor, etc.) in the size of the unit cell area of the pixel. Convert to integer.

  As a result, the imaging vibration control device 1 can suppress a prediction process residual after the motion compensation prediction process when performing video compression encoding. Therefore, the imaging vibration control device 1 can improve the compression encoding efficiency of the video, and can realize, for example, improvement in image quality, suppression of increase in bit rate, and the like.

  Further, the imaging vibration damping device 1 has excessive vibration of the subject image that is a multiple of the size of the unit cell area of the pixel because the displacement units 40a to 40d suppress the vibration only by the displacement control amount (ΔX, ΔY). Will not cancel out. As a result, the imaging vibration damping device 1 can prevent excessive cancellation of even the shake of the subject image intended for production reasons.

<Specific Example of Pixel Unit Cell Region>
Hereinafter, a specific example of a unit cell region of a pixel will be described with reference to FIGS. 3A to 3C. The “pixel unit lattice region” means a region where the center point of each unit cell (pixel) forming the image sensor can move as described above. The unit cell region of the pixel differs depending on the structure and specifications of the image sensors 10 and 60.

  FIG. 3A shows an example of a unit cell region of pixels in the unit cell 101 that forms the image sensor 100 that is configured as a monochromatic square lattice and has an aperture ratio of 100%.

  In FIG. 3A, the inside of the region indicated by the dotted line is a unit lattice region of pixels in the unit cell 101 (that is, a region where the center point of the unit cell 101 can move). In FIG. 3A, the regions indicated by dotted lines are the midpoint between the center of the unit cell 101 and the center of the adjacent upper left cell, the midpoint between the center of the unit cell 101 and the center of the adjacent upper right cell, and the center of the unit cell 101. This is a square area whose apex is the midpoint of the center of the adjacent lower right cell and the midpoint of the center of the unit cell 101 and the center of the adjacent lower left cell. The camera optically moves the unit cell 101 with respect to the object scene, and at that time, the range of the movement is limited so that the center point of the unit cell 101 does not deviate from the unit lattice region (hereinafter, referred to as the unit cell 101). The same).

  FIG. 3B shows an example of a unit lattice region of pixels in a unit cell that forms the imaging device 200 configured as a square lattice for color.

  FIG. 3B (a) shows the structure of the image sensor 200. In the example shown in FIG. 3B (a), the imaging device 200 includes two green (G) unit cells, one red (R) unit cell, and one blue (B), as indicated by a dashed line. In this structure, four cells, each of which is a unit cell, are used as one block cell for color.

  FIG. 3B (b) shows an example of the unit cell region of the pixel in the green (G) unit cell 201G that forms such an image sensor 200. In FIG. 3B (b), the area inside the grid indicated by the dotted line is the unit grid area of the pixel in the unit cell 201G. In FIG. 3B (b), a region indicated by a dotted line is a square region connecting the centers of the upper, right, lower, and left cells of the unit cell 201G.

  FIG. 3B (c) shows an example of the unit cell region of the pixel in the unit cell 201R for red (R). In FIG. 3B (c), the area inside the grid indicated by the dotted line is the unit grid area of the pixel in the unit cell 201R. In FIG. 3B (c), the area indicated by the dotted line is a square area connecting the centers of the upper left, upper right, lower right, and lower left cells of the unit cell 201R.

  Since the unit cell 201G for green (G) has two cells in one block cell for color, the unit cell region of the pixel is as shown in FIG. 3B (b). On the other hand, since the unit cell 201R for red (R) has only one cell in one block cell for color, the unit cell region of the pixel is as shown in FIG. 3B (c). Here, it is assumed that the unit cell region of the pixel of the blue (B) unit cell (not shown) is the same as the unit cell 201R for red (R). However, the unit cell region of these pixels may have different configurations depending on the structure and specifications of the image sensor 200.

  FIG. 3C shows an example of the unit cell region of the pixel in the unit cell 301 that forms the imaging device 300 configured as a honeycomb lattice. In FIG. 3C, the area inside the grid indicated by the dotted line is the unit grid area of the pixel in the unit cell 301. In FIG. 3C, a region indicated by a dotted line is a hexagonal shape obtained by obtaining six midpoints between the center of the unit cell 301 and each of the six unit cells adjacent to the unit cell 301 and connecting these six midpoints. It has become an area.

<Position of imaging after displacement when the present invention is applied>
As described above, the present invention generates a “displacement” so that the image after displacement does not protrude from the range of the unit cell region of the pixel (see FIG. 4B). Improves efficiency and realizes improvement of image quality, suppression of increase in bit rate, etc., and also prevents excessive cancellation of even the subject image shake intended for production reasons. is there.

  Hereinafter, with reference to FIG. 4, the difference in image formation position after displacement between the case where the present invention is not applied and the case where the present invention is applied will be described. Note that “when the present invention is applied” is a value that always causes the image after displacement to be within the range of the unit cell region R of the pixel (that is, the image after displacement is always in the unit cell region R of the pixel. This means that the value of “displacement control amount” is set to a value that does not protrude from the range. Further, “when not applying the present invention” means that the value of “displacement control amount” is not set to such a value.

  FIG. 4A shows an example of the position of the image after displacement when the present invention is not applied as a comparative example. On the other hand, FIG. 4B shows an example of the position of the image after displacement when the present invention is applied as the first embodiment. FIG. 4 shows an example in which the first image sensor 10 is configured as a monochromatic single plate square lattice and the aperture ratio is 100%.

  As shown in FIG. 4A, in the comparative example to which the present invention is not applied, the image after displacement extends over four pixels. That is, in the comparative example, the image after displacement is in a state of protruding from the range of the unit lattice region R of the pixel in the decimal pixel unit.

  In such a comparative example, there is a region that protrudes from the range of the unit cell region R of the pixel at the time of motion compensation prediction processing when compressing and encoding video, and motion compensation of non-integer pixels is performed in the motion compensation prediction processing. For this reason, an influence of the interpolation filter is generated, and an extra motion compensation residual is generated by the amount of the influence, so that the encoding efficiency (rate distortion characteristic) is lowered. As a result, in the comparative example, there may be a problem that the image quality is deteriorated or the bit rate is increased. In addition, the comparative example may excessively cancel even the sway of the subject image intended for production reasons.

  On the other hand, as shown in FIG. 4B, in the first embodiment to which the present invention is applied, the image after displacement is placed on a single pixel. That is, in the first embodiment, the image after displacement does not protrude from the range of the unit lattice region R of the pixel in units of decimal pixels.

  In the first embodiment as described above, there is no region that protrudes from the range of the unit cell region R of the pixel at the time of motion compensation prediction processing when compressing and encoding the image. In the compensation prediction process, it is possible to avoid an extra motion compensation residual due to the influence of the interpolation filter. That is, this Embodiment 1 can suppress the prediction process residual after the motion compensation prediction process at the time of video compression encoding. As a result, Embodiment 1 can suppress a decrease in encoding efficiency (rate distortion characteristics). As a result, the first embodiment improves the compression encoding efficiency of the video, realizes the improvement of the image quality, the suppression of the increase of the bit rate, and the like, and also up to the sway of the subject image intended for the production reason. Can be prevented from being excessively offset.

  As described above, according to the imaging vibration damping device 1 according to the first embodiment, the displacement result vector of the image after displacement of the image captured by the first imaging element 10 (that is, as a result of performing displacement control, the first The motion vector of the subject image of the output video imaged by one image sensor 10 is substantially converted to an integer value between the frames.

  Therefore, according to the image pickup vibration control device 1, when the camera performs compression encoding of the output video (in particular, the camera is, for example, MPEG-1, MPEG-2, MPEG-4, H.264 / MPEG-4). When compressing and encoding output video using a method such as AVC that compresses and encodes video using time correlation), the video compression and coding efficiency is improved, image quality is improved, and bit rate is increased. In addition to realizing suppression and the like, it is possible to prevent excessive cancellation of even the sway of the subject image intended for production reasons.

  Further, according to the imaging vibration control device 1, the first imaging element 10 is displaced only within a range of ± 0.5 pixels in the horizontal direction and the vertical direction, so that the true position of the subject and the position of the photographed subject image Is within a range of ± 0.5 pixels in the horizontal and vertical directions, and dynamic image quality degradation caused by adding the image pickup vibration control device 1 to the camera can be minimized.

[Embodiment 2]
Next, the imaging vibration damping device 1B according to the second embodiment will be described. Compared with the imaging damping device 1 according to the first embodiment, the imaging damping device 1B uses an ornamental video imaged by the imaging element 10 as a measurement video (that is, an ornamental video image). The difference is that the displacement control amount is calculated (by using it as an image for measurement). Hereinafter, the configuration of the imaging vibration damping device 1B according to the second embodiment will be described with reference to FIG. Here, the “measurement motion vector (that is, the measurement value of the motion vector of the entire image or a part thereof)” according to the second embodiment is referred to as “measurement motion vector Vd”.

  As shown in FIG. 5, the imaging vibration damping device 1 </ b> B according to the second embodiment is different from the imaging vibration damping device 1 according to the first embodiment from the imaging vibration damping device 1 according to the first embodiment (see FIG. 1). The configuration is such that the half mirror 50 and the second image sensor 60 are excluded.

  Further, in the imaging vibration damping device 1B, the displacement amount estimation unit 70B uses an ornamental video imaged by the imaging element 10 and a control signal representing the value of the displacement control amount output from the displacement control unit 80. The estimated motion vector Vc is calculated.

  The displacement amount estimation unit 70B calculates the “measurement motion vector Vd” by performing block matching on the whole or a part of the video in the video for viewing between a plurality of frames taken by the imaging device 10 at different imaging times. .

  Specifically, for example, as shown in FIG. 6, the displacement amount estimation unit 70B precedes the current frame to be photographed by the image sensor 10 from now on, and the frame-1F and the frame− captured by the image sensor 10. The “measured motion vector Vd” is calculated by performing block matching on the whole or a part of the video for viewing between two frames with 2F.

  In the example illustrated in FIG. 6, the imaging vibration control device 1 </ b> B uses the imaging device 10 as a measurement image and an ornamental image at the “accumulation” timing of the frames −2F, −1F, F1, F2,. The "image after displacement" used together is photographed. The imaging vibration control device 1B performs the following processing after the “accumulation” period of the video of the frame -1F ends until the “accumulation” period of the video of the frame F1 starts.

  In the imaging vibration control device 1B, the control unit 2 reads each video of the frames -2F and -1F at the timing of the read time t1.

  Next, at the timing of the calculation time t2, the displacement amount estimation unit 70B of the control unit 2 first calculates the above-described “measured motion vector Vd” based on the respective images of the frames −2F and −1F, and then Based on the measured motion vector Vd, the above-described “estimated motion vector Vc” is calculated, and the displacement control unit 80 of the control unit 2 calculates “displacement control amount (ΔX, ΔY)”.

  The displacement amount estimation unit 70B performs block matching with decimal pixel accuracy when calculating the “measurement motion vector Vd”. A technique for performing block matching with decimal pixel accuracy is the same as the technique described in the first embodiment.

  Here, the imaging time of these two frames will be described as time ta and time tb (where ta <tb). Here, “imaging time” is defined as, for example, accumulation start time or accumulation end time, or an average value thereof.

  Further, here, the displacement control amount instructed by the displacement control unit 80 to the displacement units 40a to 40d at the time ta is the vector “Da”, and the displacement instructed by the displacement control unit 80 to the displacement units 40a to 40d at the time tb. The control amount will be described as a vector “Db”.

Under such conditions, the displacement amount estimation unit 70B calculates the “estimated motion vector Vc” with decimal pixel accuracy based on the following equation (3), for example.

  After calculating the estimated motion vector Vc, the displacement amount estimation unit 70B outputs the calculated estimated motion vector Vc to the displacement control unit 80. Then, the displacement control unit 80 calculates the displacement control amount (ΔX, ΔY) based on the above-described equation (2) using the calculated estimated motion vector Vc.

  That is, in the imaging vibration damping device 1B, the displacement control unit 80 uses the horizontal component value uc and the vertical component value vc of the “estimated motion vector Vc” from the horizontal component value uc and the vertical component value vc of the “estimated motion vector Vc”. A horizontal component calculation value round (uc) and a vertical component calculation value round (vc) obtained by performing a carry-up operation or a carry-down operation with the decimal part of the size of the unit cell region of the pixel directed to the nearest integer value for each. By subtracting, the displacement control amount (ΔX, ΔY) is calculated.

  After calculating the displacement control amount (ΔX, ΔY), the displacement control unit 80 outputs the displacement control amount (ΔX, ΔY) to the displacement units 40a to 40d. In response to this, the displacement units 40a to 40d move the image sensor 10 by a displacement control amount (ΔX, ΔY).

  The displacement amount estimation unit 70B performs a filtering process (for example, for the three most recent time points) on the time series of the “estimated motion vector Vc” calculated based on the above-described equation (3) instead of the estimated motion vector Vc. Is calculated (ie, Vcn = (Vc1 + Vc2 + Vc3) / 3) and the like to calculate an estimated motion vector Vcn that is a result of the filter processing, and the estimated motion vector You may make it output Vcn to the displacement control part 80. FIG.

  As described above, according to the imaging damping device 1B according to the second embodiment, as in the imaging damping device 1 according to the first embodiment (see FIG. 1), the compression encoding efficiency of the video is improved and the image quality is improved. Improvement of the image quality, suppression of an increase in the bit rate, and the like, and it is possible to prevent excessive cancellation of even the sway of the subject image intended for production reasons.

  Further, according to the imaging vibration control device 1B, as in the imaging vibration suppression device 1 according to the first embodiment, the imaging element 10 is displaced only within a range of ± 0.5 pixels in the horizontal direction and the vertical direction. The error between the true position of the image and the position of the photographed subject image is within a range of ± 0.5 pixels in the horizontal and vertical directions, and the dynamic image quality resulting from adding the image pickup vibration control device 1B to the camera Degradation can be minimized.

  In addition, according to the imaging vibration damping device 1B, the configuration is obtained by removing the half mirror 50 and the second imaging element 60 from the imaging vibration damping device 1 (see FIG. 1) according to the first embodiment. The apparatus can be made smaller than the imaging vibration damping device 1.

[Embodiment 3]
Next, an imaging vibration damping device 1C according to the third embodiment will be described. Compared with the imaging damping device 1 according to the first embodiment, the imaging damping device 1C is an element that can optically displace the optical path instead of the displacement units 40a to 40d (hereinafter, “optical path changing element”). And the displacement control unit 80C according to the third embodiment is different in that the control signal corresponding to the displacement control amount is output to the optical path changing element to operate the optical path changing element. Hereinafter, with reference to FIG. 7, the configuration of the imaging vibration damping device 1 </ b> C according to the third embodiment will be described.

  As illustrated in FIG. 7, in the imaging vibration damping device 1 </ b> C, a displacement unit 90 is disposed between the half mirror 50 and the first imaging element 10. The displacement part 90 is configured by an optical path changing element that can optically displace the optical path 30. Hereinafter, the displacement portion 90 is referred to as an “optical path changing element 90”.

  As shown in FIG. 8, the optical path changing element 90 moves the optical path 30 in the horizontal direction, the vertical direction, and the horizontal direction according to a control signal corresponding to the displacement control amount output from the displacement control unit 80C according to the third embodiment. This is an element that is displaced in multiple stages with decimal pixel accuracy in any direction of the oblique direction. The optical displacement unit 90 emits outgoing light so that the optical axis direction of incident light and the optical axis direction of outgoing light are parallel to each other.

  In such a configuration, in the imaging vibration damping device 1C according to the third embodiment, similarly to the imaging vibration damping device 1 according to the first embodiment, the displacement amount estimation unit 70 includes the “measured motion vector Vs” and the “estimated motion vector Vc”. Is calculated.

  That is, in the imaging vibration damping device 1C, similarly to the imaging vibration damping device 1 according to the first embodiment, first, the displacement amount estimation unit 70 has a plurality of frames with different imaging times (for example, taken by the second imaging element 60 (for example, In an image for measurement between two frames, frame-1 and frame-2, taken by the second image sensor 60 prior to the current frame to be photographed by the first image sensor 10, The “measured motion vector Vs” is calculated by performing block matching in whole or in part. Next, in the imaging vibration damping device 1C, similarly to the imaging vibration damping device 1 according to the first embodiment, the displacement amount estimation unit 70 uses the calculated measurement motion vector Vs based on the above-described equation (1). Then, the “estimated motion vector Vc” is calculated with decimal pixel accuracy.

  Thereafter, the imaging vibration control device 1C uses the estimated motion vector Vc calculated by the displacement amount estimation unit 70 by the displacement control unit 80C, similarly to the displacement control unit 80 of the imaging vibration suppression device 1 according to the first embodiment. Then, based on the above equation (2), “displacement control amount (ΔX, ΔY)” is calculated.

  That is, in the imaging vibration damping device 1C, the displacement control unit 80C uses the horizontal component value uc and the vertical component value vc of the “estimated motion vector Vc” from the horizontal component value uc and the vertical component value vc of the “estimated motion vector Vc”. A horizontal component calculation value round (uc) and a vertical component calculation value round (vc) obtained by performing a carry-up operation or a carry-down operation with the decimal part of the size of the unit cell region of the pixel directed to the nearest integer value for each. By subtracting, the displacement control amount (ΔX, ΔY) is calculated.

  When the displacement control unit 80C calculates the displacement control amount (ΔX, ΔY), the displacement control unit 80C generates a control signal corresponding to the displacement control amount (ΔX, ΔY), and outputs the generated control signal to the optical path changing element 90. At this time, the generated control signal is a direction in which the displacement units 40a to 40d of the imaging vibration damping device 1 (see FIG. 1) according to the first embodiment displace the first imaging element 10 with respect to the optical path changing element 90. The content is an instruction to displace the optical path 30 in the opposite direction and by the same magnitude (absolute value).

  As a result, the optical path changing element 90 is in the opposite direction and the same size as the direction in which the displacement units 40a to 40d of the imaging vibration damping device 1 (see FIG. 1) according to the first embodiment displace the first imaging element 10. The optical path 30 is displaced by the length (absolute value).

  According to the imaging vibration damping device 1C according to the third embodiment, as with the imaging vibration damping device 1 according to the first embodiment (see FIG. 1), the video compression encoding efficiency is improved, the image quality is improved, and the bit rate is improved. It is possible to suppress an increase in rate and the like, and to prevent excessive cancellation of even a subject image shake intended for production reasons.

  In addition, according to the imaging vibration damping device 1C, the imaging vibration damping device 1 according to the first embodiment displaces the first imaging element 10 only within a range of ± 0.5 pixels in the horizontal direction and the vertical direction, instead of the optical path. 30 is displaced only within a range of ± 0.5 pixels in the horizontal and vertical directions, so that the error between the true position of the subject and the position of the photographed subject image is ± 0.5 pixels in the horizontal and vertical directions. The dynamic image quality degradation caused by adding the imaging vibration control device 1C to the camera can be minimized.

[Embodiment 4]
Next, an imaging vibration damping device 1D according to the fourth embodiment will be described. Compared with the imaging damping device 1B according to the second embodiment (see FIG. 5), the imaging damping device 1D uses an optical path changing element 90 instead of the displacement portions 40a to 40d, and the fourth embodiment. Similar to the third embodiment, the displacement control unit 80 </ b> C is different in that the control signal corresponding to the displacement control amount is output to the optical path changing element 90 to operate the optical path changing element 90. Hereinafter, with reference to FIG. 9, the configuration of the imaging vibration damping device 1 </ b> D according to the fourth embodiment will be described. Here, the “measurement motion vector (that is, the measurement value of the motion vector of the entire imaging or a part thereof)” according to the fourth embodiment is referred to as “measurement motion vector Vd”.

  As illustrated in FIG. 9, in the imaging vibration damping device 1 </ b> D, an optical path changing element 90 is disposed between the half mirror 50 and the first imaging device 10, similarly to the imaging vibration damping device 1 </ b> C according to the third embodiment. Yes.

  In such a configuration, in the imaging damping device 1D according to the fourth embodiment, similarly to the imaging damping device 1B according to the second embodiment, the displacement amount estimation unit 70B performs “measured motion vector Vd” and “estimated motion vector Vc”. Is calculated.

  That is, in the imaging damping device 1D, similarly to the imaging damping device 1B according to the second embodiment, first, the displacement amount estimation unit 70B has a plurality of frames (for example, imaging device) with different imaging times taken by the imaging device 10. 10, an image for viewing between two frames, frame-1F (see FIG. 6) and frame-2F (see FIG. 6) photographed by the image sensor 10, prior to the current frame to be photographed. Then, the “measured motion vector Vd” is calculated by performing block matching on the whole or a part of the video. Next, in the imaging vibration damping device 1D, similarly to the imaging vibration damping device 1B according to the second embodiment, the displacement amount estimation unit 70B calculates the calculated measurement motion vector Vd, the vector Da representing the displacement control amount at the time ta, Then, using the vector Db representing the displacement control amount at the time tb, the “estimated motion vector Vc” is calculated with the decimal pixel accuracy based on the above equation (3).

  Thereafter, in the imaging vibration damping device 1D, similarly to the imaging vibration damping device 1B according to the second embodiment, the displacement control unit 80C uses the estimated motion vector Vc calculated by the displacement amount estimation unit 70B, and uses the above-described equation. Based on (2), the “displacement control amount (ΔX, ΔY)” is calculated.

  That is, in the imaging vibration damping device 1D, the displacement control unit 80C determines that the horizontal component value uc and the vertical component value vc of the “estimated motion vector Vc” from the horizontal component value uc and the vertical component value vc of the “estimated motion vector Vc”. A horizontal component calculation value round (uc) and a vertical component calculation value round (vc) obtained by performing a carry-up operation or a carry-down operation with the decimal part of the size of the unit cell region of the pixel directed to the nearest integer value for each. By subtracting, the displacement control amount (ΔX, ΔY) is calculated.

  When calculating the displacement control amount (ΔX, ΔY), the displacement control unit 80C generates a control signal corresponding to the displacement control amount (ΔX, ΔY) and changes the generated control signal to an optical path, as in the third embodiment. Output to the element 90. At this time, the generated control signal is a direction in which the displacement units 40a to 40d of the imaging vibration damping device 1 (see FIG. 1) according to the first embodiment displace the first imaging element 10 with respect to the optical path changing element 90. The content is an instruction to displace the optical path 30 in the opposite direction and by the same magnitude (absolute value).

  As a result, similarly to the third embodiment, the optical path changing element 90 is opposite to the direction in which the displacement units 40a to 40d of the imaging vibration damping device 1 (see FIG. 1) according to the first embodiment have displaced the first imaging element 10. The optical path 30 is displaced in the direction and by the same size (absolute value).

  According to the imaging vibration damping device 1D according to the fourth embodiment, as with the imaging vibration damping device 1C according to the third embodiment (see FIG. 7), the compression encoding efficiency of the video is improved, the image quality is improved, and the bit is improved. It is possible to suppress an increase in rate and the like, and to prevent excessive cancellation of even a subject image shake intended for production reasons.

  Further, according to the imaging vibration control device 1D, as in the imaging vibration suppression device 1C according to the third embodiment, the optical path 30 is displaced only within a range of ± 0.5 pixels in the horizontal direction and the vertical direction. The error between the true position and the position of the photographed subject image is within a range of ± 0.5 pixels in the horizontal and vertical directions, and dynamic image quality degradation caused by adding the imaging vibration control device 1D to the camera Can be kept to a minimum.

  Moreover, according to the imaging damping device 1D, the configuration is obtained by removing the half mirror 50 and the second imaging element 60 from the imaging damping device 1C according to the third embodiment (see FIG. 7). The apparatus can be made smaller than the imaging vibration damping device 1C.

  In the above-described first to fourth embodiments, the control units 2, 2B, 2C, and 2D each execute a control program that is pre-stored in a storage unit (not shown) so as to be readable. It is assumed that the estimation unit 70 (or 70B) "and the" displacement control unit 80 (or 80B) "are provided.

The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the present invention.
For example, the imaging vibration damping device 1 according to the first embodiment (see FIG. 1) and the imaging vibration damping device 1C according to the third embodiment (see FIG. 7) use a half mirror 50 as a branching member that branches the optical path. ing. However, the imaging vibration damping devices 1 and 1 </ b> C may be configured to use a half prism as a branching member instead of the half mirror 50.
Further, the image sensor 10 may be used in combination for motion estimation (substitute for the second image sensor 60) and video output in a time division manner.

  It should be noted that the above-described “displacement control amount in which the value of the displacement result vector becomes a value that coincides with any vector value or zero vector value in the vector value set obtained by connecting the center points of two adjacent pixels. Is a displacement applicable not only to a square lattice (see FIGS. 3A and 3B) and a honeycomb lattice (see FIG. 3C) but also to a lattice (not shown) such as a rectangular lattice, an oblique lattice, a quinkanx lattice, etc. Control amount.

1, 1B, 1C, 1D Imaging vibration control device 2, 2B, 2C, 2D Control unit 10, 60 Imaging element 20 Lens 30 Optical axis 40a, 40b, 40c, 40d Displacement unit (actuator)
50 Half mirror 70, 70B Displacement amount estimation unit 80, 80B Displacement control unit 90 Displacement unit (optical path changing element)

Claims (7)

An image sensor for capturing images;
A displacement part for generating a displacement of the optical position of the image sensor relative to the object field;
A control unit for controlling the displacement unit,
The controller is
An estimated value of an imaging motion vector generated when the displacement is not generated is used as an estimated motion vector, and the moving amount of the imaging on the image sensor between two frames having different imaging times is matched, A displacement amount estimation unit that calculates the estimated motion vector with a finer precision than the size of the unit cell region of the pixels constituting the image sensor;
The imaging motion vector obtained as a result of generating the displacement is defined as a displacement result vector, and the center of the two adjacent pixels whose values of the displacement result vector are adjacent from the estimated motion vector calculated by the displacement amount estimation unit A displacement control unit that calculates a displacement control amount that matches a vector value or zero vector value in a vector value set obtained by connecting points and outputs the displacement control amount to the displacement unit An imaging vibration damping device comprising: The displacement control unit is configured to determine, from a horizontal component value and a vertical component value of the estimated motion vector, a fractional part in the size of the unit cell region of the pixel for each of the horizontal component value and the vertical component value of the estimated motion vector 2. The imaging according to claim 1, wherein the displacement control amount is calculated by subtracting a horizontal component calculation value and a vertical component calculation value obtained by performing a carry operation or a carry operation of a digit toward an integer value. Damping device. Furthermore, another image sensor,
A branching member that branches an optical path into the direction of the imaging element and the direction of the other imaging element;
When the estimated motion vector of the imaging on the image sensor is set to “Vc” and the motion vector of the imaging on the other image sensor is set to “Vs”, the displacement amount estimation unit The motion vector Vs is calculated based on an image between two frames imaged by the imaging element, and the estimated motion vector Vc is calculated based on the calculated motion vector Vs. Item 3. The image damping device according to Item 2. The displacement amount estimation unit sets an imaging frame time interval, a horizontal pixel pitch, and a vertical pixel pitch of the imaging device to “Tc”, “Pc”, and “Qc”, respectively, and an imaging frame time interval of the other imaging device. When the horizontal pixel pitch and the vertical pixel pitch are “Ts”, “Ps”, and “Qs”, respectively, the estimated motion vector Vc is calculated based on Expression (1). Item 4. The imaging vibration damping device according to Item 3.

The displacement amount estimation unit sets the estimated motion vector of the imaging on the image sensor between the current frame used as an ornamental image by the image sensor and a frame imaged immediately before the image to be “Vc”, The motion vector of the image formation on the image sensor between two frames imaged at the times ta and tb (where ta <tb) past the current frame by the image sensor is “Vd”, and at the time ta 3. The estimated motion vector Vc is calculated based on Expression (2) when the displacement control amount is “Da” and the displacement control amount at time tb is “Db”. The imaging vibration damping device according to 1.

The imaging vibration damping device according to claim 1, wherein the displacement unit is a unit that physically moves one of the imaging element and the optical system. 6. The optical path changing element according to claim 1, wherein the displacement part is an optical path changing element that is disposed on an optical path and optically changes a position of the image formation on the imaging element. The imaging damping device according to item.

Images