JP2015173388A - Imaging apparatus and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress the magnification of system resources required for processing to correct a posture change of an imaging apparatus body, with the reduction of processing time and power consumption.SOLUTION: An imaging apparatus 100 performs the processing of: acquiring, when generating an image from the output of an imaging device 102, the posture change information of the imaging apparatus 100; detecting each motion vector from a plurality of time continuous images; predicting a motion vector between with the subsequent image, from the motion vectors at present and in the past; estimating, from the posture change information and the predicted motion vector, a first geometrical deformation parameter to determine a readout region for reading out a signal from the imaging device 102; estimating, from the posture change information and the motion vector, a second geometrical deformation parameter for reading out the target region of the geometrical deformation processing included in the readout region; controlling signal readout from the imaging device 102, on the basis of the first geometrical deformation parameter; and executing geometrical deformation processing on the image of the target region, using the second geometrical deformation parameter.

Description

  The present invention relates to an imaging apparatus and a control method therefor, and in particular, an image area required for geometric deformation by predicting a geometric deformation amount when performing geometric deformation anti-vibration by image processing according to a blurring state of the imaging apparatus main body. It is related with the technology to control.

  In recent imaging apparatuses such as digital video cameras and digital cameras, high-quality images (images) have been obtained due to improvements in imaging sensors and signal processing techniques. Further, as the size of the image pickup apparatus main body is further reduced, an opportunity to shoot with the image pickup apparatus main body held with one hand increases, and the posture change of the image pickup apparatus main body such as camera shake at that time tends to increase. Therefore, in order to acquire a more stable image (image) regardless of whether it is moving image shooting or still image shooting, the importance of a technique for correcting the posture change of the imaging device body is increasing.

  As a method for acquiring information related to the posture change of the imaging apparatus main body, a method using a sensing device such as an angular velocity sensor (gyro sensor) or an acceleration sensor, or a so-called motion that detects motion from an input image and a past image (reference image). There is a method by vector detection. The posture change of rotation and translation of the imaging device main body affects the image as the change in the angle of view affects the image, but it is possible to appropriately suppress the blur by applying geometric deformation processing with a high degree of freedom such as projective transformation. it can.

  However, in order to perform motion vector detection processing in real time in shooting situations with high time constraints such as movie shooting and still image continuous shooting, high-speed operation is required for systems such as signal processing circuit performance, memory, and memory bus bandwidth. What is possible and high performance are required. In a system with high performance in response to such a demand, generally, heat generation and power consumption increase due to high performance of a signal processing circuit, high capacity of a memory, and the like. On the other hand, with the downsizing of the imaging apparatus main body, restrictions on heat resistance and power consumption are increasing, and such restrictions are an obstacle to mounting a high-performance signal processing circuit.

  Therefore, the system design of the entire imaging apparatus balances the demand for the degree of freedom, accuracy, performance, etc. of posture change correction of the imaging apparatus main body, which are in a trade-off relationship with each other, and the demand for the imaging apparatus system resources, power consumption, etc. It is necessary to consider this.

  For such a problem, a technique for efficiently controlling memory resources has been proposed. For example, in the technique described in Patent Document 1, when a plurality of image processing units access the image memory via the system bus, the system bus is controlled with an access format suitable for each image processing. . Further, in the technique described in Patent Document 2, pipeline processing is performed while overlapping processing time via a buffer memory for image memory access when performing hierarchization processing.

JP 2009-116763 A JP 2009-081596 A

  However, when correcting the posture change of the imaging device main body, the amount of data accessed to the image memory varies greatly according to the posture change amount of the imaging device main body. For this reason, it is difficult to reduce the calculation amount of the entire signal processing and the load on the system bus only by devising the access format as in the technique described in Patent Document 1. Further, if the buffer memory capacity is fixedly secured as in the technique described in Patent Document 2, the amount of calculation of the entire signal processing and the load on the system bus cannot be reduced.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a technique capable of suppressing an increase in system resources required for processing for correcting a posture change of an imaging apparatus body and reducing processing time and power consumption.

  An imaging apparatus according to the present invention includes a reading unit that reads a signal from an imaging element on which a subject image is formed through an optical system, and a developing unit that generates an image from the signal read from the imaging element. And a posture change detecting means for detecting a posture change of the imaging device and outputting posture change information, and a detecting means for detecting a motion vector from a plurality of temporally continuous images generated by the developing means. Predicting means for predicting a motion vector between a current image and a past image generated by the developing means based on the motion vector in the past and a first geometry for determining a reading area for reading a signal from the image sensor. First estimation means for estimating a deformation parameter from the posture change information and the predicted motion vector; and a target of geometric deformation processing included in the readout area A second estimating means for estimating a second geometric deformation parameter for reading a region from the posture change information and the motion vector, and a signal from the reading region of the image sensor based on the first geometric deformation parameter. Control means for controlling the reading means so as to read, and geometric deformation means for performing a geometric deformation process on the image of the target region using the second geometric deformation parameter.

  According to the present invention, the amount of data to be subjected to signal processing is reduced in accordance with the posture change of the imaging apparatus main body, and the processing time and memory access for posture change correction are reduced. As a result, it is possible to realize a reduction in processing time and power consumption required for the correction processing while suppressing an increase in the scale of system resources for executing the posture change correction processing.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. 3 is a flowchart illustrating an imaging operation (imaging control method) of the imaging apparatus in FIG. 1. It is a figure which shows the general relationship between a focal distance (view angle) and the geometric deformation component which appears in an image, and a figure which shows the general relationship between an objective distance and the geometric deformation component which appears in an image. It is a schematic diagram explaining the outline | summary of template matching. It is a figure which shows typically the time fluctuation of a motion vector. It is a figure explaining the image formation area of the to-be-photographed image in an image sensor, and the read-out area | region set to a part of image formation area. It is a block diagram which shows schematic structure of the imaging device which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the imaging operation of the imaging device of FIG. It is a figure which shows typically a part of data holding area | region of a memory.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram showing a schematic configuration of an imaging apparatus 100 according to the first embodiment of the present invention. The imaging apparatus 100 is typically a digital still camera or a digital video camera, but is not limited to this, and various electronic devices having a camera function (a shooting function that acquires a video (image) using an imaging device). It may be. For example, the imaging apparatus 100 may be a mobile communication terminal with a camera function (such as a mobile phone or a smartphone), a portable computer with a camera function (tablet terminal), a mobile game machine with a camera function, or the like.

  The imaging apparatus 100 includes an imaging optical system 101, an imaging element 102, a development processing unit 103, a memory 104, an attitude change information acquisition unit 105, an image stabilization processing control unit 106, a motion vector detection unit 107, and a motion vector prediction unit 108. Further, the imaging apparatus 100 includes a first geometric deformation parameter estimation unit 109, a second geometric deformation parameter estimation unit 110, a geometric deformation processing unit 111, an image storage unit 112, an image display unit 113, and a read control unit 114.

  The imaging optical system 101 captures reflected light from the subject to form a subject image (optical image), and the formed subject image is formed on the image sensor 102. The imaging element 102 is an image sensor such as a CCD sensor or a CMOS sensor, for example, photoelectrically converts the formed subject image, and supplies the generated analog signal to the development processing unit 103 and the motion vector detection unit 107. Note that in accordance with an instruction from the read control unit 114, a signal (accumulated charge) of a part of an imaging region where a subject image is formed can be sequentially read from the image sensor 102.

  The development processing unit 103 includes an A / D conversion unit (not shown), an auto gain control unit (AGC), an auto white balance unit, and the like. The development processing unit 103 A / D converts the analog signal acquired from the image sensor 102 to generate a digital signal (RAW image), performs a predetermined development process on the generated RAW image, and generates an image signal. The processed image signal is supplied to the memory 104. The imaging optical system 101, the imaging element 102, and the development processing unit 103 constitute an imaging system that acquires a video (image). In this embodiment, it is assumed that the image signal output from the development processing unit 103 is a digital signal subjected to development processing regardless of whether the imaging apparatus 100 captures moving images or still images.

  The memory 104 temporarily stores and holds an image of one frame or a plurality of frames, which is an image signal supplied from the development processing unit 103. The posture change information acquisition unit 105 includes a gyro sensor that acquires posture change (motion) of the imaging apparatus 100 such as camera shake or camera work as posture change information, and sends the acquired posture change information to the image stabilization processing control unit 106. Supply. The motion vector detection unit 107 detects a motion vector between two temporally continuous frame images acquired from the memory 104. The motion vector prediction unit 108 predicts a motion vector between the next frame image from the current and past motion vectors detected by the motion vector detection unit 107.

  The first geometric deformation parameter estimation unit 109 uses the motion vector predicted by the motion vector prediction unit 108 to calculate a correction amount of blur motion between frame images as a geometric deformation parameter (first geometric deformation parameter). . The second geometric deformation parameter estimation unit 110 uses the motion vector detected by the motion vector detection unit 107 to calculate the amount of correction of motion blur between frame images as a geometric deformation parameter (second geometric deformation parameter). . The image stabilization processing control unit 106 controls the read control unit 114 based on the geometric deformation parameter calculated by the first geometric deformation parameter estimation unit 109. Also, the image stabilization processing control unit 106 is based on the posture change information supplied from the posture change information acquisition unit 105 and the motion vector supplied from the motion vector detection unit 107, and the motion vector prediction unit 108 and the first geometric deformation parameter estimation. The processing contents of the unit 109 and the second geometric deformation parameter estimation unit 110 are changed.

  Based on the geometric deformation parameter acquired from the second geometric deformation parameter estimation unit 110, the geometric deformation processing unit 111 performs a geometric deformation process for correcting blur on the frame image, and writes the image to the memory 104. The image storage unit 112 stores an image subjected to the image stabilization process in a storage medium. The image display unit 113 displays the image subjected to the image stabilization process on a display (not shown).

  The imaging apparatus 100 includes a central control unit (not shown) that performs overall operation control of the imaging apparatus 100. The central control unit includes an arithmetic processing unit (CPU), a ROM that stores programs and the like executed by the CPU, and a RAM that is used as a work area where the CPU develops programs and that temporarily stores calculation results and parameters. . The processing unit that performs the above-described various data processing may be a functional block of the central control unit or a functional block that exhibits a predetermined function under the control of the central control unit.

  FIG. 2 is a flowchart illustrating the imaging operation (imaging control method) of the imaging apparatus 100. Each process shown in FIG. 2 is realized by causing each block of the imaging apparatus 100 to execute a predetermined operation and process when the CPU expands and executes a program stored in the ROM in the RAM.

  In step S201, an image input process is performed. Specifically, when a subject image is formed on the image sensor 102 formed by the imaging optical system 101, the image sensor 102 uses the accumulated charge of the pixels in the readout area indicated by the readout controller 114 as an analog signal. 103. The development processing unit 103 converts the acquired analog signal into a digital signal (RAW image) by an A / D conversion unit, and further performs signal level correction and white level correction by AGC and AWB to generate an image signal. The image signal is stored and held in the memory 104. In the imaging apparatus 100, frame images are sequentially generated at a predetermined frame rate, and the frame images stored and held in the memory 104 are input to the motion vector detection unit 107. Further, the frame images stored and held in the memory 104 are sequentially updated.

  When shooting is started, the posture change information acquisition unit 105 acquires posture change information of the imaging device 100 in step S202 simultaneously with step S201. In this embodiment, it is assumed that a gyro sensor is used as a method of acquiring posture change information, and posture change information in each of the roll, pitch, and yaw directions of the imaging apparatus 100 is obtained. However, the posture change information acquisition means is not limited to the gyro sensor, and other means may be used as long as the posture change information of the imaging apparatus 100 can be obtained.

  In step S203, the image stabilization processing control unit 106 narrows down the geometric deformation parameters. Specifically, the geometric deformation parameters estimated by the first geometric deformation parameter estimation unit 109 and the second geometric deformation parameter estimation unit 110 are narrowed down according to the focal length and the objective distance of the imaging optical system 101 with respect to the subject. As a result, it is possible to control so that the instruction area to the read control unit 114 does not become excessive. Here, the process of step S203 will be described with reference to FIG.

  FIG. 3A is a diagram showing a general relationship between the focal length (angle of view) and the geometric deformation component appearing in the image, and FIG. 3B is a graph showing the relationship between the objective distance and the geometric deformation component appearing in the image. It is a figure which shows a general relationship.

  As shown in FIG. 3A, the geometric deformation component appearing in the image is estimated to change according to the focal length (field angle). In wide shooting, a shooting mode such as one-handed holding or walking is often used, and as a result, many geometric deformation components appear as blur components. Therefore, it is necessary to estimate parameters in consideration of tilting. On the other hand, in tele-photographing, the object is often captured by holding with both hands and, as a result, geometric deformation components other than the translational component are reduced. Therefore, parameter estimation using only the translational motion component is sufficient. In middle shooting, it is considered that blurring occurs between wide shooting and tele shooting, so for example, a threshold is set for the focal length, and if the focal length is shorter than the threshold, the tilt is taken into account. If is greater than or equal to the threshold, only translational movements should be considered.

  As shown in FIG. 3B, the geometric deformation component appearing in the image is estimated to change according to the objective distance. In macro photography with a very short objective distance, geometric deformation components other than the translation component are reduced as a result. Therefore, parameter estimation using only the translational motion component is sufficient. On the other hand, when the objective distance is long (far-distance shooting), the geometric deformation component appears as a blur component as a result, and therefore parameter estimation considering the tilt is necessary. In short-distance shooting, it is thought that blurring between macro shooting and long-distance shooting will occur, so for example, a threshold is set for the objective distance, and if the objective distance is shorter than the threshold, only the translational movement is considered, When the objective distance is greater than or equal to the threshold value, the tilt may be considered. The relationship between FIGS. 3A and 3B may be used in combination for control, or may be used after being corrected using the shake information actually acquired by the posture change information acquisition unit 105.

  In step S204, the motion vector detection unit 107 detects a motion vector between two input frame images based on a predetermined motion vector detection region and the number of detections. In the present embodiment, template matching is used as an example of a motion vector detection method. FIG. 4 is a schematic diagram for explaining an outline of template matching.

  4A shows an original image 400, and FIG. 4B shows a reference image 450. These images are frame images input from the development processing unit 103 and the memory 104. FIG. A template block 401 is arranged at an arbitrary position in the original image 400, and a correlation value between the template block 401 and each region of the reference image 450 is calculated. At this time, if the correlation value is calculated for the entire region of the reference image 450, the amount of calculation is enormous. Therefore, in practice, a rectangular region for calculating the correlation value on the reference image 450 is used as the search range 402. Set.

  Although the position and size of the search range 402 are not particularly limited, a correct motion vector cannot be detected unless the region corresponding to the movement destination of the template block 401 is included in the search range 402. Therefore, as an example of a correlation value calculation method, a sum of absolute differences (SAD) is used. However, the present invention is not limited to this, and other correlation values such as sum of squared differences (SSD) and normalized cross correlation (NCC) may be used. The calculation formula of SAD is shown in the following formula 1.

式１において、ｆ（ｉ，ｊ）は、テンプレートブロック４０１内の座標（ｉ，ｊ）における輝度値であり、ｇ（ｉ，ｊ）は、サーチ範囲４０２において相関値算出の対象となる相関値算出ブロック４０３内の各輝度値である。ＳＡＤでは、これらの輝度値ｆ（ｉ，ｊ），ｇ（ｉ，ｊ）についての差の絶対値を計算し、その総和を求めることで、相関値Ｓ＿ＳＡＤを得る。相関値Ｓ＿ＳＡＤの値が小さいほど、テンプレートブロック４０１と相関値算出ブロック４０３との間の輝度値の差分が小さい、つまりこれらのブロック内のテクスチャが類似していることを示している。

In Equation 1, f (i, j) is a luminance value at the coordinates (i, j) in the template block 401, and g (i, j) is a correlation value for which a correlation value is calculated in the search range 402. Each luminance value in the calculation block 403. In SAD, a correlation value S_SAD is obtained by calculating an absolute value of a difference between these luminance values f (i, j) and g (i, j) and obtaining a sum thereof. The smaller the correlation value S_SAD, the smaller the difference in luminance value between the template block 401 and the correlation value calculation block 403, that is, the textures in these blocks are similar.

  The correlation value calculation block 403 is moved for the entire region of the search range 402, the correlation value with the template block 401 is calculated, and the position where the correlation is highest is determined. Thereby, it is possible to detect a motion vector between images that indicates where the template block 401 on the original image 400 has moved in the reference image 450.

  The motion vector detection unit 107 performs such a motion vector detection process in a plurality of regions between the input frame images, and uses the detected motion vector group as the motion vector prediction unit 108 and the first geometric deformation parameter estimation unit 109. To supply.

  In step S205, the motion vector prediction unit 108 determines the next motion vector group from the motion vector group of the current frame detected in step S204 and the motion vector group of the previous frame in accordance with the geometric deformation parameter narrowed down in step S203. Predict. FIG. 5 is a diagram schematically showing temporal variation of the motion vector. The motion vector changes as time passes. For example, in order to predict the motion vector m2 at time t2, it can be obtained by extrapolation using a linear expression based on the motion vectors m0 and m1 at times t0 and t1, as shown in Equation 2 below.

なお、外挿補間方法は、一次式によるものに限られず、多次式によるものを用いてもよい。ステップＳ２０５では、このような動きベクトル予測処理が所定のベクトル数だけ行われ、予測された動きベクトル群は、防振処理制御部１０６に供給される。

Note that the extrapolation method is not limited to a linear expression, and a multi-order expression may be used. In step S <b> 205, such motion vector prediction processing is performed for a predetermined number of vectors, and the predicted motion vector group is supplied to the image stabilization processing control unit 106.

  In step S206, the first geometric deformation parameter estimation unit 109 estimates the value of the geometric deformation parameter instructed by the image stabilization processing control unit 106 using the next motion vector obtained in step S205. In the present embodiment, the following description will be given assuming that a 3 × 3 determinant called a homography matrix is used as an example of the geometric deformation means.

  First, it is assumed that a certain point a on the image represented by the following expression 3 moves to a point a ′ represented by the following expression 4 in the next frame. Here, the subscript T shown in Equation 3 and Equation 4 indicates a transposed matrix. By using the homography matrix H, the correspondence relationship between the point a in Expression 3 and the point a ′ in Expression 4 can be expressed by the following Expression 5. Here, the homography matrix H is a determinant representing the amount of deformation caused by translation, rotation, scaling, shearing, and tilting between images, and is represented by the following Expression 6.

ホモグラフィ行列Ｈの各要素は、ステップＳ２０４で得られる動きベクトル群、つまりフレーム画像間における代表点の対応関係を用いて推定することにより求めることができる。上記の式６のホモグラフィ行列Ｈでは、パラメータｈ_１３，ｈ_２３が並進の動き成分を、パラメータｈ_１１，ｈ_１２，ｈ_２１，ｈ_２２が回転、変倍及びせん断の動き成分を、パラメータｈ_３１，ｈ_３２があおりの動き成分をそれぞれ表している。

Each element of the homography matrix H can be obtained by estimation using the motion vector group obtained in step S204, that is, the correspondence of representative points between frame images. In the homography matrix H of Equation 6 above, the parameters h ₁₃ and h ₂₃ are the translational motion components, the parameters h ₁₁ , h ₁₂ , h ₂₁ and h ₂₂ are the rotational, scaling and shearing motion components, and the parameter h Reference numerals ₃₁ and h ₃₂ denote the movement components of the cage, respectively.

In the case where even the motion component of the tilt is targeted for image stabilization, since all of these eight parameters must be estimated, it is necessary to use the least square method with 8 degrees of freedom. On the other hand, in the case where the rotational motion component is the target of vibration isolation, it is not necessary to estimate the parameters h ₃₁ and h ₃₂ representing the tilt motion component, so the other six parameter estimations, That is, it is sufficient to perform estimation by the least square method with six degrees of freedom. When only translational motion components are to be subject to image stabilization, only the parameters h ₁₃ and h ₂₃ in Equation 5 need to be estimated. At this time, since the translational motion component is a simple motion such as translation of the image, there is no need to use a statistical estimation method such as the least square method, and the estimation is accurately performed even with a simple processing such as a histogram processing. It can be performed. In the present embodiment, the least square method or the histogram processing is used as the geometric deformation amount estimation method, but the present invention is not limited to this, and other estimation methods may be used.

In step S206, the motion vector group calculated by the motion vector prediction unit 108 is used, and the geometric deformation amount between the frame images is determined based on the selected geometric deformation parameter described with reference to the above equations 3 to 6. Estimate and calculate the amount of blur correction. That is, the homography matrix H estimated by the above method represents the amount of deformation of the image due to the blur of the imaging apparatus 100. Therefore, in order to correct the blur of the image, the amount of image deformation cancels the deformation due to the blur. Thus, the homography matrix H may be converted. That is, the homography matrix H is converted into an inverse matrix H- ¹ . When the movement from the point a to the point a ′ is caused by the shake of the imaging apparatus 100, the correspondence relationship between the point a and the point a ′ is expressed by the following Expression 7, and the point a after the shake occurs according to the Expression 7. It becomes possible to return ′ to the same coordinates as the point a before the shake occurs.

ステップＳ２０７において、防振処理制御部１０６は、ステップＳ２０５で求められた予測された動きベクトル群とステップＳ２０６で算出された幾何変形パラメータとから、撮像素子１０２の読み出し領域を決定する。図６は、撮像素子１０２における被写体像の結像領域と、結像領域の一部に設定される読み出し領域を説明する図である。第１の領域６００は、被写体像の結像領域、つまり、撮像素子１０２の全読み出し領域であり、１フレームの画像領域である。第２の領域６０１は、ぶれが生じていないときに読み出したい領域である。第３の領域６０２は、ぶれが生じていることにより読み出しが必要となった領域を示している。第４の領域６０３は、実際に読み出す必要がある領域を示している。

In step S207, the image stabilization processing control unit 106 determines a reading area of the image sensor 102 from the predicted motion vector group obtained in step S205 and the geometric deformation parameter calculated in step S206. FIG. 6 is a diagram for explaining the imaging region of the subject image in the image sensor 102 and the readout region set as a part of the imaging region. The first area 600 is an imaging area of a subject image, that is, an entire reading area of the image sensor 102, and is an image area of one frame. The second area 601 is an area to be read when there is no blurring. A third area 602 indicates an area that needs to be read due to blurring. A fourth area 603 indicates an area that actually needs to be read.

By converting the coordinates of the second region 601 to be read when there is no blurring with the inverse matrix H ⁻¹ of Equation 7 above, the third region 602 that needs to be read due to shaking. Can be requested. That is, the third area 602 becomes a target area for geometric deformation processing in step S209 described later. At this time, if the geometric deformation is indicated only by the homography matrix H, the third region 602 can be obtained by calculating the coordinates of the four vertices of the second region 601. However, in the case of geometric deformation processing including the effects of distortion and rolling distortion, the third region 602 is not necessarily rectangular, and it is necessary to perform coordinate calculation of the four sides of the second region 601. is there. When the third region 602 is obtained, a fourth region 603, which is a rectangle inscribed in the third region 602, is obtained, and this fourth region 603 is determined as a readout region for actually reading a signal from the image sensor 102. To do.

  In step 208, the second geometric deformation parameter estimation unit 110 estimates the value of the geometric deformation parameter. In step S206, the first geometric deformation parameter estimation unit 109 uses the prediction vector output from the motion vector prediction unit 108, but the second geometric deformation parameter estimation unit 110 outputs the current vector output from the motion vector detection unit 107. Information on a frame motion vector group is used. The method of estimating the value of the geometric deformation parameter in the second geometric deformation parameter estimating unit 110 is the same as the method of estimating the value of the geometric deformation parameter in step S206 by the first geometric deformation parameter estimating unit 109. Omitted.

  In step S209, the geometric deformation processing unit 111 performs a geometric conversion process on the image of the third region 602 using the geometric deformation amount obtained in step S208, thereby completing the image stabilization process. In subsequent step S210, the image storage unit 112 stores the image subjected to the image stabilization process in a storage device (not illustrated), and the image display unit 113 displays the image subjected to the image stabilization process (not illustrated). Display on the device.

  By the way, in the image stabilization processing according to the flowchart of FIG. 2, the geometric deformation parameter used based on the posture change information of the imaging device 100 acquired by the posture change information acquisition unit 105 is switched in step S203. At this time, if the geometric deformation parameters are simply switched, there is a possibility that discontinuity occurs in the motion between the frame images before and after the switching. Therefore, in order to solve this problem, it is preferable to perform a smoothing process in the time direction (according to the passage of time) on the geometric deformation parameter and its value to be used.

  In this smoothing process, while the image stabilization process is being performed, the geometric deformation parameters for the past several frames are stored and retained, and the influence of the past geometric deformation parameters is left on the few frames immediately after switching. Vibration correction processing is performed by correcting the geometric deformation parameters. As a result, it is possible to eliminate the occurrence of discontinuous movement at the moment of switching of the geometric deformation parameters, and to gradually switch to the image stabilization processing using different geometric deformation parameters. The smoothing method is not particularly limited. For example, a method such as a moving average or an IIR (Infinite impulse response) filter can be used. The smoothing process is performed using the first geometric deformation parameter estimation unit 109 and the second geometric deformation. This is performed in the parameter estimation unit 110.

  Further, although a motion vector group is predicted in step S205, since the motion vector is detected from the image, the quality of information may vary depending on the size and motion of the subject and the motion of the background. For example, when the posture change information acquisition unit 105 is a gyro sensor, the geometric deformation parameters obtained as the homography matrix Hg when detecting rotational shakes of the rotation angles α, β, and γ in the yaw, pitch, and roll directions are as follows: Can be represented by the following formula 8.

画像のぶれを補正するには、ぶれによる変形を打ち消すような画像変形量となるようにホモグラフィ行列Ｈを変換する必要があったのと同様に、ホモグラフィ行列Ｈｇを逆行列Ｈｇ^−１に変換する必要がある。そこで、第１幾何変形パラメータ推定部１０９及び第２幾何変形パラメータ推定部１１０のそれぞれにおいて、逆行列Ｈ^−１と逆行列Ｈｇ^−１とを線形合成し、その際に、逆行列Ｈ^−１と逆行列Ｈｇ^−１との差分が大きくなったら、Ｈｇ^−１の比率を高くするようにする。これにより、動きベクトルの情報品質の変動を補正して、ベクトル予測が外れたことによる幾何変形の参照領域不足を解消することができる。

In order to correct image blurring, the homography matrix Hg is converted to the inverse matrix Hg ⁻¹ in the same manner as the homography matrix H needs to be converted so as to obtain an image deformation amount that cancels the deformation due to blurring. Need to convert. Therefore, in each of the first geometric deformation parameter estimation unit 109 and the second geometric deformation parameter estimation unit 110, the inverse matrix H ⁻¹ and the inverse matrix Hg ⁻¹ are linearly combined, and at that time, the inverse matrix H ⁻¹ and When the difference from the inverse matrix Hg ⁻¹ increases, the ratio of Hg ⁻¹ is increased. Thereby, it is possible to correct the variation in the information quality of the motion vector and solve the shortage of the reference area of the geometric deformation due to the vector prediction being deviated.

  As described above, in the present embodiment, the amount of input data from the image sensor 102 can be reduced in accordance with a change in the attitude of the image capturing apparatus 100, thereby avoiding an increase in the scale of system resources and the image stabilization process. The processing time required for the process can be reduced. Thereby, power consumption can be suppressed.

Second Embodiment
FIG. 7 is a block diagram showing a schematic structure of an imaging apparatus 700 according to the second embodiment of the present invention. In the imaging device 100 described in the first embodiment, the image stabilization processing control unit 106 is configured to control the readout control unit 114. However, in the imaging device 700 according to the second embodiment, the development processing unit is set to 103. The storage area of the memory 104 is controlled. Therefore, in FIG. 7, among the constituent elements of the imaging apparatus 700, the same constituent elements as those of the imaging apparatus 100 shown in FIG.

  In the imaging apparatus 700, in order to reduce the processing data, the development processing unit 103 performs a digital signal after A / D conversion of the analog signal acquired from the imaging element 102 in accordance with an instruction from the image stabilization processing control unit 106. Once stored in the memory 104. Then, the digital signal stored in the memory 104 is read again to the development processing unit 103, and signal level correction and white level correction are performed by the AGC and AWB in the development processing unit 103 to convert them into image signals.

  FIG. 8 is a flowchart showing the imaging operation of the imaging apparatus 700. Each process illustrated in FIG. 7 is realized by causing each block of the imaging apparatus 100 to execute a predetermined operation and process when the CPU expands and executes a program stored in the ROM in the RAM.

  In the flowchart of FIG. 8, the same processes as those in the flowchart of FIG. 2 are denoted by the same reference numerals, and description thereof is omitted here (steps S <b> 202 to S <b> 206 and S <b> 208 to S <b> 210).

  In the image input performed in step S801 instead of step S201 in FIG. 2, when the subject optical image is formed on the image sensor 102 by the imaging optical system 101, the image sensor 102 accumulates pixels in an area designated by the read control unit 114. The charge is output to the development processing unit 103 as an analog signal. The development processing unit 103 A / D-converts only the analog signal in the area instructed by the image stabilization processing control unit 106 among the analog signals acquired from the image sensor 102, and performs predetermined development processing on the generated RAW image. The generated image signal is stored in the memory 104. The image signal is stored in the memory 104 for each frame.

  In the imaging apparatus 700, frame images are sequentially generated by the processing in step S801 at a predetermined frame rate, and the frame images stored and held in the memory 104 are input to the motion vector detection unit 107. At this time, the frame images stored and held in the memory 104 are sequentially updated.

  In step S801, processing may be performed as follows. That is, first, accumulated charges are read from the pixels in the entire area of the image sensor 102, and a digital signal after A / D conversion is performed in the development processing unit 103 is temporarily stored in the memory 104. Then, the development processing unit 103 reads out the digital signal of only the area designated by the image stabilization processing control unit 106 from the memory 104, performs development processing such as signal level correction and white level correction, and stores the generated image signal in the memory 104. You may make it memorize | store. Not limited to this, the development signal after the signal level correction and the white level correction are performed on the digital signal after A / D conversion is temporarily stored in the memory 104, and the image stabilization processing control unit 106 instructs from there. The image signal of the area to be processed may be taken out. However, by writing data to the memory 104 at an early stage of processing in the development processing unit 103, the effect of suppressing the data amount of signal processing is enhanced.

  In step S802, the fourth area 603 described with reference to FIG. 6 is determined, and an image signal corresponding to the fourth area 603 is determined as an area to be written to the memory 104. FIG. 9 is a diagram schematically showing a part of the data holding area of the memory 104. The first storage area 900 is an area that can hold data of the first area 601 that is the entire reading area of the image sensor 102. The second storage area 901 is an area that can hold an image signal corresponding to the third area 602 that needs to be read from the image sensor 102 due to a shake. The third storage area 902 is an area that can hold an image signal corresponding to the fourth area 603 that actually needs to be read from the image sensor 102. In the present embodiment, by using only the third storage area 902, it is possible to reduce the use area of the memory 104 and reduce the load related to writing and reading data to the memory 104.

  The image input in step S803 refers to an image signal that has been developed (signal level correction or white level correction processed) in the fourth area 603 that the development processing unit 103 actually needs to read, determined in step S802. Is a process of writing only to the memory 104. Here, as in step S801, the accumulated charge may be read from the entire area of the image sensor 102, and only the fourth area 603 may be developed and written to the memory 104. However, by controlling the image signal at the time of writing, the effect of ensuring the margin of the bus bandwidth of the memory 104, the effect of shortening the access time, and the effect of suppressing the power consumption are maximized.

  As described above, in the present embodiment, the input / output data amount of the memory 104 is reduced in accordance with the posture change of the imaging apparatus 700. As a result, the processing time required for the image stabilization process can be reduced, the margin of the memory bus bandwidth can be ensured, and the power consumption can be suppressed.

<Other embodiments>
Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. Furthermore, each embodiment mentioned above shows only one embodiment of this invention, and it is also possible to combine each embodiment suitably.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

DESCRIPTION OF SYMBOLS 101 Image pick-up optical system 102 Image pick-up element 103 Development processing part 104 Memory 105 Attitude change information acquisition part 106 Anti-vibration process control part 107 Motion vector detection part 108 Motion vector prediction part 109 1st geometric deformation parameter estimation part 110 2nd geometric deformation parameter estimation Unit 111 geometric deformation processing unit 112 image storage unit 113 image display unit 114 read control unit

Claims (8)

An image pickup apparatus comprising: a reading unit that reads a signal from an image sensor on which a subject image is formed through an optical system; and a developing unit that generates an image from the signal read from the image sensor.
Posture change detection means for detecting posture change of the imaging device and outputting posture change information;
Detecting means for detecting a motion vector from a plurality of temporally continuous images generated by the developing means;
Prediction means for predicting a motion vector between a current image and a previous image generated by the developing means from the past motion vector;
First estimation means for estimating a first geometric deformation parameter for determining a readout region for reading a signal from the image sensor from the posture change information and the predicted motion vector;
Second estimation means for estimating a second geometric deformation parameter for reading a target region of the geometric deformation process included in the read region from the posture change information and the motion vector;
Control means for controlling the reading means so as to read a signal from the reading area of the image sensor based on the first geometric deformation parameter;
An imaging apparatus comprising: a geometric deformation unit that performs a geometric deformation process on the image of the target region using the second geometric deformation parameter. An imaging device that generates an image from a subject image formed on an imaging device through an optical system,
A developing unit that generates an image from a signal in a predetermined region among signals output from the image sensor, and stores the image in a storage unit;
Posture change detection means for detecting posture change of the imaging device and outputting posture change information;
Detecting means for detecting a motion vector from a plurality of temporally continuous images generated by the developing means;
Prediction means for predicting a motion vector between a current image and a previous image generated by the developing means from the past motion vector;
First estimation means for estimating a first geometric deformation parameter for determining a read area to be read from an image stored in the storage means from the posture change information and the predicted motion vector;
Second estimation means for estimating a second geometric deformation parameter for reading a target region of the geometric deformation process included in the read region from the posture change information and the motion vector;
Control means for reading out the image of the readout area from the image stored in the storage means based on the first geometric deformation parameter;
An imaging apparatus comprising: a geometric deformation unit that performs a geometric deformation process on the image of the target region using the second geometric deformation parameter.   The control means is configured to estimate the value by the first geometric deformation parameter and the second estimation means that the first estimation means estimates values according to a focal length and / or an objective distance of the optical system. The imaging apparatus according to claim 1, wherein the second geometric deformation parameter is changed. The first geometric deformation parameter whose value is estimated by the first estimating means is obtained by smoothing the first geometric deformation parameter used in the past in the time direction,
4. The second geometric deformation parameter whose value is estimated by the second estimating means is obtained by smoothing a second geometric deformation parameter used in the past in a time direction. The imaging device according to any one of the above. The first geometric deformation parameter includes a geometric deformation parameter obtained from the motion vector and a geometric deformation parameter obtained from the posture change information.
The control means increases the ratio of the geometric deformation parameter obtained from the posture change information as the difference between the geometric deformation parameter obtained from the motion vector and the geometric deformation parameter obtained from the posture change information increases. The imaging apparatus according to any one of claims 1 to 4, wherein the first geometric deformation parameter is set. The second geometric deformation parameter includes a geometric deformation parameter obtained from the predicted motion vector and a geometric deformation parameter obtained from the posture change information,
The control means increases the ratio of the geometric deformation parameter obtained from the posture change information as the difference between the geometric deformation parameter obtained from the predicted motion vector and the geometric deformation parameter obtained from the posture change information increases. The imaging apparatus according to claim 1, wherein the second geometric deformation parameter is set. A method for controlling an imaging apparatus,
A reading step of forming a subject image on an image sensor through an optical system and reading a signal from the image sensor;
A development step of generating an image from a signal read from the image sensor;
An acquisition step of detecting posture change of the imaging device and acquiring posture change information;
A detection step of detecting a motion vector from a plurality of temporally continuous images generated in the development step;
A prediction step of predicting a motion vector between a current image and a past image generated by the development step from the past motion vector;
A first estimation step of estimating a first geometric deformation parameter for determining a readout region for reading a signal from the image sensor from the posture change information and the predicted motion vector;
A second estimating step for estimating a second geometric deformation parameter for reading a target region of the geometric deformation process included in the read region from the posture change information and the motion vector;
A control step of controlling the readout step so as to read out a signal from the readout region of the imaging device based on the first geometric deformation parameter;
And a geometric deformation step for performing a geometric deformation process on the image of the target region using the second geometric deformation parameter. A method for controlling an imaging apparatus,
A developing step of forming a subject image on an image sensor through an optical system, and generating an image from a signal in a predetermined region among signals output from the image sensor;
A storage step of storing the image generated in the developing step in a storage unit;
An acquisition step of detecting posture change of the imaging device and acquiring posture change information;
A detection step of detecting a motion vector from a plurality of temporally continuous images generated by the developing step;
A prediction step of predicting a motion vector between a current image and a past image generated by the development step from the past motion vector;
A first estimation step for estimating a first geometric deformation parameter for determining a read area to be read from an image stored in the storage means from the posture change information and the predicted motion vector;
A second estimation step of estimating a second geometric deformation parameter for reading out a target area of the geometric deformation process included in the readout area from the storage unit, from the posture change information and the motion vector;
A reading step of reading an image of the reading area from an image stored in the storage unit based on the first geometric deformation parameter;
A control method for an imaging apparatus, comprising: a geometric deformation step of performing a geometric deformation process on the image of the target region using the second geometric deformation parameter.

Images