JP5869833B2 - Camera shake correction processing apparatus and camera shake correction processing method - Google Patents

Description

  The present invention relates to a camera shake correction processing apparatus and a camera shake correction processing method.

  If the camera moves due to camera shake when shooting a moving image with the camera in hand, the effect of camera shake appears in the acquired moving image. As an apparatus for compensating for such a camera shake, a camera shake correction processing apparatus that corrects the influence of camera shake in a moving image is known (Patent Documents 1 and 2). In the electronic camera shake correction processing for a moving image, the motion vector (direction and amount) of the image due to the camera shake is based on the pixel data of the preceding and subsequent image frames among the plurality of image frames constituting the moving image. ) Is detected, and pixel data of a partial range in the image frame is cut out based on the detection result of the motion vector, thereby performing a camera shake correction process.

JP 2007-151008 A JP 2006-246270 A

  In such a camera shake correction process based on motion vector detection, pixel data of a current image frame to be corrected needs to be held in a frame memory, and pixel data of a past image frame needs to be held in a frame memory. There is. Generally, a large capacity memory such as SDRAM (Synchronous Dynamic Random Access Memory) is used as a frame memory, and a control circuit for controlling the SDRAM is required. These cause the cost increase of the camera shake correction processing apparatus and the cost increase of the camera system equipped with the camera shake correction processing apparatus.

  Although the invention described in Patent Document 2 does not require a frame memory that holds pixel data of a current image frame to be corrected, it still needs a frame memory that holds pixel data of a past image frame. .

  The present invention has been made to solve the above-described problems, and an object thereof is to provide a camera shake correction processing apparatus and a camera shake correction processing method capable of performing camera shake correction processing without using a frame memory. To do.

A camera shake correction processing device according to the present invention is a device for correcting the influence of camera shake in a moving image composed of a plurality of image frames. (1) The nth image frame F (n) of the plurality of image frames. The pixel data of the first area in the image frame F (n) input in the initial predetermined period when the pixel data is input, and the (n−1) th image frame F () before the image frame F (n). n-1) motion for detecting a motion vector of an image during a period T (n) from the image frame F (n-1) to the image frame F (n) based on the information on the feature points of the first area A vector detection unit; (2) a correction amount calculation unit that calculates a cutout position for the image frame F (n) based on the motion vector of the image detected by the motion vector detection unit; and (3) a correction amount calculation unit. A part in the image frame F (n) based on the calculated cut-out position Cut out pixel data of the circumference, and image cropping processing unit for outputting the pixel data after the cut, (4) as information about the feature points in the image frame F (n-1), stores the coordinate values of the feature points And a feature point data memory for selectively storing pixel data of some pixel blocks including the feature point in the image frame in association with the image frame, and (5) a feature in the image frame F (n). And a search data memory that selectively stores pixel data of a part of pixel blocks including a position corresponding to a coordinate value of a point . Further, the motion vector detection unit detects information about the feature points in the image frame F (n−1) stored in the feature point data memory and a part of the image frame F (n) stored in the search data memory. A motion vector of an image in a period T (n) is detected based on pixel data of a pixel block.

The motion vector detection unit includes (1) a first extraction unit that extracts a plurality of feature points in a first area in each image frame, and (2) a coordinate value of each feature point extracted by the first extraction unit and A first storage unit that associates the pixel data of the pixel block including the feature point with each other and stores it in the feature point data memory ; and (3) an image frame F (n−1) stored in the feature point data memory by the first storage unit. ) Read out the coordinate value of each feature point in the first area and pixel data of the pixel block including the feature point, and the pixel of the pixel block including the feature point in the first area in the image frame F (n−1). Based on the data and the pixel data of the pixel block including the position corresponding to the coordinate value of the feature point in the first area in the image frame F (n) stored in the search data memory , the period T (n) First detection for detecting a motion vector of each feature point in the first area And (4) a first calculation for calculating a motion vector of the image in the period T (n) based on the motion vector of each feature point in the first area in the period T (n) detected by the first detection unit. It is preferable to include a part. At this time, it is preferable that the correction amount calculation unit calculates the cutout position for the image frame F (n) based on the motion vector of the image of the period T (n) calculated by the first calculation unit.

The motion vector detection unit includes: (1) a second extraction unit that extracts a plurality of feature points in a second area other than the first area in each image frame; and (2) each feature point extracted by the second extraction unit. A second storage unit for storing the coordinate value of the pixel block and the pixel data of the pixel block including the feature point in the feature point data memory, and (3) an image frame stored in the feature point data memory by the second storage unit The coordinate value of each feature point in the second area in F (n-1) and the pixel data of the pixel block including the feature point are read, and the feature point in the second area in the image frame F (n-1) is read. Based on the pixel data of the pixel block including the pixel block and the pixel data of the pixel block including the position corresponding to the coordinate value of the feature point in the second area in the image frame F (n) stored in the search data memory. The motion vector of each feature point in the second area of T (n) A second detection unit to detect, and (4) based on the motion vector of each feature point in the second area in the period T (n) detected by the second detection unit, the motion vector of the image in the period T (n) It is also preferable to further include a second calculation unit for calculating. At this time, the correction amount calculation unit calculates the motion vector of the image in the period T (n) calculated by the first calculation unit, and the motion vector of the image in the period T (n−1) calculated by the second calculation unit. Based on the above, it is preferable to calculate the cut-out position for the image frame F (n).

The first storage unit preferably stores the feature amount of each feature point extracted by the first extraction unit in the feature point data memory . At this time, the first calculation unit weights the motion vectors of the feature points detected by the first detection unit with the feature amounts of the feature points stored in the feature point data memory by the first storage unit. Thus, it is preferable to calculate the motion vector of the image during the period T (n).

It is preferable that the second storage unit also stores the feature amount of each feature point extracted by the second extraction unit in the feature point data memory . At this time, the second calculation unit weights the motion vectors of the feature points detected by the second detection unit with the feature amounts of the feature points stored in the feature point data memory by the second storage unit. Thus, it is preferable to calculate the motion vector of the image during the period T (n).

  The correction amount calculation unit calculates the motion vector of the image during the period T (n) calculated by the first calculation unit and the motion vector of the image during the period T (n−1) calculated by the second calculation unit. It is preferable to calculate the cut-out position for the image frame F (n) by calculating the motion vector of the image by weighting.

  The image processing system of the present invention includes (1) the above-described camera shake correction processing device of the present invention that corrects the influence of camera shake in a moving image composed of a plurality of image frames, and (2) pixel data of each of the plurality of image frames. And an image signal processing unit that performs image processing different from the camera shake correction processing performed by the camera shake correction processing device. The image processing system of the present invention outputs pixel data of each image frame that has been subjected to image processing by an image signal processing unit and camera shake correction processing by a camera shake correction processing device.

  The correction amount calculation unit of the camera shake correction processing device calculates the cutout position for the image frame F (n) until the image signal processing unit outputs pixel data after image processing for the image frame F (n). Is preferred. At this time, the image cutout processing unit of the camera shake correction processing device, based on the cutout position calculated by the correction amount calculation unit, the pixel data of the partial range in the image frame F (n) output from the image signal processing unit. It is preferable to cut out and output pixel data after the cutout.

The camera shake correction processing method of the present invention is a method for correcting the influence of camera shake in a moving image composed of a plurality of image frames. (1) The nth image frame F (n) of the plurality of image frames is corrected. The pixel data of the first area in the image frame F (n) input in the initial predetermined period when the pixel data is input, and the (n−1) th image frame F () before the image frame F (n). n-1) motion for detecting a motion vector of an image during a period T (n) from the image frame F (n-1) to the image frame F (n) based on the information on the feature points of the first area A vector detection step; (2) a correction amount calculation step for calculating a cutout position for the image frame F (n) based on the motion vector of the image detected in the motion vector detection step; and (3) a correction amount calculation step. Based on the calculated clipping position, Cut out the pixel data of sub-ranges in beam F (n), to the image cutting processing step of outputting the pixel data after the cut, characterized in that it comprises a. Further, in the motion vector detection step, the coordinate value of the feature point is stored as information on the feature point in the image frame F (n-1), and the feature point is included in the image frame in association with the coordinate value. A feature point data memory for selectively storing pixel data of some pixel blocks, and pixel data of some pixel blocks including positions corresponding to coordinate values of feature points in the image frame F (n) are selectively selected. And a search data memory stored in the image data F, information on feature points in the image frame F (n-1) stored in the feature point data memory, and an image frame F (n) stored in the search data memory. The motion vector of the image in the period T (n) is detected based on the pixel data of some of the pixel blocks.

The motion vector detection step includes (1) a first extraction step for extracting a plurality of feature points in the first area in each image frame, and (2) a coordinate value of each feature point extracted in the first extraction step, and A first storage step for storing in the feature point data memory in association with the pixel data of the pixel block including the feature point ; and (3) an image frame F (n−1) stored in the feature point data memory in the first storage step. ) Read out the coordinate value of each feature point in the first area and pixel data of the pixel block including the feature point, and the pixel of the pixel block including the feature point in the first area in the image frame F (n−1). Based on the data and the pixel data of the pixel block including the position corresponding to the coordinate value of the feature point in the first area in the image frame F (n) stored in the search data memory , the period T (n) Movement vector of each feature point in the first area A first detection step for detecting a toll; and (4) a motion of the image in the period T (n) based on the motion vector of each feature point in the first area in the period T (n) detected in the first detection step. And a first calculation step of calculating a vector. At this time, in the correction amount calculation step, it is preferable to calculate the cutout position for the image frame F (n) based on the motion vector of the image in the period T (n) calculated in the first calculation step.

The motion vector detection step includes (1) a second extraction step for extracting a plurality of feature points in a second area other than the first area in each image frame, and (2) each feature point extracted in the second extraction step. A second storing step for storing the coordinate value of the pixel block and the pixel data of the pixel block including the feature point in the feature point data memory, and (3) the image frame stored in the feature point data memory in the second storing step The coordinate value of each feature point in the second area in F (n-1) and the pixel data of the pixel block including the feature point are read, and the feature point in the second area in the image frame F (n-1) is read. Based on the pixel data of the pixel block including the pixel block and the pixel data of the pixel block including the position corresponding to the coordinate value of the feature point in the second area in the image frame F (n) stored in the search data memory. Each of the second area of T (n) A second detection step of detecting a motion vector of the feature point; and (4) a period T (n) based on the motion vector of each feature point in the second area of the period T (n) detected in the second detection step. It is preferable that the method further includes a second calculation step of calculating a motion vector of the first image. At this time, in the correction amount calculation step, the motion vector of the image in the period T (n) calculated in the first calculation step, and the motion vector of the image in the period T (n−1) calculated in the second calculation step Based on the above, it is preferable to calculate the cut-out position for the image frame F (n).

In the first storage step, it is preferable that the feature amount of each feature point extracted in the first extraction step is also stored in the feature point data memory . At this time, in the first calculation step, the motion vector of each feature point detected in the first detection step is weighted with the feature amount of each feature point stored in the feature point data memory in the first storage step. Thus, it is preferable to calculate the motion vector of the image during the period T (n).

In the second storage step, it is preferable that the feature amount of each feature point extracted in the second extraction step is also stored in the feature point data memory . At this time, in the second calculation step, the motion vector of each feature point detected in the second detection step is weighted with the feature amount of each feature point stored in the feature point data memory in the second storage step. Thus, it is preferable to calculate the motion vector of the image during the period T (n).

  In the correction amount calculation step, the motion vector of the image in the period T (n) calculated in the first calculation step and the motion vector of the image in the period T (n−1) calculated in the second calculation step. It is preferable to calculate the cut-out position for the image frame F (n) by calculating the motion vector of the image by weighting.

  The image processing method of the present invention includes (1) the above-described camera shake correction processing method of the present invention for correcting the influence of camera shake in a moving image composed of a plurality of image frames, and (2) pixel data of each of the plurality of image frames. And an image signal processing step for performing image processing different from the camera shake correction processing by the camera shake correction processing method. Furthermore, the image processing method of the present invention is characterized by outputting pixel data of each image frame that has been subjected to image processing by the image signal processing step and camera shake correction processing by the camera shake correction processing method.

  In the correction amount calculation step of the camera shake correction processing method, the cutout position for the image frame F (n) is calculated until the pixel data after image processing is output for the image frame F (n) in the image signal processing step. Is preferred. At this time, in the image cutout processing step of the camera shake correction processing method, the pixel data of the partial range in the image frame F (n) output from the image signal processing step based on the cutout position calculated in the correction amount calculation step It is preferable to cut out and output pixel data after the cutout.

  According to the present invention, it is possible to perform camera shake correction processing without using a frame memory.

It is a figure which shows the structure of the camera system 1 provided with the camera-shake correction processing apparatus 10 of this embodiment. It is a figure explaining the time order of image frame F (n-2), F (n-1), F (n), F (n + 1), ... which comprises a moving image. It is a figure explaining the 1st area and 2nd area in image frame F (n). It is a figure which shows the input timing and output timing of image frame F (n) in the image signal processing part 20. It is a figure explaining the feature point extraction method in this embodiment. It is a block diagram which shows the structure of the motion vector detection part 11 of the camera-shake correction processing apparatus 10 of this embodiment. It is a flowchart explaining the camera-shake correction processing method of this embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a diagram illustrating a configuration of a camera system 1 including a camera shake correction processing device 10 according to the present embodiment. The camera system 1 includes an image processing system 2, an imaging unit 30, and a display unit 40. The image processing system 2 includes a camera shake correction processing device 10 and an image signal processing unit 20. Further, the camera shake correction processing device 10 includes a motion vector detection unit 11, a correction amount calculation unit 12, an image cutout processing unit 13, a feature point data memory 14, and a search data memory 15.

  The imaging unit 30 converts an optical image of a subject into an electrical signal and outputs the electrical signal, and outputs pixel data of each of a plurality of image frames constituting a moving image to the camera shake correction processing device 10 and the image signal processing unit 20. To do. For example, a camera including a CMOS image sensor or a CCD image sensor is used as the imaging unit 30.

  The image signal processing unit 20 receives the pixel data of each image frame output from the imaging unit 30, and performs various types of image processing (for example, white balance adjustment, noise removal) different from the camera shake correction processing for the pixel data of each image frame. , Gamma correction, color space conversion, etc.) and output pixel data of the processed image frame.

  The camera shake correction processing apparatus 10 corrects the influence of camera shake in a moving image composed of a plurality of image frames. Specifically, the camera shake correction processing apparatus 10 receives pixel data of each image frame output from the imaging unit 30, detects a motion vector (direction and amount) of the image based on the pixel data, and detects the detected motion vector. Based on the motion vector, the cutout position for the image frame is calculated. Then, based on the calculated cutout position, the camera shake correction processing apparatus 10 cuts out the pixel data of the partial range from the pixel data of the image frame output from the image signal processing unit 20, and the pixel data after the cutout Is output to the display unit 40.

  The display unit 40 inputs pixel data of each image frame that has been subjected to various types of image processing by the image signal processing unit 20 and camera shake correction processing by the camera shake correction processing device 10, and displays a moving image.

  The details of the camera shake correction processing apparatus 10 are as follows. In the following, as shown in FIG. 2, the n-th image frame among the plurality of image frames constituting the moving image is represented as F (n), and the image frame F (n−1) to the image frame F ( The period up to n) is represented as T (n). Pixel data is output from the imaging unit 30 at regular time intervals in the order of image frames F (n-2), F (n-1), F (n), F (n + 1),. In the following, a case where the camera shake correction process is performed on the pixel data of the nth image frame F (n) will be described.

  The motion vector detection unit 11 receives the pixel data of the first area in the image frame F (n) input during the initial predetermined period when the pixel data of the image frame F (n) is input, and the image frame F (n− 1) The motion vector of the image in the period T (n) is detected based on the information on the feature points in the first area.

  The feature point data memory 14 stores data relating to feature points in the image frame F (n) extracted when the motion vector detecting unit 11 detects a motion vector of the image. The search data memory 15 stores pixel data of pixel blocks including feature points of the first area in the image frame F (n) used when the motion vector detection unit 11 detects a motion vector of the image.

  The motion vector detection unit 11 includes data relating to feature points in the image frame F (n−1) stored in the feature point data memory 14 and the first in the image frame F (n) stored in the search data memory 15. Based on the pixel data of the area, the motion vector of the image in the period T (n) is detected.

  In addition to the feature points of the first area in the image frame F (n) input in the initial predetermined period when the pixel data of each image frame F (n) is input, the motion vector detection unit 11 The motion vector of the image may be detected using the feature points in the second area other than the first area.

  The correction amount calculation unit 12 calculates a cutout position for the image frame F (n) based on the motion vector of the image detected by the motion vector detection unit 11. Then, the image cutout processing unit 13 cuts out the pixel data of the partial range in the image frame F (n) output from the image signal processing unit 20 based on the cutout position calculated by the correction amount calculation unit 12, The extracted pixel data is output.

  FIG. 3 is a diagram illustrating the first area and the second area in the image frame F (n). It is assumed that the pixel data of each image frame F (n) is output from the imaging unit 30 in the raster scan order, and the pixel data of the image frame F (n) is input to the camera shake correction processing device 10. That is, in the figure, pixel data of each row is input to the camera shake correction processing device 10 in order from the top of the image frame F (n), and pixel data is input to the camera shake correction processing device 10 in order from the left in each row. To do. At this time, the upper area in the image frame F (n) can be the first area F1 (n), and the central area in the image frame F (n) is the second area F2 (n). be able to.

  FIG. 4 is a diagram illustrating the input timing and output timing of the image frame F (n) in the image signal processing unit 20. The image processing performed by the image signal processing unit 20 includes filter processing as typified by noise removal processing, and is performed by internally holding several rows of pixel data in the image frame F (n). Therefore, the output timing of the image frame F (n) from the image signal processing unit 20 has a delay of several lines with respect to the input timing of the image frame F (n) to the image signal processing unit 20.

  Therefore, the correction amount calculation unit 12 calculates the cutout position for the image frame F (n) until the image signal processing unit 20 outputs the pixel data after image processing for the image frame F (n). Is preferred. Then, the image cutout processing unit 13 cuts out pixel data of a partial range in the image frame F (n) output from the image signal processing unit based on the cutout position calculated by the correction amount calculation unit 12, It is preferable to output the pixel data after the cutout.

  Next, the feature point extraction method in this embodiment will be described. In order to detect a motion vector of an image during a period T (n) from the image frame F (n−1) to the image frame F (n), the feature point of the image can be used. By using the feature points of the image, the amount of calculation when obtaining the motion vector can be reduced, the required memory capacity can be reduced, and the motion vector of the image can be detected with high accuracy. it can.

  For example, edge portions and corner portions in the image are extracted as feature points. Feature points can be extracted by differentiating pixel data between adjacent pixels in the image or by evaluation by a Harris operator. Further, the differential value or the Harris operator evaluation value can be used as the feature amount of the feature point. The Harris operator is a technique for extracting feature points based on the correlation between pixel data in the image, and uses the fact that the correlation output value becomes large at the feature points such as edges and corners in the image. Thus, feature points can be extracted.

  The present embodiment is basically based on extracting a plurality of feature points in the first area F1 (n) in the image frame F (n). A plurality of feature points may be extracted also in the second area F2 (n) in the image frame F (n). The size of the first area F1 (n) in the image frame F (n) and the number of extracted feature points are determined from the input timing of the image frame F (n) in the image signal processing unit 20 shown in FIG. It is preferable that the delay time is set so as to correspond to the delay time up to. The size of the first area F1 (n) in the image frame F (n) can be, for example, 10 lines.

  When the number of feature points extracted is too small, the accuracy of the motion vector of the image calculated based on the motion vector of the feature point is lowered. On the other hand, if the number of feature points extracted is too large, the required capacity of each of the feature point data memory 14 and the search data memory 15 increases. From these viewpoints, the number of extracted feature points is set to an appropriate number. For example, the number of feature points extracted in the first area F1 (n) is 10, and the number of feature points extracted in the second area F2 (n) is 50. A feature point is extracted by selecting a feature having a large feature amount. In addition, in each of the first area F1 (n) and the second area F2 (n), it is preferable that the selected feature points are not biased to the local region. For this purpose, each area is further subdivided, Preferably, one feature point is selected in each segmented area.

  FIG. 5 is a diagram for explaining a feature point extraction method according to the present embodiment. In the figure, in addition to the image frame F (n) including the first area F1 (n) and the second area F2 (n), the pixel block Bp including the feature point P in the image frame F (n-1). A pixel block Bs (n) including a position corresponding to the coordinate value of the feature point P in the image frame F (n) is also shown. The pixel block Bp (n−1) is a block of, for example, 5 × 5 pixels centered on the feature point P, and the pixel data of the pixel block Bp (n−1) is held by the feature point data memory 14. The pixel block Bs (n) is, for example, a block of horizontal 13 pixels × vertical 9 pixels centered on the corresponding position of the feature point P, and the pixel data of the pixel block Bs (n) is held by the search data memory 15.

  The motion vector detection unit 11 detects the pixel data of the pixel block Bp (n-1) including the feature point P in the image frame F (n-1) and the corresponding position of the feature point P in the image frame F (n). The motion vector of the feature point P is detected based on the pixel data of the pixel block Bs (n) including Specifically, the motion vector detection unit 11 uses a block matching method that is a generally known method for evaluating similarity between images. That is, the motion vector detection unit 11 calculates the difference between pixel data at positions corresponding to each other from the pixel block Bs (n) obtained by moving the pixel block Bp (n−1) by the vector (Vx, Vy). The absolute value sum SAD (Sum of Absolute Difference) is obtained as an evaluation value of the similarity between the two. Then, the motion vector detection unit 11 sets Vx and Vy to various values for each pixel to obtain an absolute value sum SAD, and features a vector (Vx, Vy) when the absolute value sum SAD is minimized. The motion vector of the point P is assumed.

  FIG. 6 is a block diagram illustrating a configuration of the motion vector detection unit 11 of the camera shake correction processing apparatus 10 according to the present embodiment. The motion vector detection unit 11 includes a first extraction unit 51, a first storage unit 52, a first detection unit 53, and a first calculation unit 54, which process pixel data of the first area F1 (n). Preferably, the motion vector detection unit 11 further includes a second extraction unit 61, a second storage unit 62, a second detection unit 63, and a second calculation unit 64, whereby the pixel data of the second area F2 (n) Perform the process.

  FIG. 6 shows functional blocks of the vector detection unit 11. Since the pixel data of the second area F2 (n) is processed after the pixel data of the first area F1 (n) of each image frame F (n), the first extractor 51 and the second extractor 61 are processed. May be realized by a common circuit, the first detection unit 53 and the second detection unit 63 may be realized by a common circuit, and the first calculation unit 54 and the second calculation unit 64 are It may be realized by a common circuit.

  The first extraction unit 51 extracts a plurality of feature points in the first area F1 (n) in each image frame F (n). The first storage unit 52 stores the coordinate value of each feature point extracted by the first extraction unit 51 and the pixel data of the pixel block including the feature point in the feature point data memory 14 in association with each other. Preferably, the first storage unit 52 also stores the feature amount of each feature point extracted by the first extraction unit 52 in the feature point data memory 14.

  The first detection unit 53 includes the coordinate value of each feature point of the first area F1 (n-1) in the image frame F (n-1) stored in the feature point data memory 14 by the first storage unit 52 and the coordinate value. Pixel data of a pixel block including a feature point, and pixel data of a pixel block Bp (n-1) including a feature point of the first area F1 (n-1) in the image frame F (n-1); Based on the pixel data of the pixel block Bs (n) including the position corresponding to the coordinate value of the feature point in the first area F1 (n) in the image frame F (n), the first in the period T (n). A motion vector of each feature point in the area is detected.

  The first calculator 54 calculates a motion vector of the image in the period T (n) based on the motion vector of each feature point in the first area in the period T (n) detected by the first detector 53. Preferably, the first calculation unit 54 weights the motion vectors of the feature points detected by the first detection unit 51 with the feature amounts of the feature points stored by the first storage unit 52. Then, the motion vector of the image in the period T (n) is calculated.

  The motion vector (V1x, V1y) of the image in the period T (n) calculated by the first calculator 54 is expressed by the following equation (1). Here, V1x (i) is the horizontal component of the motion vector at the feature point i in the first area, V1y (i) is the vertical component of the motion vector at the feature point i in the first area, and W1 (i) is This is the feature amount of the feature point i in the first area.

  The second extraction unit 61 extracts a plurality of feature points in the second area F2 (n) in each image frame F (n). The second storage unit 62 stores the coordinate values of the feature points extracted by the second extraction unit 61 in association with the pixel data of the pixel block including the feature points in the feature point data memory 14. Preferably, the second storage unit 62 also stores the feature amount of each feature point extracted by the second extraction unit 61 in the feature point data memory 14.

  The second detection unit 63 includes the coordinate values of the feature points of the second area F2 (n-1) in the image frame F (n-1) stored in the feature point data memory 14 by the second storage unit 62 and the coordinate values. Pixel data of a pixel block including a feature point, and pixel data of a pixel block Bp (n-1) including a feature point of the second area F2 (n-1) in the image frame F (n-1); Based on the pixel data of the pixel block Bs (n) including the position corresponding to the coordinate value of the feature point in the second area F2 (n) in the image frame F (n), the second area T (n). A motion vector of each feature point in the area is detected.

  The second calculator 64 calculates the motion vector of the image in the period T (n) based on the motion vector of each feature point in the second area in the period T (n) detected by the second detector 63. Preferably, the second calculation unit 64 weights the motion vectors of the feature points detected by the second detection unit 61 with the feature amounts of the feature points stored by the second storage unit 62. Then, the motion vector of the image in the period T (n) is calculated.

  The motion vector (V2x, V2y) of the image in the period T (n) calculated by the second calculator 64 is expressed by the following equation (2). Here, V2x (i) is the horizontal component of the motion vector at the feature point i in the second area, V2y (i) is the vertical component of the motion vector at the feature point i in the second area, and W2 (i) is This is the feature amount of the feature point i in the second area.

  The correction amount calculation unit 12 calculates the cutout position for the image frame F (n) based on the motion vector (V1x, V1y) of the image in the period T (n) calculated by the first calculation unit 54. Preferably, the correction amount calculation unit 12 includes the motion vector (V1x, V1y) of the image in the period T (n) calculated by the first calculation unit 54 and the period T (n calculated by the second calculation unit 64. Based on the motion vector (V2x, V2y) of the image of (-1), the cut-out position for the image frame F (n) is calculated. More preferably, the correction amount calculation unit 12 includes the motion vector (V1x, V1y) of the image in the period T (n) calculated by the first calculation unit 54 and the period T ( The image motion vector (Mx, My) is obtained by weighting the image motion vector (V2x, V2y) of n-1), and the cutout position for the image frame F (n) is calculated based on this. To do.

  The motion vector (Mx, My) of the image calculated by the correction amount calculation unit 12 is expressed by the following equation (3). Here, ΣW1 (i) and W2 are weighting coefficients. The weighting coefficient W2 may be a fixed value or may be set from the outside. The weighting coefficient W2 may be kΣW2 (i), and the coefficient k may be set from the outside. A fixed value may be used instead of ΣW1 (i).

  The expression (3) represents as follows. In the camera shake correction process according to the present embodiment, the motion vector of the image is basically obtained based on the motion vector of the feature point of the first area in the image frame. However, an accurate motion vector may not be detected only in the first area depending on the image pattern of the image frame. A typical example is a case where the pattern of the first area (for example, the upper part of the screen) is extremely flat and a feature point having a feature is not detected in the first area. If block matching processing is performed on such an image, there is a high possibility that matching will be performed on random noise components rather than image features, and motion vectors inappropriate for the purpose of camera shake correction processing will be obtained. You might get lost.

  In such a case, from the motion vector (V1x, V1y) of the image of the period T (n) calculated by the first calculation unit 54, the time is calculated by the second calculation unit 64 although it is past by one frame. The motion vector (V2x, V2y) of the image during the period T (n-1) is more likely to take a value close to the true motion vector in terms of probability. Therefore, in the present embodiment, weighting is performed with the feature amount with the intention of taking into account the value of the motion vector detected in the second area F2 (n-1) of the image frame F (n-1).

  According to the above equation (3), when the pattern of the first area is flat, the feature amount W1 (i) in the above equation takes a small value and the first area F1 (n) of the image frame F (n). The weight of the motion vector detected in the second area F2 (n-1) of the image frame F (n-1) is relatively larger than the weight of the motion vector detected in step 1. The weighting factor W2 may be determined appropriately on an actual system.

  Next, the operation of the camera shake correction processing apparatus 10 of the present embodiment will be described, and the camera shake correction processing method of the present embodiment will be described. FIG. 7 is a flowchart for explaining the camera shake correction processing method of the present embodiment. Here, a process performed when the pixel data of the image frame F (n) is input to the camera shake correction processing apparatus 10 will be described. The camera shake correction processing method of this embodiment includes a motion vector detection step including steps S10 to S19, a correction amount calculation step S20, and an image cutout processing step S30.

  In step S10, the pixel data of the image frame F (n) output from the imaging unit 30 is input to the motion vector detection unit 11 in raster scan order. The pixel data of the first area F1 (n) input to the motion vector detection unit 11 in the initial predetermined period of the input period of the image frame F (n) is sent to the first extraction unit 51 and the first detection unit 53. As a result, the processes of steps S11 to S14 are performed.

  In the first extraction step S11, the first extraction unit 51 extracts a plurality of feature points in the first area F1 (n) in the image frame F (n). In the first storage step S12, the coordinate value of each feature point extracted in the first extraction step S11 by the first storage unit 52, the pixel data of the pixel block including the feature point, and the feature amount of the feature point correspond to each other. Attached and stored in the feature point data memory 14.

  In the first detection step S13, each of the first areas F1 (n−1) in the image frame F (n−1) stored in the feature point data memory 14 in the first storage step S12 by the first detection unit 53. The coordinate value of the feature point and the pixel data of the pixel block including the feature point are read out, and the pixel block Bp (including the feature point of the first area F1 (n-1) in the image frame F (n-1) is read. based on the pixel data of (n-1) and the pixel data of the pixel block Bs (n) including the position corresponding to the coordinate value of the feature point in the first area F1 (n) in the image frame F (n). The motion vector of each feature point in the first area in the period T (n) is detected.

  In the first calculation step S14, each of the first areas F1 (n-1) in the image frame F (n-1) stored in the feature point data memory 14 in the first storage step S12 by the first calculation unit 54. The feature amount of the feature point is read out, the motion vector of each feature point detected in the first detection step S13 is weighted with the feature amount of each feature point, and the motion of the image in the period T (n) A vector (V1x, V1y) is calculated (the above equation (1)).

  The pixel data of the second area F2 (n) input to the motion vector detection unit 11 after the pixel data of the first area F1 (n) of the image frame F (n) is the second extraction unit 61 and the second detection unit. 63, the processes of steps S16 to S19 are performed.

  In the second extraction step S16, the second extraction unit 61 extracts a plurality of feature points in the second area F2 (n) in the image frame F (n). In the second storage step S17, the coordinate value of each feature point extracted in the second extraction step S16 by the second storage unit 62, the pixel data of the pixel block including the feature point, and the feature amount of the feature point correspond to each other. Attached and stored in the feature point data memory 14.

  In the second detection step S18, each of the second areas F2 (n-1) in the image frame F (n-1) stored in the feature point data memory 14 in the second storage step S17 by the second detection unit 63. The coordinate value of the feature point and the pixel data of the pixel block including the feature point are read, and the pixel block Bp () including the feature point of the second area F2 (n-1) in the image frame F (n-1) is read. n-1) pixel data and pixel data of the pixel block Bs (n) including the position corresponding to the coordinate value of the feature point in the second area F2 (n) in the image frame F (n). The motion vector of each feature point in the second area in the period T (n) is detected.

  In the second calculation step S19, each of the second areas F2 (n-1) in the image frame F (n-1) stored in the feature point data memory 14 in the second storage step S17 by the second calculation unit 64. The feature amount of the feature point is read out, the motion vector of each feature point detected in the second detection step S18 is weighted with the feature amount of each feature point, and the motion of the image in the period T (n) A vector (V2x, V2y) is calculated (the above equation (2)).

  The first extraction step S11 and the first storage step S12, and the first detection step S13 and the first calculation step S14 can be performed in parallel. Further, the second extraction step S16 and the second storage step S17, and the second detection step S18 and the second calculation step S19 can be performed in parallel. After the first calculation step S14, a correction amount calculation step S20 is performed.

  In the correction amount calculation step S20, the correction amount calculation unit 12 calculates the motion vector (V1x, V1y) of the image in the period T (n) calculated in the first calculation step S14 and the period calculated in the second calculation step S19. The motion vector (Mx, My) of the image is obtained by weighting the motion vector (V2x, V2y) of the image of T (n-1) (the above equation (3)), and the image is based on this. A cut-out position for the frame F (n) is calculated.

  In the image cutout processing step S30, the image cutout processing unit 13 outputs an image frame that has been subjected to image processing in the image signal processing step by the image signal processing unit 20 based on the cutout position calculated in the correction amount calculation step S20. Pixel data in a partial range in F (n) is cut out, and the pixel data after the cut out is output.

  Then, pixel data of each image frame that has been subjected to various types of image processing by the image signal processing unit 20 and camera shake correction processing by the camera shake correction processing device 10 is input to the display unit 40, and a moving image with reduced influence of camera shake is displayed. Displayed by the unit 40.

  In the present embodiment, the feature points in the image frame F (n) extracted by the first extraction unit 51 and the second extraction unit 61 of the motion vector detection unit 11 when the feature points are extracted and the camera shake correction process is performed. The data (the coordinate value of the feature point, the feature amount of the feature point, and the pixel data of the pixel block Bp (n) including the feature point) are stored in the feature point data memory 14 and the first detection unit 53 of the motion vector detection unit 11 is used. Since the pixel data of a part of the pixel blocks Bs (n) in the image frame F (n) used when the second detection unit 63 detects the motion vector of the image may be stored in the search data memory 15, the frame It is possible to perform camera shake correction processing without using a memory. Therefore, in this embodiment, it is possible to reduce the cost of the camera shake correction processing apparatus 10 and it is also possible to reduce the cost of the camera system 1 and the image processing system 2.

  In the present embodiment, when the process by the first calculation unit 54 (first calculation step S14) for the first area F1 (n) of the image frame F (n) is completed, the process (correction) by the correction amount calculation unit 12 is completed. Since the amount calculation step S20) is performed, the correction amount calculation unit 12 performs the cut-out position calculation process (correction amount calculation) until the image signal processing unit 20 outputs pixel data after image processing for the image frame F (n). Step S20) can be terminated. Therefore, in the present embodiment, real-time processing required for moving image processing in the camera system 1 is possible.

DESCRIPTION OF SYMBOLS 1 ... Camera system, 2 ... Image processing system, 10 ... Camera shake correction processing apparatus, 11 ... Motion vector detection part, 12 ... Correction amount calculation part, 13 ... Image cut-out processing part, 14 ... Feature point data memory, 15 ... Search data Memory 20, image signal processing unit 30, imaging unit 40, display unit 51, first extraction unit 52, first storage unit 53, first detection unit 54, first calculation unit 61, first 2 extraction unit, 62 ... second storage unit, 63 ... second detection unit, 64 ... second calculation unit.

Claims (16)

An apparatus for correcting the influence of camera shake in a moving image composed of a plurality of image frames,
The pixel data of the first area in the image frame F (n) input in the initial predetermined period when the pixel data of the nth image frame F (n) among the plurality of image frames is input, and the image frame Based on the information on the feature points of the first area in the (n-1) th image frame F (n-1) before F (n), the image frame F (n-1) to the image frame F ( a motion vector detection unit for detecting a motion vector of an image in a period T (n) until n);
A correction amount calculation unit that calculates a cutout position for the image frame F (n) based on the motion vector of the image detected by the motion vector detection unit;
An image cutout processing unit that cuts out pixel data of a partial range in the image frame F (n) based on the cutout position calculated by the correction amount calculation unit, and outputs the pixel data after the cutout;
The coordinate value of the feature point is stored as information on the feature point in the image frame F (n-1), and the pixel data of a part of the pixel block including the feature point in the image frame is associated with the feature point. A feature point data memory for selectively storing
A search data memory for selectively storing pixel data of some pixel blocks including a position corresponding to the coordinate value of the feature point in the image frame F (n);
Bei to give a,
The motion vector detection unit includes information on feature points in the image frame F (n−1) stored in the feature point data memory and a part of the image frame F (n) stored in the search data memory. A motion vector of an image in a period T (n) is detected based on pixel data of the pixel block of
And a camera shake correction processing apparatus. The motion vector detection unit is
A first extraction unit that extracts a plurality of feature points in a first area in each image frame;
A first storage unit that associates the coordinate value of each feature point extracted by the first extraction unit with the pixel data of a pixel block including the feature point and stores the data in the feature point data memory ;
Reading out the coordinate value of each feature point of the first area in the image frame F (n−1) stored in the feature point data memory by the first storage unit and the pixel data of the pixel block including the feature point; Pixel data of a pixel block including a feature point of the first area in the image frame F (n-1), and coordinates of the feature point in the first area in the image frame F (n) stored in the search data memory A first detector that detects a motion vector of each feature point in the first area of the period T (n) based on pixel data of a pixel block including a position corresponding to the value;
A first calculation unit that calculates a motion vector of an image of a period T (n) based on a motion vector of each feature point of the first area of the period T (n) detected by the first detection unit;
Including
The correction amount calculation unit calculates a cut-out position for the image frame F (n) based on the motion vector of the image in the period T (n) calculated by the first calculation unit;
The camera shake correction processing apparatus according to claim 1. The motion vector detection unit is
A second extraction unit for extracting a plurality of feature points in a second area other than the first area in each image frame;
A second storage unit that stores the coordinate value of each feature point extracted by the second extraction unit and the pixel data of the pixel block including the feature point in association with each other in the feature point data memory ;
Reading out the coordinate value of each feature point of the second area in the image frame F (n−1) stored in the feature point data memory by the second storage unit and the pixel data of the pixel block including the feature point; Pixel data of a pixel block including a feature point of the second area in the image frame F (n-1), and coordinates of the feature point in the second area in the image frame F (n) stored in the search data memory A second detection unit that detects a motion vector of each feature point in the second area of the period T (n) based on pixel data of a pixel block including a position corresponding to the value;
A second calculation unit that calculates a motion vector of the image in the period T (n) based on the motion vector of each feature point in the second area in the period T (n) detected by the second detection unit;
Further including
The correction amount calculation unit includes a motion vector of the image for the period T (n) calculated by the first calculation unit, and a motion vector of the image for the period T (n−1) calculated by the second calculation unit. Based on the above, a cutout position for the image frame F (n) is calculated.
The camera shake correction processing apparatus according to claim 2. The first storage unit also stores the feature amount of each feature point extracted by the first extraction unit in the feature point data memory ;
The first calculation unit weights the motion vector of each feature point detected by the first detection unit with the feature amount of each feature point stored in the feature point data memory by the first storage unit. Go to calculate the motion vector of the image of period T (n),
The camera shake correction processing apparatus according to claim 2. The second storage unit also stores the feature amount of each feature point extracted by the second extraction unit in the feature point data memory ;
The second calculation unit weights the motion vectors of the feature points detected by the second detection unit with the feature amounts of the feature points stored in the feature point data memory by the second storage unit. Go to calculate the motion vector of the image of period T (n),
The camera shake correction processing apparatus according to claim 3.   The correction amount calculation unit includes a motion vector of the image for the period T (n) calculated by the first calculation unit, and a motion vector of the image for the period T (n−1) calculated by the second calculation unit. The camera shake correction processing apparatus according to claim 3, wherein weighting is performed to obtain a motion vector of the image, and a cut-out position for the image frame F (n) is calculated. The camera shake correction processing apparatus according to any one of claims 1 to 6, which corrects an influence of camera shake in a moving image including a plurality of image frames.
An image signal processing unit that performs image processing different from the camera shake correction processing by the camera shake correction processing device for the pixel data of each of the plurality of image frames;
With
Outputting pixel data of each image frame subjected to image processing by the image signal processing unit and camera shake correction processing by the camera shake correction processing device;
An image processing system characterized by that. The correction amount calculation unit of the camera shake correction processing apparatus determines a cutout position for the image frame F (n) until the image signal processing unit outputs pixel data after image processing for the image frame F (n). Calculate
Based on the cutout position calculated by the correction amount calculation unit, the image cutout processing unit of the camera shake correction processing device outputs pixels in a partial range in the image frame F (n) output from the image signal processing unit. Cut out the data and output the pixel data after the cut out.
The image processing system according to claim 7. A method for correcting the influence of camera shake in a moving image composed of a plurality of image frames,
The pixel data of the first area in the image frame F (n) input in the initial predetermined period when the pixel data of the nth image frame F (n) among the plurality of image frames is input, and the image frame Based on the information on the feature points of the first area in the (n-1) th image frame F (n-1) before F (n), the image frame F (n-1) to the image frame F ( a motion vector detection step of detecting a motion vector of an image in a period T (n) until n);
A correction amount calculating step for calculating a cutout position for the image frame F (n) based on the motion vector of the image detected in the motion vector detecting step;
An image cutout processing step of cutting out pixel data of a partial range in the image frame F (n) based on the cutout position calculated in the correction amount calculating step, and outputting the pixel data after the cutout;
Bei to give a,
In the motion vector detection step,
The coordinate value of the feature point is stored as information on the feature point in the image frame F (n-1), and the pixel data of a part of the pixel block including the feature point in the image frame is associated with the feature point. A feature point data memory for selectively storing
A search data memory for selectively storing pixel data of some pixel blocks including a position corresponding to the coordinate value of the feature point in the image frame F (n);
Using,
Information on feature points in the image frame F (n−1) stored in the feature point data memory, pixel data of some pixel blocks in the image frame F (n) stored in the search data memory, and To detect a motion vector of an image in a period T (n) based on
And a camera shake correction processing method. The motion vector detection step comprises:
A first extraction step of extracting a plurality of feature points in a first area in each image frame;
A first storage step of storing the coordinate values of each feature point extracted in the first extraction step and the pixel data of the pixel block including the feature point in association with each other in the feature point data memory ;
Reading out the coordinate value of each feature point of the first area in the image frame F (n-1) saved in the feature point data memory in the first saving step and the pixel data of the pixel block including the feature point; Pixel data of a pixel block including a feature point of the first area in the image frame F (n-1), and coordinates of the feature point in the first area in the image frame F (n) stored in the search data memory A first detection step of detecting a motion vector of each feature point in the first area of the period T (n) based on pixel data of a pixel block including a position corresponding to the value;
A first calculation step of calculating a motion vector of an image in the period T (n) based on a motion vector of each feature point in the first area in the period T (n) detected in the first detection step;
Including
In the correction amount calculating step, a cut-out position for the image frame F (n) is calculated based on the motion vector of the image in the period T (n) calculated in the first calculating step.
The camera shake correction processing method according to claim 9 . The motion vector detection step comprises:
A second extraction step of extracting a plurality of feature points in a second area other than the first area in each image frame;
A second storage step of associating the coordinate value of each feature point extracted in the second extraction step with the pixel data of the pixel block including the feature point in the feature point data memory ;
Reading out the coordinate value of each feature point of the second area in the image frame F (n-1) saved in the feature point data memory in the second saving step and the pixel data of the pixel block including the feature point; Pixel data of a pixel block including a feature point of the second area in the image frame F (n-1), and coordinates of the feature point in the second area in the image frame F (n) stored in the search data memory A second detection step of detecting a motion vector of each feature point in the second area of the period T (n) based on pixel data of a pixel block including a position corresponding to the value;
A second calculation step of calculating a motion vector of the image in the period T (n) based on the motion vector of each feature point in the second area in the period T (n) detected in the second detection step;
Further including
In the correction amount calculating step, the motion vector of the image in the period T (n) calculated in the first calculation step, and the motion vector of the image in the period T (n−1) calculated in the second calculation step Based on the above, a cutout position for the image frame F (n) is calculated.
The camera shake correction processing method according to claim 10. In the first storage step, the feature amount of each feature point extracted in the first extraction step is also stored in the feature point data memory ,
In the first calculation step, the motion vector of each feature point detected in the first detection step is weighted with the feature amount of each feature point stored in the feature point data memory in the first storage step. Go to calculate the motion vector of the image of period T (n),
The camera shake correction processing method according to claim 10. In the second storage step, the feature amount of each feature point extracted in the second extraction step is also stored in the feature point data memory ,
In the second calculation step, the motion vector of each feature point detected in the second detection step is weighted with the feature amount of each feature point stored in the feature point data memory in the second storage step. Go to calculate the motion vector of the image of period T (n),
The camera shake correction processing method according to claim 11.   In the correction amount calculating step, the motion vector of the image in the period T (n) calculated in the first calculation step, and the motion vector of the image in the period T (n−1) calculated in the second calculation step The camera shake correction processing method according to claim 11, wherein weighting is performed to obtain a motion vector of the image, and a cut-out position for the image frame F (n) is calculated. The camera shake correction processing method according to any one of claims 9 to 14, wherein an effect of camera shake in a moving image including a plurality of image frames is corrected.
An image signal processing step for performing image processing different from the camera shake correction processing by the camera shake correction processing method for the pixel data of each of the plurality of image frames;
With
Outputting pixel data of each image frame that has undergone image processing by the image signal processing step and image stabilization processing by the image stabilization processing method;
An image processing method. In the correction amount calculation step of the camera shake correction processing method, the cut-out position for the image frame F (n) is determined until the pixel data after image processing is output for the image frame F (n) in the image signal processing step. Calculate
In the image cutout processing step of the camera shake correction processing method, pixels in a partial range in the image frame F (n) output from the image signal processing step based on the cutout position calculated in the correction amount calculation step Cut out the data and output the pixel data after the cut out.
The image processing method according to claim 15.