TW201523516A - Video frame stabilization method for the moving camera - Google Patents

Abstract

The present invention discloses a video frame stabilization method for a moving camera. The method includes the following steps: detecting corner points (also called feature points) in the frame and calculating an optical flow of each corner point; transferring the optical flow to the polar coordinate for analysis and then deriving a global motion vector; deducing a rotation angle by the corner points; and finally the global motion vector and the rotation angle are utilized to compensate the vibrant shift image for stabilizing the video frames.

Description

Mobile camera stabilization method

The present invention relates to a picture stabilization method, and more particularly to a picture stabilization method for a video image taken by a camera in motion using a software design method.

At the beginning of the picture stabilization method, it is mostly used in the fixed photographic monitoring system to avoid the situation that the external factors such as wind blowing and ground motion cause the picture to shake. The common method of image stabilization is through mechanical devices, electronic sensing devices or optical sensations. With the aid of additional components such as measuring devices, the required cost and space required for hardware are high. In recent years, related products of hand-held photographic devices or mobile photographic devices have been continuously developed, and the quality of image display and its practicality for movement have been developed. There are higher requirements, so the picture stabilization method is more oriented towards the software design to achieve the effect of picture stabilization.

The technology of video image stabilization by software design focuses on the estimation accuracy of the moving direction between screens. Most of the conventional techniques are based on block matching, with different blocks in the picture. Combined configuration as an estimate of the global motion vector. However, if the detection block contains other moving objects, the moving vector of the moving object may cause interference to the estimated motion vector of the block, thereby estimating the error, thereby causing the screen displacement compensation to be misaligned, and finally causing the image to be stable. The effect.

Therefore, the present invention provides a picture stabilization method for a mobile camera, which does not need to increase the additional space and cost of the hardware, and at the same time improves the shortcomings of the software design mode, improves the estimation accuracy of the global motion vector, and thereby increases the image stabilization effect. .

In view of the above-mentioned problems of the prior art, the object of the present invention is to provide a picture stabilization method for a mobile camera, so as to solve the conventional image stabilization compensation using a software design, the global motion vector is susceptible to interference from moving objects and affects the final picture. The result of the problem, thereby improving the stability of the mobile camera screen.

According to an aspect of the present invention, a method for image stabilization of a mobile camera is provided, which comprises the steps of: capturing a picture by a mobile camera; performing a pre-processing on the picture via the pre-processing module and detecting a corner of the picture (corner) Points), also known as feature points, use the moving vector of the coordinate change of the corners in the front and rear pictures as the optical flow of the corner points; use the displacement stability calculation module to calculate the frame shift change The amount of light is converted into a polar coordinate, and the data density of the polar coordinates is clustered, and the background light flow is generated after filtering the noise, and is used as a screen displacement variation; the rotation stability calculation module is used to calculate the image. The rotation angle change amount is obtained by taking the angle between each successive two corner points, comparing the change of the front and rear screen angles as the rotation angle, the statistical rotation angle, selecting the maximum cumulative rotation angle as the screen rotation angle change amount, and the transmission compensation ( Compensation) The output module corrects the displacement and rotation angle of the screen, and the amount of change in the rotation angle of the screen The screen is rotated in the reverse direction and reversed with the screen displacement vector. After the compensation process, the stable image result is output.

Preferably, the polar coordinates are composed of information on the optical flow length and the optical flow angle of the motion vector of the optical flow.

Preferably, the displacement stability calculation module groups the similarities of the optical flow length and the optical flow angle.

Preferably, the pre-processing may further include down-sampling to reduce the resolution and amount of information of the picture.

Preferably, the compensation process may further comprise a super-sample to recover the resolution of the picture.

Preferably, the background light stream can be smoothed by the background light stream to obtain displacement information with less variation.

Preferably, the rotation angle with the highest cumulative number can be smoothed by the rotation angle to obtain the rotation information with less variation.

According to the above, the picture stabilization method of the mobile camera according to the present invention may have one or more of the following advantages:

(1) The image stabilization method of the mobile camera does not need to be equipped with an additional sensing device, and conforms to the design direction of the camera, so that it is easy to carry and move, and the convenience and practicability thereof are increased.

(2) The image stabilization method of the mobile camera converts the feature point motion vector into polar coordinate analysis, eliminates the influence noise, and avoids the compensation misalignment caused by the moving object interference, thereby enhancing the effect of the image stabilization.

S1~S5, S21~S53‧‧‧ steps
‧‧‧Previous image screen
‧‧‧ Current image
‧‧‧corner
‧‧‧Rotation angle
‧‧‧ corner angle
‧‧‧Global rotation angle

Fig. 1 is a flow chart showing a method for stabilizing a picture of a mobile camera of the present invention.

Fig. 2 is a schematic diagram showing the optical flow distribution of the picture stabilization method of the mobile camera of the present invention.

Fig. 3 is a schematic diagram showing the polar coordinate conversion of the picture stabilization method of the mobile camera of the present invention.

Fig. 4 is a schematic diagram showing the polar coordinates of the picture stabilization method of the mobile camera of the present invention.

Fig. 5 is a schematic diagram showing the calculation method of the rotation angle of the picture stabilization method of the mobile camera of the present invention.

Fig. 6 is a histogram of the rotation angle statistics of the picture stabilization method of the mobile camera of the present invention.

The technical features, contents, advantages and advantages of the present invention will be understood by the reviewing committee, and the present invention will be described in detail with reference to the accompanying drawings. The subject matter is only for the purpose of illustration and description. It is not intended to be a true proportion and precise configuration after the implementation of the present invention. Therefore, the scope and configuration relationship of the attached drawings should not be interpreted or limited. First described.

Please refer to FIG. 1 , which is a flow chart of a method for stabilizing a mobile camera of the present invention. The picture stabilization method of the mobile camera includes the following processing steps: S1: picture input, S2: pre-processing, S3: displacement stability calculation processing, S4: rotation stability calculation, and S5: compensation output. The screen input S1 of the screen stabilization method of the mobile camera mainly uses the image captured by the moving camera as the input screen, and obtains the corner point (also referred to as "feature point") of the screen after the pre-processing S2 and The optical flow (also referred to as "moving vector"), and then the displacement stability calculation S3 and the rotation angle calculation S4 are performed on the processed information, and the screen is compensated and outputted for the calculated displacement change amount and the rotation change amount after the calculation. S5, to obtain a stable image frame result, which will be described in detail later for the main processing steps S2 to S5.

Process step S2 of the image stabilization method of the mobile camera: The pre-processing includes the following detailed steps: S21: downsampling, S22: detecting corner points, and S23: optical flow method analysis. Among them, the picture obtained by the mobile camera often has high resolution when inputting, but when processing and analyzing the picture data, it is not necessary to use so much image information quantity, but a larger amount of image information will cause calculation. The time is wasted. Therefore, when inputting the picture, the down-sample S21 is first performed by bilinear interpolation. On the one hand, the input picture is normalized and the amount of picture information is reduced. To increase the efficiency of picture processing. Furthermore, due to the analysis of the motion vector of the picture block in the past, the position of the non-corner point often deviates from the estimated value due to the aperture problem, and in order to reduce the calculation of a large number of pixel points, the present invention utilizes Harris corners detection method S22, find the corner points of the main feature points in the picture, as the main object of subsequent analysis and calculation. Finally, the detected angular point is subjected to optical flow analysis S23 to calculate the previous image frame. With the current image The motion vector of each corner is used as the optical flow of each corner to obtain the optical flow estimation value of the overall picture. As shown in Fig. 2, it is the distribution of the optical flow on the plane coordinates in the picture. Because of the previous image With the current image The position of each corner point corresponds to each other, so the calculated optical flow estimation value should have high accuracy, but since some information may be interfered with the wrong optical flow estimation value, it is necessary to further filter and filter the abnormal information. .

To obtain the correct optical flow estimation value, the optical flow information is input into the processing step S3: displacement stability calculation, which includes the following detailed steps: S31: optical current coordinate conversion, S32: density grouping, and S33: background light flow. Smoothing. Among them, the optical current coordinate conversion S31 is a reference Hough transformation principle, which converts the information that could not be detected in the coordinates of the XY plane, converts it to the polar coordinate analysis statistics, and finally converts the statistical result back to the XY plane coordinates, and draws Finally, the detected result is used to divide the similar optical flows into the same group, and the large differences are divided into different groups, for example, an optical flow vector is defined as to , which can be converted to polar coordinates by conversion according to the following formula (1) and formula (2) ,among them Represents the length of the optical flow, Represents the angle of light flow.

(1)

(2)

Referring to Figures 2 to 4, as shown in Fig. 2, the optical flow information of each corner point in the picture can be converted into a polar coordinate diagram as shown in Fig. 3 by the above formula, wherein the polar coordinate central cross scale is optical flow. The length and the outer ring scale are the optical flow angles. After the optical flow information of the original XY plane coordinates is converted into polar coordinates, it helps to filter out the interference noise and filter out the optical flow of the previously estimated error. The method is to use the DBSCAN algorithm to perform density grouping S32 on the optical information of the polar coordinates. As shown in Fig. 3, the optical flow on the polar coordinates is distributed at different lengths and angles, and the similarity between the optical flow length and the optical flow angle is attributed to the higher degree. In the same group, the optical flow information of a single or a small number is eliminated, and the polar coordinate grouping diagram as shown in FIG. 4 is obtained, and only the main group of the optical flow leaves only two groups. After grouping, the distribution of each group of optical flows in the XY plane coordinates is calculated, and the group with the widest distribution range is set as the background optical stream, which may also be called a global motion vector, and the background actual The motion vector is consistent. Since the distribution curve of the optical flow obtained above is tortuous and the amount of change is large, in order to achieve the effect of stabilizing the background of the picture, after obtaining the background optical flow, it must be brought into the Kalman filter to smooth the background optical flow. S33, to obtain a relatively smooth and stable change in the screen displacement.

Referring to Figures 5 and 6, after calculating the amount of change in the screen displacement, the screen of the mobile camera may have a change in the rotation of the screen during the movement. Therefore, it is necessary to discuss the change of the rotation angle, which is mainly based on the movement of the camera. The processing step S4 in the stabilization method is processed by the rotation stability calculation, which includes the following detailed steps: S41: rotation angle calculation, S42: statistical global rotation angle, and S43: rotation angle smoothing. Among them, the rotation angle calculation S41 is also for the previous image frame. With the current image The corner point is calculated, as shown in Figure 5, the previous image Corner point is , current image Corresponding corner point is If the number of corner points between the two screens is ,then ,E.g and ,among them and An angle can be obtained for each successive two points ,E.g and Corner angle , and You can find the angle of the corner point and then the difference between the angles of the two corner points. Get the two pairs of feature point rotation angles , and so on, you can find the rotation angle of all feature points , as in formula (3), where versus Separately The values on the X and Y coordinates, versus Separately The values on the X and Y coordinates, versus Then it can be analogized.

(3)

From the above formula, the angle change of each two adjacent corner points can be obtained, and all rotation angles are counted. The histogram as shown in Fig. 6 can be obtained, wherein the horizontal axis is the rotation angle, the vertical axis is the cumulative number of the rotation angles, and the cumulative number of the most selected rotation angles is selected as the global rotation angle. S42, that is, the angle at which the picture is rotated. Finally, since the distribution curve of the original rotation angle changes drastically, the rotation angle smoothing S43 is also required to obtain a relatively smooth and stable change in the angle of rotation of the screen.

Obtaining the amount of change in the rotation angle of the screen during the rotation stability calculation S4, and obtaining the amount of change in the screen displacement during the displacement stability calculation S3, and bringing it into the screen stabilization method of the mobile camera, the processing step S5: compensating the output to generate the processing The stable picture, wherein the compensation output S5 includes: S51: reverse rotation compensation, S52: reverse displacement compensation, and S53: output image result. The reverse rotation compensation S51 is used to inversely rotate the screen by the affine transformation to compensate for the original screen rotation. Next, a reverse displacement compensation S52 is performed, which is inversely moved to compensate for the amount of change in the screen displacement, and the global motion vector must be super-sampled to restore the resolution of the original picture. Finally, after the processing of the reverse rotation compensation of the picture and the compensation of the reverse displacement of the picture, a stable output picture is obtained by the mobile camera outputting the image result S53.

The above-mentioned mobile camera picture stabilization method is mainly used for digital cameras, digital cameras, smart phones, car cameras, hand-held camera image recognition systems, blind visual aid systems, unmanned vehicle vision systems, military and police vehicle images ( On the moving camera, such as the identification, search system, and the unmanned reconnaissance camera system, the above device is stabilized by the displacement and rotation angle analysis of the corner points in the image captured by the camera, and the original image is stabilized by reverse compensation. It can continuously display a stable picture for users to view and use.

The above is intended to be illustrative only and not limiting. Any equivalent modifications or alterations to the spirit and scope of the invention are intended to be included in the scope of the appended claims.

S1~S5, S21~S53‧‧‧ steps

Claims (7)

A method for image stabilization of a mobile camera, comprising the following steps:
Taking a picture by taking a video from a mobile camera;
Performing a pre-processing on the picture through a pre-processing module, and detecting a corner point of the picture, and using the corner point to move the vector in one of the coordinates of the front and rear pictures as one of the corner points;
Calculating a displacement variation of a picture by using a displacement stability calculation module, which converts the optical flow of the corner point into a polar coordinate, grouping the data density of the polar coordinate, filtering out the noise to generate a background optical flow, and It is used as the amount of change in the screen displacement;
The rotation stability calculation module is used to calculate the amount of change of the rotation angle of a picture, and the angle is obtained by each of the two consecutive corner points, and the change of the angle of the front and rear pictures is compared as a rotation angle, and the rotation angle is counted, and the cumulative quantity is selected to be the largest. The rotation angle is used as the amount of change of the rotation angle of the screen; and the displacement and the rotation angle of the screen are corrected by a compensation output module, and the screen is rotated in the reverse direction according to the amount of change of the rotation angle of the screen, and is inversely shifted according to the displacement vector of the screen. After a compensation process, the output stabilizes one of the image results. The method for image stabilization of a mobile camera according to claim 1, wherein the polar coordinate system is composed of information of an optical flow length and an optical flow angle of the motion vector of the optical flow. The method for image stabilization of a mobile camera according to claim 2, wherein the displacement stability calculation module groups the optical flow length and the similarity of the optical flow angle. The method for image stabilization of a mobile camera according to claim 1, wherein the pre-processing may further comprise a downsampling to reduce the resolution and the amount of information of the picture. The picture stabilization method of the mobile camera of claim 4, wherein the compensation process further comprises an oversampling to restore the resolution of the picture. The method for image stabilization of a mobile camera according to claim 1, wherein the background optical flow can obtain a displacement information with less variation through a background optical flow smoothing process. The picture stabilization method of the mobile camera according to claim 1, wherein the rotation angle having the highest cumulative number can be smoothed by a rotation angle to obtain rotation information having a small amount of change.