KR101396838B1 - Video stabilization method and system by selection one of various motion models - Google Patents

Abstract

Provided are a method and a system for stabilizing an image by selectively using multiple motion models. A method for stabilizing an image according to an embodiment of the present invention comprises creating stabilization images, of which the shaking cause by a photographing device is corrected, by using multiple motion models, by unit of motion model; comparing a key frame with the stabilization images to select one of the motion models; and outputting an stabilization image by using the selected motion model. Therefore, the distortion which can occur in the signal model estimation can be minimized by selecting the best motion model among motion models in an image with shaking phenomenon due to a camera and correcting the shaking.

Description

TECHNICAL FIELD [0001] The present invention relates to a video stabilization method and system for selectively using a plurality of motion models,

The present invention relates to image processing, and more particularly, to a method and system for stabilizing an image from camera shake.

In the conventional image stabilization method, the camera motion is estimated using the correspondence relation between the current frame and the nearest neighboring frame. However, if the movement of the moving object is not large in this process, there is a possibility that the image stabilization is performed abnormally by being included as the background

In addition, the low-complexity motion model used for image stabilization can not accurately represent the viewpoint distortion or twist of the image, and the high-complexity motion model shows the partially distorted stabilized image if the feature points are not uniformly distributed on the screen.

Accordingly, it is required to search for an optimal stabilization scheme regardless of the content and state of the input image.

SUMMARY OF THE INVENTION It is an object of the present invention to reduce visual sickness by selecting a best motion model among a plurality of motion models for an image in which camera shake occurs, And to provide an image stabilization method and apparatus for compensating for shaking.

According to an aspect of the present invention, there is provided an image stabilization method comprising: generating stabilized images compensated for shaking by a photographing apparatus for each motion model using a plurality of motion models; Comparing the key frames with the stabilized images, and selecting one of the plurality of motion models; And outputting the stabilized image using the motion model selected in the selecting step.

The plurality of motion models may be motion models having different complexities.

The selecting may further include: warping the stabilized images to the key frame; And selecting a motion model that generates a stabilized image having a minimum color difference between the warped stabilized images and the key frame.

The image stabilization method according to an embodiment of the present invention further includes optimizing a plurality of motion models, and the generation step may use the plurality of motion models optimized in the optimization step.

According to another aspect of the present invention, there is provided an image stabilization method comprising: tracking feature points of a background portion in an image; And predicting a feature point trajectory from the feature points of the background portion traced in the tracing step, wherein the optimizing step optimizes the plurality of motion models to the feature point trajectory predicted in the predicting step have.

The tracking step may track feature points extracted from the image corresponding to the feature points, and may remove feature points of the parts determined to be the mobile object.

According to another aspect of the present invention, there is provided a method of stabilizing an image, the method comprising: selecting a frame in which a camera shake is most reflected among past frames as a key frame; The selected key frame can be used.

According to another aspect of the present invention, there is provided an image stabilization apparatus including: an image input unit receiving an image generated by a photographing apparatus; And a processor for generating stabilized images compensated for shaking by the photographing apparatus for each of the motion models using the plurality of motion models and comparing the key frames and the stabilized images to select one of the plurality of motion models, And a processor for outputting a stabilized image using the selected motion model.

As described above, according to the embodiment of the present invention, it is possible to compensate for the shaking by selecting the best motion model among a plurality of motion models for an image in which camera shake occurs, thereby effectively reducing visual sickness in the image . That is, according to the embodiment of the present invention, by using various motion models, it is possible to minimize the distortion that may occur in single model estimation.

Further, by using the feature point locus information of a predetermined length, it is possible to extract reliable feature point correspondence information corresponding to inter-frame feature point correspondence. In addition, the stabilized image of the user's eye is generated through the feature point trajectory prediction.

In addition, since the stabilized images are generated after removing the moving object from the image, it is possible to accurately compensate the camera shake.

In addition, since the frame in which the camera shake is most reflected in the past frames is used as the key frame, the occurrence of distortion in the stabilization process can be minimized.

1 shows a monitoring system to which the present invention is applicable,
FIG. 2 is a detailed block diagram of the image analysis server shown in FIG. 1,
FIG. 3 is a flow chart for explaining an image stabilization method according to a preferred embodiment of the present invention,
4 is a diagram illustrating motion models that can be used in an embodiment of the present invention,
FIG. 5 is a diagram provided for a further description of the feature point trajectory prediction,
6 is a diagram provided in a side view of the motion model optimization process.

Hereinafter, the present invention will be described in detail with reference to the drawings.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary terms, and the inventor should appropriately interpret the concept of the term appropriately in order to describe its own invention in the best way. The present invention should be construed in accordance with the meaning and concept consistent with the technical idea of the present invention.

Therefore, the embodiments described in this specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and do not represent all the technical ideas of the present invention. Therefore, It is to be understood that equivalents and modifications are possible.

1. System to which the present invention is applicable

1 is a diagram showing a monitoring system to which the present invention is applicable. A surveillance system to which the present invention can be applied is a system for photographing an image to perform monitoring, analysis and search.

As shown in FIG. 1, the surveillance system that performs the above functions includes a CCTV camera 110, a control server 120, an image analysis server 130, an image DB 140, a metadata DB 150, And an image search server 160.

The CCTV camera 110 is installed indoors or outdoors, captures an image of the corresponding area and generates an image, and transmits the generated image to the control server 120 and the image analysis server 130.

The control server 120 adds a time stamp to the image received from the CCTV camera 110 and stores the time stamp in the image DB 140. In addition, the control server 120 can display an image received from the CCTV camera 110 in real time so that an administrator can monitor it.

The image analysis server 130 stabilizes the image received from the CCTV camera 110, analyzes the image to generate metadata, and stores the generated metadata in a metadata DB 150 together with a time stamp assigned to the image do. The image analysis server 130 will be described later in detail with reference to FIG.

The meta data is feature information of the object displayed on the image. When the object is a person, the face feature, clothes color, and the like are generated as metadata. When the object is a vehicle, the color, size, and number of the vehicle are generated as metadata.

The image search server 160 searches the meta data DB 150 for the meta data to be searched and searches the image DB 140 for the image to which the timestamp matched to the searched meta data is attached.

2. Image analysis server

Hereinafter, the above-described image analysis server 130 will be described in detail with reference to FIG. 2 is a detailed block diagram of the image analysis server 130 shown in FIG. The image analysis server 130 includes an image input unit 131, a processor 132, a storage unit 137, a DB interface 138, and a server interface 139, as shown in FIG.

The image input unit 131 is an interface and buffering means for receiving an image from the CCTV camera 110 and delivering the image to the processor 132.

2, the processor 132 includes an image stabilization module 133, an object extraction / tracking module 134, a feature extraction module 134, Module 135 and an event extraction module 136. [

The image stabilization module 133 stabilizes the image by compensating for the shake caused by the CCTV camera 110 in the image received through the image input unit 131. [ The image stabilization method will be described later in detail with reference to FIG.

The object extraction / tracking module 134 extracts the object in real time from the stabilized image in the image stabilization module 133, and tracks the extracted object in real time.

The feature extraction module 135 extracts features from extracted / tracked objects and generates extracted features as metadata. The metadata generated by the feature extraction module 135 is stored in the metadata DB 150 via the DB interface 138.

The event extraction module 137 extracts events occurring in the image and generates event information. For example, if a person or a car enters a danger zone or a security zone, it can be treated as an event.

The event information generated by the event extraction module 137 may be transmitted to the control server 120 via the server interface 139 to alert the administrator.

The storage unit 137 provides necessary storage space for the modules 133 to 136 of the processor 132 to perform image processing.

3. Image stabilization method

Hereinafter, the image stabilization process by the image stabilization module 133 will be described in detail with reference to FIG. FIG. 3 is a flowchart illustrating a method of stabilizing an image according to a preferred embodiment of the present invention.

In the image stabilization method according to the present embodiment, the best motion model among various motion models with different complexities is selected to stabilize the image. The concrete procedure is as follows.

As shown in FIG. 3, the feature points of the background portion are extracted and tracked in the input image (S210). Specifically, in step S210, feature points extracted by finding feature correspondences are tracked, and feature points of parts determined as a moving object are removed.

The feature points of the moving object portion can be effectively removed by using the cumulative information of the feature point change, compared with the conventional method using the inter-frame motion change.

Next, in step S220, feature trajectory prediction for a case where there is no shaking is performed from the feature points of the background part extracted / tracked (S220).

Thereafter, various motion models are optimized (Optimization for motion model) most similar to the predicted trajectory in step S220 (S230). In step S230, models such as simple translation, similarity transform, affine transform, and projective transform can be applied.

In addition, a key frame is selected as a key frame in which the camera shake is most reflected in past frames (S240).

In step S230, the stabilized images compensated for the shake are generated for each motion model using the optimized motion models, and the best motion model is selected based on the comparison result with the key frame selected in step S240 (S250).

Specifically, in step S250, a stabilized image is warped to a key frame, and a motion model that generates a stabilized image having a minimum color difference between the warped stabilized images and the key frame is selected.

Thereafter, the stabilized image generated using the motion model selected in step S250 is output as the image stabilization result.

Hereinafter, each of the steps shown in FIG. 2 will be described in detail.

4. Extraction / tracking of feature points of the background part

Most of the conventional image stabilization methods estimate the motion model by obtaining the feature point correspondence with the nearest frame. However, if the motion of the camera is small, there is a disadvantage that the moving object can not be effectively separated.

Therefore, in the embodiment of the present invention, a weighting re-projection error between a frame in a window and a current frame T using a feature point trajectory within a certain time period (window size W: observation frame length) The cumulative distance difference between a minutiae point and an actual minutiae position). If the sum exceeds the threshold value, the mobile device is separated using a method of determining the mobile device.

The motion models used in the present embodiment are shown in FIG. 4. The higher the complexity is, the more various motions of the screen can be expressed. However, if the feature points are not extracted uniformly, more distortion can be shown compared with the low complexity model.

5. Feature point trajectory prediction

In the embodiment of the present invention, the feature point can be a corner or various unfavorable feature amount based feature points, and the feature point tracking can use an optical flow type tracker. The feature point correspondence of the background from which the moving object is removed from the input image can be expressed as follows.

On the other hand, the conventional image stabilization method estimates the motion model through motion smoothing (process of softening the trajectory of the parameters of the motion model) using the inter-frame feature point correspondence. However, since it is necessary to smoothing the entire motion of the image to create a smooth camera motion, it can not be used in applications requiring real-time image stabilization.

Therefore, in the embodiment of the present invention, the motion model is estimated by using the given feature point trajectory information to predict the feature point for the current frame T and minimize the difference therebetween.

Estimation methods include the method of modeling the motion of a trajectory in a given function form and the regression technique, which is a statistical method for predicting the relationship between variables based on given data. Also, in order to reduce visual fatigue due to image stabilization and to generate a natural stabilized image, the feature trajectory R _T is a combination of the following parts when D is a differential function.

1) static camera: D ¹ T (t) = 0

2) Camera motion with constant velocity: D ² T (t) = 0

3) camera motion with constant acceleration: D ³ T (t) = 0

The following trajectory can be predicted using a linear programming technique with a weighted sum of these characteristics as an objective function, which is illustrated in FIG.

6. motion  Model optimization

The nonlinear optimization method is a method of optimizing the motion model that minimizes the difference from the predicted point of each feature point trajectory calculated above, that is, the motion model most similar to the feature point motion (yellow line) Optimization).

The nonlinear optimization technique is to find the average distance difference (d) between the feature points converted by the motion model and the predicted position as the cost function C _t and repeatedly find the motion model parameters that minimize the cost function by each model. Can be expressed.

7. Select the keyframe

In the embodiment of the present invention, a frame in which camera shake is most reflected is selected as a key frame, thereby minimizing an area not considered in generating a stabilized image.

In this case, the number of feature points n, the feature point tracking length S _k, and the feature point locus area A _k (rectangle) are used as a criterion for determining how much the shake input image is included. Therefore, it is determined that the feature points in W are more reliable as they are traced for a long time, and the region is also used as a reference of key frame selection.

In the case of fixed cameras, if there is no PTZ operation by the user, the first key frame is used continuously. If the camera moves, the key frame is selected again.

In case of mobile camera, it is assumed that the number of feature points tracked in the current frame and the average track distance change exceeds the threshold value (n _th , A _th ), and the key frame is selected again by the above method.

8. motion  Model Selection

In order to select the best motion model among the motion models optimized by the nonlinear optimization technique, a homography M _t 'from the stabilized image I' _t generated by each motion model to the key frame I _K is obtained and the warped image I _{t And} the stabilized image I ' _t of the motion model that minimizes the RGB mean color difference M _t from the key frame I _K is selected as the stabilization result.

Let P be the number of elements of the RGB image, I _{k the} key frame I _K, and the ith pixel of the input image I _{t be} I _Ki , I _ti . The color difference M of the RGB image is as follows. .

9. Variation example

The preferred embodiments of the present invention have been described in detail above.

The motion models presented in the above embodiments are exemplary and the technical idea of the present invention is also applicable when other kinds of motion models are used. Also, the number of motion models to be used is not limited to four, and even if two, three, or five or more motion models are used, the technical idea of the present invention is applicable.

In addition, the monitoring system and the image analysis server proposed in the above embodiment are all exemplary, and the image stabilization method proposed in the embodiment of the present invention can be applied to other kinds of systems and servers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

130: image analysis server 131: image input unit
132: processor 133: image stabilization module
134: object extraction / tracking module 135: feature extraction module
136: Event extraction module 137:

Claims (8)

Generating stabilized images compensated for shaking by the image pickup apparatus for each of the motion models using a plurality of motion models;
Comparing the key frames with the stabilized images, and selecting one of the plurality of motion models; And
And outputting the stabilized image using the motion model selected in the selecting step.
The method according to claim 1,
The plurality of motion models,
Wherein the motion vectors are motion models having different complexities.
The method according to claim 1,
Wherein the selecting step comprises:
Warping the stabilized images to the key frame; And
And selecting a motion model that generates a stabilized image having a minimum color difference between the warped stabilized images and the key frame.
The method according to claim 1,
Optimizing the plurality of motion models to motion models that exhibit motion in the absence of shaking,
Wherein the generating comprises:
Wherein the plurality of motion models optimized in the optimizing step are used.
5. The method of claim 4,
Tracking feature points of a background portion in an image; And
And predicting the feature point trajectory from the feature points of the background part traced in the tracing step,
Wherein the optimizing comprises:
And optimizing the plurality of motion models to the feature point trajectory predicted in the prediction step.
6. The method of claim 5,
Wherein the tracking step comprises:
Wherein the feature points corresponding to the feature points are tracked and the feature points of the parts determined as the moving object are removed.
The method according to claim 1,
Selecting a frame in which a camera shake is most reflected among past frames as a key frame,
Wherein the selecting step comprises:
Wherein the key frame selected in the selecting step is used.
An image input unit receiving an image generated by a photographing apparatus; And
A plurality of motion models are used to generate stabilized images compensated for shaking by the photographing apparatus for each of the motion models, and one of the plurality of motion models is selected by comparing the key frames and the stabilized images, And a processor for outputting the stabilized image using the generated motion model.

Images