KR101993468B1 - Apparatus and method for stabilizing multi channel image - Google Patents

Abstract

A multi-channel image stabilization apparatus according to an exemplary embodiment includes a multi-channel image generation unit for generating a multi-channel image, a stabilization period determination unit for determining a stabilization period of the multi-channel image, And a stabilization processing unit for performing stabilization processing in accordance with the section.

Description

[0001] Apparatus and method for stabilizing multi-channel image [

Embodiments relate to a multi-channel image stabilization apparatus and method.

As the use of multimedia devices increases, the demand for image enhancement technology for digital images photographed in various environments is increasing. Image quality improvement technologies include image blurring, noise cancellation, and image stabilization techniques. They are widely used for digital cameras, smart phones, home cameras or camcorders, industrial surveillance cameras, broadcast cameras, and imaging devices such as military image capture devices. . The initial imaging device only digitizes the existing analog image to produce the image. However, recently used imaging devices are equipped with a variety of line processing and post-processing correction techniques, so that digital images of clearer and clearer images than conventional analog images can be obtained.

Image stabilization technology is the most commonly used technique among digital image correction techniques. In the case of a camera installed on a moving means such as an automobile, an airplane or a helicopter, there is a possibility that vibration due to various environmental factors such as mechanical vibration or friction with the ground There will be a shake. Also, as the magnification of the zoom lens increases, there is a problem that the screen shake becomes worse even with slight movement. The image stabilization technology is a technique for obtaining a clear and sharp image when the photographing apparatus is shaken during photographing, and is applied to remove unwanted shaking that occurs in the photographed image.

Conventional image stabilization technology extracts feature points from the reference image, finds corresponding points using the feature point tracking in the current image, calculates the motion between the reference image and the current image, transforms the current image with the calculated motion, . In this image stabilization technique, since the image is stabilized by using the movement of the background image, the motion of the image is calculated using the feature points extracted from the background image. However, when the feature points extracted from the dynamic object are used for the motion computation of the image, it can cause an error that can not be neglected in the motion computation of the image, resulting in a failure to stabilize the image.

There is disclosed a video service that geometrically calibrates and synthesizes a plurality of images obtained by photographing a single subject in various channels with a plurality of cameras and provides a multi-channel image to a user. These multi-channel images provide realistic images that transcend the concept of high-quality images, thereby allowing users to feel more immersive in the media and to greatly enhance the effect of transmitting video information in advertising, education, medical care, defense and entertainment have. For such a multi-channel image, there is a separate correction after the installation of a plurality of cameras. However, there is a case where a physical impact of the camera fixed during shooting occurs, There is a case where it is not smooth and shaking, and it needs to be corrected further. In this case, there is a limitation in applying the existing image stabilization technology as it is, and there is a need to improve the image stabilization technology to fit the multi-channel image.

Embodiments relate to a multi-channel image stabilization apparatus and method.

A multi-channel image stabilization apparatus according to an exemplary embodiment of the present invention includes a camera controller for controlling a plurality of cameras to shoot a subject for a predetermined time and transmitting the multi-channel images corresponding to the plurality of cameras through a communication network; And grouping the multi-channel images transmitted from the camera controller into at least one group of time, channel, time, and channel, and performing stabilization processing on the grouped images at the request of the user terminal And a stabilization module.

A multi-channel image stabilization apparatus according to another embodiment includes a multi-channel image generation unit for generating a multi-channel image; A stabilization period determiner for determining a stabilization period of the multi-channel image; And a stabilization processor for designating a region of interest in the multi-channel image and performing the stabilization process according to the determined stabilization period.

From the video sequence information of the generated multi-channel image, it is possible to determine the continuous section and the stabilization processing application section.

The stabilization process application section can be determined based on the channel change information included in the request of the user terminal.

The stabilization device may further include a trekking unit for performing trekking on the designated area of interest.

And perform stabilization processing on the trekked ROI.

The stabilization process may perform shake correction using the displacement vector of the minutiae in the ROI.

The stabilization process may perform shake correction using the displacement vector of the feature point in the trekked region of interest.

According to another aspect of the present invention, there is provided a method of stabilizing a multi-channel image, comprising: generating a multi-channel image; Determining a stabilization period of the multi-channel image; And designating a region of interest in the multi-channel image, and performing the stabilization process according to the determined stabilization period.

And a recording medium on which a program for executing the multi-channel image stabilization method according to another embodiment is stored in a computer.

1 is a schematic diagram of a transmission system 100 for a multi-channel image according to one embodiment.
2 is an exemplary diagram for generating a multi-channel image.
3 and 4 are illustrations for explaining a multi-channel image stabilization according to an embodiment.
FIGS. 5 to 7 are exemplary diagrams illustrating a method of stabilizing a multi-channel image according to an exemplary embodiment.
8 to 11 are exemplary diagrams illustrating a method of stabilizing a multi-channel image according to an exemplary embodiment.
FIG. 12 is a schematic diagram of a stabilizer applied in a state where a region of interest according to another embodiment is fixed.
FIG. 13 is a schematic diagram of a stabilization apparatus applying a method of tracking a region of interest according to another embodiment.

Although the terms used in the present embodiments have been selected in consideration of the functions in the present embodiments and are capable of being widely used in general terms, they may vary depending on the intention or circumstance of a technician working in the art, the emergence of new technology . Also, in certain cases, there are arbitrarily selected terms, and in this case, the meaning will be described in detail in the description part of the embodiment. Therefore, the terms used in the embodiments should be defined based on the meaning of the terms, not on the names of simple terms, and on the contents of the embodiments throughout.

In the description of the embodiments, when a part is connected to another part, it includes not only a direct connection but also a case where the part is electrically connected with another part in between. In addition, when a part includes an element, it does not exclude other elements unless specifically stated otherwise, but may include other elements. Further, the term " part " described in the embodiments means a unit for processing at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

It should be noted that the terms such as " comprising " or " including ", as used in these embodiments, should not be construed as necessarily including the various elements described in the specification or the various steps, The steps may not be included, or may be interpreted to include additional components or steps.

The following description of the embodiments should not be construed as limiting the scope of the present invention and should be construed as being within the scope of the embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Exemplary embodiments will now be described in detail with reference to the accompanying drawings.

The image stabilization technique extracts feature points from the reference image, finds corresponding points using the feature point tracking in the current image, calculates the motion between the reference image and the current image, and generates a stable image by converting the current image into the calculated motion . In this image stabilization technique, since the image is stabilized by using the movement of the background image, the motion of the image is calculated using the feature points extracted from the background image.

1 is a schematic diagram of a transmission system 100 for a multi-channel image according to one embodiment.

1, a transmission system 100 controls a plurality of cameras 111 to 113, a plurality of cameras 111 to 113, processes and processes multi-channel images photographed from a plurality of cameras 111 to 113, And a video server 200 for processing and storing multi-channel images transmitted from the camera control unit 110 and the camera control unit 110 for transmitting. The image server 200 receives a request for a multi-channel image, a request for a switching image, and a request for a specific event from the user terminal 150, and transmits the stored multi-channel image to the user terminal 150.

The video server 200 may include an image processor 130 and an image storage unit 140.

As shown in Fig. 2, a plurality of cameras 1 to N may be arranged to photograph a specific object. A plurality of cameras arranged in an array around a subject receives a plurality of images of the subject photographed at multiple angles. The plurality of camera array methods may be, for example, arranging N cameras in a line on substantially the same plane with respect to any camera. In this case, N cameras may be sequentially arranged on a circle spaced a certain distance from the subject, or the distance between the two cameras disposed on both sides of the center of the camera may be matched with the distance between the subject and the camera, The distance to the subject may be different. Further, the subject may be a fixed subject or a moving subject.

The plurality of cameras 1 to N and the camera control unit 110 may be capable of communicating by wire or wireless, and may include a plurality of camera control units for controlling the plurality of cameras 1 to N, respectively.

The camera control unit 110 may control the plurality of cameras 1 through N through a synchronization signal for synchronizing the plurality of cameras 1 through N. [ The camera control unit 110 temporarily stores images photographed from the plurality of cameras 1 to N, and reduces the size of the photographed image by changing the codec, thereby enabling quick transmission.

The camera control unit 110 may include a camera drive control unit 111, an image conversion unit 112, and a transmission unit 113.

The camera drive control section 111 controls camera synchronization and shooting.

The image converting unit 112 reduces the size of the images by changing the codec so that the images generated from the plurality of cameras 1 to N can be transmitted quickly through the communication network 120. It is also possible to determine a data transmission scheme that is suitable for a shooting site or suitable for wired or wireless transmission.

The transmission unit 113 transmits the converted images to the video server 200 through the communication network 120.

The video server 200 groups and stores the multi-channel images transmitted from the camera controller 110 into at least one combination of time, channel, time, and channel, Through the communication network.

Image stabilization obtains the feature points of each frame and estimates the motion vector between the matching feature points. The feature points of the frames generated at one point of view can be applied to observe the shake correction if the conventional stabilization algorithm is applied because the size of the motion vector is not large and is almost constant.

In the case of frames extracted from multi-channel or multi-viewpoint, since the displacement vector between the object and the background feature points is not constant, it is difficult to stabilize the frames.

To solve these problems, stabilization of a multi-channel image can be performed through the following two methods.

A method of fixing the region of interest and a method of tracking the region of interest can be used.

First, in the method of fixing the ROI, a target object of a multi-channel image is designated as an ROI, a displacement vector is calculated through the ROI in the ROI, and the ROI is applied to the entire image to correct the shaking.

Next, a method of estimating a region of interest may track an object using a trekking algorithm, calculate a displacement vector through a feature point in the tracked object, and apply the calculated displacement vector to the entire image to correct the shaking.

In the image stabilization according to the embodiment, after capturing a multi-channel image, a multi-channel image is generated by transmitting image sequence information. The stabilization period can be determined from this sequence information.

Each camera for capturing a multi-channel image is operated by a fixed installation. In a section for reproducing an image shot by one camera, the shaking does not occur largely. Therefore, as shown in FIG. 5, the stabilization can be performed from the section in which the channel changes or the section in which the view point is rotated to the last section. By eliminating frames that do not require processing through such processing, the amount of computation can be reduced and the processing speed can be increased.

Feature point extraction detects feature points from the reference image, the current image (feedback stabilized current image), and the next image. These feature points can be displayed as coordinates, respectively. The image has various feature elements, and some of these various feature elements can be extracted as feature points according to the needs of the user. A corner invariant feature transform (SIFT) algorithm, a speeded up robust feature (SURF) algorithm, and the like can be used to detect feature points from an image.

The feature point extraction includes a first feature point extracting unit 121 for extracting feature points from the stabilized current image, a second feature point extracting unit 122 for extracting feature points from the reference image, and a third feature point extracting unit 122 for extracting feature points from the next image. (123). Here, the reference image may be an image having the least shake among the images included in the image data generated by photographing an object, and an image having the least shake may be set as a reference image.

In selecting the feature points, the feature points corresponding to the dynamic object are removed from the foreground images delayed by one frame from the feature points of the reference image, and the feature points corresponding to the background are selected. In this way, the feature point selection removes the feature points corresponding to the dynamic object among the foreground images from the feature points of the reference image, and selects the feature points corresponding to the background. The motion calculation is to track the selected feature points of the reference image in the next input image, find corresponding points, and calculate the motion between the reference image and the next image using the corresponding points. Motion estimation tracks the selected feature points of the reference image in the next image to obtain corresponding points. Here, acquiring the corresponding point is a task of finding where the feature points selected in the reference image are located in the next image. Corresponding point acquisition methods may utilize the known KLT (kanade-lucas-tomasi) tracker algorithm, the taylor expansion, and the hessian matrix. When the corresponding point is thus obtained, the motion calculation can calculate a motion amount indicating how far the next image has moved relative to the reference image using the corresponding point. The image correction or image stabilization process transforms the next image using the calculated motion amount to generate a stabilized image.

One embodiment may also be implemented in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

Claims (10)

A camera controller for controlling a plurality of cameras to shoot a subject for a predetermined time and transmitting the multi-channel images corresponding to the plurality of cameras through a communication network; And
Channel images transmitted from the camera control unit by at least one criterion mixed with time, channel, time, and channel, and storing the grouped images in a stabilization process Module,
The stabilization module includes:
A stabilization period determiner for determining a stabilization period of the multi-channel image; And
And a stabilization processor for designating a region of interest in the multi-channel image and performing the stabilization process according to the determined stabilization period,
The stabilization period determination unit may determine,
Wherein the channel stabilization processing section determines a channel stabilization processing application period from a period in which a channel changes or a period in which a channel is changed to a last interval in an image grouped based on channel change information included in a request of the user terminal, Device. delete The method according to claim 1,
And determines a continuous section and a stabilization processing application section from the image sequence information of the generated multi-channel image. delete The method according to claim 1,
Further comprising a trekking unit for performing trekking on the designated area of interest. 6. The method of claim 5,
And performs a stabilization process on the trekked region of interest. The method according to claim 1,
Wherein the stabilization process performs shake correction using a displacement vector of a feature point in the ROI. The method according to claim 6,
Wherein the stabilization process performs shake correction using a displacement vector of feature points in the trekked ROI. Generating a multi-channel image;
Determining a stabilization period of the multi-channel image; And
Designating a region of interest in the multi-channel image, and performing the stabilization process according to the determined stabilization period,
A channel in which a channel is changed or a channel in which a view point is rotated is divided into at least one group of time, channel, time, and channel based on the channel change information included in the request of the user terminal, Wherein the stabilization process is determined to be a stabilization process application period. 12. A recording medium on which a program for causing a computer to execute the method according to claim 9 is recorded.

Images