KR101424387B1 - Apparatus and method for processing image - Google Patents

Abstract

The present invention relates to a method and an apparatus for processing an image, which are capable of improving the quality of an image collected by transportation such as a vehicle. According to an embodiment of the present invention, an image processing apparatus for transportation includes: a first camera which includes a first lens and has a first capture speed; a second camera which includes a second lens vertical to the first lens and has a second capture speed; a beam splitter which is arranged at an angle of 45 degrees with the first lens and the second lens; an image processing unit which creates a third image using first information which is acquired from a first image photographed by the first camera and second information which is acquired from a second image photographed by the second camera; and a control unit which controls the first camera, the second camera, the beam splitter, and the image processing unit.

Description

[0001] APPARATUS AND METHOD FOR PROCESSING IMAGE [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method for improving the image quality of an image collected by a transportation means such as a vehicle.

As automobile life becomes more common in Hyundai, disputes about automobile related accidents are increasing. In the case of automobile accidents, there are many cases where the accidents are different from each other. However, recently, as the black box for vehicles has become widespread, the stored black box image is easily used for the traffic accident scene analysis and the surrounding road information analysis. A typical vehicle black box is equipped with a single camera on the front or rear of the vehicle to store forward or backward images of the vehicle in motion. Since the stored image can be checked at any time after shooting, it can be used as objective evidence for a traffic accident site, and it can be used for re-checking of the information on the road around.

However, vehicle black box images have limitations in terms of accurate information transmission. Considering the cost and usability, a typical commercial black-box image is VGA (640x480), which has sufficient resolution to understand the overall situation of the driving process but is not high resolution. Therefore, And so on. To solve this problem, a black box having a high resolution performance of more than HD (1280x720) is recently being released. However, these high-quality products are expensive and difficult to purchase, and the frame capture rate per second is as low as 30 frames.

A low-speed camera with a low frame-capturing rate per second has the disadvantage that it is difficult to obtain a clear still image for a moving object or background. That is, a moving image obtained from a vehicle traveling at a high speed or a moving object obtained by photographing a fast moving external object, even when the vehicle is stationary, generates motion blurring when each frame is extracted as a still image. To obtain still images robustly against the motion blur problem, a high-speed camera with a high frame capture rate per second is required, but the resolution can not be reduced when the storage capacity is considered. The low-resolution images have the problem of the detailed information extraction limit mentioned above. In addition, there is a problem that the image acquired by the high-speed camera has low illumination. That is, as the frame capture rate per second is fast, the time interval between still images of adjacent time becomes narrow, so that a sufficient amount of light is not included in the stored image. Therefore, there is a disadvantage in that the color information is not sufficient for the image captured by the high-speed camera.

Open No. 10-2011-0088467 discloses that a limited storage capacity of the storage can be used more efficiently by recording at least one of a recording image quality, a resolution, and a frame number of a camera image.

However, the above patent does not disclose a specific method for improving the black box image quality.

Therefore, there is a need for research on a method of improving the image quality of an image captured by a vehicle such as a vehicle.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus and method for generating an image with improved image quality than an existing image captured by a camera through fusion of images obtained from a plurality of cameras having different capture rates.

According to an aspect of the present invention, there is provided an image processing apparatus for a transportation vehicle, comprising: a first camera having a first lens and having a first capture speed; A second camera having a second lens disposed in a direction orthogonal to a direction in which the first lens is directed, and having a second capturing speed; A beam splitter disposed at an angle of 45 degrees with respect to a direction in which the first lens faces and a direction in which the second lens faces; An image processor for generating a third image using first information obtained from a first image captured through the first camera and second information obtained from a second image captured through the second camera; And a control unit for controlling the first camera, the second camera, the beam splitter, and the image processing unit.

According to an embodiment of the present invention, there is provided a camera having a first camera having a first lens and having a first capture speed, a second lens disposed in a direction orthogonal to a direction in which the first lens is directed, And a beam splitter arranged at an angle of 45 degrees with respect to a direction in which the first lens faces and a direction in which the second lens faces, The method of claim 1, further comprising: obtaining first information from a first image photographed through the first camera; Acquiring second information from a second image photographed through a second camera; And generating a third image using the obtained first information and the obtained second information.

The image processing apparatus and method according to an embodiment of the present invention can generate an image with improved image quality than an existing image captured by a camera through fusion of images obtained from a plurality of cameras having different capture rates .

According to an embodiment of the present invention, video information around the vehicle can be stably provided using a camera mounted outside the means of transportation such as a vehicle.

According to an embodiment of the present invention, when an image acquired by a vehicle such as a moving vehicle resolves the cause of deterioration in image quality, it is possible to restore the surrounding car number to a recognizable level even in a video stored during high- The effectiveness of the box can be enhanced.

1 is a view showing an image processing apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating an image processing method performed by the image processing apparatus shown in FIG.
FIG. 3 shows an example of an image acquired through the image processing apparatus shown in FIG.
4 is a diagram illustrating an example of removing motion blurring of an image in an image processing method according to an embodiment of the present invention.
5 is a diagram illustrating an example of estimating a point spread function used to remove motion blurring of an image in an image processing method according to an embodiment of the present invention.
FIG. 6 is a view showing the curve obtained in step 4 of FIG. 5 viewed vertically from above. FIG.
FIG. 7 is a diagram illustrating an example of removing motion blurring by partially estimating a point intensity distribution function for a situation where various motions exist simultaneously in an image processing method according to an embodiment of the present invention.
8 is a view showing an example of reconstructing a high-resolution image by fusing a low-resolution image of a high-speed camera.
9 is a diagram illustrating an example of generating an optimal image by applying color information of a low-speed camera to an image of a high-speed camera reconstructed using a super-resolution restoration algorithm.

Hereinafter, an image processing apparatus and an image processing method performed by the image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

As used herein, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. In this specification, the terms "comprising ", or" comprising "and the like should not be construed as necessarily including the various elements or steps described in the specification, Or may be further comprised of additional components or steps. Also, the terms "module "," module "and the like described in the specification mean units for processing at least one function or operation, which may be implemented by hardware or software or by a combination of hardware and software.

The image processing apparatus according to an exemplary embodiment of the present invention may be embedded in or attached to a transportation means to perform image processing for improving the image quality of the image collected by the transportation means. The transportation means may include a vehicle, an aircraft, a train, a subway, and the like.

Hereinafter, a vehicle image processing apparatus will be described as an embodiment of the present invention. The vehicle image processing apparatus may include a black box.

1 is a view showing an image processing apparatus according to an embodiment of the present invention.

1, the image processing apparatus 100 includes a first camera 110, a second camera 120, a beam splitter 130, an image processing unit 140, and a control unit 150 .

The first camera 110 and the second camera 120 may have different capture rates. The capture rate may include a frame capture rate per second.

In the following embodiments, a low-resolution high-speed camera is used as the first camera 110, and a low-speed camera having a high resolution is used as the second camera 120. [

The image processing apparatus 100 shows an example of registration of images obtained from the two cameras 110 and 120 using a beam splitter 130. [ In order to fuse the images obtained by the two cameras 110 and 120, a matching process for matching the views of two photographed scenes must be preceded.

Although there are various algorithms related to image matching, if the matching algorithm is used, the processing speed becomes slow and the performance of the matching algorithm can not be guaranteed depending on the situation. Therefore, in this embodiment, the hardware matching method is used .

The beam splitter 130 is a reflector device used to split the beam of light into two or more images, and is a transparent material having the characteristics of transmitting 50% of light and reflecting 50% of light.

By placing the two cameras 110 and 120 at right angles and placing the beam splitter 130 in the direction of 45 degrees, it is possible to obtain the effect that the two cameras 110 and 120 shoot the same scene. The positions of the cameras 110 and 120 and the position of the beam splitter 130 are checked so that the images of the two cameras 110 and 120 are matched so that the two images coincide with each other.

The fact that the two cameras 110 and 120 are disposed at right angles means that the angle between the direction of the lens of the first camera 110 and the direction of the lens of the second camera 120 is perpendicular It says.

The reason for placing the two cameras 110 and 120 at right angles and placing the beam splitter 130 in the direction of 45 degrees is that the direction in which the beam of the beam splitter 130 is directed by the lens of the first camera 110, May be disposed at an angle of 45 degrees with respect to the direction in which the lens of the light source 120 faces.

The image processing unit 140 may generate a third image by fusing images photographed through the first camera 110 and the second camera 120. The third image may be an image obtained by improving the image quality of the first image or the second image. In addition, the first image and the second image may be images of the same view. For example, the image processing unit 140 may generate a third image in a manner that removes motion blurring caused by movement of the vehicle and a fast frame capturing speed, or processes compensation of color loss . A detailed description of a method of generating the third image will be described later.

The control unit 150 may control the first camera 110, the second camera, the beam splitter 130, and the image processing unit 140 as a whole.

2 is a flowchart illustrating an image processing method performed by the image processing apparatus shown in FIG.

The image processor 140 may acquire a first image photographed through the first camera 110 and a second image photographed through the second camera 120. The image processing unit 140 may acquire first information from the first image and acquire second information from the second image (S210, S220).

FIG. 3 shows an example of an image acquired through the image processing apparatus shown in FIG.

3 (a) shows an image taken through the second camera 120, which is a high-resolution low-speed camera, and FIG. 3 (b) shows an image taken through the first camera 110, which is a low-

For example, assuming that the number of capture frames per second of a low-speed camera is 10 frames and the number of capture frames per second of a high-speed camera is 100 frames, a high-speed camera obtains ten frame images while obtaining one frame image from a low- have. Images obtained from a low-speed camera have high resolution, but shutter blurring may occur depending on the movement of the camera or the object because the shutter exposure time is relatively long. However, since the image obtained from the high-speed camera has a relatively low resolution but the shutter exposure time is relatively short, it is possible to capture a large number of frame images during the same time as the low-speed camera, It is easy to do. However, since the low speed camera has a long shutter exposure time, the amount of light is enough to sufficiently capture the color information of the image. However, since the shutter speed of the high speed camera is short, the amount of light is insufficient and the image is relatively dark.

The image processing unit 140 may acquire a third image having an improved image quality than the first image or the second image using the first information and the second information (S230). The third image may be generated by removing motion blurring of the second image or may be generated by a method of compensating a color of a reconstructed image created by fusing the first image, May be generated.

First, a method for generating a third image by removing motion blur will be described.

4 is a diagram illustrating an example of removing motion blurring of an image in an image processing method according to an embodiment of the present invention. In particular, FIG. 4 illustrates an example of restoring a motion blurred image using a point spread function. The point intensity distribution function refers to the response of the imaging system to a point object. That is, it represents the information of the motion of the camera during the shutter time of the camera when motion blurring occurs.

Typically, the motion blurred image can be modeled as a convoluted form by a kernel with a point intensity distribution function. Therefore, if the point intensity distribution function of the motion blurred image can be accurately estimated, it is possible to obtain the blurred image by the deconvolution operation. However, most of the motion blurred images generated by one camera have to estimate the point intensity distribution function only by the image itself, but it is difficult to estimate the point intensity distribution function for the complex motion. To solve this problem, an embodiment of the present invention uses two cameras 110 and 120 having different frame capture rates per second.

5 is a diagram illustrating an example of estimating a point spread function used to remove motion blurring of an image in an image processing method according to an embodiment of the present invention. In this embodiment, it is assumed that only global motion exists between each frame image and the frame image. That is, it is assumed that there is no movement of the object but only the movement of the camera. In this case, motion vectors representing motion between frames can be extracted through comparison between consecutive frames.

Step 1 of FIG. 5 represents the motion vector of each frame. Each point f represents the end point of the motion vector, and the start point of the motion vector is the point f of the previous frame. Step 2 in Fig. 5 is a connection of each point f with a differentiable curve. In step 3 of FIG. 5, the base of the quadrangle is obtained by connecting the midpoint between each point f on the curve of step 2, and the height is determined so that the width of the quadrangle becomes equal to the length of the base. This is based on the assumption that the camera shutter speeds are the same between each frame image, and that the same camera shutter speed includes the same amount of light, and the width of each quadrangle can be seen as the amount of light equally contained. Each point h represents the midpoint of the upper side of the rectangle. Step 4 in FIG. 5 is a curve connecting each point h to be differentiable.

FIG. 6 is a view showing the curve obtained in step 4 of FIG. 5 viewed vertically from above. FIG.

The height of h represented in step 4 of FIG. 5 can be expressed in color in FIG. The graph of FIG. 6, obtained as follows, can be a finally estimated point intensity distribution function. The point intensity distribution function can be estimated through a series of frame images obtained from the high-speed camera in the same manner as described above. Finally, the third image can be obtained through the deconvolution operation between the estimated point intensity distribution function and the motion blurred high-quality image obtained from the low-speed camera.

However, the above process is based on the assumption that global motion exists between frame images. Since the actual vehicle image includes both motion of the object and motion of the camera, it is impossible to estimate and apply only one motion to the entire image . Therefore, in order to solve this problem, it is possible to divide the image into the respective regions according to the motion, and to apply the deconvolution operation by estimating different point intensity distribution functions for each region.

FIG. 7 is a diagram illustrating an example of removing motion blurring by partially estimating a point intensity distribution function for a situation where various motions exist simultaneously in an image processing method according to an embodiment of the present invention.

First, the image processing unit 140 analyzes the motion of a given image using an optical flow, and divides the background region and the region having the moving object by using the analysis result. Then, the image processing unit 140 obtains a point intensity distribution function using the images of the high-speed camera in the manner described above for each of the background region and the object region, applies the obtained kernel of each different point intensity distribution function separately to the region, (I.e., a third image).

Next, a method of generating a third image by compensating for a color will be described. According to an embodiment of the present invention, a third image can be generated by blending color information obtained from a high-resolution image of a low-speed camera with a low-resolution image of a high-speed camera. In this case, a method of restoring a low-resolution image of a high-speed camera to a high resolution may be preceded.

8 is a view showing an example of reconstructing a high-resolution image by fusing a low-resolution image of a high-speed camera.

Resolution images can be restored to a high-resolution image through super resolution. Every time a frame image arrives at a new time, it can be merged with the restored image until the previous time. It is possible to obtain an effect in which a plurality of images are fused with only two image fusion. The super resolution restoration algorithm process may include a process of simply increasing the size of the low resolution image by interpolation using a size image to be restored when a low resolution image is input at a certain time. Thereafter, the reconstructed image can be created by matching the restored image up to the previous time and fusing the reconstructed image and the low-resolution image matched to the previous time to a weight ratio.

However, low-resolution images of a high-speed camera have insufficient amount of light due to a fast shutter speed of the camera, and thus the color information is relatively invisible because of the illumination. For this reason, these images do not contain color information sufficiently for the high-resolution reconstructed images that are fused together.

9 is a diagram illustrating an example of generating an optimal image by applying color information of a low-speed camera to an image of a high-speed camera reconstructed using a super-resolution restoration algorithm.

High-resolution images created by blending high-speed camera images have sufficient texture information, but have low illumination and low color information. To overcome this problem, color information of high-resolution images of low-speed cameras is used. Since the high-resolution image created by blending low-resolution images of high-speed cameras and the high-resolution images of low-speed cameras contain almost similar scenes, color information of high-resolution images of low-speed cameras can be sufficiently obtained.

The color of the high-resolution image of the low-speed camera is analyzed to divide it into areas having similar colors, and colors of the divided areas are superimposed on high-resolution images with low illumination. Thus, even if the image quality of the low-speed camera of the low-speed camera without motion blurring is poor, the low-resolution images of the high-speed camera are restored to the high resolution through the super-resolution restoration algorithm and the optimum image is obtained by bringing the color information of the low- .

As described above, the image processing apparatus 100 according to an exemplary embodiment of the present invention can improve the image quality of an original image captured by a camera through fusion of images obtained from a plurality of cameras having different capture rates Images can be generated. In addition, image information around the vehicle can be stably provided using a camera mounted outside the means of transportation such as a vehicle.

According to the embodiment of the present invention, when an image acquired from a vehicle such as a moving vehicle is applied to a black box by resolving causes of image quality deterioration (e.g., motion blurring, lack of color information, etc.) It is possible to restore the car number of the surrounding area to a recognizable level, thereby increasing the effectiveness of the black box.

The image processing described above may be implemented in the form of a program command that can be executed through various computer means and recorded on a computer-readable recording medium. At this time, the computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. On the other hand, the program instructions recorded on the recording medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software.

The computer-readable recording medium includes a magnetic recording medium such as a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, a magnetic disk such as a floppy disk, A magneto-optical media, and a hardware device specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

The recording medium may be a transmission medium, such as a light or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, and the like.

The program instructions also include machine language code, such as those generated by the compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above-described image processing apparatus and method are not limited to the configuration and method of the embodiments described above, but the embodiments may be modified so that all or some of the embodiments are selectively combined .

100: image processing device
110: First camera
120: Second camera
130: beam splitter
140:
150:

Claims (10)

A video processing apparatus for transportation means,
A first camera having a first lens and having a first capture speed;
A second camera having a second lens disposed in a direction orthogonal to a direction in which the first lens is directed, and having a second capturing speed;
A beam splitter disposed at an angle of 45 degrees with respect to a direction in which the first lens faces and a direction in which the second lens faces;
An image processor for generating a third image using first information obtained from a first image captured through the first camera and second information obtained from a second image captured through the second camera; And
And a control unit for controlling the first camera, the second camera, the beam splitter, and the image processing unit,
Wherein the image processing unit updates the restored image by fusing an image of the current time matched with the restored image up to a previous time. The method according to claim 1,
Wherein the first capture rate is faster than the second capture rate and the resolution of the first camera is lower than the resolution of the second camera. 3. The method of claim 2,
Wherein the image processing unit generates the third image by removing motion blurring of the second image using the first information,
Wherein the first information includes a point intensity distribution function. 4. The apparatus of claim 3, wherein the image processing unit
And removes motion blurring separately from the background region and the motion object region. 3. The apparatus of claim 2, wherein the image processing unit
And restores the first image using a resolution restoration technique and compensates the color of the first image in which the resolution is restored by using the color information of the second image. A second camera including a first lens and having a first capture speed and a second lens disposed in a direction orthogonal to a direction in which the first lens faces, And a beam splitter disposed at an angle of 45 degrees with respect to a direction in which the first lens faces the first lens and a direction in which the first lens faces the second lens, the image processing method comprising:
Acquiring first information from a first image photographed through the first camera;
Acquiring second information from a second image photographed through a second camera;
Generating a third image using the obtained first information and the obtained second information; And
And updating the restored image by fusing an image of the current time matched with the restored image up to a previous time. The method according to claim 6,
Wherein the first capture rate is faster than the second capture rate and the resolution of the first camera is lower than the resolution of the second camera. 8. The method of claim 7,
The generating of the third image may include removing motion blurring of the second image using the first information,
Wherein the first information comprises a point spread function. 9. The method of claim 8, wherein the motion blurring removing step
And separating the background region and the motion object region to remove motion blurring separately. 8. The method of claim 7, wherein the third image generation step comprises:
Reconstructing the first image using a resolution restoration technique;
And compensating a color of the first image in which the resolution is restored by using the color information of the second image.

Images