KR101378337B1 - Apparatus and method for processing image of camera - Google Patents

Abstract

The present invention relates to an image processing apparatus of a camera and a method of the same comprising the steps of: generating a camera image taken at a random point in time with at least one camera installed in a vehicle; affine transforming a previous camera image taken before the random point in time using a mobility model of the vehicle; and synthesizing the camera image and the previous camera image. According to the present invention, the dead zone around the vehicle can be eliminated and drivers can accurately recognize the surrounding situations of the vehicle. [Reference numerals] (AA) Start; (BB) End; (S710) Generating a camera image taken at a random point in time with at least one camera installed in a vehicle; (S720) Affine transformating a previous camera image taken before the random point in time using a mobility model of the vehicle; (S730) Synthesizing the camera image and the previous camera image; (S740) Outputting by synthesizing the synthesized camera image with a vehicle image

Description

Apparatus and Method for Processing Image of Camera

The present invention relates to an image processing apparatus and method of a camera, and more particularly, to an image processing apparatus and method of a camera capable of capturing and displaying an image of the surrounding environment of a vehicle through a camera installed in a vehicle.

In general, a camera imaging system that allows the driver to conveniently check the surrounding environment of the vehicle by visually photographing and displaying the surrounding environment of the vehicle for the convenience of the driver is commonly used.

The camera imaging system photographs the surrounding environment through cameras installed at the front, rear, left and right sides of the car, and displays the synthesized image of the surrounding environment so that the driver can accurately recognize the surrounding situation of the car. Allows convenient parking without looking at the side mirrors or back mirrors.

However, since there are blind spots in which obstacles are not visible depending on the structure of the vehicle, the position or posture at which the camera is installed, the conventional camera imaging system has a problem in that it cannot provide the driver with the obstacles located in the blind spots.

Accordingly, a driver who wants to park a car has a risk of an accident that may not be able to find an obstacle around him and collide with the obstacle, and even a driver who is used to driving raises his head and looks directly at the surrounding environment. Parking can be done by avoiding obstacles, but there is a problem of difficulty in parking because only partial visibility can be obtained.

Korean Patent Publication No. 1997-0038091 (Published: 1997. 07. 24)

The problem to be solved by the present invention is to provide a driver by synthesizing a plurality of camera images taken at different times in real time to the driver to eliminate blind spots around the car and to accurately recognize the situation around the car An image processing apparatus and method are provided.

The image processing method of the camera according to an embodiment of the present invention for solving this problem comprises the steps of generating a camera image photographed at any time by at least one camera installed in the vehicle; Affine converting a previous camera image captured by the camera before an arbitrary time point using a mobility model of the vehicle; Compositing the camera image with the previous camera image.

After the synthesizing of the camera image, the method may further include synthesizing and outputting a car image to the synthesized camera image.

The camera image generating step may include receiving an original image image captured by the camera at an arbitrary time point; The method may include converting the original image image into a view mode to generate the camera image.

Affine converting a previous camera image photographed before a certain point of time by the camera may include affine conversion of the previous camera image corresponding to the camera image using a mobility model of the vehicle based on the amount of movement of the vehicle. Can be performed.

The movement amount of the vehicle may include speed information and steering angle information of the vehicle.

The synthesizing of the camera image and the previous camera image may be performed by stitching the camera image and the previous camera image.

The vehicle image may be output in outline or translucent.

On the other hand, the image processing apparatus of the camera according to an embodiment of the present invention includes a generation unit for generating a camera image photographed at any point of time at least one camera installed in the vehicle; A converting unit for affine converting a previous camera image photographed before a certain point in time by using the mobility model of the vehicle; And a synthesizer configured to synthesize the camera image and the previous camera image.

The synthesizing unit may output a car image by synthesizing the synthesized camera image.

The generation unit may receive the original image photographed by the camera at an arbitrary time point, and convert the original image image into a view mode to generate the camera image.

The converting unit may perform an affine transformation of the previous camera image corresponding to the camera image by using the mobility model of the vehicle based on the movement amount of the vehicle.

The synthesizing unit may synthesize the camera image and the previous camera image by stitching.

The vehicle image may be output in outline or translucent.

As described above, according to the image processing apparatus and method of the camera according to an embodiment of the present invention, the camera synthesizes a plurality of camera images taken at different times in real time to the driver to eliminate blind spots around the car and the driver There is an advantage to be able to accurately recognize the surrounding situation.

More specifically, by synthesizing and outputting the camera image photographed at the present point in time and the camera image photographed at the past point of view, there is an advantage that it is easier to provide the driver with information about obstacles located in the blind spot.

In addition, by affine-converting camera images taken in the past using the vehicle's mobility model through the speed and steering angle of the car, it is possible to more naturally synthesize camera images taken in the past and cameras taken in the past. There is an advantage.

In addition, when synthesizing the camera image and the car image, there is an advantage that the driver can recognize the obstacle and the direction guidance information located on the lower part of the car by displaying the car image in outline or translucent.

Accordingly, it is possible to more easily confirm a blind spot where a driver can hardly obtain a view, thereby providing a safe driving environment for the driver and a safe parking environment.

1 is a block diagram of an image processing apparatus of a camera according to an embodiment of the present invention.
2 is an exemplary diagram of an original image image photographed by each of a plurality of cameras.
FIG. 3 is an exemplary diagram of a camera image obtained by converting the original image image shown in FIG. 2 into a view mode.
4 is a view for explaining a mobility model of a vehicle according to an embodiment of the present invention.
5 is a camera image A1 of a past view, a converted camera image A1 'and a camera image B1 of a current view, and a camera image D1 synthesized thereof according to an embodiment of the present invention. An illustration of the.
6 illustrates a first and a second camera image A2 and A3 of a past viewpoint, a first and a second camera image B2 of a current viewpoint, and a first view of a current viewpoint according to another embodiment of the present invention. And an example of the second rearward image E2 and E3.
7 is a flowchart illustrating an image processing process of a camera according to an embodiment of the present invention.

The terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary terms and the inventor may appropriately define the concept of the term in order to best describe its invention It should be construed as meaning and concept consistent with the technical idea of the present invention.

Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiments of the present invention and do not represent all of the technical idea of the present invention, various equivalents that may be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an image processing apparatus of a camera according to an embodiment of the present invention.

As shown in FIG. 1, an image processing apparatus 100 of a camera includes a camera 110, a generation unit 120, a storage unit 130, a conversion unit 140, a synthesis unit 150, and an output unit 160. It is configured to include).

The camera 110 is configured of at least one, and performs a function of capturing a three-dimensional object as a two-dimensional image through a lens. If there are two cameras, the cameras may be installed at the front and rear (or left and right) of the vehicle, respectively. If the cameras are four, the cameras may be installed at the front, rear, left and right of the vehicle.

Referring to the position where the camera 110 is installed in more detail, the front camera is installed in the center of the bonnet of the car, the cameras installed on the left and right are the edges or the bottom of both side mirrors of the car, respectively It can be installed to be located on the side. In addition, the camera installed in the rear may be installed in the center of the rear bumper, the cameras installed in the front and rear may be installed so that 170 degrees or more based on the vertical line in the ground direction.

In addition, the camera 110 may include a lens having a large angle of view, such as a wide angle lens or a fish eye lens, and may be a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor or a CCD (Charge). Coupled Device) type image sensor or the like.

2 is an exemplary diagram of an original image image photographed by each of a plurality of cameras, and FIG. 3 is an exemplary diagram of a camera image obtained by converting the original image image illustrated in FIG. 2 to a view mode.

As shown in FIG. 2, the generation unit 120 may receive the original image captured by the camera 110 at an arbitrary point in time (hereinafter, referred to as the present point of view). The unit 120 may receive four original image images D1, D2, D3, and D4 photographed by four cameras 110a, 110b, 110c, and 110d through respective channels.

In addition, the generation unit 120 may generate a camera image by converting the input original image image into a view mode. In more detail, as illustrated in FIG. 3, the generation unit 120 may display four original image images in an around view monitor (AVM) mode, a top view mode, or a specific view mode. By converting, the camera image D11 in which the four original image images are combined may be generated. In this case, the view mode may be set or stored in advance by the user.

Meanwhile, in the exemplary embodiment of the present invention, four cameras for capturing images of the front, rear, left, and right sides of the vehicle are described as an example, but two images for photographing the front and rear (or left and right) images of the vehicle are illustrated. Even in the case of a large camera, the technical features of the present invention may be equally applied.

The storage unit 130 may store a camera image generated by the generation unit 120, and may store a plurality of camera images photographed at a predetermined time interval (for example, 1 second) through the camera 110.

For example, the first camera image captured by the camera 110 at 11:07:00 AM, the second camera image captured by the camera 110 at 11:07:01 AM, and the camera 110 may be displayed in the morning. The third camera image photographed at 11:07:02 may be sequentially generated and stored in the storage 130.

The transform unit 140 performs affine transformation on the previous camera image photographed by the camera 110 before an arbitrary viewpoint (hereinafter, referred to as a past viewpoint) by using a kinematic vehicle model. Can be.

In more detail, the conversion unit 140 may use the mobility model of the vehicle so that the previous camera image photographed at a past point in time corresponds to a camera image captured at the present point in time (hereinafter, referred to as a current camera image). An affine transformation can be performed that maps the camera image onto the space of the current camera image.

Here, the mobility model of the vehicle is a model for predicting the movement of the vehicle based on the amount of movement of the vehicle, including the speed and steering angle of the vehicle, and the affine transformation is a transformation that corresponds one vector space to another vector space, and is a linear transformation. And parallel translation transformation.

4 is a view for explaining a mobility model of a vehicle according to an embodiment of the present invention. Referring to FIG. 4, in order to perform an affine transformation in the conversion unit 140, a previous camera is performed using a mobility model of the vehicle. You must move the previous camera image by the number of pixels so that the image corresponds to the current camera image. To this end, the vehicle's speed (υ) and steering angle (δ) detected by the speed sensor 172 and the steering angle sensor 174 of the vehicle, and the length of the vehicle (b) are substituted into Equation 1 below to display the previous camera image. The amount of movement (x, y, ψ (twisted)) can be calculated, and affine transformation can be performed through this calculation process.

(Equation 1)

If the previous camera image taken from the past viewpoint is composed of a plurality of previous camera images taken from different past viewpoints, the conversion unit 140 affines each of the plurality of previous camera images to correspond to the camera image of the current viewpoint. You can perform the conversion.

The combining unit 150 may synthesize the current camera image and the previous camera image, and more specifically, may adjust and synthesize the size of the camera image and the previous camera image through a stitching method.

In addition, when the previous camera image captured from the past viewpoint consists of a plurality of previous camera images taken from different past viewpoints, the synthesis unit 150 converts the plurality of previous camera images and the current camera image into one camera image. Can be synthesized.

The synthesizer 150 may apply different weights to pixel values when synthesizing the camera image. For example, the brightness of the previous camera image is applied at 4 and the brightness of the camera image at the current view is 5, so that the driver can recognize the current view and the past view when the camera images are synthesized.

5 is a camera image A1 of a past view, a converted camera image A1 'and a camera image B1 of a current view, and a camera image D1 synthesized thereof according to an embodiment of the present invention. An illustration of the is shown.

For example, referring to FIG. 5, the obstacle a on the camera image A1 captured by the camera at a past viewpoint is present in a blind spot, as in b1 on the camera image B1 at the current viewpoint. Since the image is not displayed on the camera image, the exemplary embodiment of the present invention performs an affine transformation on the camera image A1 of the past view to correspond to the camera image of the current view, thereby converting the camera image A1 'of the past view. The camera image D1 may be generated by combining the converted camera image A1 ′ with the current camera image B1 and the obstacle a.

In addition, the combiner 150 may output the car image by combining the synthesized camera image, and more specifically, may synthesize the synthesized camera image and the car image by overlaying and output the synthesized camera image. That is, the synthesizer 150 may match and output the virtual camera image so that the current position of the car appears in the camera image obtained by synthesizing the camera image photographed at the past and the camera image photographed at the present time.

As such, by matching and outputting the virtual car image indicating the current position of the vehicle to the camera image synthesized by the synthesizing unit 150, the driver can recognize that the vehicle is passing through the section in which the speed bump is installed. The blind spots can be easily identified around the image.

6 illustrates a first and a second camera image A2 and A3 of a past viewpoint, a first and a second camera image B2 of a current viewpoint, and a first view of a current viewpoint according to another embodiment of the present invention. And an exemplary view of the second rear image image E2 (E3).

In FIG. 6, the first camera image B2 of the current view is an image generated by converting the first rear view image E2 taken by the camera installed at the rear of the vehicle to the Top View mode. The second camera image B3 is an image generated by converting the second rear image image E3 photographed after a predetermined time elapses from the time when the first rear image image E2 is captured to the top view mode. After a predetermined time has elapsed, the first camera image corresponding to the P area among the first camera images B2 of the viewpoint is converted into a second camera image corresponding to the R region, and the first camera image A2 of the past viewpoint is By being synthesized, it may be used as the second camera image A3 of a past viewpoint.

As such, the second camera image A3 corresponding to an arbitrary viewpoint (past viewpoint) may be a camera image (R region) photographed before an arbitrary viewpoint and a camera image (S region photographed earlier than an arbitrary viewpoint). ) May be generated in combination.

In addition, the synthesis unit 150 may display a blind spot covered by the car by displaying the car image as an outline or translucent, and the blind spot covered by the car may include a lower part of the car. That is, the synthesizing unit 150 displays the vehicle image in outline or translucent so that the driver can recognize obstacles and direction guide information located in the surrounding environment of the vehicle.

The output unit 160 displays a camera image obtained by synthesizing a camera image of a past point in time, a camera image of a present point in time, and a car image, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), and the like. Display on the display module.

For example, if the camera takes an image of where an obstacle is installed and then the vehicle continues to drive and the image is taken by the camera at the current position after 1 second, the obstacle is installed. The place may be located in a blind spot.

Accordingly, the image processing apparatus 100 of the camera according to the present exemplary embodiment displays a previous camera image of a past view taken one second ago on a camera image taken at a present time so that the driver may recognize information about an obstacle located in a blind spot. By compositing, the driver can recognize where the obstacles are installed, and by composing the car image to the synthesized camera image, the position of the current car can be recognized.

In addition, when synthesizing the camera image and the previous camera image by using the mobility model of the car affine conversion and combining the previous camera image to control the camera image is more naturally synthesized.

Hereinafter, an image processing process of a camera according to an embodiment of the present invention will be described.

7 is an operation flowchart showing an image processing process of a camera according to an embodiment of the present invention.

As illustrated in FIG. 7, at least one camera installed in the vehicle generates a camera image photographed at an arbitrary time point (S710).

In more detail, in order to generate a camera image, the camera 110 may receive an original image photographed at an arbitrary point in time (hereinafter, referred to as a present point of view). The 120 may receive four original image images D1, D2, D3, and D4 photographed by four cameras 110a, 110b, 110c, and 110d through respective channels.

Thereafter, the input original image image may be converted into a view mode to generate a camera image. In more detail, the four original video images may be converted into an around view monitor (AVM) mode or a specific view mode to generate a camera image D11 in which the four original video images are combined.

Next, the camera affine-converts the previous camera image photographed before an arbitrary time point using the mobility model of the vehicle (S720).

In more detail, the previous camera image captured in the past using the mobility model of the vehicle corresponds to the camera image captured in the current viewpoint (hereinafter referred to as the current camera image), that is, the previous camera image is replaced with the current camera image. The affine transformation can be performed to correspond in space.

Next, the camera image and the previous camera image are synthesized (S730). In this case, the synthesized camera image and the car image may be synthesized and output in an overlay method.

Then, the vehicle image is synthesized and output to the synthesized camera image (S740).

As such, by matching and outputting a virtual car image indicating the current position of the vehicle to the synthesized camera image, the driver can recognize that the vehicle is passing through the section where the speed bumps are currently installed, and the blind spot centered on the car image. It can be easily confirmed.

In addition, the blind spots covered by the car may be displayed by displaying the car image in outline or translucent, and the blind spots covered by the car may include a lower part of the car. That is, the synthesizing unit 150 displays the vehicle image in outline or translucent so that the driver can recognize obstacles and direction guide information located in the surrounding environment of the vehicle.

One embodiment of the present invention includes a computer-readable medium including program instructions for performing various computer-implemented operations. This medium records a program for executing the image processing method of the camera described above. The medium may include program instructions, data files, data structures, etc., alone or in combination. Examples of such media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD and DVD, programmed instructions such as floptical disk and magneto-optical media, ROM, RAM, And a hardware device configured to store and execute the program. Or such medium may be a transmission medium, such as optical or metal lines, waveguides, etc., including a carrier wave that transmits a signal specifying a program command, data structure, or the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

The foregoing detailed description is illustrative of the present invention. It is also to be understood that the foregoing is illustrative and explanatory of preferred embodiments of the invention only, and that the invention may be used in various other combinations, modifications and environments. That is, it is possible to make changes or modifications within the scope of the concept of the invention disclosed in this specification, the disclosure and the equivalents of the disclosure and / or the scope of the art or knowledge of the present invention. The foregoing embodiments are intended to illustrate the best mode contemplated for carrying out the invention and are not intended to limit the scope of the present invention to other modes of operation known in the art for utilizing other inventions such as the present invention, Various changes are possible. Accordingly, the foregoing description of the invention is not intended to limit the invention to the precise embodiments disclosed. It is also to be understood that the appended claims are intended to cover such other embodiments.

100: image processing device of the camera
110: camera 120: generation unit
130: storage unit 140: conversion unit
150: synthesis unit 160: output unit

Claims (13)

Generating at least one camera image photographed at an arbitrary time by at least one camera installed in the vehicle;
Affine converting a previous camera image captured by the camera before an arbitrary time point using a mobility model of the vehicle;
Compositing the camera image with the previous camera image,
The camera image generating step,
Receiving, by the camera, an original image photographed at an arbitrary point in time;
And converting the original image image into a top view mode to generate the camera image. The method of claim 1,
And synthesizing and outputting a car image to the synthesized camera image after the camera image synthesizing. The method of claim 1,
The image processing method of the camera synthesized by applying different brightness of the camera image and the previous camera image. The method of claim 1,
Affine converting a previous camera image photographed before a certain point in time by the camera,
And performing affine transformation of the previous camera image corresponding to the camera image using the mobility model of the vehicle based on the movement amount of the vehicle. 5. The method of claim 4,
The amount of movement of the car,
An image processing method of a camera comprising speed information and steering angle information of the vehicle. The method of claim 1,
Comprising the camera image and the previous camera image,
And a stitching method for synthesizing the camera image with the previous camera image. 3. The method of claim 2,
The car image is,
Image processing method of the camera output in the contour or translucent. A generator configured to generate a camera image photographed at an arbitrary time point by at least one camera installed in the vehicle;
A converting unit for affine converting a previous camera image photographed before a certain point in time by using the mobility model of the vehicle;
Comprising a synthesis unit for synthesizing the camera image and the previous camera image,
Wherein the generation unit comprises:
And the camera receives an original image photographed at an arbitrary time point, and converts the original image image into a top view mode to generate the camera image. The method of claim 8,
The synthesizing unit,
An image processing apparatus of a camera for synthesizing and outputting a car image to the synthesized camera image. The method of claim 8,
The synthesizing unit,
The image processing apparatus of the camera synthesized by applying different brightness of the camera image and the previous camera image. The method of claim 8,
Wherein,
And an affine transformation of the previous camera image corresponding to the camera image using the mobility model of the vehicle based on the amount of movement of the vehicle. The method of claim 8,
The synthesizing unit,
And an image processing apparatus for synthesizing the camera image and the previous camera image by stitching. The method of claim 9,
The car image is,
An image processing apparatus of a camera, which is output in outline or translucent to display a blind spot covered by the vehicle.

Images