KR102371588B1 - System and method for recognizing obstacle based on image - Google Patents

Abstract

An obstacle recognition system related to an embodiment of the present invention includes an image acquisition unit that acquires an image of the exterior of the vehicle from a camera mounted on the vehicle, a vehicle speed acquisition unit that acquires the speed of the vehicle, and when the speed of the vehicle is less than or equal to a preset speed , a control unit for generating a plurality of difference frames composed of difference values between the current frame and each of a plurality of previous frames of the image, generating an overlapping frame by overlapping the plurality of difference frames, and recognizing an obstacle from the overlapping frame can

Description

Image-based obstacle recognition system and method {System and method for recognizing obstacle based on image}

The present invention relates to an image-based obstacle recognition system and method.

Various types of obstacles exist around a vehicle, and various methods for recognizing obstacles have been proposed in order to be protected from such a potential collision risk factor.

Although there is a method of using an image, ultrasound, laser, etc. to recognize an obstacle, a method using an image is preferable as a method for a driver to directly identify an obstacle.

Among the methods of recognizing obstacles using images, there is a method of applying vehicle behavior information to continuous image changes using a wide-angle fisheye camera and estimating the existence of obstacles in areas where changes occur that are different from the vehicle behavior.

This method removes some of the changes in the image in consideration of the vehicle's behavior, that is, the speed of the vehicle, and recognizes the remaining parts as obstacles. ) and so on.

However, the existing method has a problem in that the performance of recognizing obstacles is poor because there is insufficient change in the image when the speed of the vehicle, which is the standard for judgment, is slow, and requires a lot of calculation processes to not guarantee real-time performance.

The present invention has been devised to solve the problems of the prior art, and has an object to improve obstacle recognition performance and reduce unnecessary operations in an image-based obstacle recognition method.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned are clearly to those of ordinary skill in the art to which the present invention belongs from the description below. can be understood

An obstacle recognition system related to an example of the present invention for realizing the above-described problem includes an image acquisition unit for acquiring an image outside the vehicle from a camera mounted on the vehicle, a vehicle speed acquisition unit for acquiring the speed of the vehicle, and When the speed is less than or equal to a preset speed, a plurality of difference frames composed of difference values between the current frame of the image and each of a plurality of previous frames are generated, and an overlapping frame is generated by overlapping the plurality of difference frames, and an obstacle from the overlapping frame is generated. It may include a control unit for recognizing

Also, the controller may correct spatial positions of the plurality of previous frames using the vehicle speed, and generate the difference frame using the corrected plurality of previous frames.

In addition, when the speed of the vehicle is 0 or when the speed of the vehicle exceeds the preset speed, the controller compares the current frame with at least one of the plurality of previous frames without generating the overlapping frame. obstacles can be recognized.

Also, when the camera is a fisheye lens camera, the controller may correct distortion of the image by using a distortion rate of the fisheye lens camera.

Also, when the vehicle speed is 0, the controller may not correct the distortion of the image.

Embodiments of the present invention can improve obstacle recognition performance and reduce unnecessary computation in an image-based obstacle recognition method.

In addition, by applying an embodiment of the present invention, an obstacle can be recognized in real time even using a low-performance controller.

On the other hand, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be able

The following drawings attached to the present specification illustrate a preferred embodiment of the present invention, and serve to further understand the technical idea of the present invention together with the detailed description of the present invention, so the present invention is limited to the matters described in those drawings It should not be construed as being limited.
1 is a block diagram of an obstacle recognition system according to an embodiment of the present invention.
FIG. 2A is an image captured by a fisheye lens camera according to an embodiment of the present invention, and FIG. 2B is an image in which distortion of an image captured by a fisheye lens camera according to an embodiment of the present invention is corrected.
3A shows a difference frame when the vehicle speed is 5 km/h according to an embodiment of the present invention, and FIG. 3B shows a difference frame when the vehicle speed is 1 km/h according to an embodiment of the present invention. indicates.
4A is a current frame of an image according to an embodiment of the present invention, and FIGS. 4B to 4D are previous frames of an image arranged in reverse chronological order according to an embodiment of the present invention.
5A is a current frame of an image according to an embodiment of the present invention, and FIGS. 4B to 5D are previous frames of an image whose spatial positions are corrected and arranged in reverse chronological order according to an embodiment of the present invention.
6A to 6C are differential frames derived from each of a current frame and a previous frame of an image according to an embodiment of the present invention.
7 is an overlapping frame according to an embodiment of the present invention.
8 is a flowchart illustrating an obstacle recognition method according to an embodiment of the present invention.
9 is a block diagram illustrating a computing system executing an obstacle recognition method according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment of the present invention, the detailed description thereof will be omitted.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the component from other components, and the essence, order, or order of the component is not limited by the term. In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

1 is a block diagram of an obstacle recognition system according to an embodiment of the present invention.

FIG. 2A is an image captured by a fisheye lens camera according to an embodiment of the present invention, and FIG. 2B is an image obtained by correcting distortion of an image captured by a fisheye lens camera according to an embodiment of the present invention.

3A shows a difference frame when the vehicle speed is 5 km/h according to an embodiment of the present invention, and FIG. 3B shows a difference frame when the vehicle speed is 1 km/h according to an embodiment of the present invention. indicates.

4A is a current frame of an image according to an embodiment of the present invention, and FIGS. 4B to 4D are previous frames of an image arranged in reverse chronological order according to an embodiment of the present invention.

5A is a current frame of an image according to an embodiment of the present invention, and FIGS. 4B to 5D are previous frames of an image whose spatial positions are corrected and arranged in reverse chronological order according to an embodiment of the present invention.

6A to 6C are differential frames derived from each of a current frame and a previous frame of an image according to an embodiment of the present invention.

7 is an overlapping frame according to an embodiment of the present invention.

First, referring to FIG. 1 , the obstacle recognition system according to the present invention may include an image acquiring unit 100 , a vehicle speed acquiring unit 200 , and a controller 300 .

However, since the components shown in FIG. 1 are not essential, an obstacle recognition system having more or fewer components may be implemented.

The image acquisition unit 100 is configured to acquire an image outside the vehicle.

The image acquisition unit 100 may be a camera for photographing the exterior of the vehicle, or may be an image receiving unit for acquiring an image from a camera photographing the exterior of the vehicle.

In addition, the camera may be a fisheye lens camera for capturing an omnidirectional image, and the image obtained from the fisheye lens camera may be a distorted image because it is a wide-angle image obtained with one lens.

The vehicle speed obtaining unit 200 is a component for obtaining the speed of the vehicle, and may obtain information about the direction in which the vehicle moves as well as the speed of the vehicle. For example, the vehicle speed obtaining unit 200 may be an On-Board Diagnostics (OBD).

The control unit 300 is a configuration for recognizing an obstacle from the image obtained by the image acquisition unit 100, and may recognize the obstacle from a plurality of frames of the image by performing different calculations according to the speed of the vehicle.

Hereinafter, a method for the controller 300 to recognize an obstacle will be described in detail with reference to FIGS. 2A to 7 .

When the obtained image is an image obtained by a fisheye lens camera, the controller 300 may correct distortion of the image by using a distortion rate of the fisheye lens camera. In order to apply the vehicle behavior information to the continuously inputted image change, the actual distance information and the image information are matched, such as IPM (Inverse Perspective Mapping).

That is, as shown in FIG. 2A , the image acquired by the fisheye lens camera is distorted compared to the actual image. The controller 300 may derive a corrected image as shown in FIG. 2B by performing a calculation method such as IPM on the distorted image.

However, when the obtained image is an undistorted image, there is no need to perform a separate correction operation.

Also, even when the vehicle speed is 0, it is not necessary to correct image distortion. This is because, when the vehicle is stopped, it is not necessary to perform an operation for applying vehicle behavior information to the image, and only the moving obstacle needs to be recognized through comparison between a plurality of frames of the image.

The controller 300 may determine whether the speed of the vehicle is slower than the preset speed Vth and, when the speed of the vehicle is less than or equal to the preset speed, may perform an operation for strengthening the characteristics of the obstacle.

Although the characteristics of the obstacle are clearly revealed in the differential frame derived using the frame of the image obtained from the vehicle moving at 5 km/h as shown in FIG. 3A, the frame of the image obtained from the vehicle moving at 5 km/h as shown in FIG. 3B This is because the characteristics of obstacles are not clearly revealed in the difference frame derived using .

That is, when the speed of the vehicle is fast, the size or position change of the obstacle between adjacent frames of the image is noticeable, but when the speed of the vehicle is slow, the change in the movement of the obstacle expressed in the adjacent frame is insignificant.

First, the controller 300 may extract a current frame and a plurality of previous frames from an image in order to perform an operation for reinforcing the characteristics of the obstacle.

4A shows a current frame in an image, and FIGS. 4B to 4D are frames captured temporally with respect to the current frame.

Since the vehicle moves slowly but the speed is not 0, the position of the fixed obstacle changes as much as the moving distance of the vehicle in the image.

Referring to FIGS. 4A to 4D , the position of the obstacle gradually rises with reference to the red line from FIG. 4A to FIG. 4D . Since FIG. 4D of FIGS. 4A to 4D is the first frame taken temporally, the vehicle approaches the obstacle.

After extracting the current frame and the plurality of previous frames, the control unit 300 extracts the current frame and the plurality of previous frames, and before generating a differential frame between each of the plurality of previous frames, according to an embodiment of the present invention, a plurality of previous frames using the vehicle speed can correct the spatial position of

That is, the plurality of previous frames is corrected by applying the vehicle behavior information, and the positions of the obstacles present in each of the plurality of previous frames are corrected to be the same as the positions of the obstacles present in the current frame.

5B to 5D show frames in which spatial positions of a plurality of previous frames are corrected according to an embodiment of the present invention, and the positions of obstacles appearing in the plurality of frames are the same based on a red line.

The controller 300 may generate a plurality of difference frames including a difference value between each of a plurality of previous frames whose spatial position is corrected and a current frame.

That is, a frame in which only a portion having a difference between each of the current frame and the previous frame is generated is generated.

6A is a differential frame derived from FIGS. 5A and 5B, FIG. 6B is a differential frame derived from FIGS. 5A and 5C, and FIG. 6C is a differential frame derived from FIGS. 5A and 5D.

Referring to FIGS. 6A to 6C , it can be seen that the obstacle appears more clearly in FIG. 6C , which is a difference frame derived from FIGS. 5A and 5D that is further away in time.

If the above-described spatial position is not corrected, the differential frame may be derived from FIGS. 4A and 4B, from FIGS. 4A and 4C, and from FIGS. 4A and 4D.

The controller 300 may create an overlapping frame as shown in FIG. 7 by overlapping the derived FIGS. 6A to 6C .

Although obstacles can be recognized in FIG. 6C, which is a difference frame derived from FIGS. 5A and 5D that is farther away in time, there may be moving obstacles instead of only fixed obstacles around the vehicle, so obstacles are removed from one difference frame. If it does, it cannot recognize fast-moving obstacles.

Therefore, when a plurality of difference frames are overlapped, an overlapping frame with high temporal resolution is derived, and a fast moving obstacle may also appear in the overlapping frame.

The controller 300 may recognize an obstacle from the overlapping frame.

On the other hand, when the speed of the vehicle exceeds the preset speed, as shown in FIG. 3A , a change in size or position of an obstacle between adjacent frames of an image is conspicuous.

Accordingly, the controller 300 may recognize an obstacle from a difference frame between adjacent frames of an image without performing the obstacle feature enhancement operation described with reference to FIGS. 4A to 7 . Also, even when the vehicle speed is 0, the controller 300 may recognize an obstacle from a difference frame between adjacent frames of an image without performing an obstacle feature enhancement operation.

Hereinafter, an obstacle recognition method will be described in detail based on the above-described configurations with reference to FIG. 8 .

8 is a flowchart illustrating an obstacle recognition method according to an embodiment of the present invention.

8 , the obstacle recognition method according to an embodiment of the present invention includes acquiring an image (S100), determining whether the vehicle speed is greater than zero (S200), and correcting image distortion (S300), determining whether the vehicle speed is less than or equal to a preset speed (Vth) (S400), generating an overlapping frame (S510), generating a difference frame (S520), and recognizing an obstacle (S600) may be included.

Hereinafter, steps S100 to S600 described above will be described in detail with reference to FIGS. 1 to 7 .

In step S100 , the image acquisition unit 100 may acquire an image outside the vehicle. The image acquisition unit 100 may photograph the exterior of the vehicle as a camera, and may acquire an image from a camera photographing the exterior of the vehicle.

Also, the obtained image may be an omnidirectional image photographed with a fisheye lens camera, and an image photographed with a fisheye lens camera may be a distorted image because it is a wide-angle image.

In step S200 , the controller 300 may determine whether the vehicle speed is greater than zero.

As described above, when the vehicle speed is 0, that is, when the vehicle is stationary, the control unit 300 may perform the step of generating a difference frame from adjacent frames of an image, which will be described later, without performing an obstacle feature enhancement operation. .

When the vehicle speed is greater than 0, in step S300 , the controller 300 may correct the distortion of the image.

In step S300, when the image obtained in step S100 is a distorted image as an image obtained by a fisheye lens camera, etc., in order to apply vehicle behavior information from continuously input phase changes, This is a step in which the actual distance information and the image information are matched. That is, the image shown in FIG. 2A is corrected as shown in FIG. 2B.

In this case, the controller 300 may correct the distortion of the image by using the distortion rate of the camera that captured the image.

In step S400 , the controller 300 may determine whether the vehicle speed is slower than a preset speed Vth.

When the speed of the vehicle is slower than the preset speed, the controller 300 may generate an overlapping frame by performing an obstacle feature reinforcement operation as described with reference to FIGS. 4A to 7 ( S510 ).

When the speed of the vehicle is faster than the preset speed, as shown in FIG. 3A , the characteristic of the obstacle is clearly revealed in the difference frame, so a separate obstacle feature reinforcement operation is not performed, and the controller 300 controls the difference between each frame from the adjacent frame of the image. A difference frame composed of values may be generated (S520).

In step S600 , the controller 300 may recognize an obstacle from the overlapping frame or the difference frame.

9 is a block diagram illustrating a computing system executing an obstacle recognition method according to an embodiment of the present invention.

Referring to FIG. 8 , the computing system 1000 includes at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , and storage connected through a bus 1200 . 1600 , and a network interface 1700 .

The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and/or the storage 1600 . The memory 1300 and the storage 1600 may include various types of volatile or nonvolatile storage media. For example, the memory 1300 may include read only memory (ROM) and random access memory (RAM).

Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by the processor 1100 , or a combination of the two. A software module resides in a storage medium (ie, memory 1300 and/or storage 1600 ) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM. You may. An exemplary storage medium is coupled to the processor 1100 , the processor 1100 capable of reading information from, and writing information to, the storage medium. Alternatively, the storage medium may be integrated with the processor 1100 . The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

In the obstacle recognition system and method described above, the configuration and method of the above-described embodiments are not limitedly applicable, but all or part of each embodiment is selectively combined so that various modifications can be made. may be configured.

100: image acquisition unit
200: vehicle speed acquisition unit
300: control unit
1000: computing system
1100: Processor
1200: system bus
1300: memory
1310: ROM
1320: RAM
1400: user interface

Claims (5)

an image acquisition unit for acquiring an image outside the vehicle from a camera mounted on the vehicle;
a vehicle speed obtaining unit which obtains the speed of the vehicle;
When the speed of the vehicle is less than a preset speed,
correcting the spatial position of a plurality of previous frames of the image by using the vehicle speed, generating a plurality of difference frames composed of difference values between the current frame of the image and each of the corrected plurality of previous frames, and Obstacle recognition system comprising a; a control unit for generating an overlapping frame by overlapping the difference frames of , and recognizing an obstacle from the overlapping frame. delete The method of claim 1,
When the speed of the vehicle is 0 or when the speed of the vehicle exceeds the preset speed,
The control unit recognizes the obstacle by comparing the current frame with at least one of the plurality of previous frames without generating the overlapping frame. The method of claim 1,
When the camera is a fisheye lens camera,
The control unit is an obstacle recognition system, characterized in that for correcting the distortion of the image by using the distortion rate of the fisheye lens camera. 5. The method of claim 4,
When the vehicle speed is 0,
The obstacle recognition system, characterized in that the control unit does not correct the distortion of the image.

Images