WO2016076449A1

WO2016076449A1 - Method and system for detecting an approaching obstacle based on image recognition

Info

Publication number: WO2016076449A1
Application number: PCT/KR2014/010786
Authority: WO
Inventors: Jae Min Ban
Original assignee: Movon Corporation
Priority date: 2014-11-11
Filing date: 2014-11-11
Publication date: 2016-05-19

Abstract

This invention is about detecting method and system of an approaching obstacle based on image recognition by only using a single wide-angle camera of around view system particularly that could decipher image correctly also could detect the obstacle from input image by it. Detecting method of an approaching obstacle according to this invention includes below characteristics, step 1 which is correcting radial distortion of recorded image from vehicle camera; step 2 which is setting the region has collision risk with the vehicle as the ROI in above corrected image; step 3 of extracting the road marker at least from above ROI and process them weakened; step 4 which is extracting the boundary between the bottom of obstacle and the ground surface (hereinafter referred to as 'bottom boundary of obstacle') from above image with the weakened road markers; step 5 of detecting obstacle in the movement (hereinafter referred to as 'moving obstacle') by using above bottom boundary of obstacle; step 6 which is Deciding the bottom boundary of moving obstacle invaded into above ROI or not; and step 7 of generating the warning event when the result of step 6 is comprising above invasion case.

Description

[Title of Invention]

METHOD AND SYSTEM FOR DETECTING AN APPROACHING OBSTACLE BASED ON IMAGE RECOGNITION

[Technical Field]

[0001] This invention relates method and system for detecting an approaching obstacle, it more particularly is about the method and system for detecting an approaching obstacle based on image recognition that detects an obstacle approaching the vehicle and deciphers recorded image from the camera (in particular, 180 degree field of view camera) installed on the vehicle and alerts the driver.

[Background Art]

[0002] There are various types of obstacles around the vehicle and it tries to detect these obstacles in order to minimize the risks of potential collisions.

[0003] Typically, an ultrasonic sensor or a laser sensor is used for detecting an approaching obstacle. This moving object detecting system recognizes the approaching obstacles and alerts the driver of this impending danger.

[0004] However, in case of using camera installed on vehicle to detect an approaching obstacle instead of ultrasound or laser sensor, it needs different method with these sensors.

[0005] Camera based sensor is different from the ultrasonic or laser sensor, it can not only detects the presence of obstacles but also provides the detailed information about the obstacles.

[0006] Particularly in case of 360 degree's around view system, it generally uses the wide-angle camera to obtain a wide field of view.

[0007] But the wide-angle camera has radial distortion depending on a degree of distortion in refractive index of a convex lens inevitably, therefore it might lead to visual distortion of displayed image and serious errors on the result of image processing apparatus (that is, image recognition processing section) as well.

[0008] As described, in case using wide-angle camera, features change according to the location of image on the camera. To recognize an obstacle from the image obtained by wide-angle camera needs a prescribed technique.

[0009] If looking over the prior system to detect obstacles through image recognition, like the prior patent no. 1 and the prior patent no. 3 it is common to use a set of two cameras, known as stereo vision, in order to detect obstacles and measure the distance.

[0010] it is efficient and normal that Around View system is configured of four cameras which are installed on the front and rear of the vehicle and at the bottom of side mirrors. But to accomplish the image recognition for obstacle detection, it is very inefficient to install total eight cameras which means two pairs of cameras in the four corners of the vehicle, so the technology of using single wide-angle camera for image recognition and obstacle detection is desperately needed.

[001 1] Prior Art Document

1. Korea Utility Model Publication No. 20-1999-0026799 (Publication date: July 15, 1999)

2. Korea Patent No. 10-0882670 (Release Date: February 2, 2009)

3. Korea Patent Publication No. 10-2000-0037604 call (Publication Date: July 5, 2000) [Disclosure of Invention]

[Technical Problem]

[0012] This invention was made to solve the above problems, the goal of this invention is particularly to provide the method and system for detecting an approaching obstacle based on image recognition in spite of the configuration of single wide-angle camera from around view system. In addition, this_, configuration can not only decipher input image correctly but also can detect an obstacle.

[Solution to Problem]

[0013] The method for detecting an approaching obstacle based on image recognition to achieve above goal in this invention is characterized in these steps, step 1 of correcting the radial distortion from recorded image by the camera installed on vehicle; and step 2 of setting the region of collision risk in the above corrected image as region of interest; step 3 of extracting road markers in above region of interest at least and processing of weakening them; step 4 of extracting the boundary (hereinafter referred to as obstacle's bottom boundary) between the bottom of the obstacle and the ground surface in above image with weakening-processed road markers; step 5 of detecting an moving obstacle (hereinafter referred to as 'moving obstacle') by using above bottom boundary of obstacle; step 6 of deciding if the above bottom boundary of moving obstacle has gotten into region of interest; step 7 of generating an warning event when the result through step 6 confirms that the moving object invades Region of Interest. [Advantageous Effects of Invention]

[0014] According to the method and system for detecting an approaching obstacle based on image recognition in the present invention, around view system with single wide-angle camera could decipher the input images correctly and detect an approaching obstacle, so it could maximize the economic side and spatial efficiency of a detecting obstacle device. In addition, it not only detects the presence of obstacles but also provides the detailed information. Therefore, it could prevent the accident from the collision caused by an immature or careless driving at low speed.

[Brief Description of Drawings]

[0015] Fig. 1 is overall configuration of system for detecting an approaching obstacle based on image recognition in the present invention; Fig. 2 is a detailed block configuration diagram of Fig. 1 of image recognition processing section; Fig. 3 is a block diagram to illustrate the flow of method for detecting an approaching obstacle based on image recognition in the present invention; Fig. 4 is a conceptual diagram to explain an image correction algorithm of present invention; (a) of Fig. 5 is an example of a distortion image before image correcting process in the present invention; (b) of Fig. 5 is an example of corrected image by the image correction processing in the present invention; Fig. 6 is an example of setting the region of interest in accordance with a preferred embodiment in the present invention; (a) of Fig. 7 is an example of an initial input image which is recorded in accordance with the present invention; (b) of Fig. 7 is an example of showing weakening-processed road markers in the present invention; Fig. 8 is an example of a horizontal component size image processed by the bottom boundary detection method in the present invention; Fig. 9 is an example of a binary coded image that shows binary coded obstacle's bottom boundary area according to the present invention; Fig. 10 is an example of detection of obstacle's bottom boundary according to the present invention; Fig. 1 1 is an example of the detection result of moving obstacle in case of stationary vehicle according to the present invention; Fig. 12 is an example of showing the extracted image of obstacle's candidate region base on Entropy according to the present invention; Fig. 13 is an example of absolute value difference image in accordance with the present invention; and Fig. 14 is an example of showing the detection result of moving obstacle on system for detecting an approaching obstacle in case a vehicle is running in the present invention.

[Description of part numbers] 10: Camera 20: Warning output

30: Image recognition processing section 31 : Image correcting section

32: Image dividing section 33: Road marker processing section

34: Lower boundary of obstacle extraction section

35: Moving obstacle detection 36: Collision risk determination section

40: Region of interest 50, 51 : Road marker

62: Lower boundary of obstacle

[Mode for the Invention]

[0016] As terms used in the present invention are the purpose of describing particular embodiments, they are not intended to limit this invention. A singular includes a plural unless the context apparently points out the different meanings. In this specification, Since terms such as 'include' or 'have' is to show the presence of the characteristics, numbers, steps, operations, elements, components or those combinations described in this specification, it has to be considered that one or more other features or other characteristics, numbers, steps, operations, elements, components, or those combinations or additional possibilities are not previously excluded.

[0017] Further, in the present specification, when the certain verbs such as "is connected to" or "is accessed to" or etc. are used between one component and other components, one component may be connected to the other component directly or accessed with other components directly. But, unless there are clear other comments about them, it should be understood that the parameters may be connected or accessed through another component.

[0018] In the followings, preferred embodiments, advantages and features of the present invention will be described in detail, with reference to the accompanying drawings.

[0019] Fig. 1 is a configuration diagram on the entire system of the method for detecting an approaching obstacle based on image recognition in accordance with this invention, and Fig. 2 is a block configuration diagram in detail of an image recognition processing section illustrated in Fig. 1 .

[0020] If referring to figs. 1 and 2, the system for detecting an approaching obstacle based on image recognition is made up of the camera (10 ), the image recognition processing section (30) and the warning output section (20) according to the present invention.

[0021] Camera (10) of this invention is mounted on the vehicle to obtain and video the road around the vehicle; CMOS camera or CCD camera are preferred for this component.

[0022] In the present invention, camera (10) could be used for the Around View System in particular, this Around View System uses a wide-angle camera generally to ensure a wide field of view. Here, the configuration of Around View System includes at least four cameras which are installed on the front and rear of the vehicle and the right and left under each side mirror.

[0023] Image recognition processing section (30) in the present invention recognizes and detects obstacles from images taken by the camera (10) and then carries out the calculation process to estimate the risk of collision. It serves to prevent the collision from a driving or stationary vehicle.

[0024] The detailed configuration of the image recognition processing section(30) can be functionally divided into; Image correcting section (31), Image dividing section (32), Road markers' processing section (33), Bottom boundary of obstacle extracting section(34), Moving obstacle detecting section (35), Collision risk's deciding section (36).

[0025] The function of Image correcting section (31) in the present invention is correcting the radial distortion on the input images from the camera (10).

[0026] According to a preferred embodiment, Image correcting section (31 ) is configured to correct the radial distortion by using horizontal and longitudinal length of the above recorded images.

[0027] Dividing image section (32) in the present invention defines the region at the risk of collision as the region of interest from corrected image via Image correction section (31 ).

[0028] According to a preferred embodiment, dividing image section (32) makes ROI set as trapezoid shape. The upper side of trapezoid is set to be placed on the lower region of vanishing line within the corrected image, and the upper side of trapezoid is divided into an upper region and a lower region.

[0029] Road marker processing section (33) in the present invention performs to function as weakening process of extracted road marker at least within the region of interest.

[0030] According to a preferred embodiment, Road marker's processing section (33) is configured to perform the weakening process of resetting or replacing the pixel value of extracted road markers. [0031] Further, Road marker processing section (33) is using the difference in brightness between the ground surface and the road marker. But the region of interest is configured to extract the road marker, dividing into plural regions by setting a different critical value for the respective regions.

[0032] Obstacle's bottom boundary extracting section (34) in the present invention performs to detect the boundary between the ground surface and the lower side of obstacle (hereinafter referred to as 'lower boundary of obstacle¹) from the image of weakening-processed road marker by Road marker's processing section (33).

[0033] According to a preferred embodiment, Obstacle's bottom boundary extracting section(34) is configured to extract the horizontal component from the image of weakening-processed road marker by using Sobel Operator.

[0034] Also, Obstacle's bottom boundary extracting section (34) is based on the horizontal component of obstacle's bottom boundary extracted by Sobel Operator. It is configured to extract'the final bottom boundary of obstacle whose residual noise by road marker is removed

[0035] Here, the residual noise means that road markers might be not removed completely even though they have been removed or weakened through Road marker processing section(33). Thus, the boundary of road marker area within the region of interest is still present to a low value. As it is recognized as a noise, it is preferred to further remove these residual noises caused by road markers.

[0036] In this case, residual noise removal is carried out by Smoothing and Morphology Operator.

[0037] Moving obstacle detecting section (35) in this invention is using extracted lower boundary via Obstacle's lower boundary extracting section (34) to detect the obstacles which are moving or drifting (hereinafter referred to as 'moving obstacle ')

[0038] According to a preferred embodiment, Moving obstacle detecting section (35) is configured to tell whether the corresponding obstacle based on the upper pixel of the above obstacle's lower boundary is moving or not within an arbitrary area of ROI, using extracted lower boundary of obstacle via Obstacle's lower boundary extracting section (34).

[0039] Here, if the vehicle is stationary, it is configured that moving obstacle is detected by using the difference of absolute value image between the current image and a moving average image.

[0040] If the vehicle is, travelling, it is configured that moving obstacle is detected by using the texture information which is calculated by Local Entropy for the upper region of the above obstacle's lower boundary, the difference of absolute value image between the current image and the previous image, and motion information which is calculated by the above difference of absolute value image.

[0041] Collision risk deciding section (36) in the present invention plays a role to decide if the lower boundary of moving obstacle which is detected through moving obstacle detection section (35) has invaded into ROI or not.

[0042] According to a preferred embodiment, collision risk deciding section (36) is configured to decide if there is the invasion by using the position information of longitudinal axis of moving obstacle's lower boundary extracted through Moving obstacle detecting section (35)

[0043] In addition, it's configured to decide the distance between the vehicle (degree of access to the obstacle) and moving obstacle which is invaded into the region of interest after the calculation of the horizontal width's size in the region of interest corresponding to the longitudinal axis coordinate's height of the detected moving obstacle.

[0044] Warning output section (20) of this invention outputs a visual or audible alert according to the signal which Collision risk deciding section(36) issues. It plays a role to have a driver notice the corresponding fact.

[0045] According to a preferred embodiment, Warning output section (20) is configured that when the vehicle enters the upper area of the region of interest, the appearance of collision-zone-approaching obstacle is pre-warned and then when the vehicle invade the lower area as well as the upper area, the collision-impending warning is issued finally. Here, the format of output corresponding to the pre-alert warning and final warning should be differently in order to have a driver figure out the current status exactly.

[0046] Hereinafter, the method to detect an approaching vehicle from input image based on Obstacle image recognition processing section (30) as described above and the method to alert the driver of the danger will be described in detail.

[0047] Fig. 3 is a block diagram which illustrated the procedure flow of the method to detect an approaching obstacle based on image recognition.

[0048] Referring to Fig. 3, the method for detecting an approaching obstacle based on image recognition in accordance with the present invention includes these characteristics, (stepl ) of correcting the radial distortion from the image taken by camera installed on the vehicle; and (step2) of setting the colliding-risk area against the vehicle from above corrected image as the Region of Interest; (step3) of extracting the road marker at least within ROI and performing weakening process; (step 4) of extracting the boundary between obstacle's bottom part and ground surface in the above weakening processed image; (step 5) of detecting the moving obstacle by using the above lower boundary of obstacle; (step 6)of decision-making if the above lower boundary of moving obstacle invade into ROI or not; and step 7 of generating the warning event in the case of invasion as a result of step 6.

[0049] Preferably, above step 5 is configured to remove a noise caused by the above road markers from the weakening-processed image and then to detect the bottom boundary of obstacle.

[0050] Preferably, road markers' weakening process of above step 3 is configured to modify and replace the pixel value of the extracted road markers.

[0051] Hereinafter, embodiments of the each step will be described in detail.

[0052] <Step 1 (S10)>

[0053] The camera of the present invention could be the around view system in particular, these around view systems are common to use wide-angle camera to ensure the wide field of view. However, the wide-angle camera inevitably leads to the radial distortion which is decided according to the degree of distortion in the refractive index of a convex lens and therefore the camera has the image distorted and get the image recognition by image processing apparatus (that is, the image recognition processing section) to be able to bring about the serious errors.

[0054] Step 1 of the present invention corrects the radiation distortion from the image taken by vehicle camera ( especially around view system ) so that it makes Image recognition processing section (30) to recognize recorded image more accurately.

[0055] The former image distortion correction is performed to substitute measured parameters for distortion correction polynomial, using a well-noticed pattern such as chess board. But, there is disadvantage to be sensitive to the image noise.

[0056] In contrast, image distortion correction in the present invention features image correction algorithm which corrects the distortion region by the radial distortion, using horizontal and longitudinal length of the recorded image.

[0057] Fig. 4 is a conceptual diagram of an image correction algorithm of the present invention. Relationship between the distortion of the image coordinate (x, y) in Fig. 4, and the distortion correction of the image coordinate (x ', y¹) according to the image correction algorithm of the present invention is as following equation 1 and 2. On the other hand, as the ratio of image's horizontal length (horizontal width) and longitudinal length (longitudinal width) is not necessarily a 1 : 1 ratio, a different proportion equation to each X-axis and Y-axis is applied.

[0058] Equation [1]

[0059] Equation [2]

[0060] in the (equation 1 and 2,

[0061] x, y: coordinates of distortion image,

[0062] x ', y': corrected distortion image coordinates (corrected image coordinates),

[0063] xu, yu: center coordinates of the image,

[0064] W, H: horizontal and longitudinal length of the image,

[0065] LX, LY: x, y direction, three-dimensional distance between the center coordinates of the image and coordinates of the distorted image.)

[0066] Preferably, in order to guarantee real-time processing for the image correction of step 1 , the memory to store the correction coordinates calculated by the equation 1 and 2 is allocated. And, correction image coordinates corresponding to the distortion i

[0067] (a) of Fig. 5 is an example showing a distortion image before image correction, (b) of Fig. 5 is an example showing the corrected image by the image correction.

[0068] Refer to Fig. 5, it shows that the distortion of the image recorded by around system is more severe when it is farther from the center of the image by radial distortion, corrected image according to the image correction algorithm, in contrast to the distortion image, it could maintain the horizontal and longitudinal structure without distortion (almost) even the distance away from the center, using these characteristics < to estimate road surface or ground surface (hereinafter, referred to as ground surface), and could detect a non-specific obstacles that exist on the ground surface through the following steps.

[0069] <Step 2 (S20)>

[0070] Step 2 of this invention is a step of dividing corrected image obtained by stepl into small areas and setting the ROI, it performed by Image dividing section (32). Here, (ROI) within corrected image is randomly divided area from the entire image was taken by the camera, which means the area that can occurs a collision with the front vehicle which is moving forward in particular (it means the zone has a risk of collision with the vehicle)

[0071] According to a preferred embodiment, set the upper boundary of ROI in the , lower area of vanishing line, and set the boundary between ground surface and front end of vehicle (or rear end) (hereinafter referred to as 'vehicle boundary') as the lower boundary of ROI. Here, the vanishing line means a line corresponding to the horizon point (the vanishing point) from many lines of division in the horizontal direction along with the longitudinal axis for the entire image, vanishing line in the image can be calculated by using the height of installed camera , tilt angle and focus distance.

[0072] It means that the lower boundaries of all the obstacles exist in the obtained image from recording exist on the upper boundary of the vanishing line's lower part and vehicle boundary (e.g., vehicle hood), in case of the obstacle existing in the vicinity of vanishing line is very far away, so it is not necessary to detect in around view system such in a low-speed driving condition.

[0073] Fig. 6 shows the ROI set in accordance with a preferred exemplary embodiment.

In case of exemplary embodiment of Fig. 6, ROI (40) configure to a trapezoidal shape, but the upper side of the trapezoid (that is, the upper boundary of the ROI (40)) is set at the lower side of the vanishing line, and the lower boundary (that is, lower boundary of the ROI) is configured to be set on the boundary of a front end of vehicle (or rear end) or it's vicinity.

[0074] In addition, the ROI (40) divide into two regions of the upper region (41 ) and the lower region (42), and when the vehicle enters the upper region (41 ), warns the appearance of obstacle with collision risk, when the vehicle invades into the lower region (42) beyond the upper region (42), warns an impending collision risk.

[0075] Preferably, the upper region and a lower region is in the trapezoidal shape of the ROI, set the region close to the vanishing line based on the half-point of the height of the trapezoid as the upper region(41 ), and the region near the vehicle could be set as lower region(42).

[0076] On the other hand, the degree of access obstacles to driving passenger vehicles, that is, the Entry of upper region (41 ) and Invasion of lower region (42) could be estimated by calculating the size of vehicle width (i.e., horizontal width of the ROI (40)) which corresponds to the detected obstacle's height of longitudinal coordinate within ROI (40).

[0077] <Step 3 (S30)>

[0078] Step 3 in this invention is a step of extracting Road markers (50) (e.g., lane marker, direction guide line, etc.) on the present image and performing weakening process, road marker (50) subjected extraction is preferably target is the road marker set by via step2 existing within in ROI, set the target for the road marker within ROI

(40) via step 2 the entire image, so above case is also not excluded.

[0079] Step 3 performed by the Road marker processing section (33), this process, especially the weakening process of above Road marker (50) is characterized by using these method with modification of a pixel value or replacement.

[0080] In the ROI set by step 2, the road marker exists on the ground surface can be assumed the height "0" and road marker could be extracted within the ROI by using those characteristics.

[0081] In other words, the Road marker (50) is always placed in the lower area of vanishing line. Also, the road marker does not always face the vanishing line (vanishing point) in the Around View system, road marker is displayed in a variety of colors. Also, in most cases, road marker compare to the ground surface always has lighter value. In addition, between the boundary of ground surface and obstacle always has very dark value.

[0082] Therefore, the characteristics of the road marker as described above, that is, extract the road markers by using the difference in brightness between the ground surface and the road marker but in consideration of the various road markers according to the camera and the road status, divide the ROI into plural areas (preferably three) and extract the road marker of candidate region by setting different critical values for respective each areas.

[0083] In detail, since the darkest part of the road marker assumes as yellow color, it is preferable to extract the pixel values by experiment obtains the various critical values. This may vary depending on the performance of the camera sensor.

[0084] Further, the reason of dividing ROI into a plural of regions and applies different critical value to each area is even though the same obstacle; it has different pixel value according to the coordinates in the wide-angle camera.

[0085] After extraction of road marker as described above method, weakening process performs on this image, it performs by operating pixel value of extracted road marker. Operates pixel value means that modify or replace the initial pixel value of road marker to other pixel value, to be replaced pixel value of the road marker is pixels of candidate region with reference to the nearest pixel to allocates the value.

[0086] (a) of Fig. 7 is the initial input image of recording and (b) of Fig. 7 is an example showing the weakening processed image of road marker in accordance with this invention. As seen in Fig. 7, recorded initial input image through the camera is displayed sharply and strongly of its road marker on the ground surface, but weakening processed image in accordance with this invention shows that the road marker (51 ) is almost removed.

[0087] <Step 4 (S40)>

[0088] Step 4 is extracting the boundary between the lower part of obstacle marker and ground surface (hereinafter referred to as "lower boundary of obstacle") from the image which is readjusted the pixel value of the road marker via Step 3.

[0089] According to a preferred embodiment, method for detecting an obstacle's lower boundary in Step 4 is performed by the Sobel Operator within the image of weakening processed road marker via Step3, whereby generate the size image of horizontal component with corresponding image, thus generated horizontal component is used for extracting the lower' boundary of obstacle.

[0090] Here, for calculating the horizontal components, the Sobel Operator is the equation 3 as follows,

[0091] Equation [3]

/ ^¾ \ _x\

[0092] (in the equation 3,

[0093] f: horizontal component size image,

[0094] Gx: horizontal Sobel image)

[0095] Sobel Operator could use a variety size and shape of the mask, meets the realtime but since it is approximating so there is inaccuracy. Thus, according to a preferred embodiment, the mask for Sobel Operator has to be used as follows.

[0096] First, big size mask use a large number of pixels to obtain more accurate results but the removed area could be seen again. Here, big size mask means more than 5x5 masks.

[0097] Next, since small size mask is sensitive to the noise, it could extract a small discrete area presenting on the ground surface, but it does not stand out significantly, so it is preferable to use small size mask. Here, small size mask means the mask having a size of Ί ^χ 3' or '3 * 3'.

[0098] Fig. 8 is an example of processed image of horizontal element size image which is processed in accordance with a method for detecting lower boundary of obstacle as described previously. Horizontal component has high value in horizontal area of interest of Lower boundary of obstacle (60). Thus, as shown in Fig. 8,

, the size of horizontal component in step 3 could be calculated via the Sobel Operator calculation of step4

[0099] Horizontal component calculated by Sobel operation of step 4 could be configured to be performed as Lower boundary of obstacle (60), in above case, there is inaccuracies due to the noise caused by road marker (55) .

[0100] Accordingly, in order to ensure more accurate image recognition and detection of the lower boundary of obstacle as described above, it is preferable to be configured after removing the residual noise.

Wherein residual noise (55) means that although road marker has been removed (i.e. weakening process) via step 3 but corresponding road marker is still not completely removed and affects as a noise. [0101] in the ROI, the boundary of road marker area still exist a low value, so that affects as a noise,

Therefore, more removing operation of residue noise by such road marker should be comprised. Here, step of removing residual noise referred as 'Step 4-1 ".

[0102] <Step 4-1 (S41 )>

[0103] Step 4-1 is based on the horizontal component of obstacle's lower boundary extracted by Sobel Operator of step 4, step of extracting the lower boundary of obstacle from the image with noise caused by road marker(55) removal lastly.

[0104] According to a preferred embodiment, the noise reduction of the step 4-1 , in particular, has characteristics performed via Smoothing and Morphology Operator, the specific flow of the processing is as follows.

[0105] (1) Using a simple blur mask for calculating the average value in the mask to make the pixel values planarization of the ROI. In this case, overall pixel value of the image is lowered so that discrete point having a low value will be removed.

[0106] (2) Using morphology operation to remove the noise components of the road marker and surface boundary.

Here, the horizontal components of the road marker area does not remain constant, it can't be removed completely only via simple blur operation, using Closing operation to remove the small area with less connectivity.

[0107] (3) Maintain lower boundary of obstacle's area for the retention and remove other regions via Gaussian Blur. Since the Gaussian Blur considers the correlation with adjacent pixels a lot, so it sets the mask size larger that the Edge with low correlation with adjacent values appears to be flat.

[0108] Here, since the. affect of Gaussian blur is noticeable at more than size '5 χ 5', so that is desirable to use the mask size greater than '5x5'.

[0109] (4) Generate the cumulative histogram and perform binarization for the image of horizontal components in the ROI. In the cumulative histogram of the "0" from "255" calculating a pixel value equal to the value selected in any of one from the lower 70% to 80% range (preferably 75%), and thus calculated value of the pixel image is used as a threshold for binarization image generating.

[01 10] (5) Examining the pixel of longitudinal axis having the lowest value is non-zero value in each horizontal pixel basis on ROI. That is, moving the horizontal pixel from the left to the right (preferably from the left side of top to right side of top) and detecting corresponding to the longitudinal axis which is non-zero and the lowest value. And store detected the lowest values in buffer. In this case, if the value does not exist or the value location is lower than vanishing line, it doesn't display.

[01 1 1] Stored value in the buffer via step 4 as above process corresponds lower boundary of obstacle, and it is displayed in the form of images such as drawing. As a reference, Fig. 9 is an sample of a binary coded image showing the binary coded lower boundary of obstacle, in Fig. 9, black color corresponds to the pixel value "0", white color, that is lower boundary of obstacle corresponds to pixel value '255;

[01 12] Fig. 10 is an example showing a lower boundary detection of obstacle. Extracted lower boundary of obstacles (62) by above described steps not only applies detection of a stationary obstacle but also detection of moving obstacle. If there is an obstacle in the voluntary area (preferably, ROI) generates a event to decide the extent of the subsequent collision.

[01 13] In the followings, will be described in detail with the moving obstacle detecting step (step 5, S50)

[0 14] <Step 5 (S50)>

[01 15] Step 5 of is detecting the obstacle that has a movement (hereinafter, referred to as 'moving obstacle') by using the lower boundary of the obstacle. Detecting moving obstacle step is using extracted lower boundary of obstacle, in detail, upper pixel of obstacle's lower boundary in the voluntary area of the ROI detects the targeted obstacle in the movement or not. At this time, if the vehicle is stationary, use a moving average image and the difference of absolute value image and if the vehicle is traveling, use the information of image texture and motion, the difference of absolute value image to detect a moving obstacle.

[01 16] (1) obstacle detection in case of stationary

[01 17] In case of stationary vehicle, detecting obstacle operates base on image difference.

In case of obtained image from outdoor environment, the previous image and current image could be changed differently by environmental factors such as changes in lighting, weather etc., so in order to minimize the change, performs Gaussian blur.

[01 18] Then, in order to operate strongly by background elements such as swaying trees, the moving average image is used. Moving average image defines as follow equation 4,

[01 19] Equation [4]

[0120] (in Equation 4,

[0121] St: Moving average image,

[0122] 1 St-: t-1 accumulated average image,

[0123] lt-1 : t-1 image

[0124] a: weight of the t-th image)

[0125] Here, value 'a' is a control influence parameter that accumulated average image to the previous frame which makes current value to have a larger weight than the previous value and have a greater value than normal average value.

[0126] Calculates moving average image St, current image and difference of absolute value in each pixel then generates different image. After subtract the current image from the average image, extract a foreground which is the part has big changes. Here, above foreground includes the obstacle in the movement.

[0127] Then, using dilation of morphology expansion operation, Gaussian blur, median filter to remove noise components and detect the movement of the obstacle in the extracted foreground area.

[0128] Fig. 1 1 is a sample of moving obstacle detection in case of stationary vehicle, it shows that detected obstacle in the square block (65) of Fig. 1 1 is a moving obstacle.

[0129] (2) obstacle detection in vehicle travelling

[0130] Obstacle detection in vehicle travelling is using motion information and texture information, the absolute value of the difference image. While the vehicle is driving applies method of stationary vehicle case, then background area will be expressed as well, so extracts candidate area by using texture information, and applies weighted by absolute value difference image on motion information.

[0131] Using Texture to extract candidate region of obstacle presence, calculate the Local Entropy for the area of upper than the area of obstacle's lower boundary. Entropy shows higher value in the non-uniform region and most of the obstacles on the road can be assumed that contains non-uniform region, so calculate entropy in the voluntary size of bock.

[0 32] Here, above block of random size can be changed according to the image size, and preferably, can be set by calculating the common divisor of the width and height of the image. For example, '320 χ 240' if the image size, can be of a group consisting of '16 x 16 ',' 8 x 8 ',' 4 x 4 'the size of the block. [0133] Method for calculating entropy in the block as following equation 5

[0134] Equation 05

Ά iogPs

[0135] (in Equation 5,

[0136] Pi: probability of Gray Level for i,

[0137] ni: frequency of Gray Level for i,

[0138] M, N: horizontal and vertical size of the block

[0139] e: entropy in block)

[0140] Fig. 12 is an example showing an image of obstacle candidate region extraction which is entropy-based ; image is divided into blocks (70) of voluntary size and displayed the area with high entropy. Referring to Fig. 12, to have a robust characteristic for size change of the obstacle, more have high entropy is divided into blocks of smaller size and recalculate the entropy within divided block (70). Area with high entropy is set as obstacle candidate region and extracts motion information only for set as candidate region.

[0141] Using Lucas-Kanade Optical Flow to extract motion information. Lucas-Kanade Optical Flow only calculates the velocity vector for the pixels that appear below the minimum error in the sequence of frames, and exclude the others from the calculation.

[0142] Lucas-Kanade Optical Flow is contrary to the assumption that small and consistent movements in actual vehicle images with less accuracy use difference image of absolute value with the motion information to extract independently moving obstacles.

[0143] Fig. 13 is an example of difference image of absolute value, the difference image of absolute value which is not using moving average opposite with the case of stationary vehicle.

In the difference image of Fig. 13, extracts a pixel value only for obstacle candidate region extraction region using by the entropy to detect and display moving object. For a case of Fig. 13 indicated by the white line (80) is the equivalent.

[0144] At the time, when the vehicle is moving in the area corresponding to the background present the motion, and according to the accumulated result but appears as a noise, therefore use only the difference between the current frame (image) and previous frame (image) to use as a weight for the motion information, equation of applying the weights as following equation 6.

[0146] (in equation (6),

[0147] w (x, y): Weighting of the motion information for (x, y) coordinates,

[0148] d (x, y): difference image of the absolute value for (x, y) coordinates of the pixel values,

[0149] a: constant)

[0150] Then, for the weight computation by the equation 6, calculate by multiplying the velocity vector for each pixel as shown in Equation 07. Motion information magnifies the value of velocity vector of moving obstacle and the other background area is calculated by control the velocity vector.

[0151] [Equation 7]

mv(x,y) = v(x,y) * w(x,y)

[0152] If extract the velocity vector of certain over critical value among the velocity vector calculated as described method, moving obstacle while vehicle travelling is detected as Fig. 14, in 'case of Fig. 14, the block marked as '90' is corresponding to a moving obstacle when vehicle is travelling.

[0153] Here, extract only certain critical value of velocity vector could be extracted the velocity vector greater than . In case of obstacle is moving, it will have more than velocity vector by weight equation of equation 6.

[0 54] < Step 6 (S60)>

[0155] Step 6 of this invention is Deciding whether detected lower boundary of moving obstacle via step 5 is affected ROI or not, specifically, by using the longitudinal axis position information of detected obstacle's lower boundary to decide the invasion, and further is configured to generate the segmental warning events according to the distance between the vehicle and the obstacle.

[0156] According to a preferred embodiment, the distance between the vehicle and the obstacle can be configured calculating via the vehicle-width calculation method as follows: the degree of approach of an obstacle to a vehicle, that, the distance between the obstacle and the vehicle could be decided after calculating the size of detected obstacle's horizontal width (that, vehicle width) corresponds to the height of longitudinal axis coordinate, and then using the variation of ROI width.

[0157] <Step 7 (S70)>

[0158] Step 7 is a phase of indicating audible or (and) visual to the driver in case of invasion the lower boundary of moving obstacle into ROI via step 6.

[0159] In case of stationary obstacle, if it's lower boundary does not move to the bottom of the image means there is no risk of collision, so only display the boundary of the ground surface, and informs that there is an obstacle in the event a certain distance.

[0160] Since the moving obstacle changes its location broadly, so it configured when the lower boundary of moving obstacle invades into arbitrarily defined area, (for e.g. ROI) a warning event works.

[0161] According to a preferred embodiment, it could be configured that the ROI (40) as shown in Fig. 6 divide into 2 regions which contains upper region (41) and lower region (42), when the vehicle invades into the upper region pre-alert the appearance of obstacle with collision risk, when the vehicle invades beyond upper region (41) into lower region, warns of impending collision risk finally. Here, decision of invasion into upper or lower region of ROI (ROI) could be estimated by vehicle width calculation method as described previously.

[0162] On the other hand, object which is moving in same speed can be configured not to perform warning event, in case of the object decided going up by using motion information, then it decided as less dangerous of collision.

[0163] The above example of the preferred embodiment of this invention using specific terms is described and shown, but such terms are only intended to clarify this invention, and embodiments and described technical terms of this invention could be applied a variety of technological change and modification obviously without depart from the scope of the following claims. These modified embodiments should not be understood individually and belongs to the concept and scope of this invention.

Claims

[CLAIM 1]

Detecting method of an approaching obstacle based on image recognition which characterized in comprising below steps;

Step 1 of correcting radial distortion from the recorded image via vehicle camera;

Step 2 of setting the ROI which has collision risk with the vehicle from the corrected image above;

Step 3 of extracting the road marker from at least above ROI and processing them weakened;

Step 4 of extracting the boundary between the bottom of an obstacles and the ground surface (hereinafter referred to as 'lower boundary of obstacle') from the weakening processed image of road marker.

Step 5 of detecting an obstacle in the movement (hereinafter referred to as "moving obstacle ') using the lower boundary of the obstacle;

Step 6 of deciding whether the lower boundary of above moving obstacle has invaded into ROI.

Step 7 of generating warning event when it's decided as the case of invasion by step 6.

[CLAIM 2]

In the Claim 1 ,

Above weakening process in step 3 is a detection method of an approaching obstacle based on image recognition which is characterized in configuration of processing pixel value of above extracted road marker in a manner of modification or replacement.

[CLAIM 3]

In the Claim 1 ,

The extraction of road marker in the step 3 is the detection method based of an approaching obstacle based on image recognition which is characterized in configuration of using the brightness difference between ground surface and road marker, but it is divided above ROI into plural of regions and setting a different critical value for each area to extract the road marker.

[CLAIM 4]

In the Claim 1 ,

The extraction of lower boundary of obstacle in step 4 is the method of an approaching obstacle based on image recognition which is characterized in configuration of using the horizontal components calculated by Sobel Operator of image with weakened road marker via above step 3.

[CLAIM 5]

In the Claim 4,

The above step 4 is

Detecting method of an approaching obstacle based on image recognition which is characterized including below steps,

After calculation of the horizontal components by above Sobel Operator, The 4-1 step of planarization process, on the pixel values in ROI by using the simple blur mask;

The 4-2 step of removing the noise components on the ground surface and boundary of road marker through the morphology operation;

The 4-3 step of preserving the bottom boundary area also removing other area by using the Gaussian blurs;

The 4-4 step of generating the cumulative histogram also performing the binarization on above horizontal components; and

The 4-5 step of detecting the lowest value among the pixel on vertical axis which is non zero-value was set on the basis of each horizontal pixel in above ROI and storing in the buffer.

[CLAIM 6]

In the claim 5,

The above step 4-4 is, detecting method of an approaching obstacle based on image recognition which is characterized in configuration of calculating at least 1 pixel value correspond to any of level in the cumulative histogram from "0" to "255" with the lower range of 70% to 80%, also using above calculated pixel value as the critical value for generating the binary image.

[CLAIM 7]

In the Claim 1 ,

Above step 5 is,

Method for detecting an approaching obstacle based on image recognition which is characterized in configuration of using the lower boundary of obstacle extracted via above step 4, and in the voluntary area of above ROI and targeted at upper pixel of lower boundary of obstacle to decide and detect the present moving obstacle has a movement or not

[CLAIM 8]

In the Claim 7,

Detecting method of approaching obstacle based on image recognition which configured by using these characteristics,

If the vehicle is in the stop mode, using the difference between moving average image and absolute value of current image to detect above moving obstacle,

If the vehicle is in the driving mode,

Using texture information extracted by Local entropy calculation about the upper region of above obstacle's lower boundary, image of absolute value difference between current image and previous image, motion information calculated by above image of absolute value difference.

[CLAIM 9]

In the Claim 1 ,

Above step 6 is,

Detecting method of approaching obstacle based on image recognition which is characterized in configuration of determination of presence of above invasion by using the location information of vertical axis of moving obstacle's bottom boundary detected via above step5.

[CLAIM 10]

In the claim 9,

Detecting method of approaching obstacle based on image recognition which is characterized in configuration of determinations the distance between the moving obstacle which is invading to the ROI and vehicle (access degree of obstacle for approaching) after calculating the size of the horizontal width of above ROI corresponding to the height of the vertical axis represents the coordinates of above detected moving obstacle.

[CLAIM 1 1]

In the claim 1 ,

Above step 1 is, method of an approaching obstacle based on image recognition, characterized by being configured using the horizontal and vertical length of recorded image to correct the radial distortion.

[CLAIM 12]

In the claim 1 ,

The detection method of an approaching obstacle based on image recognition, characterized by being configured that the ROI of above step 2 set to the trapezoid shape but the upper area is set to be positioned on the lower side of the vanish line in above corrected image and the trapezoid shape is divided into two regions consisting of the upper region and the lower region.

[CLAIM 13]

In the claim 12,

Above step7 is,

the detection method of an approaching obstacle based on image recognition, characterized by being configured that the vehicle entered above upper region, pre- warns about a appearance of obstacle which has collision risk and the vehicle invaded lower region over upper region makes a final warning of impending crash risk.

[CLAIM 14]

Detecting method of approaching obstacle based on image recognition which configured includes below characteristics,

The camera records around the vehicle for taking images;

Correcting radial distortion section from the input images taken by above camera;

Image division, section of setting and dividing the region has a collision risk with the vehicle as the ROI in above corrected image; Road marker processing section extracts the road marker from the ROI at least and processes them weakened;

Extracting lower boundary of obstacle section detects the boundary between lower boundary of obstacle and ground surface from above processed image with road marker weakened;

Detecting moving obstacle section uses above lower boundary of obstacle to detect obstacle in movement (hereinafter, referred to 'lower boundary of obstacle);

Deciding collision risk section distinguish whether the lower boundary of moving obstacle has invaded into above ROI or not; and

Generating warning section generates a warning event when the result of above deciding collision risk section distinguished in case of invasion.