WO2013022159A1

WO2013022159A1 - Traffic lane recognizing apparatus and method thereof

Info

Publication number: WO2013022159A1
Application number: PCT/KR2012/000190
Authority: WO
Inventors: Youngkyung Park; Jonghun Kim; Joongjae LEE; Hyunsoo Kim; Junoh PARK; Andreas PARK; Chandra Shekhar DHIR
Original assignee: Lg Electronics Inc.; Lee, Jeihun
Priority date: 2011-08-05
Filing date: 2012-01-09
Publication date: 2013-02-14
Also published as: KR101641251B1; KR20130015978A

Abstract

Disclosed is a traffic lane recognizing apparatus and method capable of accurately recognizing a travel traffic lane of a vehicle. The traffic lane recognizing apparatus includes: a camera module; a display unit configured to display an image captured by the camera module; and a controller configured to detect candidate traffic lanes based on traffic lane conformance from among traffic lanes detected based on traffic lane feature points detected from the image captured by the camera module and different traffic lane equations and display traffic lanes adjacent to a vehicle, as current traffic lanes along which the vehicle is traveling (or moving) on, among the detected candidate traffic lanes, on the image.

Description

TRAFFIC LANE RECOGNIZING APPARATUS AND METHOD THEREOF

The present invention relates to a traffic lane recognizing apparatus and a method thereof.

In general, a traffic lane recognizing apparatus is an apparatus for recognizing a traffic lane included in a certain image input from a camera, or the like, or a certain image received from an external terminal. A related art traffic lane recognizing apparatus is disclosed in Korean Patent Publication Laid Open No. 1995-0017509.

According to an aspect of the present invention, there is provided a traffic lane recognizing apparatus including: a camera module; a display unit configured to display an image captured by the camera module; and a controller configured to detect candidate traffic lanes based on traffic lane conformance from among traffic lanes detected based on traffic lane feature points detected from the image captured by the camera module and different traffic lane equations and display traffic lanes adjacent to a vehicle, as current traffic lanes along which the vehicle is traveling (or moving) on, among the detected candidate traffic lanes, on the image.

In an example related to the present disclosure, the traffic lane conformance may be previously determined based on the sum of differences in distances between a traffic lane obtained by substituting the traffic lane feature points to the different traffic lane equations, and the traffic lane feature points.

In an example related to the present disclosure, the traffic lane conformance may be determined based on the sum of differences in distances between a traffic lane obtained by substituting the traffic lane feature points to the different traffic lane equations, and the traffic lane feature points, the number of the traffic lane feature points, and a range in which the traffic lane feature points are distributed.

In an example related to the present disclosure, the controller may detect traffic lanes, whose difference in distance between themselves and the traffic lane feature points is a threshold value or smaller, as the candidate traffic lanes, from among the traffic lanes detected based on the traffic lane feature points detected from the image captured by the camera module and the different traffic lane equations.

In an example related to the present disclosure, the controller may detect the traffic lane feature points from the captured image, and sequentially substitute the different traffic lane equations to the detected traffic lane feature points to thereby detect traffic lanes, whose difference in distance smaller than the threshold value, as the candidate traffic lanes, from among the detected traffic lanes.

In an example related to the present disclosure, the controller may detect traffic lane feature points from the captured image, convert the traffic lane feature points into world coordinates, and sequentially substitute the traffic lane feature points which have been converted into the world coordinates to the different traffic lane equations, to thereby detect traffic lanes, whose difference in distance is smaller than the threshold value, as the candidate traffic lanes, from among the detected traffic lanes.

According to another aspect of the present invention, there is provided a traffic lane recognizing method including: displaying an image captured by a camera module on a display unit; detecting candidate traffic lanes based on traffic lane conformance from among traffic lanes detected based on traffic lane feature points detected from the image captured by the camera module and different traffic lane equations; and displaying traffic lanes adjacent to a vehicle as current traffic lanes along which the vehicle is traveling (or moving) on, among the detected candidate traffic lanes, on the image.

FIG. 1 is a schematic block diagram showing the configuration of a traffic lane recognizing apparatus according to an embodiment of the present invention.

FIG. 2 is a view showing an image captured by a camera according to an embodiment of the present invention.

FIG. 3 is a view showing guide lines according to an embodiment of the present invention.

FIG. 4 is a view showing feature points of traffic lanes according to an embodiment of the present invention.

FIG. 5 is a view showing feature points of traffic lanes converted into world coordinates according to an embodiment of the present invention.

FIG. 6 is a view showing travel traffic lanes according to an embodiment of the present invention.

FIG. 7 is a view showing an image and traffic lanes displayed on a display unit according to an embodiment of the present invention.

FIG. 8 is a flow chart illustrating a process of a traffic lane recognizing method according to an embodiment of the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains, and should not be interpreted as having an excessively comprehensive meaning nor as having an excessively contracted meaning. If technical terms used herein is erroneous that fails to accurately express the technical idea of the present invention, it should be replaced with technical terms that allow the person in the art to properly understand. The general terms used herein should be interpreted according to the definitions in the dictionary or in the context and should not be interpreted as an excessively contracted meaning.

In the present application, it is to be understood that the terms such as "including" or "having," etc., are intended to indicate the existence of the features, numbers, operations, actions, components, parts, or combinations thereof disclosed in the specification, and are not intended to preclude the possibility that one or more other features, numbers, operations, actions, components, parts, or combinations thereof may exist or may be added.

While terms such as "first" and "second," etc., may be used to describe various components, such components must not be understood as being limited to the above terms. The above terms are used only to distinguish one component from another. For example, a first component may be referred to as a second component without departing from the scope of rights of the present invention, and likewise a second component may be referred to as a first component.

The exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which like numbers refer to like elements throughout.

In describing the present invention, if a detailed explanation for a related known function or construction is considered to unnecessarily divert the gist of the present invention, such explanation has been omitted but would be understood by those skilled in the art. The accompanying drawings of the present invention aim to facilitate understanding of the present invention and should not be construed as limited to the accompanying drawings.

Hereinafter, the configuration of a traffic lane recognizing apparatus according to an embodiment of the present invention will be described with reference to FIG. 1. Here, the traffic lane recognizing apparatus illustrated in FIG. 1 may be configured as a stand alone device or may be applicable to various terminals such as a mobile terminal, a telematics terminal, a smart phone, a portable terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, a Wibro terminal, a navigation terminal, an AVN (Audio Video Navigation) terminal, and the like.

As shown in FIG. 1, a traffic lane recognizing apparatus 10 according to an embodiment of the present invention includes a camera module 110, a display unit 130 configured to display an image captured by the camera module 110, and a controller 120 configured to detect candidate traffic lanes based on traffic lane conformance from among traffic lanes detected by substituting different traffic lane equations (or curve equations) in the image captured by the camera module 110 and display traffic lanes adjacent to a vehicle (e.g., a traffic lane adjacent to the left side of the vehicle and a traffic lane adjacent to the right side of the vehicle) as current traffic lanes along which the vehicle is traveling (or moving) on, among the detected candidate traffic lanes, on the image. The traffic lane conformance may be previously determined based on the sum of differences in distances between a traffic lane obtained by substituting the traffic lane feature points to the different traffic lane equations, and the traffic lane feature points.

The controller 120 may detect traffic lanes, whose difference in distance between themselves and the traffic lane feature points is a threshold value or smaller, as the candidate traffic lanes, from among the traffic lanes detected based on the traffic lane feature points detected from the image captured by the camera module and the different traffic lane equations. For example, the controller 120 may detect traffic lane feature points (support points) from the captured image, and sequentially (e.g., in order from a low-order traffic lane equation (a linear traffic lane equation) to a high-order traffic lane equation (a cubic traffic lane equation) or in order from the high-order traffic lane equation (the cubic traffic lane equation) to the low-order traffic lane equation (the linear traffic lane equation)) substitute different traffic lane equations for detecting a traffic lane (a straight line and/or a curved line) to the detected traffic lane feature points, to thereby detect traffic lanes whose difference in distance is smaller than the threshold value (or traffic lanes having a high traffic lane conformance (e.g., 90% or more)), as the candidate traffic lanes, from among the detected traffic lanes.

The components of the traffic lane recognizing apparatus 10 illustrated in FIG. 1 are not all essential components; the traffic lane recognizing apparatus 10 may be implemented by more components or less components.

As shown in FIG. 2, the camera module 110 receives an image 310 captured by a single camera. For example, the camera module 110 may receive an image 210 including traffic lanes corresponding to first road, second road, third road, and the like.

The controller 120 receives the image 210 through the camera module 110, and extracts feature points of traffic lanes within the captured image 210 based on pre-set guide lines for extracting the support points (e.g., traffic lane feature points) from the image 210. Here, as for the guide lines, as shown in FIG. 3, when a lower portion of the image 210 is converted into world coordinates, it represents a closer region, and when middle and upper portions of the image 210 are converted into world coordinates, they represent a distant region. Thus, in order to obtain point intervals as uniform as possible when the data of the image 210 is converted into the world coordinates, the interval between the guide lines 310 is set to be large at the lower portion of the image 210, and the interval between the guide lines 310 set to be gradually narrowed toward the upper portion of the image 210. Here, a change width of the interval between lines of the guide lines 410 may be variably set according to a design of a designer and may be set to maintain the equal interval between lines when the data of the image 310 is converted into world coordinates. The guide line refers to a virtual line used to obtain a point interval as uniform as possible when the support points (feature points) are converted into the world coordinates, rather than being actually displayed on the image.

As shown in FIG. 4, the controller 120 extracts a plurality of traffic lane feature points from the image 310 based on the pre-set guide lines, and displays the plurality of extracted traffic lane feature points 410 on an image domain of the display unit 130. Namely, the controller 120 displays the traffic lane feature points corresponding to the traffic lanes 401 on the image domain. Here, the interval between the plurality of traffic lane feature points in a vertical direction based on the horizontal axis (x axis) is gradually narrowed from a lower side to an upper side of the display unit 130 in the vertical direction.

The controller 120 converts the plurality of extracted traffic lane feature points into world coordinates. Namely, the controller 120 may convert the plurality of extracted traffic lane feature points into world coordinates by using a conversion matrix (e.g., a homographic matrix), or the like) previously stored in the storage unit 140.

For example, as shown in FIG. 5, the controller 120 converts the plurality of extracted traffic lane feature points into world coordinates based on the homographic matrix previously stored in the storage unit 140, and displays a plurality of traffic lane feature points 510 which have been converted into the world coordinates on the display unit 130. Here, based on the horizontal axis, the interval between the plurality of traffic lane feature points which have been converted into the world coordinates in the vertical direction is maintained to be equal.

The controller 120 may detect (or check) the plurality of traffic lane feature points corresponding to the curve among the traffic lane feature points which have been converted into the world coordinates based on the traffic lane feature points which have been converted into the world coordinates and the linear, quadratic, and cubic, n-order equations previously stored in the storage unit 140. Namely, the controller 120 may sequentially substitute the plurality of traffic lane feature points which have been converted into the world coordinates to the curve equations previously stored in the storage unit 140 and determine (or check) whether the plurality of traffic lane feature points which have been converted into the world coordinates make a curve based on the substitution results. Here, the curve equation may be a linear, quadratic, and cubic, or n-order equation.

The controller 120 substitutes the plurality of traffic lane feature points which have been converted into the world coordinates to a linear traffic lane equation (e.g., y=c₁x+c₀, wherein c₁ is a tilt (or a heading angle of the vehicle), and c₀ is an offset between the traffic lane and the vehicle) previously stored in the storage unit 140. When c₁=0, the controller 120 recognizes the plurality of traffic lane feature points as a straight line, and when c ₁ 0, the controller 120 recognizes the plurality of traffic lane feature points as a curve (traffic lane detection). Namely, the linear traffic lane equation may be used to detect a traffic lane on a general straight road.

The controller 120 substitutes the plurality of traffic lane feature points which have been converted into the world coordinates to a quadratic traffic lane equation (e.g., y=c₂x²+c₁x+c₀, wherein c₂ is a curvature of the traffic lane, c₁ is a tilt of the vehicle (or a heading angle of the vehicle), and c₀ is an offset between the traffic lane and the vehicle) previously stored in the storage unit 140. When c₂=0, the controller 120 recognizes the plurality of traffic lane feature points as a straight line, and when c ₂ 0, the controller 120 recognizes the plurality of traffic lane feature points as a curve (traffic lane detection). Namely, the quadratic traffic lane equation may be used to detect a traffic lane on a curved road in which a curvature radius of a traffic lane is large and a curvature change is uniform.

The controller 120 substitutes the plurality of traffic lane feature points which have been converted into the world coordinates to a cubic traffic lane equation (e.g., y=c₃x³+ c₂x²+c₁x+c₀, wherein c₃ is a curve derivative, c₂ is a curvature of the traffic lane, c₁ is a tilt of the vehicle (or a heading angle of the vehicle), and c₀ is an offset between the traffic lane and the vehicle) previously stored in the storage unit 140, to check whether or not there is a traffic lane (curve) (traffic lane detection). When c₃=0, in the cubic traffic lane equation, c₂ is a curvature of the traffic lane, c₁ is a tilt of the vehicle (or a heading angle of the vehicle), and c₀ is an offset between the traffic lane and the vehicle, and when both c₃ and c₂ are 0, it indicates a linear detection, and c₁ is a heading angle of the vehicle and c₀ is an offset between the traffic lane and the vehicle. Namely, the cubic curve equation may be used to detect a traffic lane on a curved road in which a curvature radius of a traffic lane is small and a curvature change is not uniform. c₁ may be a direction of a traffic lane and a direction of the vehicle or a direction of the traffic lane with respect to the vehicle.

The controller 120 may check whether or not there is a traffic lane (curve) by substituting the plurality of traffic lane feature points which have been converted into the world coordinates to nth-order equation (e.g., y=c_nxⁿ+c_n-1x^n-1 +,....., +c₃x³+c₂x²+c₁x+c₀). Namely, the nth-order equation may be used to detect a traffic lane on a curved road which can be hardly modeled by the (n-)th-order equation

The controller 120 detects a traffic lane by substituting the traffic lane feature points which have been converted into the world coordinates to the curve equations (e.g., the linear equation) previously stored in the storage unit 140, and determines whether or not a difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller than the pre-set threshold value (reference distance value or a traffic lane conformance). When the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller than the pre-set threshold value (e.g., 1 to 5 cm), the controller 120 displays the traffic lane detected according to the linear equation, on the image.

When the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is equal to or greater than the pre-set threshold value (e.g., 5 to 10 cm), the controller 120 substitutes the traffic lane feature points which have been converted into the world coordinates to the curve equations (e.g., the quadratic equation) previously stored in the storage unit 140, and determines whether or not the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller, or equal to or greater than the pre-set threshold value (reference distance value or a traffic lane conformance). When the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller than the pre-set threshold value, the controller 120 displays the traffic lane detected according to the quadratic equation, on the image.

When the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is equal to or greater than the pre-set threshold value, the controller 120 substitutes the traffic lane feature points which have been converted into the world coordinates to the curve equations (e.g., the cubic equation) previously stored in the storage unit 140, and determines whether or not the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller, or equal to or greater than the pre-set threshold value (reference distance value or a traffic lane conformance). When the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller than the pre-set threshold value, the controller 120 displays the traffic lane detected according to the cubic equation, on the image.

Based on the traffic lane conformance obtained based on the sum of differences in distances between the traffic lane feature points and the traffic lane according to the substitution result, the controller 120 may detect a traffic lane having high traffic lane conformance from among the curve equations (e.g., the linear, the quadratic, and the cubic equations) previously stored in the storage unit 140.

The traffic lane conformance may be calculated in various manners, and may be calculated through sum of squares according to Equation 1 shown below:

[Equation 1]

SSE=

Here, n is the number of traffic lane feature points used for a traffic lane detection, and y_i is y value of ith traffic lane feature point,

is a y value of an equation estimated with respect to x value of the ith traffic land feature point, and w_i is a weight value with respect to each traffic lane feature point.

The controller may additionally use the number of the traffic lane feature points, the range in which the traffic lane feature points are distributed, and the like, in order to determine the conformance of a model on the current road. The controller 120 may detect the candidate traffic lanes by substituting traffic lane feature points whose distribution range is equal to or greater than a pre-set range to the different traffic lane equations. Here, the pre-set reference number and the pre-set range may be variably defined by a designer.

The controller 120 may detect traffic lanes in real time by tracking traffic lane feature points substituted to the curve equations (e.g., the linear equation) previously stored in the storage unit 140, and display the detected traffic lanes.

The controller 120 may also calculate the curve information that follows a virtual central point of a traffic lane based on the plurality of points corresponding to the detected traffic lane (a straight line or a curve). Here, the calculated curve information may be used to enhance traffic lane maintaining performance on the world coordinates by minimizing the influence of a calibration state of the camera. Namely, the controller 120 may calculate curve information following the central point of the traffic lane by applying any one of a least square method, a random sample consensus (RANSAC), a general hough transform method, a spline interpolation method, and the like, with respect to the plurality of points corresponding to the detected curve.

The controller 120 may overlap the calculated curve information following the central point of the traffic lane, the information such as the detected curve, or the like, with the captured image, and display the same on the display unit 130. For example, the controller 120 may convert (or map) the calculated curve information following the central point of the traffic lane and the detected curve information, or the like, into coordinates on an image domain, respectively, overlap the respective converted coordinates with the captured image, and display the same on the display unit 130.

As shown in FIG. 6, the controller 120 may select a first traffic lane most adjacent to the left side of the vehicle and a second traffic lane most adjacent to the right side of the vehicle based on a moving direction of the vehicle from the detected candidate traffic lanes, and display the first traffic lane and the second traffic lane as travel traffic lanes 610 of the vehicle on the image 210. For example, the controller 120 may convert (or map) the detected first traffic lane and the second traffic lane 610 into coordinates on the image domain, and overlap the respective converted coordinates with the image 210.

FIG. 7 is a view showing travel traffic lanes displayed on an image according to an embodiment of the present invention.

As shown in FIG. 7, the controller 120 may select a first traffic lane most adjacent to the left side of the vehicle and a second traffic lane most adjacent to the right side of the vehicle based on a moving direction of the vehicle from the detected candidate traffic lanes, detects the first traffic lane and the second traffic lane as travel traffic lanes 610 of the vehicle, and overlap the travel traffic lanes 610 with the image.

The camera module 110 may include at least a pair of cameras (e.g., a stereo camera, a stereoscopic camera), installed to be spaced apart horizontally on the same plane of the traffic lane recognizing apparatus 10, or a single camera. Here, the fixed horizontal interval may be set in consideration of the distance between ordinary humans two eyes. Also, the camera module 110 may be any camera modules that can capture an image.

The camera module 110 may receive a first image (e.g., a left image captured by a left camera included in the pair of cameras) and a second image (e.g., a right image captured by a right camera included in the pair of cameras) which are simultaneously captured by the pair of cameras.

The camera module 110 may be an image sensor such as a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like.

When the traffic lane recognizing apparatus 10 is installed in a vehicle, the camera module 110 may be fixed to a certain position (e.g., a room mirror of the vehicle) of the vehicle to capture an image of a front side in the traveling direction of the vehicle. The camera module 110 may be fixedly installed at certain positions (e.g., a side mirror of the vehicle, a rear bumper of the vehicle) in order to capture images of the side and the rear side of the vehicle.

The controller 120 performs functions (including a traffic lane deviation warning message function, an automatic traffic lane maintaining function, and the like) in relation to maintaining a traffic lane based on the position of the traffic lane recognizing apparatus 10 (or a vehicle including the traffic lane recognizing apparatus) and the detected curve (or the traffic lane) checked through a certain GPS module (not shown).

The display unit 130 displays various contents such as various menu screen images, or the like, by using a user interface and/or a graphic user interface included in the storage unit 140 under the control of the controller 120. Here, the contents displayed on the display unit 130 includes menu screen images such as various text or image data (including various information data) and data such as an icon, a list menu, a combo box, and the like.

The display unit 130 includes a 3D display or a 2D display. Also, the display unit 130 may include at least one of a Liquid Crystal Display (LCD), a Thin Film Transistor-LCD (TFT-LCD), an Organic Light Emitting Diode (OLED) display, a flexible display, and an LED (Light Emitting Diode).

The display unit 130 displays the 3D image (or a 2D image) under the control of the controller 120.

The traffic lane recognizing apparatus 10 may include two or more display units 130 according to its particular desired embodiment. For example, a plurality of display units may be separately or integrally disposed on a single face (the same surface) of the traffic lane recognizing apparatus 10, or may be disposed on mutually different faces of the traffic lane recognizing apparatus 10.

Meanwhile, when the display unit 130 and a sensor sensing a touch operation (referred to as a 'touch sensor', hereinafter) are overlaid in a layered manner (referred to as a 'touch screen', hereinafter), the display unit 130 may function as both an input device and an output device. The touch sensor may have the form of, for example, a touch film, a touch sheet, a touch pad, a touch panel, and the like.

The touch sensor may be configured to convert the pressure applied to a particular portion of the display unit 130 or a change in capacitance generated at a particular portion of the display unit 130 into an electrical input signal. Also, the touch sensor may be configured to detect a touch input pressure as well as a touch input position and a touch input area. When there is a touch input with respect to the touch sensor, the corresponding signal(s) are sent to a touch controller (not shown). The touch controller processes the signal(s) and transmits corresponding data to the controller 120. Accordingly, the controller 120 can recognize a touched region of the display unit 151.

The display unit 130 may include a proximity sensor. The proximity sensor may be disposed in an internal region of the traffic lane recognizing apparatus 10 covered by the touch screen or in the vicinity of the touch screen.

The proximity sensor may be disposed within the mobile terminal covered by the touch screen or near the touch screen. The proximity sensor refers to a sensor for detecting the presence or absence of an object that accesses a certain detect surface or an object that exists nearby by using the force of electromagnetism or infrared rays without a mechanical contact. Thus, the proximity sensor has a longer life span compared with a contact type sensor, and it can be utilized for various purposes. Examples of the proximity sensor may include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror-reflection type photoelectric sensor, an RF oscillation type proximity sensor, a capacitance type proximity sensor, a magnetic proximity sensor, an infrared proximity sensor, and the like. When the touch screen is an electrostatic type touch screen, an approach of the pointer is detected based on a change in an electric field according to the approach of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Recognition of a pointer positioned to be close to the touch screen without being contacted may be called a proximity touch , while recognition of actual contacting of the pointer on the touch screen may be called a 'contact touch'. In this case, when the pointer is in the state of the proximity touch, it means that the pointer is positioned to correspond vertically to the touch screen.

The proximity sensor detects a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, or the like), and information corresponding to the detected proximity touch operation and the proximity touch pattern can be outputted to the touch screen.

When the display unit 130 is used as an input device, it may receive a user s button manipulation or receive a command or a control signal generated according to a manipulation such as touching/scrolling a displayed screen image.

The traffic lane recognizing apparatus 10 according to an embodiment of the present invention may include the storage unit 140 storing a program for detecting the image and the traffic lane, traffic lane with information calculated periodically or in real time, and the like.

The storage unit 140 may further store various menu screen images, a user interface (UIs), and/or a graphic user interface (GUI).

The storage unit 140 may further store mathematical equations such as a conversion matrix (e.g., homographic matrix, and the like), a curve equation, the least square method, and the like.

The storage unit 140 may further store data, programs, and the like, required for operating the traffic lane recognizing apparatus 10.

The storage unit 140 may include at least one type of storage mediums including a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., SD or DX memory, etc), a Read-Only Memory (ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Programmable Read-Only memory (PROM), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a magnetic memory, a magnetic disk, and an optical disk.

The traffic lane recognizing apparatus 10 may further include a communication unit (not shown) performing a communication function with a certain terminal or a server under the control of the controller 120. Here, the communication unit may include a wired/wireless communication module. Here, a wireless Internet technique may include a wireless local area network (WLAN), Wi-Fi, wireless broadband (WiBro), world interoperability for microwave access (WiMAX), high speed downlink packet access (HSDPA), IEEE802.16, long-term evolution (LTE), a wireless mobile broadband service (WMBS), and the like, and a short-range communication technology include Bluetooth^TM, Radio Frequency IDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand (UWB), ZigBee^TM, and the like. Also, the wired communication technique may include USB (Universal Serial Bus) communication, and the like.

The communication unit may include CAN communication, vehicle Ethernet, flexray, LIN (Local Interconnect Network), and the like, for communication with a certain vehicle in which the traffic lane recognizing apparatus 10 is provided.

The communication unit may transmit curve information, or the like, that follows a central point of a traffic lane calculated based on a plurality of support points extracted from a certain image under the control of the controller 120, points obtained by converting the plurality of support points into world coordinates, a plurality of points corresponding to a curve among points which have been converted into the world coordinates, and a plurality of curves corresponding to the curve, to the certain terminal or server.

The communication unit may receive a first image and a second image, which were simultaneously captured by a pair of stereo cameras, transmitted from the certain terminal or server.

The traffic lane recognizing apparatus 10 may further include an input unit (not shown) including one or more microphones (not shown) for receiving an audio signal.

The microphone receives an external audio signal (including a user s voice (voice signal or voice information)) in a phone call mode, a recording mode, a voice recognition mode, and the like, and processes the audio signal into electrical voice data. The processed voice data processed by the microphone may be output through a voice output unit (not shown) or converted into a format that is transmittable and output to an external terminal through the communication unit. The microphone may implement various types of noise canceling algorithms to cancel noise generated in a procedure of receiving the external audio signal.

The input unit receives a signal according to a user's button manipulation or receives a command or a control signal generated according to a manipulation such as touching/scrolling a displayed screen image.

The input unit receives a signal corresponding to information input by the user, and as the input unit, various devices such as a keyboard, a keypad, a dome switch, a touch pad (pressure/capacitance), a touch screen, a jog shuttle, a jog wheel, a jog switch, a mouse, a stylus, pen, a touch pen, a laser pointer, and the like, may be used. Here, the input unit receives signals corresponding to inputs by various devices.

The traffic lane recognizing apparatus 10 may further include a voice output unit (not shown) outputting voice information included in the signal processed by the controller 120. Here, the voice output unit may be a speaker.

In the traffic lane recognizing apparatus and method according to an embodiment of the present invention, support points (feature points) as a candidate group of a traffic lane are extracted from an image and converted into world coordinates, and a traffic lane is recognized on the converted world coordinates. Thus, a possibility of an accumulated error can be reduced compared with the method of directly recognizing a traffic lane from an image in an error transition of calibration between camera information and the world coordinates.

In the traffic lane recognizing apparatus and method according to an embodiment of the present invention, information regarding a traffic lane recognized from the world coordinates is displayed, based on which a warning message is generated and output, thereby enhancing accuracy/sensitivity and user convenience.

First, the camera module 110 receives a first image and a second image captured by at least a pair of cameras (e.g., a stereo camera or a stereoscopic camera) installed separately by a horizontal interval on the same central axis of the same surface of the traffic lane recognizing apparatus 10, or receives an image captured by a single camera. Here, the first image may be a left image captured by a left camera included in the pair of cameras and the second image may be a right image captured by a right camera included in the pair of cameras. Also, the camera module 110 may receive any one of the first image and the second image captured by the pair of cameras.

The camera module 110 receives the image 210 captured by a single camera. For example, the camera module 110 receives an image 310 captured by a single camera. For example, the camera module 110 may receive the image 210 including traffic lanes corresponding to a first lane, a second lane, a third lane, and the like, and/or double lines of white or yellow solid lines (or double lines of white or yellow solid lines and dotted lines).

The controller 120 receives the image 210 through the camera module 110 (S11), and extracts a plurality of feature points of a traffic lane from the captured image 210 based on pre-set guide lines for detecting (extracting) traffic lane feature points from the image 210. Here, as for the guide lines, as shown in FIG. 4, when a lower portion of the image 210 is converted into world coordinates, it represents a closer region, and when middle and upper portions of the image 210 are converted into world coordinates, they represent a distant region. Thus, in order to obtain point intervals as uniform as possible when data of the image 210 is converted into the world coordinates, the lower portion of the image 210 is set to have a wider interval between lines and the interval between lines of the guide lines 310 is gradually narrowed toward the upper portion of the image 210. Here, a change width of the interval between lines of the guide lines 310 may be variably set according to a design of a designer and may be set to maintain the equal interval between lines when the data of the image 210 is converted into world coordinates. The guide line refers to a virtual line used to obtain a point interval as uniform as possible when the traffic lane feature points are converted into the world coordinates, rather than being actually displayed on the image.

As shown in FIG. 5, the controller 120 extracts a plurality of traffic lane feature points from the image 210 based on the pre-set guide lines, and displays the plurality of extracted traffic lane feature points 510 on an image domain of the display unit 130. Namely, the controller 120 displays the traffic lane feature points corresponding to the traffic lanes 501 and the traffic lane feature points corresponding to the traffic lanes 501 on the image domain. Here, the interval between the plurality of traffic lane feature points in a vertical direction based on the horizontal axis (x axis) is gradually narrowed from a lower side to an upper side of the display unit 130 in the vertical direction.

For example, as shown in FIG. 6, the controller 120 converts the plurality of extracted traffic lane feature points into world coordinates based on the homographic matrix previously stored in the storage unit 140, and displays a plurality of traffic lane feature points 610 which have been converted into the world coordinates on the display unit 130. Here, based on the horizontal axis, the interval between the plurality of traffic lane feature points which have been converted into the world coordinates in the vertical direction is maintained to be equal.

The controller 120 detects traffic lanes based on the plurality of traffic lane feature points which have been converted into the world coordinates and the curve equation previously stored in the storage unit 140 (S12). Namely, the controller 120 may detect the traffic lanes by substituting the plurality of traffic lane feature points which have been converted into the world coordinates to the curve equations previously stored in the storage unit 140 and determining (or checking) whether the plurality of traffic lane feature points which have been converted into the world coordinates make a straight line or a curve based on the substitution result. Here, the curve equation may be a linear equation, a quadratic equation, or a cubic equation, or higher.

The controller 120 detects the candidate traffic lanes from the detected traffic lanes based on the traffic lane conformance (S13). For example, the controller 120 detects traffic lanes having a high traffic lane conformance (e.g., traffic lanes having a small difference in distance) as the candidate traffic lanes, from among the detected traffic lanes. Also, among the traffic lanes detected based on the traffic lane feature points detected from the image captured by the camera module and the different traffic lane equations, the controller may detect traffic lanes, whose difference in distance between themselves and the traffic lane feature points is smaller than a threshold value, as the candidate traffic lanes.

When the controller 120 detects a traffic lane by substituting the traffic lane feature points which have been converted into the world coordinates to the linear equation previously stored in the storage unit 140, if the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is equal to or greater than the pre-set threshold value (reference distance value), the controller 120 detects the traffic lane by substituting the traffic lane feature points which have been converted into the world coordinates to the quadratic equation previously stored in the storage unit 140, and if the difference in distances between the traffic lane feature points and the traffic lane obtained from the quadratic equation substitution result is equal to or greater than the pre-set threshold value, the controller 120 detects a traffic lane by substituting the traffic lane feature points which have been converted into the world coordinates to the cubic equation previously stored in the storage unit 140. Meanwhile, if the differences in distances between traffic lanes according to the linear to cubic traffic lane equations and the traffic lane feature points which have been converted into the world coordinates are all equal to or greater than the pre-set threshold value, the controller 120 may determine traffic lanes, whose difference in distances between the traffic lanes according to the linear to cubic traffic lane equations and the traffic lane feature points which have been converted into the world coordinates are the smallest, as the candidate traffic lanes.

The controller 120 determines a first traffic lane most adjacent to the left side of the vehicle and a second traffic lane most adjacent to the right side of the vehicle based on the direction in which the vehicle is traveling on, as travel traffic lanes of the vehicle, among the detected candidate traffic lanes (S14).

The controller 120 displays the detected travel traffic lanes on the image (S15). For example, the controller 120 converts (or maps) the detected travel traffic lanes into coordinates on the image domain, respectively, and overlaps the respective converted coordinates on the image 210.

As described above, in the traffic lane recognizing apparatus and method according to embodiments of the present invention, traffic lanes are detected based on traffic lane conformance from among traffic lanes detected based on the traffic lane feature points detected from an image captured by a camera module and different traffic lane equations. Thus, traffic lanes can be accurately detected.

Claims

A traffic lane recognizing apparatus comprising:

a camera module;

a display unit configured to display an image captured by the camera module; and

a controller configured to detect candidate traffic lanes based on traffic lane conformance from among traffic lanes detected based on traffic lane feature points detected from the image captured by the camera module and different traffic lane equations and display traffic lanes adjacent to a vehicle, as current traffic lanes along which the vehicle is traveling on, among the detected candidate traffic lanes, on the image.
The apparatus of claim 1, wherein the traffic lane conformance is previously determined based on the sum of differences in distances between a traffic lane obtained by substituting the traffic lane feature points to the different traffic lane equations, and the traffic lane feature points.
The apparatus of claim 2, wherein the traffic lane conformance is determined based on the sum of differences in distances between a traffic lane obtained by substituting the traffic lane feature points to the different traffic lane equations, and the traffic lane feature points, the number of the traffic lane feature points, and a range in which the traffic lane feature points are distributed.
The apparatus of claim 1, wherein the controller detects traffic lanes, whose difference in distance between themselves and the traffic lane feature points is a threshold value or smaller, as the candidate traffic lanes, from among the traffic lanes detected based on the traffic lane feature points detected from the image captured by the camera module and the different traffic lane equations.
The apparatus of claim 1, wherein the controller detects the traffic lane feature points from the captured image, and sequentially substitutes the different traffic lane equations to the detected traffic lane feature points to thereby detect traffic lanes, whose difference in distance is smaller than the threshold value, as the candidate traffic lanes, from among the detected traffic lanes.
The apparatus of claim 1, wherein the controller detects traffic lane feature points from the captured image, convert the traffic lane feature points into world coordinates, and sequentially substitutes the traffic lane feature points which have been converted into the world coordinates to the different traffic lane equations, to thereby detect traffic lanes, whose difference in distance is smaller than the threshold value, as the candidate traffic lanes, from among the detected traffic lanes.
The apparatus of claim 1, wherein the different traffic lane equations are linear, quadratic, and cubic traffic lane equations, and the linear traffic lane equation is y=c₁x+c₀ wherein c₁ is a heading angle of the vehicle and c₀ is an offset between the traffic lane and the vehicle, the quadratic traffic lane equation is y=c₂x²+c₁x+c₀ wherein c₂ is a curvature of the traffic lane, c₁ is a heading angle of the vehicle, and c₀ is an offset between the traffic lane and the vehicle, and the cubic traffic lane equation is y=c₃x³+ c₂x²+c₁x+c₀ wherein c₃ is a curve derivative, c₂ is a curvature of the traffic lane, c₁ is a heading angle of the vehicle, and c₀ is an offset between the traffic lane and the vehicle.
The apparatus of claim 1, wherein when the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is equal to or greater than the pre-set threshold value, the controller converts the traffic lane feature points into world coordinates, and substitutes the traffic lane feature points to any one of the different traffic lane curve equations, and when the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller than the pre-set threshold value, the controller displays the traffic lane detected according to the quadratic equation, on the image.
The apparatus of claim 8, wherein the any one traffic lane curve equation is a quadratic traffic lane equation which is y=c₂x²+c₁x+c₀ wherein c₂ is a curvature of the traffic lane, c₁ is a heading angle of the vehicle, and c₀ is an offset between the traffic lane and the vehicle.
A traffic lane recognizing method comprising:

displaying an image captured by a camera module on a display unit;

detecting candidate traffic lanes based on traffic lane conformance from among traffic lanes detected based on traffic lane feature points detected from the image captured by the camera module and different traffic lane equations; and

displaying traffic lanes adjacent to a vehicle as current traffic lanes along which the vehicle is traveling on, among the detected candidate traffic lanes, on the image.
The method of claim 10, wherein the traffic lane conformance is previously determined based on the sum of differences in distances between a traffic lane obtained by substituting the traffic lane feature points to the different traffic lane equations, and the traffic lane feature points.
The method of claim 11, wherein the traffic lane conformance is determined based on the sum of differences in distances between a traffic lane obtained by substituting the traffic lane feature points to the different traffic lane equations, and the traffic lane feature points, the number of the traffic lane feature points, and a range in which the traffic lane feature points are distributed.
The method of claim 10, wherein the detecting of the candidate traffic lanes comprises detecting traffic lanes, whose difference in distance between themselves and the traffic lane feature points is a threshold value or smaller, as the candidate traffic lanes, from among the traffic lanes detected based on the traffic lane feature points detected from the image captured by the camera module and the different traffic lane equations.
The method of claim 10, wherein the detecting of the candidate traffic lanes comprises:

detecting the traffic lane feature points from the captured image; and

detecting traffic lanes, whose difference in distance is smaller than the threshold value, as the candidate traffic lanes, from among the detected traffic lanes detected by sequentially substituting the different traffic lane equations to the detected traffic lane feature points.
The method of claim 10, wherein the detecting of the candidate traffic lanes comprises:

detecting traffic lane feature points from the captured image;

converting the traffic lane feature points into world coordinates; and

sequentially substituting the traffic lane feature points which have been converted into the world coordinates to the different traffic lane equations, to thereby detect traffic lanes, whose difference in distance is smaller than the threshold value, as the candidate traffic lanes, from among the detected traffic lanes.
The method of claim 10, wherein the different traffic lane equations are linear, quadratic, and cubic traffic lane equations, and the linear traffic lane equation is y=c₁x+c₀ wherein c₁ is a heading angle of the vehicle and c₀ is an offset between the traffic lane and the vehicle, the quadratic traffic lane equation is y=c₂x²+c₁x+c₀ wherein c₂ is a curvature of the traffic lane, c₁ is a heading angle of the vehicle, and c₀ is an offset between the traffic lane and the vehicle, and the cubic traffic lane equation is y=c₃x³+ c₂x²+c₁x+c₀ wherein c₃ is a curve derivative, c₂ is a curvature of the traffic lane, c₁ is a heading angle of the vehicle, and c₀ is an offset between the traffic lane and the vehicle.
The method of claim 10, wherein the detecting of the candidate traffic lanes comprises:

when the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is equal to or greater than the pre-set threshold value, converting the traffic lane feature points into world coordinates;

substituting the traffic lane feature points to any one of the different traffic lane curve equations; and

when the difference in distances between the traffic lane feature points and the traffic lane obtained from the substitution result is smaller than the pre-set threshold value, displaying the traffic lane detected according to the quadratic equation, on the image.
The method of claim 17, wherein the any one traffic lane curve equation is a quadratic traffic lane equation which is y=c₂x²+c₁x+c₀ wherein c₂ is a curvature of the traffic lane, c₁ is a heading angle of the vehicle, and c₀ is an offset between the traffic lane and the vehicle.