KR101257871B1 - Apparatus and method for detecting object based on vanishing point and optical flow - Google Patents

Abstract

The present specification minimizes an error due to a pitch motion of a camera based on at least one of a vanishing point detected in at least one image among a plurality of images acquired with parallax through the camera and an optical flow existing between the plurality of images. Then, a detection apparatus and a detection method for detecting any object are provided.
To this end, the detection method according to an embodiment includes a method of detecting any dynamic or static object, the method comprising: obtaining a plurality of images with parallax through a camera; And detecting the arbitrary object through at least one of the plurality of images, wherein when a pitch motion with respect to the camera occurs while acquiring the plurality of images, at least one of the plurality of images. The detection of the arbitrary object may be performed by minimizing an error caused by the pitch motion of the camera based on at least one of a vanishing point detected in an image of the optical signal and a light flow existing between the plurality of images.

Description

Detecting device and detection method based on vanishing point and optical flow {APPARATUS AND METHOD FOR DETECTING OBJECT BASED ON VANISHING POINT AND OPTICAL FLOW}

The present disclosure relates to an apparatus and a method for detecting an object through a camera.

Recently, development of a detection apparatus for detecting an object such as a lane, a signal, a vehicle, or other indices through a camera mounted on a vehicle is actively progressing. Information about the object detected through such a detection device may be used to drive the vehicle.

However, while the vehicle is driving, pitch motion may occur according to the state of the road surface. Pitch motion of a vehicle refers to an up-down movement of the vehicle due to a moving road surface condition.

The pitch motion of such a vehicle may cause an error in the detection device detecting the object. Thus, there is a need for a technique that minimizes errors for the pitch motion.

An existing correction method for pitch motion may include a method of changing a photographing direction of a camera mounted on a vehicle or a position of the camera in a reverse direction of the pitch motion.

However, in the method of changing the photographing direction or position of the camera, the response speed to the pitch motion is slow, and the movement of the vehicle is larger than that of the general camera. There may be a problem that the correction for the pitch motion becomes inaccurate.
{Republic of Korea Patent No. 10-0746351 "Method of measuring real-time height using geometric information (2007.07.30)",
Republic of Korea Patent No. 10-0666276, "lane departure warning method of a vehicle using a vanishing point (2007.01.03)", etc.}

The present disclosure provides a method of minimizing the influence of the pitch motion of the camera based on at least one of a vanishing point detected in at least one image among a plurality of images acquired with parallax through the camera, and an optical flow existing between the plurality of images. It is an object of the present invention to provide a detection apparatus and a detection method capable of detecting any object accurately and stably.

A detection method according to the present disclosure for achieving the above object, the method of detecting any dynamic or static object, comprising the steps of: obtaining a plurality of images with a parallax through a camera; And detecting the arbitrary object through at least one of the plurality of images, wherein when a pitch motion with respect to the camera occurs while acquiring the plurality of images, at least one of the plurality of images. The detection of the arbitrary object may be performed by minimizing an error caused by the pitch motion of the camera based on at least one of a vanishing point detected in an image of the light and a light flow existing between the plurality of images.

As an example related to the present specification, the minimization of the error may include extracting information on the pitch motion of the camera based on at least one of the vanishing point and the optical flow, and extracting the pitch motion of the camera. It may be made based on the information about.

As an example related to the present specification, the information about the pitch motion of the camera may be at least one of a direction of the pitch motion and an angle of the pitch motion.

As an example related to the present specification, the information about the pitch motion of the camera includes at least one of first information extracted based on the vanishing point and second information extracted based on the optical flow, and minimizes the error. May be based on at least one of the first information and the second information.

As an example related to the present specification, the information about the pitch motion of the camera may be extracted based on a value obtained by multiplying a weight corresponding to each of the first information and the second information.

As an example related to the present specification, detecting the arbitrary object may include extracting a support point that is a candidate group of the arbitrary object in the captured image; Converting the support points to world coordinates; And detecting the arbitrary object on the converted world coordinates, and minimizing the error may be performed by converting the support point into world coordinates based on at least one of the vanishing point and the optical flow.

As an example related to the present specification, the conversion of the support point into world coordinates is based on a transformation matrix, and the minimization of the error is based on at least one of the vanishing point and the optical flow coefficient of the transformation matrix. It may be made by changing the.

As an example related to the present specification, the camera may be mounted on a vehicle, and the arbitrary object may have a relative speed compared to the moving speed of the vehicle.

As an example related to the present specification, the parallax for acquiring the plurality of images may be variable according to the speed of the vehicle.

As an example related to the present specification, the pitch motion of the camera may be caused by acceleration or deceleration of the vehicle or a state of a road surface on which the vehicle travels.

As an example related to the present specification, the minimization of the error may be continued until the pitch motion of the camera due to the elastic modulus and the damping coefficient of the shock absorber of the vehicle is minimized.

A detection apparatus according to the present specification for achieving the above objects, the device for detecting any dynamic or static object, the camera module for obtaining a plurality of images at a parallax through the camera; And a controller configured to detect the arbitrary object through at least one of the plurality of images, wherein when a pitch motion with respect to the camera occurs while acquiring the plurality of images, at least one of the plurality of images. The detection of the arbitrary object may be performed by minimizing an error caused by the pitch motion of the camera based on at least one of a vanishing point detected in an image of the light and a light flow existing between the plurality of images.

As an example related to the present specification, minimizing the error may be performed by extracting information on the pitch motion of the camera based on at least one of the vanishing point and the optical flow, and based on the information on the pitch motion of the extracted camera. It may be done.

As an example related to the present specification, the information about the pitch motion of the camera may be at least one of a direction of the pitch motion and an angle of the pitch motion.

According to an embodiment disclosed in the present specification, a pitch motion of a camera is based on at least one of a vanishing point detected in at least one of a plurality of images acquired with parallax through a camera, and light flow existing between the plurality of images. The present invention provides a detection apparatus and a detection method for detecting an arbitrary object after minimizing an error caused by the error.

In particular, according to the detection apparatus and the detection method according to an embodiment disclosed herein, it is possible to detect any object more accurately and stably by minimizing the influence of the pitch motion of the camera based on at least one of the vanishing point and the light flow. There is an advantage.

1 is a block diagram showing the configuration of a detection apparatus according to the embodiments disclosed herein.
2 is a flow chart illustrating a detection method according to embodiments disclosed herein.
FIG. 3 illustrates a change in the visible area according to the pitch motion of a camera mounted on a vehicle.
4 is an exemplary view illustrating a change of an image according to a pitch motion of a camera.
5 is an exemplary view showing a vanishing point grasped from an image acquired from a camera.
6 is an exemplary diagram illustrating an optical flow existing between a plurality of images acquired with parallax.
7 is a flowchart showing a detection method according to the first embodiment disclosed in the present specification.
8 is an exemplary view showing a detection method according to a second embodiment disclosed herein.
9 is an exemplary view illustrating a process of minimizing an error due to pitch motion according to a third embodiment disclosed herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals are used to designate identical or similar elements, and redundant description thereof will be omitted.

Further, in the description of the technology disclosed in this specification, a detailed description of related arts will be omitted if it is determined that the gist of the technology disclosed in this specification may be obscured. It is to be noted that the attached drawings are only for the purpose of easily understanding the concept of the technology disclosed in the present specification, and should not be construed as limiting the spirit of the technology by the attached drawings.

It is noted that the technical terms used herein are used only to describe specific embodiments and are not intended to limit the scope of the technology disclosed herein. Also, the technical terms used herein should be interpreted as being generally understood by those skilled in the art to which the presently disclosed subject matter belongs, unless the context clearly dictates otherwise in this specification, Should not be construed in a broader sense, or interpreted in an oversimplified sense. In addition, when a technical term used in this specification is an erroneous technical term that does not accurately express the concept of the technology disclosed in this specification, it should be understood that technical terms which can be understood by a person skilled in the art are replaced. Also, the general terms used in the present specification should be interpreted in accordance with the predefined or prior context, and should not be construed as being excessively reduced in meaning.

Also, the singular forms "as used herein include plural referents unless the context clearly dictates otherwise. In this specification, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some steps It should be construed that it may not be included or may further include additional components or steps.

In addition, the suffixes "module", "group", and "unit" for the components used in the present specification are given or used in consideration of the ease of writing the specification, and have a meaning or role that is distinct from each other. no. The suffixes can also be implemented in hardware or software or a combination of hardware and software.

Furthermore, terms including ordinals such as first, second, etc. used in this specification can be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, the configuration of a detection apparatus according to the embodiments disclosed herein will be described with reference to FIG. 1.

The In embodiments  Description of the configuration of the detection device according to

Here, the detection device of FIG. 1 may be configured not only as stand alone, but also as a mobile terminal, a telematics terminal, a smart phone, and a portable terminal. , Personal Digital Assistant (PDA), Portable Multimedia Player (PMP), Notebook Computer, Tablet PC (Tablet PC), Wibro Terminal, IPTV (Internet Protocol Television) Terminal, Television ), 3D televisions, video devices, telematics terminals, navigation terminals, AVN terminals, and the like.

1 is a block diagram showing a configuration of a detection apparatus according to the embodiments disclosed in the present specification. As shown in FIG. 1, the detection apparatus 10 includes a camera module 110, a control unit 120, and a display unit 130. And a storage unit 140. Not all components of the detection device 10 shown in FIG. 1 are essential components, and the detection device 10 may be implemented by more components than those shown in FIG. 1, and fewer components. The detection apparatus 10 may also be implemented.

The camera module 110 may include at least one pair of cameras (eg, a stereo camera or a stereo course) that are spaced apart from each other at horizontal intervals on the same central axis of the same plane of the detection apparatus 10. A stereoscopic camera or a single camera. In this case, the fixed horizontal interval may be set in consideration of a distance between two eyes of a general person, and is set when configuring the detection apparatus 10. In addition, the camera module 110 may be any camera module capable of image capturing.

In addition, the camera module 110 may include a first image (eg, a left image captured by a left camera included in the pair of cameras) and a second image captured by the pair of cameras simultaneously. For example, the right image captured by the right camera included in the pair of cameras) is received.

In addition, the camera module 110 may be an image sensor such as a CCD device and a CMOS device.

In addition, when installed in a vehicle, the camera module 110 may be fixed to a predetermined position of the vehicle, and may be configured to photograph the front, the side, the rear of the driving direction, and the like to receive the photographed image.

According to the embodiments disclosed herein, the camera module 110 may acquire a plurality of images with a parallax through the camera. The plurality of images may be used to extract at least one of vanishing point or light flow for analyzing the effect on the pitch motion of the camera. The plurality of images may be frame images of a video.

The pitch motion may generally refer to an up-down movement of the object due to noise generated from a moving object. For example, in driving of the vehicle, the pitch motion may be vibration in the up-down direction of the vehicle due to the state of the road surface on which the vehicle travels.

The vanishing point means that the extension lines meet by the perspective effect when an extension line is drawn on the object. For example, in the case of a car, a parking line or a lane may be used as a straight line component in the front / rear camera image, and an outer line of the building may be used as the straight line component in the landscape image. Hough transform may be used as a straight line detection method in an image.

The vanishing point can be used to analyze the effect of the pitch motion. That is, the pitch motion of the camera may be detected by tracking the vanishing point detected from the image acquired from the camera. For example, when the vanishing point is extracted from the acquired image and the temporal change of the vanishing point is tracked, the vanishing point may move up or down according to the pitch motion of the camera. Therefore, the pitch motion of the camera can be detected by observing the movement of the vanishing point in the vertical direction.

In addition, the optical flow refers to the velocity distribution of the apparent movement in the image generated by gradually changing the brightness pattern. In other words, the optical flow refers to a vector representing the apparent motion appearing in the image from two different temporally different image data photographed and input by the camera.

Also, an object (for example, a lane or a vehicle) may be detected based on the light flow. Compare or divide the previous image data into a plurality of unit blocks having a predetermined pixel, and move the current image data in pixel units and divide the previous image data into unit blocks having the same size as the unit block of the previous image data. Each unit block of the previous image data is compared with a plurality of unit blocks of the current image data to obtain a difference value such as luminance and chromaticity of the pixels in the unit block, and the position where the pixel of the previous image data is moved by the obtained difference value. Is expressed as a vector, and when a vector of a certain size or more occurs in a specific region, To detect the object.

The controller 120 controls the overall operation of the detection device 10.

According to the embodiments disclosed herein, the control unit 120 may control the overall operation of the detection apparatus 10 such that any object is detected through at least one image of the plurality of images obtained from the camera module. Can be.

For example, when the detection apparatus 10 performs an operation of detecting a lane, the controller 120 may control the overall operation of the detection apparatus 10 to detect the lane as follows.

In this case, the controller 120 supports a plurality of support points in any one of the first image and the second image received by the at least one pair of cameras based on a preset guide line. point) or a plurality of support points in the image received by the single camera. Here, the guide line is set in the left and right directions with respect to the corresponding image with respect to the horizontal axis, and the spacing between the vertical axes of the plurality of guide lines is greater than the spacing between the guide lines at the bottom of the image than the spacing between the guide lines at the top of the image. Set wider. That is, the guide line may be configured such that the guide line is spaced from the bottom of the image to the top in order to obtain the most uniform point spacing when converting data (including support points, etc.) of the original image (or the original image) into world coordinates. Can be set narrow.

In addition, the controller 120 converts the extracted plurality of support points into world coordinates. That is, the controller 120 converts the extracted plurality of support points into world coordinates, respectively, using a transformation matrix (eg, a homographic matrix, etc.) stored in the storage 140 in advance. do. Here, the plurality of support points respectively converted into the world coordinates maintain the same interval in the vertical direction, and thus, even if there are some errors among the plurality of support points respectively converted into the world coordinates, checking the error Can increase the accuracy. Here, the error may mean an error with an actual lane.

In addition, the controller 120 may select a plurality of points corresponding to a curve among points converted into the world coordinates based on a curve equation stored in the storage 140 in advance for the points converted into the world coordinates. Check (or extract).

In addition, the controller 120 performs tracking based on a plurality of points corresponding to the identified curve in order to reduce the calibration time and reduce the noise.

In addition, the controller 120 calculates curve information following a center point of the lane based on a plurality of points corresponding to the identified curve. In this case, the calculated curve information may be used to improve the lane keeping performance in world coordinates by minimizing the influence of the calibration state of the camera. That is, the controller 120 may include a least square method, a random sample consensus (RANSAC), a general hough transform method, and a spline for a plurality of points corresponding to the identified curve. (spline) Calculates curve information following a center point of a lane by applying one of interpolation methods and the like.

In addition, the controller 120 overlaps the received original image with curve information following the calculated center point of the lane, the checked curve, and the like and displays the information on the display unit 130. That is, the controller 120 converts (or maps) curve information for tracking the center point of the calculated lane, which is the world coordinate, and information such as the identified curve into coordinates on the image domain, respectively. Each of the converted coordinates is overlapped with the received original image and displayed on the display unit 130.

In addition, the control unit 120 may determine the position of the detection device 10 (or a vehicle provided with the detection device 10) and the checked curve (or any other GPS module (not shown)). Based on lanes, it performs functions related to lane keeping (including warning message function and auto lane keeping function).

According to embodiments disclosed herein, the control unit 120 may detect the arbitrary object while minimizing the influence on the pitch motion of the object in detecting the arbitrary object (eg, the lane). Can be. For example, the arbitrary object may be a lane or a vehicle.

In this case, the controller 120 detects a vanishing point detected in at least one of the plurality of images when pitch motion with respect to the camera occurs while acquiring the plurality of images to detect the arbitrary object. The error due to the pitch motion of the camera may be minimized based on at least one of the optical flows existing between the plurality of images.

In addition, the controller 120 may extract information on the pitch motion of the camera based on at least one of the vanishing point and the optical flow. In addition, the controller 120 may minimize the error based on the information on the pitch motion of the camera.

The information about the pitch motion of the camera may be at least one of a direction of the pitch motion and an angle of the pitch motion.

For example, the controller 120 may detect the angle of the pitch motion of the object (for example, the lane) based on the vanishing point in 18 degrees in the above direction (for pitch-up), and detect the The object can be detected through correction of 18 degrees, which is the angle of the pitch motion.

Also, for example, the controller 120 may detect the angle of the pitch motion of the object (for example, the lane) based on the optical flow in the above direction (in the case of pitch-up) by 20 degrees. The object may be detected through correction of 20 degrees, which is an angle of the detected pitch motion.

Alternatively, the controller 120 may select or combine the angles of the respective pitch motions detected based on the vanishing point and the optical flow, and may be used to correct the pitch motion, thereby detecting the object.

For example, the selection of any one of the angles of the respective pitch motions may be made depending on the detection state of the detection device 10 of the detection device 10. For example, when the detection state of the detection apparatus 10 is an outdoor environment, a pitch motion correction method based on light flow may be selected in consideration of various lighting conditions and artificial structures. Further, for example, when the pitch motion correction method based on the light flow is inappropriate according to the kind, color, or background color of an object present on a plurality of images selected from a camera (for example, the object and If the contrast between the plurality of images is small due to similar colors of the background image), a pitch motion correction method based on the vanishing point may be used.

In this case, the controller 120 may check the detection state and select a pitch motion correction method suitable for the detection state according to a predetermined criterion.

The display unit 130 displays various contents such as various menu screens using a user interface and / or a graphic user interface included in the storage unit 140 under the control of the controller 120. Here, the content displayed on the display unit 130 includes a menu screen including various text or image data (including various information data) and data such as an icon, a list menu, a combo box, and the like.

In addition, the display unit 130 includes a 3D display or a 2D display. In addition, the display unit 130 may include a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. The display device may include at least one of a flexible display and a light emitting diode (LED).

In addition, the display unit 130 displays the 3D image (or 2D image) under the control of the controller 120.

In addition, two or more display units 130 may exist according to an embodiment of the detection device 10. For example, a plurality of display units may be spaced apart or integrally disposed on one surface (same surface) of the detection device 10, or may be disposed on different surfaces, respectively.

On the other hand, when the display unit 130 and a sensor for detecting a touch operation (hereinafter, referred to as a touch sensor) form a mutual layer structure (hereinafter referred to as a touch screen), the display unit 130 outputs the touch panel. In addition to the device can also be used as an input device. The touch sensor may have, for example, a form of a touch film, a touch sheet, a touch pad, or a touch panel.

In addition, the touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 130 or capacitance generated at a specific portion of the display unit 130 into an electrical input signal. In addition, the touch sensor may be configured to detect not only the position and area of the touch but also the pressure at the time of touch. If there is a touch input to the touch sensor, the corresponding signal (s) is sent to a touch controller (not shown). The touch controller processes the signal (s) and then transmits the corresponding data to the controller 120. As a result, the controller 120 may determine which area of the display unit 130 is touched.

In addition, the display unit 130 may include a proximity sensor. In addition, the proximity sensor may be disposed in the inner region of the detection device 10 surrounded by the touch screen or near the touch screen.

In addition, the proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object present in the vicinity without using a mechanical contact by using an electromagnetic force or infrared rays. The proximity sensor has a longer life and higher utilization than a contact sensor.

Examples of the proximity sensor include a transmission photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. And to detect the proximity of the pointer by the change of the electric field along the proximity of the pointer when the touch screen is electrostatic.

In this case, the touch screen (touch sensor) may be classified as a proximity sensor. Hereinafter, for convenience of explanation, the act of recognizing that the pointer is positioned on the touch screen while the pointer is not in contact with the touch screen is referred to as "proximity touch" The act of actually touching the pointer on the screen is called "Contact Touch ". A position where the pointer is proximally touched on the touch screen means a position where the pointer corresponds to the touch screen vertically when the pointer is touched.

In addition, the proximity sensor detects a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, and a proximity touch movement state). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

As such, when the display unit 130 is used as an input device, a command or control signal may be received by an operation such as receiving a button operation by a user or touching / scrolling the displayed screen.

The storage unit 140 stores various menu screens, various user interfaces (UIs), and / or graphical user interfaces (GUIs).

According to the exemplary embodiments disclosed herein, the storage unit 140 may detect from at least one of the plurality of images in detecting the object (eg, the lane) through minimizing the influence on the pitch motion. When at least one of the vanishing point and the optical flow representing the difference between the plurality of images is extracted by the controller 120, the extracted vanishing point position (or coordinates on the image) or the optical flow value (or motion vector) may be stored. have.

In addition, the storage unit 140 stores equations such as transformation matrices (eg, homographic matrices), curve equations, least squares methods, and the like.

In addition, the storage unit 140 stores data, a program, and the like necessary for the detection apparatus 10 to operate.

In addition, the storage unit 140 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), ROM (Read-Only Memory: ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), RAM (Random Access Memory: RAM), SRAM (Static Random Access Memory) ), A magnetic memory, a magnetic disk, and an optical disk.

In addition, the detection apparatus 10 may further include a communication unit (not shown) that performs a communication function with any terminal or server under the control of the controller 120.

In this case, the communication unit may include a wired / wireless communication module. Here, the wireless Internet technology, Wireless LAN (WLAN), Wi-Fi (Wi-Fi), Wireless Broadband (Wibro), WiMAX (World Interoperability for Microwave Access (Wimax), High-speed downlink packet access (High) Speed Downlink Packet Access (HSDPA), IEEE 802.16, Long Term Evolution (LTE), and Wireless Mobile Broadband Service (WMBS), and the like, and may include the short range communication. The technology may include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like. The wired communication technology may include universal serial bus (USB) communication.

The communication unit may include CAN communication, vehicle Ethernet, flexray, LIN (Local Interconnect Network), and the like, for communication with any vehicle provided with the detection device 10.

In addition, the communication unit may be a curve among a plurality of support points extracted for an image under the control of the controller 120, points converted from the plurality of support points into world coordinates, and points converted into the world coordinates. And a plurality of points corresponding to the curve information, curve information following the center point of the lane calculated based on the plurality of points corresponding to the curve, and the like, to the arbitrary terminal or server.

In addition, the communication unit receives the first image and the second image photographed simultaneously through any pair of stereo cameras transmitted from the arbitrary terminal or server.

In addition, the detection apparatus 10 may further include an input unit (not shown) including at least one microphone (not shown) for receiving an audio signal. The microphone is configured to receive an external sound signal (including a user's voice (voice signal or voice information)) by a microphone in a call mode, a recording mode, a voice recognition mode, a video conference mode, a video call mode, and the like. Process with voice data. In addition, the processed voice data may be output through a voice output unit (not shown) or may be converted into a form transmittable to an external terminal through the communication unit and output. In addition, the microphone may be implemented with various noise removal algorithms for removing noise generated in the process of receiving an external sound signal.

In addition, the input unit receives a command or control signal generated by an operation such as receiving a signal according to a button operation by a user or touching / scrolling a displayed screen.

In addition, the input unit receives a signal corresponding to information input by a user, and includes a keyboard, a key pad, a dome switch, a touch pad (static / electrostatic), and a touch screen ( Various devices such as a touch screen, a jog shuttle, a jog wheel, a jog switch, a mouse, a stylus pen, a touch pen, and a laser pointer may be used. In this case, the input unit receives a signal corresponding to input by the various devices.

In addition, the detection apparatus 10 may further include a voice output unit (not shown) for outputting voice information included in a signal processed by the controller 120 by a predetermined signal. The voice output unit may be a speaker.

In this way, a support point which is a candidate group of lanes in the image is extracted and converted into world coordinates, the lane is recognized on the converted world coordinates, and the lane is directly transferred in the image in the error transition of calibration between the camera information and the world coordinates. Compared to the recognition method, the possibility of cumulative error can be reduced.

In addition, as described above, information on the lanes recognized on the world coordinates may be displayed and a warning message may be generated and output based on the information, thereby improving accuracy / sensitivity and improving user convenience.

Hereinafter, a detection method according to embodiments disclosed herein will be described with reference to FIG. 2.

The In embodiments  Of detection methods according to

2 is a flow chart illustrating a detection method according to embodiments disclosed herein.

Referring to FIG. 2, the detection method according to the embodiments disclosed in the present specification may include the following steps.

First, the detection apparatus 10 may acquire a plurality of images with a parallax through the camera (S110).

Next, the detection apparatus 10 may determine whether pitch motion with respect to the camera has occurred while acquiring the plurality of images (S120).

As a result of the determination, when the pitch motion with respect to the camera occurs, the detection device 10 detects at least one of a vanishing point detected in at least one of the plurality of images and an optical flow existing between the plurality of images. Based on the error due to the pitch motion of the camera can be minimized (S130).

Next, the detection device 10 may detect the arbitrary object through at least one image of the plurality of images by minimizing the error (S140).

As described above, the information on the pitch motion of the camera may be at least one of the direction of the pitch motion and the angle of the pitch motion.

In addition, the detection apparatus 10 may be mounted on a vehicle and serve to detect the arbitrary object.

In this case, the camera may be mounted on a vehicle, and the arbitrary object may have a relative speed compared to the moving speed of the vehicle. The detection device 10 can detect the relative speed and use the relative speed to detect the arbitrary object.

In addition, the parallax for acquiring the plurality of images may be variable according to the speed of the vehicle. For example, the detection apparatus 10 may bring the parallax smaller when the vehicle speed is 100 km / h than when the vehicle speed is 80 km / h.

The pitch motion of the camera may be caused by acceleration or deceleration of the vehicle or a state of a road surface on which the vehicle travels.

In addition, the minimization of the error may be continued until the pitch motion of the camera due to the elastic coefficient and the damping coefficient of the shock absorber of the vehicle is minimized. By doing so, the detection device 10 can detect the arbitrary object without being affected by the pitch motion of the camera.

The shock absorber may serve to absorb the free vibration of the spring to alleviate the shock received from the road surface while the vehicle is driving.

1st Example

Hereinafter, a detection method according to the first embodiment disclosed herein will be described with reference to FIGS. 3 to 7.

FIG. 3 illustrates a change in the visible area according to the pitch motion of a camera mounted on a vehicle.

Referring to FIG. 3, in order to detect the arbitrary object (eg, a lane or another vehicle), the detection device 10 may photograph the front surface of the vehicle through a camera mounted on the vehicle.

3 (a) shows a visible area of the camera when the vehicle is driving on a road having a normal road surface state. This case may be referred to as a normal state.

However, in the case of FIG. 3B, the pitch motion of the camera may occur while the rear wheels of the vehicle pass over the protruding obstacle existing on the road. This case may be referred to as a pitch-down state. In the case of the pitch-down state, since the camera photographs the road downward from the normal state, the image photographed from the camera may be an image that is raised upward than the case of the normal state.

In addition, in the case of FIG. 3 (c), the pitch motion of the camera may occur while the front wheel of the vehicle passes through the protruding obstacle existing on the road. This case may be referred to as a pitch-up state. In the case of the pitch-up state, since the camera photographs a road upward from the normal state, the image photographed from the camera may be an image that is lowered than the case of the normal state.

Therefore, when the pitch-down or the pitch-up state occurs, the detection apparatus 10 may minimize the influence on the pitch motion to correctly detect the arbitrary object.

4 is an exemplary view illustrating a change of an image according to a pitch motion of a camera.

Referring to FIG. 4, the detection apparatus 10 may detect various objects existing in the captured image. For example, the object may be a lane or a vehicle. However, when the pitch motion of the camera occurs as described above, the detection device 10 may not correctly detect the object by the pitch motion.

4A shows an image acquired by the camera module 110 in a normal state. The detection apparatus 10 may correctly detect a lane using the image in the steady state.

For example, when the detection target is a lane, the controller 120 extracts a plurality of support points corresponding to the lane through the obtained image and converts the extracted plurality of support points into world coordinates. . In this case, the controller 120 may use a transformation matrix (eg, including a homographic matrix) stored in the storage 140 in advance. In addition, the controller 120 may determine a curve equation corresponding to the lane through the points converted into the world coordinates, and detect the lane based on the curve equation.

4 (b) shows an image obtained when the pitch motion is pitch-up. In the case of Figure 4 (b), it can be seen that the obtained image is a lower image than the case of Figure 4 (a).

In this case, an error may occur in lane detection by the image noise generated by the pitch-up. Therefore, the detection apparatus 10 may need to detect the lane after removing the image noise due to the pitch-up.

According to the first embodiment disclosed herein, in order to reduce the effect of the pitch-up, the control unit 120 is a light loss existing between the vanishing point and the plurality of images identified from at least one of the plurality of images. Information on the pitch motion may be extracted based on at least one.

5 is an exemplary view showing a vanishing point grasped from an image acquired from a camera.

Referring to FIG. 5, since vanishing point is a point where extension lines meet by a perspective effect when an extension line is drawn to an object, in FIG. 5, the vanishing point 200 is extended to a lane existing in the acquired image. It was found to meet when drawing.

In FIG. 5, the vanishing point 200 is extracted using the extension line for the lane, but there may be various methods of extracting the vanishing point. For example, a plurality of vanishing point candidates are selected using extension lines of various objects (for example, a parking line or an outline of a building, etc.) existing in an image obtained from a camera, and a position (or an image having the most vanishing point candidates). Coordinates of the image) may be determined as the vanishing point. This is called a voting scheme. In addition, it will be apparent to those skilled in the art that various methods for determining the vanishing point may be applied to the present pitch motion correction method.

6 is an exemplary diagram illustrating an optical flow existing between a plurality of images acquired with parallax.

Referring to FIG. 6, there is shown a light flow 300 representing the difference in contrast between the images shown in FIGS. 4 (a) and 4 (b) above. The optical flow 300 represents a pixel difference between the plurality of images in a vector form and may be a kind of motion vector.

As in the case of FIG. 4, when pitch-up is performed, the light flow 300, which is a difference in contrast between the images shown in FIGS. 4A and 4B, may be illustrated as in FIG. 6.

As shown in FIG. 6, the optical flow 300 has various directions, such as a first optical flow 300a having only a component in a vertical direction or a second optical flow 300b having both a mathematical and a horizontal component according to a change direction of the pixel. It can be represented as a vector.

Hereinafter, a detection method based on at least one of the vanishing point and the light flow will be described with reference to FIG. 7.

7 is a flowchart showing a detection method according to the first embodiment disclosed in the present specification.

Referring to FIG. 7, the detection method according to the first embodiment disclosed in the present specification may include the following steps.

First, the detection apparatus 10 may acquire a plurality of images with a parallax through the camera (S210).

Next, the detection apparatus 10 may determine whether pitch motion with respect to the camera has occurred while acquiring the plurality of images (S220).

Next, the detection apparatus 10 may extract a vanishing point from at least one of the plurality of images (S230).

Next, the detection apparatus 10 may extract the light flow existing between the plurality of images (S240).

Next, the detection apparatus 10 may extract information about the pitch motion of the camera based on at least one of the vanishing point and the optical flow (S250).

Next, the detection device 10 may minimize the error due to the pitch motion based on the information on the pitch motion (S260).

Next, the detection apparatus 10 may detect an arbitrary object through at least one image of the plurality of images after the error is minimized (S270).

Second Example

According to the second embodiment disclosed herein, the detection apparatus 10 extracts information on the pitch motion of the camera based on at least one of the vanishing point and the light flow extracted from the plurality of images, and the pitch motion of the camera. The error can be minimized based on the information on.

The second embodiment disclosed herein may be implemented as a part or a combination of the configurations or steps included in the above-described embodiments, or may be implemented as a combination of the embodiments. Hereinafter, a clear description of the second embodiment disclosed in this specification And duplicate parts are omitted.

The information about the pitch motion of the camera may be at least one of a direction of the pitch motion and an angle of the pitch motion.

The information on the pitch motion of the camera may include at least one of first information extracted based on the vanishing point and second information extracted based on the optical flow, and the minimization of the error may include the first information. And at least one of the second information.

8 is an exemplary view showing a detection method according to a second embodiment disclosed herein.

According to the second embodiment, the controller 120 may minimize or correct the pitch motion by selecting or combining the first information and the second information extracted based on the vanishing point and the optical flow. have.

To this end, the controller 120 may extract the information about the pitch motion of the camera based on a value obtained by multiplying the first information and the second information by weights corresponding to the first information and the second information. For example, when the controller 120 selects one of the first information and the second information, one of the weights corresponding to each of the controller 120 may be set to '0'.

Also, for example, the weight corresponding to the second information may be greater than the weight corresponding to the first information. For example, the weight of the second information may be 0.8, and the weight of the first component may be 0.2.

Referring to FIG. 8, when the angle of the pitch motion using the vanishing point tracking is 20 degrees in the pitch up direction and the angle of the pitch motion using the optical flow extraction is 10 degrees in the pitch up direction, the control unit 120 determines the vanishing point. The weight corresponding to the tracking may be set to p, and the weight corresponding to the extracted optical flow may be set to q. In addition, the control unit 120 may determine the angle of the pitch motion based on the vanishing point tracking and the optical flow and multiply the sum of p and q as an angle of the final pitch motion. When the p and q values are 0.5, the controller 120 may determine the angle of the final pitch motion as 15 degrees in the pitch up direction as shown in FIG. 8.

Third Example

According to the third embodiment of the present disclosure, in detecting the arbitrary object, the controller 120 may extract a support point that is a candidate group of the arbitrary object in the captured image. .

In addition, the controller 120 may convert the support point into world coordinates and detect the arbitrary object on the converted world coordinates.

In addition, the controller 120 may minimize the error due to the pitch motion by converting the support point into world coordinates based on at least one of the vanishing point and the optical flow.

The third embodiment disclosed herein may be implemented as a part or a combination of the configurations or steps included in the above-described embodiments, or may be implemented as a combination of the embodiments, and a description of the third embodiment disclosed herein And duplicate parts are omitted.

The above-described transformation of the support point into world coordinates may be performed based on a transformation matrix.

Accordingly, the controller 120 may minimize the error by changing a coefficient of the transformation matrix based on at least one of the vanishing point and the optical flow.

9 is an exemplary view illustrating a process of minimizing an error due to pitch motion according to a third embodiment disclosed herein.

Referring to FIG. 9, a first point 410 present on an image acquired from a camera is located at (1,1) coordinates on the image plane.

In this case, the controller 120 may convert the first point into world coordinates using the transformation matrix 500 stored in the storage 140 in advance. For example, the transformation matrix may be a homographic matrix.

A second point 420 in which the first point is converted to world coordinates is shown in FIG. 9.

인 경우, 변경된 행렬은

일 수 있다.The controller 120 may extract information about the pitch motion based on the extracted optical flow, and change the coefficient of the transformation matrix 500 based on the extracted information. By doing so, the detection device 10 can detect the arbitrary object while minimizing the influence on the pitch motion. For example, the transformation matrix

If is, the changed matrix is

Lt; / RTI >

The method described above can be implemented in a recording medium that can be read by a computer or similar device using, for example, software, hardware, or a combination thereof.

According to the hardware implementation, the methods described so far are application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and processors ( It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. It may be implemented in the control unit 120 of the device 10.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software code may be stored in the storage 140 of the detection apparatus 10 and may be executed by the controller 120.

The scope of the present invention is not limited to the embodiments disclosed herein, and the present invention may be modified, changed, or improved in various forms within the scope of the spirit and claims of the present invention.

10: detection device 110: camera module
120: control unit 130: display unit
140:

Claims (14)

A method of detecting a dynamic or static object,
Acquiring a plurality of images with parallax through a camera; And
Detecting the object through at least one image of the plurality of images,
If pitch motion with respect to the camera occurs while acquiring the plurality of images,
The object is detected by minimizing an error due to pitch motion of the camera based on at least one of a vanishing point detected in at least one of the plurality of images and an optical flow existing between the plurality of images. But
Minimization of the error is
Extracting information about the pitch motion of the camera based on at least one of the vanishing point and the optical flow,
The detection method, characterized in that based on the information on the pitch motion of the extracted camera. delete The method of claim 1, wherein the information about the pitch motion of the camera,
And at least one of a direction of the pitch motion and an angle of the pitch motion. The method of claim 1, wherein the information about the pitch motion of the camera,
At least one of first information extracted based on the vanishing point and second information extracted based on the optical flow;
Minimization of the error is
Based on at least one of the first information and the second information,
The first information is,
Information related to at least one of the direction of the pitch motion and the angle of the pitch motion extracted based on the vanishing point,
The second information is,
And information related to at least one of the direction of the pitch motion and the angle of the pitch motion extracted based on the optical flow. The method of claim 4, wherein the information about the pitch motion of the camera,
And extracting the first information and the second information based on a value obtained by multiplying a weight corresponding to each of the first information and the second information. The method of claim 1, wherein the detecting of the object comprises:
Extracting a support point that is a candidate group of the object in at least one of the plurality of images;
Converting the support points to world coordinates; And
Detecting the object on the transformed world coordinates;
Minimization of the error is
By converting the support point into world coordinates based on at least one of the vanishing point and the optical flow,
The conversion of the support point to world coordinates,
Based on a homographic matrix,
Minimization of the error is
Detecting a coefficient of the homographic matrix based on at least one of the vanishing point and the optical flow. delete The method of claim 1,
The camera is mounted on a vehicle, and the object has a relative speed relative to the moving speed of the vehicle. 9. The method of claim 8,
Parallax for acquiring the plurality of images,
The detection method is variable according to the speed of the vehicle. 9. The method of claim 8,
The pitch motion of the camera is caused by the acceleration or deceleration of the vehicle or the state of the road surface on which the vehicle travels. The method of claim 8, wherein the minimization of the error is
And the pitch motion of the camera due to the elastic and damping coefficients of the shock absorber of the vehicle is continued until it is minimized. A device for detecting a dynamic or static object,
A camera module for acquiring a plurality of images with parallax through a camera; And
A control unit for detecting the object through at least one image of the plurality of images,
If pitch motion with respect to the camera occurs while acquiring the plurality of images,
The object is detected by minimizing an error due to pitch motion of the camera based on at least one of a vanishing point detected in at least one of the plurality of images and an optical flow existing between the plurality of images. But
Minimization of the error is
Extracting information about the pitch motion of the camera based on at least one of the vanishing point and the optical flow,
The detection device is based on the information on the pitch motion of the extracted camera. delete The method of claim 12,
Information about the pitch motion of the camera,
And at least one of a direction of the pitch motion and an angle of the pitch motion.

Images