KR20180041525A - Object tracking system in a vehicle and method thereof - Google Patents

Abstract

According to the present invention, disclosed are an object tracking control method for a vehicle and an object tracking system for a vehicle. According to the embodiment of the present invention, disclosed are a pedestrian sensing system and a control method for the same. According to the embodiment of the present invention, the pedestrian sensing method for a vehicle includes: a step of photographing an outside image of a vehicle; a step of detecting an object included in the photographed image; a step of detecting the object by transmitting and receiving a radar signal to and from the front of the vehicle; a step of firstly tracking the object based on three-dimensional information on the object detected based on the radar signal; and a step of secondly tracking the object based on two-dimensional information on the object detected in the photographed image. According to the embodiments of the present invention, the object tracking control method for a vehicle and the object tracking system for a vehicle can continuously track an object even in case detecting the object is failed, by using both a radar sensor and a camera sensor.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object tracking system for a vehicle,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for recognizing an object ahead of a vehicle based on at least one sensor installed in a vehicle and tracking the recognized object and a control method thereof.

A system has been developed to notify the driver that a pedestrian is located in front of the vehicle by recognizing the pedestrian from the image photographed by the camera installed in the front of the vehicle in case the driver can not detect the external situation.

For example, in the nighttime, the driver's view may suddenly decrease so that the driver may not see the pedestrian. In order to prevent such a case, researches on a system that enables a driver to secure a view by using a sensor such as a near infrared ray camera or a far-infrared ray camera, or a warning system by recognizing a pedestrian at night,

In addition, with respect to the pedestrian protection, a system has been developed to prevent a pedestrian accident in advance by recognizing a pedestrian using a camera mounted on the vehicle, and giving an alarm to the driver or controlling the vehicle to brak have.

In addition, a system for detecting an object in front of the vehicle using a radar installed in front of the vehicle and notifying the driver that a pedestrian is located in front of the vehicle is being developed.

However, in the case of a radar installed in front of a vehicle, two-dimensional information is obtained through an image obtained through a camera, unlike a case where an object is detected using a three-dimensional coordinate system (X, Y, Z) Techniques for improving the accuracy of tracking through position and movement of an object including a pedestrian through fusion of sensors are continuing.

Therefore, embodiments of the present invention use both the radar sensor and the camera sensor to continuously track the object even when the object detection fails.

According to an aspect of the present invention, there is provided a method of detecting an object, comprising: capturing an external image of a vehicle; detecting an object included in the captured image; sensing an object by transmitting and receiving a radar signal in front of the vehicle; First tracking the object based on three-dimensional information of the object sensed based on the radar signal; And performing second tracking on the object based on the two-dimensional information of the sensed object in the photographed image; A method of controlling an object tracking of a vehicle can be provided.

The first tracking of the object based on the three-dimensional information of the object sensed based on the radar signal may include a step of detecting a distance between the sensed object and the vehicle and a Kalman It can be tracked by applying a filter (Kalman-Filter).

A second step of performing tracking of the object based on the two-dimensional information of the sensed object in the photographed image may include: detecting the sensed object based on a CSK (Circulant Tracking by Detection with Kernels) technique; Tracking; And tracking and applying a Kalman filter including a noise model for the position of the sensed object and the speed of the vehicle.

Further, the first tracking information and the second tracking information can be exchanged.

According to another aspect of the present invention, there is provided a vehicular navigation system comprising: an image capturing unit capturing an external image of a vehicle and sensing an object included in the image; a radar sensor unit transmitting and receiving a radar signal in front of the vehicle to detect an object; And a control unit for first tracking the object on the basis of the three-dimensional information of the object sensed based on the radar signal and secondly tracking the object based on the two-dimensional information of the sensed object in the sensed image An object tracking system of the vehicle can be provided.

The display unit may further include a display unit for displaying an object included in the image in a bounding box and displaying the object to a driver.

The controller may further include a first Kalman filter converting unit for applying and tracking a Kalman filter including a noise model for the distance between the vehicle and the resolution of the radar sensor based on the three- And a tracking unit tracking the object based on the two-dimensional information of the sensed object by a CSK (Circulant Tracking by Detection with Kernels) technique. And a second Kalman filter converting unit for applying and tracking a Kalman filter including a noise model for the two-dimensional position of the sensed object and the speed of the vehicle.

The first tracking information of the first Kalman filter conversion unit and the second tracking information of the second Kalman filter conversion unit may be interchanged.

Therefore, the embodiments of the present invention can continuously track the object even when the object detection fails using both the radar sensor and the camera sensor.

1 is a block diagram of an object tracking system according to an embodiment of the present invention.
2 is an external perspective view of a vehicle equipped with an object tracking system according to an embodiment of the present invention.
3 is a schematic diagram illustrating the recognized walk of a vehicle shown to illustrate a method of controlling an object tracking system in accordance with an embodiment of the present invention.
4 is a block diagram illustrating the operation of an object tracking system according to an embodiment of the present invention.
5 is a schematic diagram showing a coordinate system for explaining a tracking method according to an embodiment of the present invention.
6 is a flowchart illustrating an object tracking method according to an embodiment of the present invention.
7 is a flowchart illustrating an object tracking method according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments are provided to fully convey the spirit of the present invention to a person having ordinary skill in the art to which the present invention belongs. The present invention is not limited to the embodiments shown herein but may be embodied in other forms. For the sake of clarity, the drawings are not drawn to scale, and the size of the elements may be slightly exaggerated to facilitate understanding.

Fig. 1 is a block diagram showing the configuration of an object tracking system 1 according to the present invention, Fig. 2 is an external perspective view of a vehicle 2 equipped with an object tracking system 1 according to an embodiment of the present invention, 3 is a schematic diagram illustrating the recognized walk of the vehicle shown to illustrate a method of controlling an object tracking system in accordance with an embodiment of the present invention.

Referring to FIG. 1, an object tracking system 1 includes a radar sensor unit 10, an image capturing unit 20, and a control unit 30 display unit 40.

The radar sensor unit 10 includes a radar module 11 and the radar module 11 detects a vehicle or a pedestrian in the vehicle 2 ) May be located below the front hood (hood).

The radar module 11 detects a vehicle or a pedestrian present in front of the vehicle 2 and includes a radar sensor (not shown). At this time, a 77 GHz radar is typically used for the radar sensor, and the radar is transmitted, and the received time is measured to calculate the distance to one or more objects such as a vehicle or pedestrian sensed ahead.

Particularly, the radar module 11 including the radar sensor (not shown) includes three-dimensional position information about the detected front object. Specifically, when the traveling direction of the vehicle 2 is the Y-axis, The position information of the object can be obtained with the Y axis and each horizontal axis as the X axis and the Z axis.

For example, as shown in FIG. 3, when the traveling direction of the vehicle 2 is the Y-axis, the X, Y, and Z coordinates can be calculated with respect to the positional information of the pedestrian recognized through the radar sensor.

Accordingly, the position information including the coordinates of the object acquired through the radar sensor can be transmitted to the control unit 30. [

Next, the image capturing section 20 acquires an image in front of the vehicle 2. [ 2, the camera sensor 21 is mounted on the lower or upper side of the rearview mirror 22 located behind the front window 17 of the vehicle, So that a forward image can be captured.

The camera module 21 includes a camera sensor (not shown), and usually uses a camera having one or more channels, and usually uses CMOS as the image sensor. A CMOS image sensor is a semiconductor device that converts exposed images into electrical form and transmits them. However, the present invention is not limited to this, and may be realized by a CCD (Change Coupled Device) image sensor (not shown).

At this time, the camera module 21 can detect an object from the image acquired by the camera sensor. Specifically, it is possible to detect the movement of an object photographed from a camera sensor (not shown), to extract an object region, and to distinguish a plurality of objects. At this time, the corresponding object is an object to be recognized of the image processing. For example, it may be an object such as a landscape, a foreground, a specific object, a pedestrian, and the like.

First, the outline of the object is extracted from the acquired image using the deviation of neighboring RGB values, and the motion of the object can be determined by analyzing the change of the outline in comparison with the previous image.

In addition, an area where motion of an object occurs can be obtained by using the detected area period as a set of setting intervals.

In addition, two or more objects may be merged or divided in one image. In order to distinguish an area of an accurate object, an estimated position and size of the object through object tracking may be calculated to extract the position and area of the object .

At this time, the camera module 21 transmits information of the extracted object including the acquired image information to the control unit 20 at predetermined time intervals.

Therefore, the camera module 21 transmits the detection of the motion of the object photographed in the image, the region of the object, and the plurality of object overlapping information to the control unit 20, Information can be calculated.

For example, as shown in FIG. 3, a pedestrian included in the image acquired through the camera module 21 can be obtained as two-dimensional U, V-axis coordinate information, as shown in FIG.

Next, the control unit 30 of the tracking system 1 of the object according to the present invention controls the overall tracking system 1 of the object.

Specifically, the control unit 30 mediates data input / output between the various constituent devices included in the object tracking system 1 and the control unit 20, and transmits the data through the radar sensor unit 10 or the image pickup unit 20 A main processor 31 for performing trekking based on coordinate information of the sensed object, and a memory 32 for storing a main processor, a program, and data.

In particular, FIG. 4 is a block diagram illustrating the operation of the object tracking system according to an embodiment of the present invention.

As shown in FIG. 4, the controller 30 performs tracking of the sensed object in stages. The controller 30 includes a first Kalman filter converting unit 33, a tracking unit 34, And a second Kalman filter conversion unit 35. [ The first Kalman filter conversion unit 33, the tracking unit 34, and the second Kalman filter conversion unit 35 proceed to the step after the detection of the object.

First, the first Kalman filter converting unit 33 proceeds to track an object using a simple Kalman filter.

Specifically, the first Kalman filter conversion unit 33 can perform tracking using the three-dimensional coordinates of the object through the following [Expression 1] to [Expression 4].

[Equation 1]

Here, _Xk denotes a state vector of the object detected at a specific time K [sec], where x is the position in the X axis at the time point K [sec] and X 'is the position in the X axis direction at the time point K [sec] . In addition, the acceleration applied to the object is a , The following equations (2) to (4) can be established by Newton's law of motion.

[Formula 2]

 [Formula 3]

[Formula 4]

및 는

시간 K[sec]에 관계없이 일정한 것으로,

는 상태 변경 매트릭스(State transition matrix)이고, β는 평균이 0이고 표준편차가 일정한 가속도에 대한 노이즈(Random White Gaussian Noise)를 의미하고,

는 가속도에 곱하여지는 곱수(Multiplier)이다.only,

And

Is constant regardless of the time K [sec]

Is a state transition matrix, and β is a random white Gaussian noise with zero mean and constant standard deviation,

Is the multiplication multiplied by the acceleration.

Although the state vectors of the object are described with respect to the X axis in [Expression 1] to [Expression 4], there is a state vector of the sensed object at the specified time K [sec] for the Y axis and the Z axis .

Therefore, the first Kalman filter conversion unit 33 can perform tracking on the object through [Equation 1] to [Equation 4].

The first Kalman filter conversion unit 33 includes a noise model that depends on the distance to the object acquired through the radar module 11 included in the radar sensor unit 10 and the radar resolution.

At this time, the radar resolution is defined by azimuth, elevation, and distance, which means a volume element of the radar module 11, and the minimum voxel size is defined as the resolution capability of the radar it means.

When tracking is performed based on the three-dimensional coordinates of the object acquired through the first Kalman filter conversion unit 33 through the radar sensor unit 10, the tracking unit 34 acquires the existing CSK

Tracking By Detection with kernels.

Specifically, the tracking unit 34 extracts an object by applying a Haar-Like feature technique to an object sensed by the camera module 21 included in the image sensing unit 20, and then performs a Kernelized least method Squares are used to track objects through a classifier.

In this case, Haar-Like Feature technique refers to a method of extracting features of objects in various positions and sizes by combining a plurality of basic features according to a rectangular shape using brightness difference between regions and regions in an image.

In addition, a classifier using kernelized least squares means a method of classifying objects or backgrounds of a rectangular shape including features of objects of various positions and sizes.

Accordingly, the tracking unit 34 can perform tracking on a two-dimensional object sensed by the camera module 21 included in the image capturing unit 20. [

Next, the second Kalman filter conversion unit 35 performs tracking according to the Kalman filter applied in the two-dimensional plane.

5 is a schematic diagram showing a coordinate system for explaining a tracking method according to the second Kalman filter conversion unit 35. In FIG.

That is, a bounding box 100 of a rectangular model is shown for a sensed object when an image is a U and V axis in the horizontal and vertical axes of the plane. Specifically, (a) shows the position of the bounding box 100 of the object detected at time K-2 [sec] in the plane coordinate system of the image, (b) (Including distance information from the center of the image) of the bounding box 100 of the object sensed at the time K-1 [sec], and (c) The bounding box 100 of the object sensed by the user.

At this time, the coordinate of the upper left vertex of the bounding box 100 at time K-2 [sec] is U1 _k-2 , and the velocity of the corresponding vertex is bounded at V1 _{k -2} and at time K-2 [sec] The coordinate of the vertex at the lower right end of the box 100 is U2 _{k -2} and the velocity of the corresponding vertex is V2 _{k -2} .

Further, K-1 time [sec] from the vertex coordinates of La at the top of the left side of the bounding box (100) U1 _k-1, and the speed of the vertices V1 _{k -1,} hours K-1 bounding box in [sec] The coordinate of the vertex of the lower right end of the vertex 100 is U2 _{k -1} and the velocity of the vertex is V2 _{k -1} .

At this time, the coordinate of the vertex at the upper left of the bounding box 100 at time K [sec] is U1 _k , and the velocity of the corresponding vertex at V1 _k and the time K [sec] referred to the coordinates of the point _k and U2, it may represent the speed of the vertex V2 is called _k.

Therefore, the second Kalman filter conversion unit 35 can perform tracking based on the position information and the velocity information of the bounding box 100. [ Unlike the Kalman filter applied in the first Kalman filter converting unit 35, the second Kalman filter converting unit 35 according to this step converts the speed of the vehicle 2 and the distance information from the center of the image of the sensed object And a noise model that is dependent on the noise model.

In the object tracking system 1 according to the present invention, the controller 30 controls the radar sensor unit 10 (not shown) in the first and second Kalman filter units 33, 34, ) Of the radar sensor unit 10 or the camera module 21 of the image capturing unit 20 according to the tracking of the object based on the position information of the object detected by the image capturing unit 20 or the position information of the object sensed by the image capturing unit 20 It can be replaced in case of failure or tracking failure.

Next, the information processed by the main processor 31 can be stored in the memory 32. [

That is, the memory 32 stores control programs and control data for controlling the operation of the object tracking system 1, information obtained from the radar sensor unit 10 and the image capturing unit 20, And various control signals including object tracking information can be temporarily stored.

In particular, the memory 32 may be a flash memory, a read only memory, an erasable programmable read only memory (EPROM), or the like, as well as a volatile memory such as an S-RAM and a D- And non-volatile memory such as Electrically Erasable Programmable Read Only Memory (EEPROM).

 Specifically, the non-volatile memory may semi-permanently store a control program and control data for controlling the operation of the object tracking system 1, and the volatile memory temporarily loads the control program and control data from the non- And can temporarily store the information acquired by the image capturing unit 20 and the radar sensor unit 10 and the tracking information processed by the control unit 30. [

Next, the display unit 40 can display the object information recognized and tracked by the control unit 30 to the driver. Although not shown, the object tracking system 1 according to the present invention includes a warning unit (not shown) For example, a warning using sound may be possible, for example.

The configuration of the object tracking system 1 according to the embodiment has been described above.

Hereinafter, the operation of the object tracking system 1 according to one embodiment will be described.

6 and 7 are flowcharts showing an operation method of the object tracking system 1 according to an embodiment of the present invention.

6 is a flowchart illustrating a method of tracking an object sensed by the radar module 11 through the first Kalman filter conversion unit 33 in the control unit 30 and FIG. And tracking the object through the tracking unit 34 and the second Kalman filter conversion unit 35 in the control unit 30 according to an embodiment of the present invention.

As shown in FIG. 6, the radar module 11 operates to start the object tracking control method according to the present invention (S10).

At this time, the radar module 11 detects an object in front of the vehicle (S20). Specifically, the radar module 11 transmits the radar to the forward object, and calculates the three-dimensional coordinates of the forward object based on the time and direction information required for returning (S30). Accordingly, the three-dimensional coordinates of the detected forward object are transmitted to the control unit 30, and the object tracking system 1 performs tracking of the object based on the position and velocity information of the object (S40).

Specifically, the first Kalman filter transformer 33 included in the object tracking system 1 of the present invention uses a simple Kalman filter to track an object. This first Kalman filter transform The unit 33 can track an object including a noise model depending on the distance to the object acquired through the radar module 11 included in the radar sensor unit 10 and the radar resolution.

Next, as shown in FIG. 7, the camera module 21 in the object tracking system 1 operates, and the object tracking control method according to the present invention starts operation (S15).

At this time, the camera module 21 obtains the forward image (S25). Specifically, the forward image obtained by the object tracking system 1 has two-dimensional information, and detects an object contained in the acquired image (S35).

For example, when a sensed object is a pedestrian, a bounding box of a rectangular model may be displayed on the outside of the pedestrian to track the movement of the bounding box.

That is, the object tracking system 1 according to the present invention performs tracking of an object sensed based on the position of the sensed object and the speed of the vehicle 2 on which the object tracking system 1 is mounted (S45). Specifically, in the tracking unit 34 included in the control unit 30, the object sensed by the camera module 21 included in the image capturing unit 20 is extracted by applying the Haar-Like Feature technique, The object is tracked through a classifier using Kernelized least squares.

The second Kalman filter conversion unit 35 included in the control unit 30 performs tracking according to the Kalman filter applied in the two-dimensional plane. In the second Kalman filter conversion unit 35, Tracking can be performed based on position information and speed information.

6 and 7 are flowcharts showing a method of tracking an object sensed through the radar module 11 or the camera module 21 in parallel, but tracking is performed in the same control unit 30, and the radar module 11 The tracking unit 34 and the second Kalman filter conversion unit 35 can not detect the object through the camera module 21 or fail to detect the object through the camera module 21. [ Tracking of the object can be replaced by the tracking method performed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limited to the embodiments set forth herein; It will be understood that various modifications may be made without departing from the spirit and scope of the invention.

1: Object tracking system
2: vehicle

Claims (8)

Capturing an external image of the vehicle;
Detecting an object included in the photographed image;
Transmitting and receiving a radar signal in front of the vehicle to detect an object;
First tracking the object based on three-dimensional information of the object sensed based on the radar signal; And
Performing a second tracking of the object based on the two-dimensional information of the sensed object in the sensed image; Wherein the object tracking control method comprises: The method according to claim 1,
First tracking the object based on three-dimensional information of the object sensed based on the radar signal;
And a Kalman filter including a noise model for the distance between the sensed object and the vehicle and the resolution of the radar sensor. The method according to claim 1,
Performing second tracking of the object based on the two-dimensional information of the sensed object in the photographed image;
Tracking the sensed object based on a CSK (Circulant Tracking by Detection with Kernels) technique; And
And tracking and applying a Kalman filter including a noise model for the position of the sensed object and the speed of the vehicle. 4. The method according to any one of claims 1 to 3,
Wherein the first tracking information and the second tracking information are interchanged. An image capturing unit capturing an external image of the vehicle and detecting an object included in the image;
A radar sensor unit for transmitting / receiving a radar signal to the front of the vehicle and sensing an object;
And a controller for first tracking the object on the basis of the three-dimensional information of the object sensed based on the radar signal and secondly tracking the object based on the two-dimensional information of the sensed object in the sensed image The object tracking system of the vehicle. 6. The method of claim 5,
And a display unit for displaying an object included in the image by a bounding box and displaying the object to a driver. The method according to claim 6,
The control unit
A first Kalman filter converting unit for applying and tracking a Kalman filter including a noise model for the distance between the vehicle and the resolution of the radar sensor based on the three-dimensional information of the sensed object;
A tracking unit tracking based on the two-dimensional information of the sensed object by the CSK (Circulant Tracking by Detection with Kernels) technique; And
And a second Kalman filter converting unit for applying and tracking a Kalman filter including a noise model for the two-dimensional position of the sensed object and the speed of the vehicle. 8. The method of claim 7,
Wherein the first tracking information of the first Kalman filter conversion unit and the second tracking information of the second Kalman filter conversion unit are interchanged.

Images