KR102469836B1 - Vehicle tracking method and device considering occlusion - Google Patents

Abstract

The present invention discloses a vehicle tracking method and apparatus considering occlusion. According to the present invention, it includes a processor and a memory connected to the processor, wherein the memory, when a plurality of vehicle lights are detected in a front image input through a camera, two vehicle lights that satisfy a preset condition are displayed on the vehicle. determining as a light pair, generating a plurality of vehicle light pair trackers representing the trajectories of each of the plurality of vehicle light pairs, and matching the vehicle light pair detected in the current frame with the plurality of vehicle light pair trackers through a preset matching condition and if the vehicle light pair detected in the current frame does not match the plurality of vehicle light pair trackers in the previous frame, it is determined whether an occluded situation occurs, and whether the matching occurs and the occurrence of the occluded situation. There is provided a vehicle tracking device storing program instructions executed by the processor to determine whether to update the plurality of vehicle light pair trackers through determination of whether or not.

Description

Vehicle tracking method and device considering occlusion {Vehicle tracking method and device considering occlusion}

The present invention relates to a vehicle tracking method and apparatus considering occlusion.

Recently, various driver assistance systems are being applied to vehicles, and even self-driving vehicles are provided.

In vehicles equipped with such driver assistance systems or self-driving vehicles, it is important to detect a vehicle located in front to prevent a collision or collision.

In general, vehicle exterior feature information is mainly used for vehicle detection. In daylight conditions, vehicles can be detected based on their color, shape, shadows and corners. However, there is a problem in that it is difficult to use the vehicle exterior feature information in a dark night situation.

Recently, a method using an Adaboost classifier or a Haar feature for detecting headlights or taillights of a vehicle has been proposed.

However, the method using the conventional Adaboost classifier and the Haar feature has a problem in that it is only possible to detect headlights and difficult to detect taillights.

In addition, a method using a Hue-Saturation-Value (HSV) color space has a problem in that it is possible to detect only a tail light and difficult to detect a headlight.

Even if the vehicle light is detected, when the vehicle light is tracked in a single frame, there is a problem in that the accuracy of the vehicle tracking is greatly reduced.

Furthermore, vehicle light pairing to describe one vehicle should be performed for vehicle identification, but conventionally, vehicle light pairing cannot be accurately performed in consecutive frames because only spatial features are used.

In addition, for accurate vehicle tracking, a situation in which vehicle lighting is obscured by the approach of two different vehicles must be considered.

Korean Patent Publication No. 10-2015-0002038

In order to solve the above problems of the prior art, the present invention is to propose a vehicle tracking method and apparatus considering occlusion that can increase the tracking accuracy of a vehicle light pair.

In order to achieve the above object, according to an embodiment of the present invention, as a vehicle tracking device considering vehicle lighting occlusion. processor; And a memory connected to the processor, wherein the memory determines two vehicle lights that satisfy a preset condition as a vehicle light pair when a plurality of vehicle lights are detected in a front image input through a camera, Creating a plurality of vehicle light pair trackers representing the trajectories of each of the plurality of vehicle light pairs, determining whether the vehicle light pair detected in the current frame matches the plurality of vehicle light pair trackers through a preset matching condition, and When the vehicle light pair detected in the frame is not matched to the plurality of vehicle light pair trackers in the previous frame, it is determined whether an occlusion situation occurs, and the matching is determined and the occlusion situation occurs. A vehicle tracking device is provided that stores program instructions executed by the processor to determine whether to update a plurality of vehicle light pair trackers.

The program instructions determine whether the predicted position of the left or right vehicle light included in the first vehicle light pair tracker in the previous frame matches the remaining vehicle light trackers except for the first vehicle light pair tracker. It is possible to determine whether a load situation has occurred.

The program instructions are such that the predicted positions of the left and right vehicle lights included in the first vehicle light pair tracker in the previous frame are two vehicle light trackers included in the remaining vehicle light trackers excluding the first vehicle light pair tracker. It is possible to determine whether the occluded situation has occurred through whether or not it matches to .

The program instructions may group a plurality of vehicle light pairs into one vehicle light pair describing one vehicle using a preset grouping condition.

The program instructions include whether the horizontal distances of the first vehicle light pair tracker and the second vehicle light pair tracker are within a predetermined range, the minimum value of the widths of the first vehicle light pair tracker and the second vehicle light pair tracker, and their horizontal distances. Is greater than a preset threshold, and whether the ratio of the maximum value and the minimum value of the widths of the first vehicle light pair tracker and the second vehicle light pair tracker is greater than the preset threshold, so that a plurality of vehicle light pairs are divided into one vehicle light pair. You can decide whether or not to group by .

The program instructions generate a vehicle light tracker for tracking each of the plurality of vehicle lights when a plurality of vehicle lights are detected in a front image input through a camera, and in frame t-1, among the plurality of vehicle lights Predicting the position of the first vehicle light tracker for tracking the first vehicle light in frame t, and overlapping scores of the predicted first vehicle light tracker with the second vehicle light detected in the t frame (current frame) It is possible to determine whether or not exceeds a preset threshold, and determine whether or not to update the first vehicle light tracker through the determination.

The overlap score may be defined as a ratio of an overlapping area to an area of a larger bounding box among a first bounding box defining the first vehicle light tracker and a second bounding box defining the second vehicle light.

The program commands search for vehicle lighting candidate regions in the input front image, and input feature values of the searched vehicle lighting candidate regions to a pre-learned classifier so that the vehicle lighting candidate regions are classified as headlights, taillights, and similar lights. However, the pre-learned classifier may take as input the Haar feature value extracted from the image included in the training dataset and the average value of the a* channel of the Lab color space.

According to another aspect of the present invention, a method for tracking a vehicle considering occlusion of vehicle lights in a device including a processor and a memory, when a plurality of vehicle lights are detected in a front image input through a camera, in advance determining two vehicle lights that satisfy a set condition as a vehicle light pair; generating a plurality of vehicle light pair trackers representing trajectories of each of the plurality of vehicle light pairs; determining whether a vehicle light pair detected in a current frame matches a plurality of vehicle light pair trackers through a preset matching condition; and determining whether an occluded situation occurs when the vehicle light pair detected in the current frame does not match the plurality of vehicle light pair trackers in the previous frame. and determining whether to update the plurality of vehicle light pair trackers by determining whether the match and whether the occlusion situation occurs are provided.

According to another aspect of the present invention, a computer readable program for performing the method described above is provided.

According to the present invention, since the vehicle light pair is tracked in consideration of occlusion, it is possible to increase the accuracy of vehicle tracking.

1 is a diagram showing the configuration of a vehicle tracking device according to a preferred embodiment of the present invention.
2 is a diagram showing a classifier learning process according to a preferred embodiment of the present invention.
3 shows the average value of the a* channel obtained from 300 headlamp images and taillight images, respectively.
4 is a diagram illustrating a process of detecting a headlight and a taillight according to the present embodiment.
5 is a diagram illustrating a virtual horizontal line for setting a region of interest according to the present embodiment.
6 illustrates a contour of a vehicle lighting candidate region according to an exemplary embodiment.
7 is a diagram illustrating a bounding box according to the present embodiment.
8 shows a class classification result according to this embodiment.
9 illustrates horizontal/vertical distance, width, and height when two vehicle lights exist according to the present embodiment.
10 is a diagram illustrating a vehicle light pair verification result according to an embodiment of the present invention.
11 is a diagram illustrating an occlusion situation according to an embodiment of the present invention.
12 is a diagram showing a result of grouping vehicle light pairs according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The present invention acquires a front image from a camera installed in a vehicle at night, detects and tracks the headlights and taillights of the vehicle in the obtained front image.

Here, the headlights and taillights of the vehicle are defined as vehicle lights, and the vehicle tracking device detects the vehicle lights in the front image using a pre-learned classifier based on machine learning, tracks the detected vehicle lights, and Carry out vehicle light pairing corresponding to the vehicle of

After vehicle light pairing, the vehicle is tracked considering the occlusion situation.

1 is a diagram showing the configuration of a vehicle tracking device according to a preferred embodiment of the present invention.

Referring to FIG. 1 , the vehicle tracking device according to the present embodiment may include a processor 100 and a memory 102 .

The processor 100 may include a central processing unit (CPU) capable of executing a computer program or other virtual machines.

Memory 102 may include a non-volatile storage device such as a non-removable hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. Memory 102 may also include volatile memory, such as various random access memories.

Program instructions executable by the processor 100 are stored in such a memory 102 .

Program instructions according to this embodiment determine, when a plurality of vehicle lights are detected in a front image input through a camera, two vehicle lights that satisfy a preset condition as a vehicle light pair, and each of the plurality of vehicle light pairs Generates a plurality of vehicle light pair trackers representing the trajectory of, determines whether the vehicle light pair detected in the current frame matches the plurality of vehicle light pair trackers through a preset matching condition, and detects the vehicle light in the current frame If the pair does not match the plurality of vehicle light pair trackers in the previous frame, it is determined whether an occlusion situation has occurred, and the plurality of vehicle light pair trackers decides whether to update

In addition, in the program instructions according to the present embodiment, the predicted position of the left or right vehicle light included in the first vehicle light pair tracker in the previous frame is matched to the remaining vehicle light trackers excluding the first vehicle light pair tracker. It is determined whether the occlusion situation occurs through whether or not the occlusion occurs.

And, whether the predicted positions of the left and right vehicle lights included in the first vehicle light pair tracker in the previous frame match two vehicle light trackers included in the remaining vehicle light trackers excluding the first vehicle light pair tracker. It is determined whether the occlusion situation has occurred through whether or not the occlusion occurs.

The program instructions perform grouping of multiple vehicle light pairs into one vehicle light pair describing one vehicle using a preset grouping condition.

In grouping, whether the horizontal distance of the first vehicle light pair tracker and the second vehicle light pair tracker is within a predetermined range, the minimum value of the width of the first vehicle light pair tracker and the second vehicle light pair tracker and their horizontal distance A plurality of vehicle light pairs are converted into one vehicle light pair through whether the ratio of the maximum value and the minimum value of the widths of the first vehicle light pair tracker and the second vehicle light pair tracker is greater than a preset threshold value. Decide whether or not to group.

For vehicle tracking according to the present embodiment, detection, tracking, and pairing of vehicle lights must be preceded, and therefore, detection, tracking, and pairing of vehicle lights will be described first.

Vehicle light detection according to the present embodiment is performed through a pre-learned classifier, and the pre-learned classifier takes the Haar feature value extracted from the images included in the training dataset and the average value of the a* channel of the Lab color space as input. to carry out learning.

Hereinafter, the classifier learning process according to the present embodiment will be described in detail.

2 is a diagram showing a classifier learning process according to a preferred embodiment of the present invention.

Referring to FIG. 2 , a training dataset including headlights, taillights, and negative images is constructed (step 200).

Here, the pseudo lighting is arbitrarily taken rather than a headlight or a tail light, and may include lighting by a vehicle body, a road surface, a road sign, or the like.

Prior to feature extraction from this training dataset, the training images are resized to the same dimension.

The RGB values of each image in the above training dataset are converted into grayscale images (step 202), and at the same time, the RGB values of each image are converted into Lab images (step 204).

Then, Haar features are extracted from the grayscale image converted in step 202 (step 206).

Here, Haar features can be extracted using two rectangles and three rectangles of the same size and shape.

Since Haar feature extraction is known to those skilled in the art, a detailed description thereof will be omitted.

Also, Lab color features are extracted from the Lab image (step 208).

Lab images (L*a*b* color space) are also denoted as L*a*b*. The L* value indicates brightness. L* = 0 indicates black, and L* = 100 indicates white.

a* indicates which side is biased between red and green. If a* is negative, the color leans toward green, and if a* is positive, the color leans toward red/purple. b* represents yellow and blue. If b* is negative, it is blue, and if b* is positive, it is yellow.

Step 208 is a process of calculating an average value of the a* channel in the Lab image for detecting a taillight at night.

The average value of the a* channel according to the present embodiment may be used to distinguish a front light from a tail light.

3 shows the average value of the a* channel obtained from 300 headlamp images and taillight images, respectively.

Referring to FIG. 3 , it can be seen that the average value of the a* channel of the headlight image is lower than 140, whereas the average value of the a* channel of the taillight image is higher than 140. Therefore, the average value of the a* channel can be a useful feature for distinguishing both headlights and taillights.

As described above, the Haar features and Lab color features extracted from the grayscale image are combined (step 210), and the combined features are input to the multiclass Adaboost classifier (step 212).

Step 212 means that preprocessed data like steps 202 to 208 are input for learning of the Adaboost classifier.

According to the present embodiment, it is determined whether a front light or a tail light is located in a front image photographed through a camera through a classifier for which learning has been completed.

4 is a diagram illustrating a process of detecting a headlight and a taillight according to the present embodiment.

Referring to FIG. 4 , the vehicle tracking device receives a front image through a camera mounted on the vehicle (step 400).

Here, the forward image is a visible image of RGB channels.

The vehicle tracking device searches for vehicle lighting candidate regions in the front image (step 402).

Step 402 may be a process of searching for bright blobs in the front image.

In the present embodiment, since only vehicle lighting is selectively detected and road lighting and traffic lights other than the vehicle are to be removed, a bright blob can be searched for within a preset region of interest (ROI) in the forward image.

As shown in FIG. 5 , the ROI is an area that can affect the vehicle and may be defined as an area below a virtual horizon.

In addition, in step 402, a threshold value such as an existing automatic multilevel thresholding or HSV color thresholding technique may be applied for bright blob search.

Then, a binary image including bright blobs is generated using a preset threshold (step 404).

5 is a diagram illustrating a binary image through application of a threshold value according to the present embodiment.

Step 404 converts a bright area that can be a vehicle lighting candidate into 1 and the remaining areas into 0 by a brightness value greater than a threshold value in the ROI area of the front image.

The vehicle tracking device searches for contour lines using the binary image generated in step 404 (step 406).

6 illustrates a contour of a vehicle lighting candidate region according to an exemplary embodiment.

Steps 400 to 406 described above may be defined as a bright blob segmentation process.

After the contour line search, the vehicle tracking device detects the headlights and taillights using the pre-learned classifier (step 408).

After the bright blob segmentation process, the contour and location of the bright blob within the region of interest can be determined by the position of the bounding box surrounding the bright blob.

7 is a diagram illustrating a bounding box according to the present embodiment.

Referring to FIG. 7 , the location of the bounding box is defined by left, top, right, and bottom coordinates.

In step 408, the position of the bounding box surrounding the extracted bright blob is used to extract a subimage from the input frontal image.

Then, the bright blob features are calculated using the feature extraction process of FIG. 3 .

Finally, a pretrained classifier performs class labeling. Here, class labeling determines whether a bright blob included in the front image is a front light, a tail light, or a similar light.

FIG. 8 shows a class classification result according to the present embodiment. In FIG. 8, a white rectangle indicates a front light and a yellow rectangle indicates a tail light.

After the vehicle lighting is detected, the vehicle tracking device according to the present embodiment performs vehicle lighting tracking in consecutive frames.

To describe vehicle light tracking, the following terms are predefined.

및

는 각각 관심 영역(ROI)에서 i번째 차량 조명 및 프레임 t에서 검출된 차량 조명의 개수를 나타낸다.

and

denotes the number of vehicle lights detected in the ith vehicle light and the frame t in the region of interest (ROI), respectively.

는 차량 조명

의 위치로서,

를 포함하는 경계 상자의 상측(top), 좌측(left), 하측(bottom) 및 우측(right) 좌표를 나타낸다.

the vehicle lights

As the position of

Indicates the top, left, bottom, and right coordinates of the bounding box containing

및

는

의 폭과 높이를 나타낸다.

and

Is

represents the width and height of

와

는 프레임 t에서 두 개의 차량 조명

와

사이의 수평 거리 및 수직 거리를 나타내며, 이는 다음과 같이 정의된다.

Wow

is the two vehicle lights in frame t

Wow

represents the horizontal distance and vertical distance between

9 illustrates horizontal/vertical distance, width, and height when two vehicle lights exist according to the present embodiment.

9, the horizontal distance of the two vehicle lights is the value obtained by subtracting the smaller right coordinate (r) from the larger left coordinate (l) of the two vehicle lights, and the vertical distance is the larger lower coordinate ( b) minus the smaller upper coordinate (t).

Here, if the horizontal or vertical projection of the two vehicle lights overlaps, the above distance value has a negative number.

와

의 수평 또는 수직 투영에서의 오버랩 정도는 다음과 같이 정의된다. Both vehicle lights in frame t

Wow

The degree of overlap in horizontal or vertical projection of is defined as:

The vehicle light tracker is initialized when a vehicle light appears for the first time in the region of interest to obtain position, size, and speed information of the same vehicle light in successive frames.

는 연속되는 프레임에서 차량 조명

의 궤적(차량 조명에 대한 경계 상자의 궤적)을 나타내는 차량 조명 추적기를 나타낸다.

vehicle lights in successive frames.

represents the vehicle light tracker that represents the trajectory of (the trajectory of the bounding box for the vehicle light).

를 추적한다. The vehicle light tracker tracks the vehicle lights as they appear at frame t from the initial frame they first appear.

to track

는 프레임 t에서 검출된 일련의 차량 조명을 나타낸다.

represents a series of vehicle lights detected in frame t.

는 프레임 t-1에서 일련의 차량 조명 추적기를 나타낸다.

denotes a series of vehicle light trackers at frame t-1.

와

에 대한 오버랩 스코어

를 정의한다. 여기서, 이들 차량 조명은 서로 다른 프레임 t 및 t'에서 검출된다. According to the present embodiment, two vehicle lights (two bounding boxes) so that the vehicle lights can be tracked even when the predetermined speed is exceeded,

Wow

Overlap score for

define Here, these vehicle lights are detected at different frames t and t'.

where A is the area of the bounding box containing the illuminated area.

는 새로 들어오는 프레임에서 새로 검출된 차량 조명의 추적 상태를 결정하기 위해 사용된다.

is used to determine the tracking state of a newly detected vehicle light in a new incoming frame.

The time interval between two consecutive frames is assumed to be short so that the vehicle speed does not change significantly, for example, the time interval may be 33 ms, considering 30 frames per second.

Therefore, the motion information is used to predict the position of the input vehicle light.

For vehicle lights tracked in at least two frames, the motion vector can be calculated as:

는 다음 프레임에서 차량 조명 추적기

의 예측된 위치를 나타낸다.

is the vehicle light tracker in the next frame

represents the predicted position of

During the light tracking process, the vehicle light tracker can be in one of three possible states.

이 이전 프레임X에서 기 존재하는 추적기

의 예측된 위치

에 매칭되면, 추적기

는

와 연관되고, 상기한 추적기는 현재 프레임에서 일련의 조명 추적기

에 추가된다. 1. Update: If the vehicle light detected in the current frame t

Pre-existing tracker in this previous FrameX

the predicted position of

If matched to , the tracker

Is

, wherein said tracker is a series of light trackers in the current frame.

is added to

Matching conditions are as follows.

는

이

와 연관되는지 여부를 확인하기 위한 매칭 임계치이다. here,

Is

this

It is a matching threshold for checking whether it is associated with.

의 값은 실험적으로 결정되며 0.2일 수 있다.

The value of is determined experimentally and may be 0.2.

이 이전 프레임에서 기 존재하는 차량 조명 추적기

의 예측된 위치

에 매칭되지 않으면, 새로운 차량 조명 추적기가 생성되고, 생성된 차량 조명 추적기가 현재 프레임에서 일련의 차량 조명 추적기

에 추가된다. 2. Appear: If the vehicle light detected in the current frame t

Pre-existing vehicle light tracker from this previous frame

the predicted position of

, a new vehicle light tracker is created, and the created vehicle light tracker is the set of vehicle light trackers in the current frame.

is added to

의 예측된 위치

와 현재 프레임 t에서 새로 검출된 차량 조명

와 매칭되지 않는 상태이다. 3. Disappear: if Existing vehicle lighting tracker

the predicted position of

and the newly detected vehicle lights in the current frame t

It is a state that does not match with

Since the existing vehicle light tracker may be temporarily obscured, the apparatus according to the present embodiment maintains the existing vehicle light tracker in a preset number of consecutive frames.

가 일시적으로 사라지는 경우, 이의 위치는 상기한 수학식 7에 의해 업데이트된다. 반면, 만일 차량 조명 추적기

가 3개 이상의 연속된 프레임에 대해 매칭되지 않으면 차량 조명이 사라진 것으로 간주하고, 일련의 조명 추적기

을 제거한다. vehicle light tracker

When n temporarily disappears, its position is updated by Equation 7 above. On the other hand, if the vehicle light tracker

is not matched for 3 or more consecutive frames, the vehicle lights are considered gone, and the sequence of light trackers

Remove

After the vehicle light tracking process, the vehicle tracking device according to the present embodiment performs a vehicle light pairing process for vehicle identification.

Vehicle light pairing is the process of determining a vehicle light pair to describe one of a plurality of vehicle lights detected within a region of interest and tracked in successive frames.

Since vehicle lights are arranged symmetrically in a pair (pair) in one vehicle, vehicle light pairing is performed to determine the position of each vehicle located on the road.

Conventionally, only spatial characteristics such as area, width, height, vertical coordinate, and correlation are used to determine similarity between two vehicle lights within one frame, but this is not sufficient for pairing vehicle lights.

If there are multiple vehicles with the same distance in front of the host vehicle, erroneous light pairing may occur because the spatial characteristics of the lights of these vehicles are similar.

According to an embodiment of the present invention, vehicle light pairing using spatiotemporal information is proposed in order to reduce detection errors.

The vehicle light pairing method according to the present embodiment includes two steps of generating and verifying a candidate vehicle light pair.

The following pairing conditions are set to create a set of candidate vehicle light pairs.

1. At frame t the two vehicle lights are highly overlapped in vertical projection. That is, two vehicle light trackers (first and second vehicle light trackers) corresponding to a vehicle light pair (first vehicle light and second vehicle light) included in one vehicle have a preset degree of overlap in vertical projection. A first threshold must be exceeded.

Here, the degree of overlap of the different vehicle lights is defined in Equation 4.

2. The two vehicle lights have similar heights, and the vehicle light pair included in one vehicle must have a ratio (ratio of heights) of a vehicle light having a large height and a vehicle light having a small height exceed a second predetermined threshold value. do.

3. The ratio of the width and height of the pair must be within the range of the third threshold value or more and the fourth threshold value or less as follows.

및

는 차량 조명 페어링의 특성을 결정하기 위한 임계치이다. here,

and

is the threshold for determining the characteristics of the vehicle light fairing.

Statistically these values are chosen to be 0.7, 0.7, 2.0 and 14.0, respectively.

The result of the first step for generating candidate vehicle light pairs is a set of candidate vehicle light pairs, but may contain some degree of error because one vehicle light may form a pair with another vehicle light.

Therefore, according to the present embodiment, verification of the candidate vehicle light pair is performed.

The pairing score of the candidate vehicle light pair may be used to eliminate erroneous pairs.

One of the plurality of candidate vehicle light pairs with the highest pairing score is retained to describe the vehicle, and the remaining pairs are removed from the set of candidate vehicle light pairs.

는 두 개의 조명

및

의 시공간적 특징의 조합으로, 시공간적 특징은 추적되는 프레임의 개수, 변위(displacement), 크기 및 상관관계를 포함하며, 다음과 같이 정의된다. pairing score

has two lights

and

As a combination of spatiotemporal features of , the spatiotemporal features include the number, displacement, size, and correlation of frames to be tracked, and are defined as follows.

는 두 개의 차량 조명 각각에 상응하는 차량 조명 추적기의 최대 개수와 최소 개수의 비율이다. here,

is the ratio of the maximum number to the minimum number of vehicle light trackers corresponding to each of the two vehicle lights.

는 제1 차량 조명을 추적하는 제1 차량 조명 추적기 및 제2 차량 조명을 추적하는 제2 차량 조명 추적기가 유지되는 프레임의 개수(즉, 제1 및 제2 차량 조명이 전방 영상에서 검출되는 개수)의 최대값 및 최소값의 비율로 정의될 수 있다. For example, when a first vehicle light and a second vehicle light are determined as a candidate vehicle light pair,

Is the number of frames in which the first vehicle light tracker tracking the first vehicle light and the second vehicle light tracker tracking the second vehicle light are maintained (ie, the number of first and second vehicle lights detected in the front image) It can be defined as the ratio of the maximum and minimum values of

는 가장 최근 복수의(예를 들어, 4개) 프레임에서 두 개의 차량 조명 추적기

및

의 전체 변위 d의 비율, 즉 변위의 최대값 및 최소값의 비율이다.

is the two vehicle light trackers in the most recent multiple (e.g., four) frames.

and

is the ratio of the total displacement d of , i.e. the ratio of the maximum and minimum values of the displacement.

Because the two vehicle lights in a pair must move coherently over time, a motion vector is used to calculate the displacement of the vehicle lights.

는 후보 차량 조명 페어에 대응되는 두 개의 차량 조명 추적기의 크기의 비율이고,

는 두 개의 차량 조명 추적기

및

의 히스토그램 사이의 상관관계이다.

Is the ratio of the sizes of the two vehicle light trackers corresponding to the candidate vehicle light pair,

has two vehicle light trackers

and

is the correlation between the histograms of

Here, the Bhattacharyya coefficient is used to compare 3D histograms of left and right illumination.

와

는 실험적으로 결정되며, 0.2, 0.2, 0.3 및 0.3일 수 있다. In Equation 12, the coefficient

Wow

is determined experimentally and may be 0.2, 0.2, 0.3 and 0.3.

10 is a diagram illustrating a vehicle light pair verification result according to an embodiment of the present invention.

After using the pairing score to remove pairs with low pairing scores, a series of vehicle light pairs describing the vehicle ahead are retrieved, and all vehicle light trackers not used for pairing are retained.

Hereinafter, a vehicle tracking process considering occlusion will be described.

Through the vehicle light pairing process described above, all vehicles that may be positioned in front of the host vehicle can be discovered, which can reduce erroneous vehicle detection.

On the other hand, there is still the possibility of not performing the vehicle light pairing properly, and the system cannot correct this considering only a single frame.

Missing vehicle light detection can occur due to several problems, which can be due to the disappearance of the vehicle light or the distorted appearance (size, shape and color) of one lamp in the light pair.

The following issues can cause pairing errors.

(1) Partial vehicle occlusion

(2) When one vehicle approaches another vehicle too closely;

(3) Vehicles turning left or right

(4) Blooming effect of only one light in an oncoming vehicle

To solve this problem, a vehicle tracking method considering occlusion is proposed.

In the vehicle tracking process, the detection result is refined using spatiotemporal features of the vehicle in successive frames.

To facilitate the vehicle tracking process, the following terms are defined.

,

는 차량 조명 페어의 세트이고,

는 검색된 잔여 차량 조명 추적기를 나타낸다.

는 차량 조명 페어의 세트를 의미하며,

는 프레임 t에서 i번째 차량 조명 페어이고,

및

는 각각 하나의 차량 i의 서로 페어링된 조명을 나타낸다.
여기서, Pt는 차량 조명 페어, Pi(1)와 Pi(2)는 서로 페어를 이루는 제1 차량 조명 및 제2 차량 조명이다.

,

is a set of vehicle light pairs,

denotes the retrieved remaining vehicle light trackers.

Means a set of vehicle light pairs,

Is the ith vehicle light pair in frame t,

and

denotes lights paired with each other of one vehicle i, respectively.
Here, Pt is a vehicle light pair, and Pi(1) and Pi(2) are a first vehicle light and a second vehicle light forming a pair.

는 차량 조명 페어

의 궤적(차량 조명 페어에 대한 경계 상자의 궤적)을 나타내는 차량 조명 페어 추적기이다.

is a vehicle light pair

is a vehicle light pair tracker that represents the trajectory of (the trajectory of the bounding box for the vehicle light pair).

의 위치는

의 경계 상자에 의해 결정된다. vehicle light pair

the location of

is determined by the bounding box of

는 이전 프레임 t-1에서 차량 조명 페어 추적기의 세트를 나타낸다.

denotes the set of vehicle light pair trackers in the previous frame t-1.

During the vehicle tracking process, the vehicle light pair tracker can be in one of the following four states.

가 이전 프레임 t-1에서의 차량 조명 페어 추적기

에 매칭되면, 차량 조명 페어 추적기

는

와 연관되며, 현재 프레임에서의 차량 조명 페어 추적기의 세트

에 추가된다. 1. Update: If the vehicle light pair at the current frame t

is the vehicle light pair tracker at the previous frame t-1

If matched to, Vehicle Light Pair Tracker

Is

Associated with is the set of vehicle light pair trackers in the current frame.

is added to

A matching condition for vehicle lighting pairs is defined by the overlap score in Equation 5 above and the ratio of pair widths in two frames t-1 and t.

That is, if the overlap score of the vehicle light pair in the current frame t and the vehicle light pair tracker in the previous frame is greater than a preset threshold, and the ratio of the maximum value to the minimum width of the width is greater than the preset threshold, vehicle light pair matching It is judged that the condition is satisfied.

와

는

가

와 연관될 수 있는 여부를 검증하기 위한 매칭 임계치이고, 실험적으로 0.3 및 0.7로 설정될 수 있다. here,

Wow

Is

go

It is a matching threshold for verifying whether it can be associated with, and can be experimentally set to 0.3 and 0.7.

가 이전 프레임 t-1에서의 차량 조명 페어 추적기

에 매칭되지 않으면 새로운 차량 조명 페어 추적기

가 생성되고 현재 프레임에서 차량 조명 페어 추적기 세트

에 추가된다. 2. Appear: If the vehicle light pair at the current frame t

is the vehicle light pair tracker at the previous frame t-1

If not matched to the new vehicle light pair tracker

is created and sets the vehicle light pair tracker in the current frame.

is added to

3. Disappear: If the following two conditions are satisfied, the vehicle light pair tracker determines that the previously tracked vehicle light has disappeared.

가 현재 프레임의 어떠한 차량 조명 페어

와도 매칭되지 않고, 두 번째로,

에서 좌우 조명의 예측된 위치가 어떠한 차량 조명 추적기

와도 매칭되지 않는 경우이다. First, the existing (old frame) vehicle light pair tracker

is any vehicle light pair in the current frame

does not match even with, and secondly,

What vehicle light tracker is the predicted position of the left and right lights in

In case it does not match with .

는 현재 프레임에서 차량 조명 페어 추적기 세트

에 추가되지 않는다. At this time, vehicle light pair tracker

sets the vehicle light pair tracker in the current frame

not added to

4. Occlusion: If the following two conditions are satisfied, the vehicle light pair tracker determines that the vehicle lights are occluded.

가 현재 프레임의 어떠한 차량 조명 페어

와도 매칭되지 않고, First, the existing (old frame) vehicle light pair tracker

is any vehicle light pair in the current frame

does not match with

에서 좌측 또는 우측 차량 조명의 예측된 위치(수학식 8에 의해 예측된 위치)가 잔여 차량 조명 추적기에 매칭(

)되거나, Second, as shown in FIG. 11A, in a situation where pairing fails due to the disappearance of one vehicle light in one vehicle light pair.

The predicted position of the left or right vehicle light (the position predicted by Equation 8) matches the remaining vehicle light tracker in (

) or

의 좌측 및 우측 차량 조명 모두의 예측된 위치가 두 개의 차량 조명 추적기에 매칭(

및

)되는 경우이다. Otherwise, in a situation where there is distortion of one vehicle light in one vehicle light pair, as shown in FIG. 11B,

The predicted positions of both the left and right vehicle lights of match the two vehicle light trackers (

and

) is the case.

는

를

에서 하나 또는 두 개의 차량 조명 추적기와 연관지음으로써 업데이트되고, 차량 조명 페어 추적기

에 추가된다. Depending on whether the above first and second situations occur, the vehicle light pair tracker

Is

cast

updated by associating with one or two vehicle light trackers in the vehicle light pair tracker

is added to

가 검색된다. 다음으로, 두 개의 차량 조명 페어를 갖는 차량에 대해, 두 개의 차량 조명 페어를 하나의 차량 조명 페어로 그룹화하기 위한 규칙을 적용한다. At the end of the vehicle tracking process, the vehicle light pair tracker at the current frame t

is searched. Next, for vehicles having two vehicle light pairs, a rule for grouping the two vehicle light pairs into one vehicle light pair is applied.

와

는 이들의 다음의 규칙을 만족하면 하나의 차량 조명 페어 추적기로 그룹화된다. Two light pair trackers

Wow

are grouped into one vehicle light pair tracker if they satisfy the following rules.

In grouping, whether the horizontal distance of the first vehicle light pair tracker and the second vehicle light pair tracker is within a predetermined range, the minimum value of the width of the first vehicle light pair tracker and the second vehicle light pair tracker and their horizontal distance A plurality of vehicle light pairs are converted into one vehicle light pair through whether the ratio of the maximum value and the minimum value of the widths of the first vehicle light pair tracker and the second vehicle light pair tracker is greater than a preset threshold value. Decide whether or not to group.

1. Two vehicle light pairs should be vertically close to each other

Two vehicle light pairs are vertically close to each other when the vertical distance of the first vehicle light pair tracker and the second vehicle light pair tracker is within a predetermined range.

2. The two vehicle light pairs will overlap high in horizontal projection

Two vehicle light pairs are highly overlapped in horizontal projection when the minimum widths of the first vehicle light pair tracker and the second vehicle light pair tracker and their horizontal distance are greater than a preset threshold.

3. Both vehicle light pairs have similar widths

Here, the two vehicle light pairs have similar widths when the ratio between the maximum and minimum widths of the first vehicle light pair tracker and the second vehicle light pair tracker is greater than a preset threshold.

와

는 하나의 차량이 형성하는 복수의 페어들을 그룹화하기 위한 임계치이고, 통계적으로 각각 0.9 및 0.7로 설정된다. In post-processing,

Wow

is a threshold value for grouping a plurality of pairs formed by one vehicle, and is statistically set to 0.9 and 0.7, respectively.

12 is a diagram showing a result of a post-processing process according to the present embodiment.

As shown in FIG. 12 , if two or more vehicle light pairs are determined to describe one vehicle, these vehicle light pairs may be grouped into one.

The embodiments of the present invention described above have been disclosed for illustrative purposes, and those skilled in the art having ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions will be considered to fall within the scope of the following claims.

Claims (10)

A vehicle tracking device considering vehicle lighting occlusion,
processor; and
Including a memory coupled to the processor,
the memory,
When a plurality of vehicle lights are detected in the front image input through the camera, two vehicle lights that satisfy a preset condition are determined as a vehicle light pair;
Creating a plurality of vehicle light pair trackers representing the trajectories of each of the plurality of vehicle light pairs;
Determining a plurality of candidate vehicle light pairs using at least one of coordinates, heights, and degree of overlap of each of the plurality of vehicle light trackers;
Calculating a pairing score using at least one of the number of frames maintained by a vehicle light tracker for tracking each vehicle light included in the plurality of candidate vehicle light pairs, displacement, size, and correlation;
Using the calculated pairing score, determining one or more vehicle light pairs among the plurality of candidate vehicle light pairs as a vehicle light pair corresponding to one vehicle;
Determine whether a vehicle light pair detected in a current frame matches a plurality of vehicle light pair trackers through a preset matching condition;
When the vehicle light pair detected in the current frame does not match the plurality of vehicle light pair trackers in the previous frame, determining whether an occlusion situation occurs;
To determine whether to update the plurality of vehicle light pair trackers by determining whether the matching and whether the occlusion situation occurs,
storing program instructions executed by the processor;
The program instructions are
Of the plurality of vehicle light trackers, the degree of overlap in the vertical projection of the first vehicle light tracker tracking the first vehicle light and the second vehicle light tracker tracking the second vehicle light, a height ratio, and a width-to-height ratio Creating a plurality of candidate vehicle lighting pairs using
The pairing score is the ratio of the maximum value and the minimum value of the number of frames in which the first vehicle light tracker and the second vehicle light tracker are maintained, the first vehicle light tracker and the second vehicle in the most recent plurality of frames. It is determined by summing the ratio of the maximum and minimum displacement of the light tracker, the ratio of the size, and the value obtained by multiplying the histogram correlation by a preset weight, respectively, the ratio of the maximum and minimum values of the number of frames, and the maximum value of the displacement. and a weight multiplied by the ratio of the magnitude and the histogram correlation is set higher than a weight multiplied by the ratio of the minimum value,
The program instructions are
Whether or not the occluded situation occurs through whether the predicted positions of the left or right vehicle lights included in the first vehicle light pair tracker in the previous frame are matched to the remaining vehicle light trackers excluding the first vehicle light pair tracker. A vehicle tracking device that determines the delete According to claim 1,
The program instructions are
Whether the predicted positions of the left and right vehicle lights included in the first vehicle light pair tracker in the previous frame match the two vehicle light trackers included in the remaining vehicle light trackers excluding the first vehicle light pair tracker Vehicle tracking device for determining whether the occluded situation occurs through. According to claim 1,
The program instructions are
A vehicle tracking device that groups a plurality of vehicle light pairs into one vehicle light pair describing one vehicle by using a preset grouping condition. delete According to claim 1,
The program instructions are
When a plurality of vehicle lights are detected in the front image input through the camera, a vehicle light tracker for tracking each of the plurality of vehicle lights is generated;
In frame t-1, predicting a position in frame t of a first vehicle light tracker for tracking a first vehicle light among the plurality of vehicle lights;
determining whether an overlap score of the second vehicle light detected in the t frame (current frame) and the predicted first vehicle light tracker exceeds a preset threshold;
The vehicle tracking device for determining whether to update the first vehicle light tracker through the determination. According to claim 6,
The overlap score is defined as a ratio of an overlapping area to an area of a larger bounding box of a first bounding box defining the first vehicle light tracker and a second bounding box defining the second vehicle light. According to claim 1,
The program instructions are
Searching for vehicle lighting candidate regions in the input front image;
Classifying whether the vehicle lighting candidate region is a headlamp, a taillight, and similar lighting by inputting a feature value of the searched vehicle lighting candidate region to a pre-learned classifier,
The pre-learned classifier is learned by taking as input the Haar feature value extracted from the image included in the training dataset and the average value of the a* channel of the Lab color space. A method for tracking a vehicle considering vehicle light occlusion in a device including a processor and a memory, comprising:
determining two vehicle lights that satisfy a preset condition as a vehicle light pair when a plurality of vehicle lights are detected from a front image input through a camera;
generating a plurality of vehicle light pair trackers representing trajectories of each of the plurality of vehicle light pairs;
determining a plurality of candidate vehicle light pairs using at least one of coordinates, heights, and degree of overlap of each of the plurality of vehicle light trackers;
Calculating a pairing score using at least one of the number of frames maintained by a vehicle light tracker for tracking each vehicle light included in the plurality of candidate vehicle light pairs, displacement, size, and correlation;
determining at least one vehicle light pair among the plurality of candidate vehicle light pairs as a vehicle light pair corresponding to one vehicle by using the calculated pairing score;
determining whether a vehicle light pair detected in a current frame matches a plurality of vehicle light pair trackers through a preset matching condition; and
determining whether an occluded situation occurs when the vehicle light pair detected in the current frame does not match the plurality of vehicle light pair trackers in the previous frame; and
Determining whether to update the plurality of vehicle light pair trackers by determining whether the matching and whether an occluded situation occurs,
The determining of the vehicle light pair may include an overlap degree in vertical projection of a first vehicle light tracker tracking the first vehicle light and a second vehicle light tracker tracking the second vehicle light, among the plurality of vehicle light trackers. , generating a plurality of candidate vehicle light pairs using a height ratio and a width-to-height ratio;
The pairing score is the ratio of the maximum value and the minimum value of the number of frames in which the first vehicle light tracker and the second vehicle light tracker are maintained, the first vehicle light tracker and the second vehicle in the most recent plurality of frames. It is determined by summing the ratio of the maximum and minimum displacement of the light tracker, the ratio of the size, and the value obtained by multiplying the histogram correlation by a preset weight, respectively, the ratio of the maximum and minimum values of the number of frames, and the maximum value of the displacement. and a weight multiplied by the ratio of the magnitude and the histogram correlation is set higher than a weight multiplied by the ratio of the minimum value,
Determining whether the occluded situation occurs may include determining that the predicted position of the left or right vehicle lights included in the first vehicle light pair tracker in the previous frame is the remaining vehicle lights other than the first vehicle light pair tracker. A vehicle tracking method for determining whether the occluded situation occurs through whether the vehicle is matched with a tracker. A computer program stored in a computer readable storage medium for performing the method according to claim 9.

Images