KR101761604B1 - Multiple pedestrians detecting, tracking system and control method thereof - Google Patents

Abstract

A multi-pedestrian recognition and tracking system for a vehicle and a control method thereof are disclosed. A multi-pedestrian recognition and tracking method of a vehicle according to an embodiment of the present invention includes a step of photographing an external image of a vehicle, a step of continuously recognizing a plurality of pedestrians from a photographed image, A step of tracking the pedestrian and a step of replacing the pedestrian recognized as the tracked pedestrian when the pedestrian recognition fails.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multi-pedestrian recognition system,

The present invention relates to a pedestrian recognition and tracking system and a control method thereof. More particularly, the present invention relates to a pedestrian recognition system, a pedestrian recognition system, and a control method thereof. More particularly, Pedestrian recognition and tracking system capable of quickly grasping the case.

A system has been developed to notify the driver that a pedestrian is located in front of the vehicle by recognizing the pedestrian from the image photographed by the camera installed in the front of the vehicle in case the driver can not detect the external situation.

Haar Wavelet Feature based method, Histogram of Oriented Gradient (HOG) method and Local Receptive Field (LRF) are mainly studied as pedestrian feature extraction methods for pedestrian detection.

In the HOG method, the normalized HOG is obtained by rotating the histogram of the direction of the HOG feature. This can more reliably characterize the object in comparison with the conventional HOG feature, such as the skew and the size change.

However, although the pedestrian detection method using the HOG has been proved to be good, there is a problem in that it is not suitable for real-time pedestrian tracking in a high-resolution image due to low throughput.

In addition, there are obstacles such as a plurality of artifacts, for example, guard rails, columns, and poles, in an actual road environment requiring multiple pedestrian recognition. There may be a case where a plurality of artefacts overlap, or an obstacle may be extracted as a candidate for a pedestrian, and a detected pedestrian may be missed.

Therefore, in the embodiments of the present invention, when the pedestrian is missed due to the movement of the pedestrian or the movement of the vehicle during the detection of the plurality of pedestrians, the pedestrian detection indicator is displayed using the tracking information generated using the sensing information of the pedestrian We want to continue the pedestrian recognition by replacing the bounding box.

Further, it is intended to reduce the error of the multi-pedestrian recognition system that may occur when a plurality of pedestrians overlap (close).

In addition, when the acceleration or the speed of the pedestrian is not constant during the pedestrian tracking, the accuracy of the pedestrian tracking may be degraded.

According to an aspect of the present invention, there is provided a method for controlling a pedestrian, comprising the steps of: photographing an external image of a vehicle; continuously recognizing a plurality of pedestrians from the photographed image; displaying the recognized pedestrian by a bounding box, Tracking the pedestrian based on a change in the position of the bounding box continuously displayed; And displaying a bounding box by replacing the pedestrian with the tracked pedestrian when the pedestrian recognition fails.

In addition, the change in the position of the bounding box can be calculated based on the speed and acceleration of the recognized pedestrian.

The step of continuously recognizing a plurality of pedestrians from the photographed image can be continuously recognized when the probability that an object recognized in the photographed image is determined as a pedestrian is equal to or greater than a preset threshold value.

In addition, if it is determined that a plurality of pedestrians are blocked, the pedestrian tracking can be terminated.

The determination that the plurality of pedestrians are blocked may be determined based on the Haar-Like Feature technique.

According to another aspect of the present invention, there is provided an image display apparatus including: an image capturing unit capturing an external image of a vehicle; a display unit displaying the captured image; And a step of recognizing a plurality of pedestrians continuously from the photographed image, displaying the recognized pedestrian on the display unit with a bounding box, and tracking the pedestrian based on the position change of the continuously displayed bounding box, And a controller for controlling the display of the bounding box by replacing the pedestrian with the tracked pedestrian when the pedestrian recognition fails.

In addition, the change in the position of the bounding box can be calculated based on the speed and acceleration of the recognized pedestrian.

In addition, the control unit can recognize the recognized pedestrian continuously when the probability that an object recognized in the photographed image is determined as a pedestrian is equal to or greater than a preset threshold value.

In addition, if it is determined that a plurality of pedestrians are blocked, the pedestrian tracking can be terminated.

The determination that the plurality of pedestrians are blocked may be based on the Haar-Like Feature value.

In the embodiments of the present invention, when a pedestrian is missed due to a movement of a pedestrian while detecting a plurality of pedestrians or a movement of a vehicle that detects multiple pedestrians, a detection of a pedestrian through a tracking using a detection information of a pedestrian detected in the past bounding box.

In addition, the pedestrian recognition can be canceled when a plurality of pedestrians overlap each other (when the pedestrians overlap each other), thereby reducing errors in the system that may occur when multiple pedestrians are recognized.

Further, the embodiments of the present invention can improve the accuracy of the pedestrian tracking when the acceleration or the speed of the pedestrian is not constant during the pedestrian tracking.

1 is a block diagram of a multi-pedestrian recognition and tracking system in accordance with an embodiment of the present invention.
2 is a block diagram of a control unit according to an embodiment of the present invention.
3 is a schematic diagram illustrating multiple pedestrian recognition of a multi-pedestrian recognition and tracking system in accordance with an embodiment of the present invention.
4 is a schematic diagram illustrating pedestrian recognition and tracking in accordance with an embodiment of the present invention.
FIG. 5 is a schematic diagram illustrating a pedestrian recognition replacement in accordance with an embodiment of the present invention. FIG.
6 is a flowchart of a method of controlling a multi-pedestrian recognition and tracking system according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments are provided to fully convey the spirit of the present invention to a person having ordinary skill in the art to which the present invention belongs. The present invention is not limited to the embodiments shown herein but may be embodied in other forms. For the sake of clarity, the drawings are not drawn to scale, and the size of the elements may be slightly exaggerated to facilitate understanding.

FIG. 1 is a block diagram of a multi-pedestrian recognition and tracking system 1 of a vehicle according to the present invention, and FIG. 2 is a block diagram of a controller of a multi-pedestrian recognition and tracking system 1 according to an embodiment of the present invention .

Referring to FIG. 1, a multi-pedestrian recognition and tracking system 1 of a vehicle includes an image capturing unit 10, an image processing unit 11, a control unit 20, a display unit 30, and a warning unit 40 .

The image capturing unit 10 may include a camera sensor (not shown) for capturing an image of the front of the vehicle 2, thereby capturing an external image of the vehicle and generating shooting information thereon.

Camera sensors usually use cameras with more than one channel, but usually CMOS (Complementary Metal Oxide Semiconductor) is used as the image sensor. A CMOS image sensor is a semiconductor device that converts exposed images into electrical form and transmits them. However, the present invention is not limited thereto, and may be implemented by a CCD (Change Coupled Device) image sensor. Thus, the image obtained through the camera sensor is transmitted to the image processing unit 11.

That is, an image captured through a camera sensor in the image capturing unit 10 is captured and transmitted to the image processing unit 11.

The image processing unit 11 processes the image received from the image capturing unit 10. It is possible to detect the motion of the object photographed in the image, to extract the region of the object, and to distinguish a plurality of objects. At this time, the corresponding object is an object to be recognized of the image processing. For example, it may be an object such as a landscape, a foreground, a specific object, a pedestrian, and the like.

First, the outline of the object is extracted from the acquired image using the deviation of neighboring RGB values, and the motion of the object can be determined by analyzing the change of the outline in comparison with the previous image.

In addition, an area where motion of an object occurs can be obtained by using the detected area period as a set of setting intervals.

In addition, two or more objects may be merged or divided in one image. In order to distinguish an area of an accurate object, an estimated position and size of the object through object tracking may be calculated to extract the position and area of the object .

In this case, it is possible to distinguish whether the object is a pedestrian according to the analysis of the horizontal and vertical components constituting the contour of the object and the size of the area, and whether the object classified as the pedestrian is detected as a pedestrian with a certain probability is referred to as a score ) Value can be calculated.

The image processing unit 11 transmits information of the extracted object including the image information acquired by the image capturing unit 10 to the control unit 20 at predetermined time intervals.

Accordingly, the image processing unit 11 can detect the motion of the object photographed in the image, the region of the object, and the plurality of object overlapped information to the control unit 20 using the image signal received by the image capturing unit 10 .

The control unit 20 performs overall control of the multi-pedestrian recognition and tracking system 1.

Specifically, the control unit 20 mediates data input / output between various constituent devices included in the multi-pedestrian recognition and tracking system 1 and the control unit 20, and includes a memory 22 for storing programs and data, A main processor 31 for recognizing and tracking a pedestrian using the image information received by the main processor 31, a display unit 30 for displaying pedestrian recognition and tracking results in the main processor 31, And a communication unit (not shown) for communicating with the warning unit 40 that warns the user.

The main processor 21 displays a bounding box on the outer periphery of the pedestrian if a score indicating a degree of accuracy classified as a pedestrian received by the image processing unit 11 is greater than a preset threshold value, . At this time, when a plurality of pedestrians are recognized on the acquired image, as shown in Fig. 3, a bounding box can be displayed with a square on the outer periphery of a plurality of pedestrians.

The main processor 21 receives the information of the object extracted by the image processing unit 11 at predetermined time intervals and continuously displays the bounding box when the object detected from the acquired image is a pedestrian, , The position information of the bounding box is stored in the memory 22 together with the position information of the pedestrian obtained to replace the pedestrian bounding box with the tracked pedestrian.

At this time, the pedestrian can be tracked using the acquired pedestrian recognition information. In addition, when missed pedestrian recognition through tracking, the bounding box may be substituted based on the information tracked by pedestrian recognition.

As described above, the pedestrian tracking using the sensing information of the pedestrian and the path prediction of the pedestrian can be specifically explained with reference to FIG. 2 to FIG.

FIG. 2 is a block diagram of a control unit according to an embodiment of the present invention. FIG. 3 is a schematic view illustrating recognition of multiple pedestrians in a multi-pedestrian recognition and tracking system according to an embodiment of the present invention. FIG. 5 is a schematic diagram illustrating a pedestrian recognition substitution in a pedestrian tracking according to an embodiment of the present invention. FIG. 5 is a schematic diagram illustrating pedestrian recognition and tracking of one of a plurality of pedestrians according to an exemplary embodiment.

Referring to FIG. 2, the controller 20 of the multi-pedestrian recognition and tracking system according to the present invention includes a pedestrian recognition unit 210 and a pedestrian tracking unit 220.

As shown in FIG. 3, if the score that indicates the degree of accuracy classified as a pedestrian received by the image processing unit 11 is greater than a preset threshold value, the pedestrian recognition unit 210 determines whether the pedestrian To display a Bounding Box. At this time, when a plurality of pedestrians are recognized on the acquired image, a bounding box can be displayed with a square on the outer periphery of a plurality of pedestrians.

The pedestrian tracking unit 220 tracks the pedestrian using the accumulated position information of the bounding box of the pedestrian recognized by the pedestrian recognition unit 210. [

Specifically, the information of the bounding box of the pedestrian recognized by the pedestrian recognition unit 210 can track the pedestrian through N frame information set in advance. In this case, the N frames set in advance mean the minimum number of frames necessary for predicting the route of the pedestrian.

However, since only the frame corresponding to a case where a score classified as a pedestrian is larger than a predetermined threshold value is included in N frames, the time interval between N frames may be different.

Next, the pedestrian tracking unit 220 can track the pedestrian by P frames using the frame including N pedestrian recognition information. In this case, the P frames mean the minimum number of frames that can replace the pedestrian recognition in the case where the pedestrian recognition is missed in the image processing unit 11. The P frames indicate the number of frames classified as pedestrians in N frames The number of P frames that can be generated can vary.

Specifically, P may be varied according to a ratio of a frame count (Framecount) to a hit count (Hitcount). Here, the frame count means the number of frames after the pedestrian is first detected, and the hit count means the number of frames when the score classified as a pedestrian after the pedestrian is first detected is equal to or greater than a preset threshold value.

Therefore, as the ratio of the frame count to the hit count is closer to 1, the number of P frames that can replace the pedestrian recognition may increase.

In addition, the pedestrian tracking unit 220 can track the pedestrian in consideration of acceleration, deceleration, constant velocity, and stop of the sensed pedestrian. That is, the speed of the pedestrian can be weighted in order to accurately track the pedestrian in the acceleration situation and the deceleration situation.

Specifically, the weight of the position information of the first recognized pedestrian is weighted by putting weight on the position information of the most recent pedestrian in the N frames in which the pedestrian is recognized.

At this time, if the pedestrian recognition unit 210 misses the pedestrian detection, the pedestrian tracking unit 220 may replace the bounding box with the tracked pedestrian.

4 is a schematic diagram illustrating pedestrian recognition and tracking of one of a plurality of pedestrians according to one embodiment of the present invention.

As shown in FIG. 4, a plurality of bounding boxes are displayed. However, for convenience of explanation, the bounding box of the outside of the recognized N number of pedestrians is overlaid and the pedestrian can be traced based on N bounding boxes. Therefore, the bounding box displayed on the display unit 30 shows only one of the N bounding boxes.

FIG. 5 shows a schematic diagram illustrating a pedestrian recognition replacement in a pedestrian tracking in accordance with an embodiment of the present invention. the bounding box displayed in (a) and (b) is displayed when the pedestrian recognition unit 210 recognizes the pedestrian. In the case of (c), if the pedestrian recognition score is less than the predetermined threshold, Is a schematic view showing a screen in which a bounding box is replaced with a pedestrian tracking based on pedestrian recognition information (including (a) and (b)) detected in the past.

When the two pedestrians are overlapped (blocked) in the N frames received by the image processing unit 11, the pedestrian tracking unit 220 ends the pedestrian tracking.

Specifically, a Haar-Like Feature technique can be used to determine when two or more objects are combined.

The Haar-Like Feature technique refers to a method of extracting features of an object at various positions and sizes by combining a plurality of basic features according to a square shape using brightness differences between regions and regions in an image.

That is, according to an embodiment of the present invention, a plurality of quadrangles are combined to extract pedestrian characteristics at various positions and sizes for pedestrian recognition, and when it is determined that the pedestrians are overlapped in the image, the pedestrian tracking is terminated .

Next, the memory 22 stores control programs and control data for controlling the operation of the multi-pedestrian recognition and tracking system 1, images acquired by the image processing unit 11, pedestrian recognition processed by the main processor 21, Various control signals including tracking information can be temporarily stored.

Particularly, the memory 22 may be a flash memory, a read only memory, an erasable programmable read only memory (EPROM), or the like, as well as volatile memories such as S-RAM and D- And non-volatile memory such as Electrically Erasable Programmable Read Only Memory (EEPROM).

 Specifically, the non-volatile memory may semi-permanently store control programs and control data for controlling the operation of the multi-pedestrian recognition and tracking system 1, and the volatile memory may invoke the control program and control data from the non-volatile memory Temporarily store the image received by the image processing unit 11 and the position information of the pedestrian processed by the control unit 20 by the number of the preset images.

The display unit 30 can display the pedestrian information recognized and tracked by the control unit 20 to the driver through display and the warning unit 40 can alert the pedestrian when the pedestrian recognized and tracked is close to the vehicle. Can be used.

The configuration of the multi-pedestrian recognition and tracking system 1 according to the embodiment has been described above.

Hereinafter, the operation of the multi-pedestrian recognition and tracking system 1 according to one embodiment will be described.

FIG. 6 is a flowchart illustrating a multi-pedestrian recognition and tracking method according to the present invention. The operation of the automatic parking assist system 1 of the vehicle will be described.

The multi-pedestrian recognition and tracking system 1 according to the present embodiment detects pedestrian candidates (S61). Specifically, the image processing unit 10 can detect a plurality of pedestrian candidates through the images photographed by the image photographing unit 10.

At this time, the image processing unit 11 can discriminate whether the object is a pedestrian or not, and it is possible to determine a certain degree of probability that an object classified as a pedestrian can be judged as a pedestrian, ≪ / RTI >

That is, the control unit 20 can secure the pedestrian and the score value detected from the image of the image processing unit 11 (S62).

Next, the pedestrian recognition unit 210 in the control unit 20 recognizes the pedestrian candidate as a pedestrian and displays a bounding box on the outskirts of the pedestrian if the acquired score value is larger than a preset threshold value (YES in S63) (S64).

However, if the obtained score value is smaller than the preset threshold value (NO in S63), the pedestrian recognition unit 210 in the control unit 20 determines that the pedestrian is not the pedestrian and continues to detect the pedestrian candidate (S61).

When the pedestrian recognition unit 210 in the control unit 20 recognizes the pedestrian, the frame storing the position information of the pedestrian is stored in the memory 22 (S65).

Thereafter, the control unit 20 tracks the pedestrian based on the pedestrian recognition information (S66). Specifically, based on the N pedestrian recognition information stored in the memory 22, the pedestrian can be tracked in consideration of the speed information such as acceleration, deceleration, and uniform velocity of the pedestrian.

Specifically, P frames are predicted based on the number of frames after the first detection of the pedestrian in the N frames and the ratio value of the number of frames when the score classified as the pedestrian after the pedestrian is first detected exceeds the preset threshold value .

Therefore, as the ratio is closer to 1, the number of P frames that can replace the pedestrian recognition may increase.

The pedestrian recognition unit 210 continues to recognize the pedestrian (NO in S67). However, if the pedestrian recognition unit 210 misses the pedestrian recognition (YES in S67), the bounding box can be replaced with the predicted pedestrian information through the pedestrian tracking.

Further, although not shown, when the pedestrian recognition unit 210 determines that a plurality of pedestrians are overlapped, the pedestrian recognition can be ended.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limited to the embodiments set forth herein; It will be understood that various modifications may be made without departing from the spirit and scope of the invention.

1: Multi-pedestrian recognition and tracking system

Claims (12)

Capturing an external image of the vehicle;
Continuously recognizing a plurality of pedestrians from the photographed image if the probability that an object recognized in the photographed image is determined as a pedestrian is equal to or greater than a preset threshold value;
Tracking the pedestrian based on the positional change of the continuously recognized pedestrian; And
And replacing the pedestrian recognized as the tracked pedestrian when the pedestrian recognition fails. The method according to claim 1,
Wherein the positional change of the recognized pedestrian is calculated based on the speed and acceleration of the recognized pedestrian. delete 3. The method of claim 2,
And terminating the pedestrian tracking if it is determined that a plurality of pedestrians are blocked. 5. The method of claim 4,
And determining that the plurality of pedestrians are blocked based on the Haar-Like Feature technique. 6. The method of claim 5,
And displaying the recognized pedestrian or tracked pedestrian in a bounding box and displaying the same on a display unit. An image capturing unit for capturing an external image of the vehicle; And
Wherein when a probability that an object recognized by the photographed image is determined as a pedestrian is equal to or greater than a preset threshold value, a plurality of pedestrians are consecutively recognized from the photographed image, and based on the positional change of the continuously recognized pedestrian, And a controller for replacing the pedestrian recognized as the tracked pedestrian when the pedestrian recognition failure occurs. 8. The method of claim 7,
Wherein the positional change of the recognized pedestrian is calculated based on the speed and acceleration of the recognized pedestrian. delete 9. The method of claim 8,
Wherein the controller terminates the pedestrian tracking if it is determined that a plurality of pedestrians are blocked. 11. The method of claim 10,
Wherein the plurality of pedestrians are determined to be closed based on Haar-Like Feature values. 12. The method of claim 11,
And a display unit for displaying the recognized pedestrian or the tracked pedestrian as a bounding box.

Images