KR102103770B1 - Apparatus and method for pedestrian detection - Google Patents

Abstract

The present invention relates to a technology for detecting a pedestrian in an image, and more particularly, to a technology for detecting a pedestrian in an image based on faster R-CNN-based deep learning. According to an embodiment of the present invention, after generating each feature map of a continuous image using R-CNN, the generated feature maps can be combined to improve pedestrian recognition performance through spatial and temporal summation.

Description

Pedestrian detection device and method {APPARATUS AND METHOD FOR PEDESTRIAN DETECTION}

The present invention relates to a technology for detecting a pedestrian in an image, and more particularly, to a technology for detecting a pedestrian in an image based on faster R-CNN-based deep learning.

In recent years, with the rapid development of unmanned vehicles and artificial intelligence surveillance systems, the importance of research to detect an accurate human area within an image acquired from a long distance is increasing. Previous studies using visible light cameras mainly studied methods for detecting a person in the daytime when there is external light, but because it is difficult to detect a person in the nighttime when there is no external light, additional Methods using near-infrared illumination and near-infrared cameras, or thermal imaging cameras were mainly used. However, in the case of near-infrared lighting, there is a limit to the irradiation angle and distance, and there is a difficulty in adaptively adjusting the lighting power when the object is near and far. In addition, thermal imaging cameras are still expensive and difficult to install and use in various locations. Considering this, there are studies that detect people at night time using a visible light camera, but this mainly targets indoor environments with a close distance to the object.

Background art of the present invention is published in Korea Patent Registration No. 10-1818129.

The present invention is to provide a more accurate pedestrian detection device and method for recognizing people even at night time.

According to an aspect of the present invention, a pedestrian detection device is provided.

Pedestrian detection apparatus according to an embodiment of the present invention is a continuous image input unit that receives a plurality of consecutive images, an image normalizing unit that normalizes the input multiple consecutive images, and machine learning for a plurality of normalized consecutive images It may include a modified R-CNN unit to classify the candidate group of pedestrians by applying the modified R-CNN.

According to another aspect of the present invention, a pedestrian detection method is provided.

Pedestrian detection method according to an embodiment of the present invention comprises the steps of receiving a plurality of consecutive images, normalizing the input consecutive plurality of images, the modified R- machine-learned for a plurality of normalized consecutive images The step of extracting the spatial feature map of each image by applying CNN, the step of estimating the candidate candidate group by temporally combining the feature maps from the spatial feature maps extracted from each image, and the spatial feature map extracted from each image, and And classifying a predicted pedestrian using the estimated pedestrian candidate group.

According to an embodiment of the present invention, it is possible to accurately detect a pedestrian in an image of a night time obtained from a visible light camera.

According to an embodiment of the present invention, after generating each feature map of a continuous image using R-CNN, the generated feature maps can be combined to improve pedestrian recognition performance through spatial and temporal summation.

1 is a view for explaining a pedestrian detection device according to an embodiment of the present invention.
2 to 7 are views for explaining a modified R-CNN unit of the pedestrian detection apparatus according to an embodiment of the present invention.
8 and 9 are diagrams for explaining a pedestrian detection method according to an embodiment of the present invention.
10 to 13 are views for explaining the machine learning of the pedestrian detection method according to an embodiment of the present invention.
14 to 17 are views for explaining the effect of the present invention.

The present invention can be variously changed and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, the singular expressions used in the specification and claims should be interpreted to mean “one or more” in general unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in describing with reference to the accompanying drawings, identical or corresponding components are assigned the same reference numbers, and redundant description thereof will be omitted. Shall be

1 is a view for explaining a pedestrian detection device according to an embodiment of the present invention.

Referring to FIG. 1, a pedestrian detection apparatus according to an embodiment of the present invention includes a continuous image input unit 110, an image normalization unit 120, a modified R-CNN unit 130, and a pedestrian detection unit 140.

In general, the night image is an image of the night time zone, and most of the pixel values are concentrated in low intensity, so the contrast is low, and further, the light is insufficient to obtain enough noise. If you increase the exposure value of the camera and get enough light for a short period of time, the noise can be greatly reduced, but in this case, motion blurring increases, so objects such as moving pedestrians are blurry to an unrecognizable level, especially moving. The camera cannot be used. As a result, if you keep the exposure value of the camera as it is, there will be scattered true edges caused by the actual dark image and false edges caused by the noise. It is difficult to extract the form of pedestrians from the information. The pedestrian detection apparatus according to the present invention can generate a strong signal by collecting weak signals at night through spatio-temporal summation in order to detect pedestrians at night images having the above-described features. The pedestrian detection apparatus according to the present invention generates deep features (using spatial information) from consecutive video frames at night using CNN, and then combines them at the feature level (using temporal information) to achieve the final pedestrian recognition performance. Can be increased.

The continuous image input unit 110 receives a plurality of continuous images. The continuous image input unit 110 may be a visible light camera and may be used as a surveillance camera depending on the application.

The image normalization unit 120 normalizes a plurality of consecutive images. The image normalization unit 120 may normalize through appropriate pre-processing because the contrast and illuminance fluctuate greatly depending on the image acquisition time zone when the input multiple consecutive images are used as they are. In more detail, the illuminance (average of the image light intensity) and contrast (variance of the image light intensity) of each of the input successive image pixels are significantly different. For example, the night image has a relatively small illuminance and contrast compared to the day image. If machine learning is performed using these various input images, it is difficult to converge to an appropriate model because it is necessary to learn to be robust to various illuminances and contrasts in addition to detecting pedestrians that are difficult to discriminate due to noise. Therefore, the image normalization unit 120 preprocesses all input images to have similar illuminance and contrast.

The image normalization unit 120 may apply the following two pre-processing methods.

The first method is pixel normalization, and for each input image, the following equation (1) can be performed for each RGB channel.

수식 (1)

Equation (1)

는 스케일을 맞추기 위한 값 (255/2)이며,

는 미니-배치 이미지의 각 채널 (R, G, B를 위한)의

위치에서의 픽셀 값이며,

은 픽셀의 개수이다. 즉, 각 채널의 픽셀 값들을 제로 평균 및 단위 표준 편차를 갖는 분포로 만든 후 스케일링 하여, 각 입력 이미지의 조도와 콘트라스트를 정규화시킬 수 있다. here,

Is the value to scale (255/2),

Of each channel (for R, G, B) of the mini-batch image

Pixel value at position,

Is the number of pixels. That is, it is possible to normalize the illuminance and contrast of each input image by scaling the pixel values of each channel into a distribution having a zero average and a unit standard deviation.

The second method, histogram equalization (HE) and mean subtraction, converts the RGB color of each input image to HSV color and then applies histogram normalization to the value channel image. Way. In general, since the brightness value of the night image is low overall, and it can be assumed that the original color is preserved to some extent, the histogram normalization is applied only to the value channel. Therefore, since the pixel brightness values of each image become closer to a nearly uniform distribution through histogram normalization, it can have a normalized contrast. Thereafter, the image pixel values were zero-centered by changing to an RGB channel again and subtracting the average RGB value of the entire image set.

The modified R-CNN unit 130 extracts the spatial feature map of each image by applying the machine-modified modified R-CNN to a plurality of normalized consecutive images. The modified R-CNN unit 130 estimates and classifies a pedestrian candidate group through temporal feature map combining from spatial feature maps extracted from each image. The modified R-CNN unit 130 will be described in more detail with reference to FIGS. 2 to 7 below.

The pedestrian detection unit 140 detects a pedestrian from the identified pedestrian candidate group.

2 to 7 are diagrams for explaining a modified R-CNN unit of a pedestrian detection device according to an embodiment of the present invention.

2, the modified R-CNN unit 130 of the pedestrian detection device includes a feature map extraction unit 132, a feature map combining unit 134, a classifier unit 136, and a deep learning unit 138. .

In the modified R-CNN unit 130 according to the present invention, since the feature resolution of the final convolution layer of the feature map extraction unit of the faster R-CNN is low, a region of interest (ROI) is not distinguished by the pooling layer, resulting in a boring feature. In order to solve the problem that it is difficult to detect a particularly small object due to this, the fourth max pooling layer is additionally removed to increase the resolution of the final feature map.

The feature map extractor 132 extracts the spatial feature map of each image by applying the machine-modified modified R-CNN to a plurality of normalized consecutive images.

Referring to FIG. 3, the feature map extracting unit 132 receives an image including a plurality of pedestrians as input, and extracts a feature map by passing 13 convolutional layers and a ReLU activation function through three max pooling layers. The feature map extractor 132 used the layers up to the last max pooling layer of the VGG-net 16 network as a feature extraction network, and additionally, a fourth max pooling layer was used to solve the low resolution feature map problem as described above. By removing it, the resolution of the last feature map was increased. As shown in Table 1 of FIG. 3, since the max pooling layer based on stride 2 ×× 2 is performed three times, the entire stride of the final feature extraction network is 8 ×× 8 ((2 ×× 2) 3). do. The feature map extracting unit 132 is used to input the extracted feature map to the feature map combining unit 134 of FIG. 4 and the classifier unit 136 of FIG. 6.

The feature map combining unit 134 estimates a pedestrian candidate group by performing feature map combining temporally on spatial feature maps extracted from each image.

)는 아래 수식 (2),(3)와 같이 앵커 박스와 제안 박스(보행자를 둘러싼다고 예상하는 경계 박스)사이의 변환을 파라미터로 표현 한 값이다.The feature map combining unit 134 is a complete convolutional network, and through the last 1x1 convolution layer, centers on the location of the image corresponding to each location of the feature map, pedestrian probability and boundary box regression for a total of 9 anchor boxes Get a vector The feature map coupling unit 134 used nine anchor boxes of different scales, for example, with the same vertical aspect ratio of a long vertical shape in accordance with the shape of the pedestrian. Where the bounding box regression vector (

) Is a value expressing the conversion between the anchor box and the suggestion box (the bounding box that is expected to surround the pedestrian) as a parameter, as in Equations (2) and (3) below.

수식 (2)

Equation (2)

수식 (3)

Equation (3)

는 박스의 중심 좌표

와 너비 및 높이를 각각 나타낸다. 또한,

는 각각 제안 박스의 중심 좌표 x와 앵커 박스의 중심 좌표 x를 나타낸다 (

도 마찬가지임). here,

Is the center coordinate of the box

And width and height respectively. In addition,

Denotes the center coordinate x of the proposal box and the center coordinate x of the anchor box, respectively, (

The same is true).

Equation (2) represents a transition in which the scale between center coordinates does not change, and Equation (3) represents a transition in log space between width and height.

를 다시 정의된 제안 박스

로 전환할 수 있다. 최종적으로, 각 제안 박스의 보행자 확률로 비최대 서프레션(non-maximum suppression;NMS)을 수행하여, 임계 값 이상 IoU(Intersection Over Union)가 겹치면서 확률이 낮은 제안 박스들을 제거하고, 보행자 확률 상위 N(N은 자연수)개 (e.g. 100개)의 제안 박스들을 선별한다. Therefore, it is possible to machine learn with a target having a similar scale (boundary box regression vector), and when a boundary box regression vector is obtained, the anchor box

Redefine the proposal box

You can switch to Finally, non-maximum suppression (NMS) is performed with the pedestrian probability of each suggestion box to remove the proposal boxes with low probability while overlapping the Intersection Over Union (IoU) above the threshold value, and the pedestrian probability upper N (N is a natural number) (eg 100) suggest boxes are selected.

Since one night image contains extremely little information about the target area due to low brightness and contrast values, it is possible to improve the target detection performance by combining a plurality of consecutive image information. That is, the true edge of the object occurs almost the same with a slight position change within a continuous frame, but the false edge caused by random noise is changed every frame, so it is true through combining. You can increase the influence of the edge. However, when the camera moves or the object moves, it may be difficult to estimate the exact bounding box because the positions of the objects are different when multiple images are combined. For this reason, the feature map combining unit 134 does not combine images of consecutive frames, and performs combining through weighting summation between the feature maps obtained from the feature map extraction unit 132 as shown in Equation (4) below. In the case of a feature map obtained through the feature map extraction unit 132 (a feature map of a size of 90 ×× 113 ×× 512 in the 5_3rd convolution layer (ReLU) in FIG. 3), the input image (720 × in FIG. 3) This is because it has a more robust characteristic to the change in object conversion through contrast, horizontal and vertical extraction. This is because CNN generally has characteristics of feature extraction that is robust to small conversion.

수식 (4)

Equation (4)

는 연속적인 프레임 인덱스를 나타내며,

은 결합에 사용된 모든 프레임의 갯수이고,

는 i번째 이미지 프레임을 입력으로 도 3의 특징맵추출부(132)로부터 얻은 90××113××512크기의 특징 맵을 의미한다. (

는 특징 맵에서의 수평, 수직 및 채널 방향 위치를 나타낸다.

는 1-D 이산 가우시안 분포 계수들을 사용하였다. 따라서, 연속적인 프레임들의 특징 맵들 (

)들을 동일한 위치끼리 가중치를 합산하여, 결합한 특징 맵들 (

)를 얻은 후, 도 4의 특징맵결합부(134)와 도 6의 분류기부(136)의 입력으로 사용된다.here,

Denotes a continuous frame index,

Is the number of all frames used for joining,

Denotes a feature map having a size of 90 ×× 113 ×× 512 obtained from the feature map extracting unit 132 of FIG. 3 as an input of the i-th image frame. (

Indicates the horizontal, vertical and channel direction position in the feature map.

Used the 1-D discrete Gaussian distribution coefficients. Thus, feature maps of successive frames (

Feature maps that combine weights between the same locations and combine them (

), And is used as an input of the feature map combining unit 134 of FIG. 4 and the classifier unit 136 of FIG. 6.

) 들은 서로 유사한 콘텐츠를 갖는 수용영역 이미지를 갖고, 동일한 CNN을 거친다. 하지만 서로 다른 노이즈를 갖고 있기 때문에, 수식 (4)와 같이 가중치 합산했을 때 노이즈의 영향력을 약화 시킬 수 있게 된다. Specifically, the reason for combining continuous frame information on the feature maps is that the deeper the feature of the layer, the larger the receptive field on the actual image is, so that the same location scalar features of successive frames contain similar content. This is because there is an increased possibility of having receiving areas (see FIG. 5). That is, the scalar feature of the same location of each frame feature map (

) Have an image of a receiving area having similar contents to each other and go through the same CNN. However, since they have different noises, it is possible to weaken the influence of noise when summing the weights as in Equation (4).

The classifier 136 classifies the predicted pedestrian using the spatial feature map extracted from each image and the estimated pedestrian candidate group.

Referring to FIG. 6, the classifier 136 uses as inputs feature maps obtained from the feature map extractor 132 and suggestion boxes obtained from the feature map combiner 134. First, the corresponding position on the feature maps is cut according to each suggestion box. At this time, since the cut sizes are all different, it is necessary to have a uniform size through ROI pooling. As the ROI pooling size used in the present invention, a rectangular shape (e.g. 3x7) having a long vertical shape considering the shape of a pedestrian was used. This is a size designed to fit this, because the shape of the anchor box is redefined little by little by the boundary box regression vector of the feature map combining unit 134. In addition, by reducing the size of the ROI pooling by more than half, the total number of parameters of the network can be reduced by half, thereby reducing over-fitting and reducing memory usage. This is because most of the weights are connected to the first fully connected layer, so the ROI pooling size can be reduced by half, reducing the number of parameters in the entire network by about half. After ROI pooling, the boring features go through a fully connected layer to get the bounding box regression vector and pedestrian probabilities. Once again, the proposed boxes are redefined as predicted boxes using the bounding box regression vector, and then the predicted boxes overlapped with the NMS are removed to obtain a final detection result. In the classification complete connection layer of FIG. 6, 2 ×× 100 is obtained, where 2 indicates a probability of being a pedestrian or a background, and 100 indicates a number of candidates.

The deep learning unit 138 machine-learns the changed R-CNN unit 130 using a database.

는 달라질 수 있음) 손실 함수를 최소화하도록 가중치가 기계 학습된다.The deep learning unit 138 uses a 4-step cross-learning method that alternately performs feature map combining learning and classifier learning using a database. The deep learning unit 138 performs two-class classification in both the feature map combining learning and the classifier learning in the changed R-CNN unit 130. Therefore, the feature map combination learning and classifier learning are the same (for each anchor box or suggestion box in the mini-batch).

Can vary) Weights are machine-learned to minimize the loss function.

수식 (5)

Equation (5)

는 앵커 박스 (특징 맵 결합의 경우) 혹은 제안 박스 (분류기의 경우)가 보행자를 둘러싸고 있다고 예상하는 확률이다.

는 대응되는 실제 지상 검증 레벨이다 (보행자=1, 배경=0). 그리고

(분류 손실 함수)는 로그 손실 함수이다.

는 해당 앵커 박스 혹은 제안 박스의 경계 박스 회귀 벡터이고,

는 대응되는 실제 지상 검증 경계 박스와의 경계 박스 회귀 벡터 이다.

(회귀 손실 함수)는 강건한 손실 함수 (매끄러운

)이다. 또한, 회귀 손실은 실제 지상 검증이 보행자인 경우 (

)에 만 발생한다. 따라서, 보행자 샘플의 경우 분류 손실과 회귀 손실을 합쳐서 주고, 배경 샘플의 경우 분류 손실만 준다. 마지막으로,

를 통해, 분류와 회귀 손실의 가중치를 조절한다.here,

Is the probability that you expect the anchor box (for feature map combination) or the proposed box (for classifier) to surround the pedestrian.

Is the corresponding actual ground verification level (pedestrian = 1, background = 0). And

(Classification loss function) is a log loss function.

Is the bounding box regression vector of the corresponding anchor box or suggestion box,

Is the bounding box regression vector with the corresponding actual ground verification bounding box.

(Regression loss function) is a robust loss function (smooth

)to be. In addition, the regression loss is when the actual ground verification is pedestrian (

). Therefore, in the case of a pedestrian sample, the classification loss and the regression loss are combined, and in the background sample, only the classification loss is given. Finally,

Through, the weighting of classification and regression loss is adjusted.

The deep learning unit 138 may perform machine learning for daytime pedestrians as well as nighttime pedestrians. This is usually because there is not enough night pedestrian image to be used for machine learning, so data expansion that applies brightness reduction, noise addition and vertical flipping for daytime pedestrian imagery and vertical flipping for nighttime machine learning image respectively (data expansion) To increase the number of machine learning images at night. Here, when the brightness is reduced and the noise is added, the following algorithm 1 is performed to have characteristics similar to the actual night image.

Algorithm 1

을 범위 내에서 얻는다 (

은 어두운 정도를 나타낸다). RGB 컬러의 주간 이미지를 HSV 컬러로 변환한 후, Schedule random variable

Get within the range (

Indicates the degree of darkness). After converting the weekly image of RGB color to HSV color,

에 따른

로 나눠줌으로써 영상을 어둡게 만든다. HSV 컬러 이미지를 다시 RGB 컬러 이미지로 변환한다. 각 픽셀의 각 RGB채널에 추가적 화이트 가우시안 노이즈 (AWGN) (

에 따른 분산 (

)를 기반으로 한 노이즈 이미지 (

))를 더한 후, 전체 이미지를 가우시안 블러링하여 야간 이미지와 유사하게 변환한다. The pixel values of the value channel image are normalized using a minimum and maximum scaling method. Each pixel in the value channel image

In accordance

Divided by to darken the video. Convert the HSV color image back to an RGB color image. Additional white Gaussian noise (AWGN) for each RGB channel of each pixel (

Variance according to (

Noise image based on ()

After adding)), the entire image is Gaussian-blurred to convert it similar to a night image.

(level of darkness)에 따라 상술한 알고리즘1에 의해 얻어진 야간 이미지의 예를 나타낸다.Figure 7 is using the Caltech database weekly image according to the data expansion,

An example of a night image obtained by the above-described algorithm 1 according to (level of darkness) is shown.

8 and 9 are diagrams for explaining a pedestrian detection method according to an embodiment of the present invention. Each step described below will be referred to collectively as a pedestrian detection device as a subject performed in each step for a concise and clear description of a process or invention performed through each functional unit constituting the pedestrian detection device.

Referring to FIG. 8, in step S810, the pedestrian detection apparatus 100 receives a plurality of consecutive images.

In step S820, the pedestrian detection apparatus 100 normalizes a plurality of consecutive images. Pedestrian detection device 100, pixel normalization (pixel normalization) for each input image, for each of the RGB channels, pixel values of each channel to a distribution with a zero average and a unit standard deviation, and then scaled, each input image The illuminance and contrast can be normalized. In addition, the pedestrian detection apparatus 100 converts the RGB color of each input image to HSV color by using histogram equalization (HE) and mean subtraction, and then to a value channel image. By applying the histogram normalization, and then changing to the RGB channel again, subtracting the average RGB value of the entire image set, the image pixel values can be zero centered.

8 and 9, in step S830, the pedestrian detection apparatus extracts the spatial feature map of each image by applying the machine-modified modified R-CNN to a plurality of normalized consecutive images.

In step S840, the pedestrian detection apparatus estimates a candidate pedestrian group by temporally combining feature maps from spatial feature maps extracted from each image.

In step S850, the pedestrian detection device classifies the predicted pedestrian using the spatial feature map extracted from each image and the estimated pedestrian candidate group.

The pedestrian detection device may further include performing machine learning using a database. The steps for performing machine learning will be described in detail with reference to FIGS. 10 to 13 below.

10 to 13 are views for explaining the machine learning of the pedestrian detection method according to an embodiment of the present invention.

In order to perform a pedestrian detection method according to an embodiment of the present invention, for example, a KAIST database and a Caltek database may be used.

640 크기 이므로 동일한 비율로 크기를 조정하여 720

900 크기로 사용할 수 있다. Referring to FIG. 10, the KAIST database is a set of pedestrian images having daytime and nighttime images, as well as a visible light camera image and a thermal image. In the present invention, only visible light camera images are used in machine learning and verification. KAIST database has 512 images

It is 640 size, so resize it at the same ratio to make it 720

Available in 900 sizes.

640 크기 이므로 일단, 512

640로 세로를 늘린 후, 동일한 비율로 크기를 조정하여 720

900 크기로 사용했다. The Caltech database is a weekly image-set containing many pedestrians. Because the KAIST database contains night images, it was sampled every 10 frames to train with various noises, and the Caltech database was sampled every 30 frames. All images in the Caltech database are 480

640 size, so once, 512

After increasing the height to 640, resize it to the same ratio and adjust it to 720

Used in 900 size.

]를 사용해 성능 평가 했다.Referring to FIG. 11, a flipping method was used as a data extension during machine learning, and a method of making the image darker to a random degree was also used. In addition, about 10% of night machine learning images were subtracted from the validation set. Pedestrian actual ground verification criteria (reasonable) were used for verification, and the evaluation metric was log-average Miss Rate on False Positive Per Image (FPPI) in [

] To evaluate performance.

)가 해당 사전 학습된 모델의 가중치 개수(

)와 맞지 않는다. 따라서, 너비축의 중앙부분과 연결되는 가중치만 잘라서 초기화에 사용했다. 또한, VGG-16 모델의 첫 번째부터 네 번째 콘볼루션 레이어까지의 사전 학습된 가중치는 기계 학습시 프리징(freezing) 했다. In machine learning, weights were initialized using a VGG-16 model that was pre-trained with the ImageNet dataset. However, since the feature map size after pooling the ROI is 7x3 (long vertical), the number of weights connected to the first layer (

) Is the number of weights of the pre-trained model (

) Does not fit. Therefore, only the weight connected to the central part of the width axis was cut and used for initialization. In addition, the pre-trained weights from the first to fourth convolutional layers of the VGG-16 model were freezing during machine learning.

The machine learning process will be described with reference to FIG. 12. To give a suggestion box of combining feature maps to the machine learning process is indicated by a solid arrow, and to initialize the weights by giving machine-learned weights is indicated by a dotted arrow. Feature map combining and classifier (solid box) in the first and second row learn end-to-end with feature map combining, and feature map extraction to share feature map combining and classifier (dashed box) feature map extraction in the last row. Except for learning unique networks.

For the specific machine learning process, the feature map extraction and feature map combination were finely tuned using the Caltech pedestrian database. Then, the proposed boxes were generated for each machine learning image using the combined feature maps, and the feature map extraction and classifier parts were fine-tuned (learning rate = 0.001). Then, using the KAIST database, it was fine-tuned once again using a 4-step cross-machine learning method (learning rate = 0.0001). Therefore, by using the Caltech database with a large number of pedestrians, feature map extraction and classifier parts for general pedestrians were machine-trained, and finally fine-tuned through the KAIST database to train a robust network at night.

Stochastic Gradient Descent (SGD) method was used for machine learning, 0.9 for the momentum and 0.0005 for the weight decay. SGD repetition was performed 80 times each for a total of 3 feature map combining (RPN) machine learning steps, and SGD repetition was performed for 40 repetitions for a total of 3 classifier machine learning steps.

Referring to FIG. 13, a graph is shown in which the average of mini-batch loss decreases during the final feature map combination (RPN) (a) and classifier (b) machine learning step. Here, the reason for the loss of feature map combining compared to the loss of the classifier is that feature map combining averages the losses for all anchor boxes in the image, and thus, there are many relatively easy samples.

After the machine learning was completed for every 6 stages (2 stages for the Caltech database and 4 stages for the KAIST database), for the next stage of machine learning, the model with the lowest validation error was selected from the stored models of the current stage. In addition, in machine learning, only images in which at least one pedestrian is present are used, so that the gradient is not too focused on background loss. It took about 2.5 days for all machine learning to be completed.

14 to 17 are views for explaining the effect of the present invention.

Referring to FIG. 14, performance of various feature combining methods is compared using a proposed model machine-trained using the pixel normalization pre-processing method of the present invention. 1-D discrete Gaussian filter coefficients are used to increase the weight of the current frame (image to obtain the actual detection result) when combining consecutive frame features (weighted summation). Theoretically, it is best to combine all the weights equally (appendix), but in practice, the current frame weight should be the largest to obtain an accurate bounding box. In this study, [0.25, 0.5, 0.25] was used as the default so that the weight of the current frame plus the weight of consecutive frames became the same.

In continuous frame combining, how many consecutive frames are combined is an important variable. Theoretically, (appendix) the more frames are combined, the better the discernment of the features is, but in reality, the corresponding number of frames is shifted more and more, so it is necessary to combine the appropriate number of frames. As shown in the table below, the best performance came out when the three frames were combined, and the performance decreased gradually from five or more. This is interpreted because the receptive field of each frame feature is greatly deviated. In the case of a fixed camera such as a surveillance camera, it is assumed that when the number of consecutive frames is increased, there will be a greater performance improvement.

This time, we examined the effect of successive frame feature combinations on classification and boundary box regression. Here, the boundary box regression is correctly regressed as the intersection over union (IoU) between the actual pedestrian's actual ground verification boundary box and the expected boundary box is larger.

Referring to FIG. 15, when the IoU threshold is small (regarded as a true positive even when the IoU is small), when the consecutive frames are combined (1510) than the uncombined graph (1520), the loss rate (MR) is remarkably Decreases. However, when the IoU threshold is high, there is little difference in performance between when combined and not. It is interpreted that by combining the characteristics of consecutive frames, the classifier performance increases, but does not show a significant effect on regression performance. Because, when the IoU tolerance is low, the classifier performance greatly affects the loss rate because the expected boundary box is considered to be true positive even if it roughly intersects the actual ground verification boundary box. Therefore, as the classifier performance was increased through the combination, the loss rate was relatively low. On the other hand, if the IoU tolerance is high, the expected box becomes true positive only when it matches the actual ground verification boundary box.

Referring to FIG. 16, the performance between the proposed model using the continuous frame feature combination and the KAIST database baseline was compared.

The algorithm used for the baseline is ACF-RGB (1620) using RGB channel, RGB channel, thermal channel (T), and ACF-RGB + T + THOG (1630) using HOG feature of the thermal image (THOG) channel. to be. Of course, comparing the performance of the two baseline algorithms, ACF-RGB + T + THOG with more information showed excellent performance at all times. In particular, ACF-RGB has very poor performance at night. This is because noise is very strong at night, so it is difficult to distinguish pedestrians with hand-crafted features. However, the model 1610 according to the present invention uses a deep feature, and because it combines such feature information for a short time, it shows a detection performance of 30% higher than that of the ACF-RG 1620 in the KAIST database night image (FIG. 16). (c)). In particular, it showed more than 5% higher performance than ACF-RGB + T + THOG (1630) using even thermal image information. In addition, it also showed stable performance in the KAIST database weekly image (Fig. 16 (b)). This is interpreted as machine learning with various images through data expansion and normalizing contrast and lighting with proper pre-processing. As a result, all of the KAIST databases (Fig. 16 (a)) exceeded the performance of the baseline.

 Referring to FIG. 17, results of detection of sample images of the KAIST database are shown.

The first row of FIG. 17 is the detection result for the daytime image, and the remaining row is the detection result for the nighttime image. The bounding boxes of green, yellow (1520) and red (1530) represent true positive boxes, false negative boxes (lost actual ground verification), and false positive boxes, respectively.

The pedestrian detection method according to the present invention was applied to images of all time zones including nighttime obtained from a visible light camera. A faster R-CNN model with slight changes to pedestrian detection was machine trained using day and night images. At this time, the intensity of the daytime image was randomly reduced and a random AWGN was added to make the model robust against noise and lighting. In addition, a pre-processing method was used to normalize the lighting and contrast of all machine learning images to ensure stable machine learning. The same pre-processing method was applied at the time of verification, and additional temporal information was used by adding weights to consecutive frame features. As a result, it exceeded the performance of the baseline in all time zones of the KAIST database, and in particular, it showed higher performance than the method used from night to thermal imaging camera information.

Because the KAIST database used for performance measurement is an image obtained from a moving vehicle, the distance and lighting from pedestrians varied. As a result, most of the error cases occurred in pedestrians or backgrounds that were too dark or far away (not visible to the human eye). However, if the surveillance system environment has certain constraints (levels such as distance, lighting, etc. can be discerned), it is expected that the effect of combining continuous frame features will be increased, resulting in high-accuracy detection performance.

Pedestrian detection method according to an embodiment of the present invention is implemented in the form of program instructions that can be performed through various computer means may be recorded on a computer-readable medium. Computer-readable media may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes (magnetic media), optical media such as CD-ROMs, DVDs, and floptical disks. Hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like are included. In addition, the above-described medium may be a transmission medium such as an optical or metal wire or a waveguide including a carrier wave that transmits a signal specifying a program command, data structure, or the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

So far, the present invention has been focused on the embodiments. Those skilled in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be interpreted as being included in the present invention.

Claims (11)

In the pedestrian detection device for detecting a pedestrian at night time,
A continuous image input unit that receives a plurality of consecutive images;
An image normalization unit that normalizes a plurality of input consecutive images; And
It includes a modified R-CNN unit for classifying a pedestrian candidate group by applying a machine-learned modified R-CNN to a plurality of normalized consecutive images,
Further comprising a deep learning unit for machine learning by performing an algorithm so that the image of the daytime environment has characteristics similar to the image of the nighttime environment,
The plurality of images
It is a night image with a large change in contrast and illuminance compared to a daytime image, and three consecutive images for combining continuous frame features.
The image normalization unit
For each input image, for each of the RGB channels, the pixel values of each channel are made into a distribution with zero average and unit standard deviation, and then scaled to normalize the illuminance and contrast of each input image.
After converting the RGB color of each input image to HSV color, apply histogram normalization to the value channel image, and then change it back to the RGB channel to normalize by subtracting the RGB average value of the entire image set,
The modified R-CNN unit
A feature map extracting unit that extracts a spatial feature map of each image by applying a machine-learned modified R-CNN to a plurality of normalized consecutive images;
A feature map combining unit for estimating a candidate group of pedestrians by performing feature map combining temporally on spatial feature maps extracted from each image; And
A pedestrian detection device including a classifier for classifying predicted pedestrians using a spatial feature map extracted from each image and an estimated pedestrian candidate group.
delete delete delete According to claim 1,
The feature map extraction unit,
A pedestrian detection device that receives images containing a plurality of pedestrians as input, and passes 13 convolution layers and ReLU activation functions through three max pooling layers to extract feature maps.
The method of claim 5,
The feature map extraction unit,
In order to solve the low-resolution feature map problem in the VGG-Net 16 network, a pedestrian detection device that further increases the resolution of the last feature map by removing the fourth max pooling layer.
According to claim 1,
The feature map coupling unit
Pedestrian detection device that performs combining by adding weights between spatial feature maps extracted from each image.
In the pedestrian detection method performed in the pedestrian detection device for the detection of pedestrians at night time,
Receiving a plurality of consecutive images;
Normalizing the input consecutive plurality of images;
Extracting a spatial feature map of each image by applying a machine-learned modified R-CNN to a plurality of normalized consecutive images;
Estimating a candidate candidate group by temporally combining feature maps from spatial feature maps extracted from each image; And
And classifying the expected pedestrian using the spatial feature map extracted from each image and the estimated pedestrian candidate group,
Further comprising the step of performing the machine learning by performing an algorithm so that the image of the daytime environment has characteristics similar to the image of the nighttime environment,
The plurality of images are night images in which contrast and illuminance fluctuate significantly compared to day images, and three consecutive images for combining continuous frame features,
The step of normalizing the input multiple consecutive images,
For each input image, for each of the RGB channels, the pixel values of each channel are made into a distribution with zero average and unit standard deviation, and then scaled to normalize the illuminance and contrast of each input image.
Pedestrian detection method that converts the RGB color of each input image to HSV color, applies histogram normalization to the value channel image, and then changes to the RGB channel again to subtract the average RGB value of the entire set of images. .
delete delete A computer program recorded on a computer-readable recording medium that executes the pedestrian detection method of claim 8.

Images