KR101908481B1 - Device and method for pedestraian detection - Google Patents

Abstract

The present invention relates to a pedestrian detecting device and a method thereof and, more specifically, to a pedestrian detecting device and a method thereof which adaptively select an image advantageous in detecting a pedestrian in an image obtained by a visible ray camera and an infrared ray camera so as to increase pedestrian detection accuracy. The present invention analyzes features of a visible ray image and an infrared ray image from a real-time input image based on a fuzzy logic so as to select an image advantageous in detection and applies a convolutional neural network to the selected image so as to increase pedestrian detection performance.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a pedestrian detection device,

The present invention relates to a pedestrian detection apparatus and method, and more particularly, to a pedestrian detection apparatus and method for improving the pedestrian detection accuracy by adaptively selecting an image favorable for pedestrian detection in an image acquired from a visible light camera and an infrared camera .

Detecting pedestrians in video is used for various purposes such as security and surveillance. However, it is difficult to accurately detect a pedestrian in an environment where there is a limit such as a part where a shadow is large, or in a dark night or in an uneven lighting environment. In addition, it is difficult to distinguish the infrared image acquired by the infrared camera when the temperature around the pedestrian becomes similar to that of the pedestrian.

Background Art [0002] The background art of the present invention is disclosed in Korean Patent Publication No. 2010-0010734 (published on Feb. 02, 2010, a platform monitoring system using a stereo camera and an infrared camera and a method thereof).

The present invention analyzes a characteristic of a visible light image and an infrared image based on a fuzzy logic, selects an image advantageous for detecting a pedestrian according to an analysis result, and verifies a pedestrian detection result through a line neural network model, Apparatus, and method.

According to one aspect of the present invention, a pedestrian detecting apparatus is provided.

A pedestrian detecting apparatus according to an embodiment of the present invention includes a video input unit for receiving a visible light image and an infrared image, a candidate region detecting unit for detecting candidate regions in the input visible light ray image and infrared ray image, and extracting feature values and weights from the detected candidate regions Based on the two fuzzy logic sets, the image analysis part applies the image selection part to select the candidate area which is advantageous for the detection of the pedestrian by using the feature value and the weight of the candidate area, and the line neural network model that learns the selected candidate area And a pedestrian detection unit for verifying whether the selected image corresponds to a pedestrian and for detecting the visible light image and the pedestrian in the infrared image according to the verification result.

According to another aspect of the present invention, there is provided a pedestrian detection method and a computer program for executing the same.

A pedestrian detection method and a computer program for executing the method of detecting a pedestrian according to an embodiment of the present invention include the steps of receiving a visible light image and an infrared image, detecting a candidate region in the input visible light ray image and infrared ray image, Selecting a candidate region which is advantageous for detecting a pedestrian using feature values and weights of the candidate region based on the two set fuzzy logic, and applying a line-by-line neural network model to the selected candidate region And verifying whether the selected image corresponds to a pedestrian, and detecting the pedestrian in the visible light image and the infrared image according to the verification result.

A pedestrian detection apparatus according to an embodiment of the present invention analyzes characteristics of a visible light image and an infrared image based on fuzzy logic, selects an image advantageous for pedestrian detection according to the analysis result, and verifies the pedestrian detection result through a line neural network model So that the pedestrian detection accuracy can be improved.

1 to 3 are views for explaining a pedestrian detection apparatus according to an embodiment of the present invention.
FIGS. 4-7 illustrate a fuzzy rule table of two fuzzy logic, an input membership function, and an output membership function according to an embodiment of the present invention.
8 and 9 are diagrams illustrating a circuit neural network model according to an embodiment of the present invention.
10 is a diagram illustrating a final pedestrian detection result through learning based on a circuit neural network according to an embodiment of the present invention.
11 is a view for explaining a pedestrian detection method according to an embodiment of the present invention.
12 to 14 are diagrams showing the results of measuring the detection accuracy of a pedestrian detection method according to an embodiment of the present invention.

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and similarities. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In addition, the singular phrases used in the present specification and claims should be interpreted generally to mean "one or more " unless otherwise stated.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or corresponding components throughout. .

1 to 3 are views for explaining a pedestrian detection apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a pedestrian detection apparatus 100 includes a dual camera 10, an image input unit 110, an image analysis unit 120, an image selection unit 130, and a pedestrian detection unit 140.

The dual camera 10 acquires an image using at least one of a visible light camera and an infrared camera in which optical axes are coupled in parallel.

The image input unit 110 receives a visible light image and an infrared image.

The image analysis unit 120 detects candidate regions from the input visible light ray image and infrared ray image, and extracts feature values and weights from the detected candidate regions.

2, the image analysis unit 120 includes a candidate region detection unit 121, a feature value extraction unit 122, and a weight extraction unit 123.

The candidate region detection unit 121 generates a difference image between the visible light background image and the input visible light image to detect a candidate region of the visible light image and generates a difference image between the infrared background image and the input infrared image to generate a candidate region of the infrared image .

3, the candidate region detecting unit 121 receives the infrared image and the visible light image as shown in FIGS. 3A and 3B, generates a difference image as shown in FIGS. 3C and 3D, ) And (f). At this time, non-pedestrians such as 310 and 330 as well as pedestrians may be detected in the candidate region as shown in 320 and 340 of FIG. Accordingly, in the present invention, the characteristics of the corresponding region are analyzed through the feature value extracting unit 122 and the weight extracting unit 123 to distinguish the pedestrian from the non-pedestrian.

The feature value extracting unit 122 extracts the first feature value and the second feature value by calculating a pixel average value in the candidate region of the visible light image and the candidate region of the infrared image.

Referring to FIG. 5, the first characteristic value and the second characteristic value are different depending on whether the candidate region is a pedestrian or a non-pedestrian. FIG. 5A shows a feature value correlation of a candidate region extracted from each of a visible light image and an infrared image when the candidate region corresponds to a pedestrian. FIG. 5B shows a feature value correlation of a candidate region extracted from each of the visible light image and the infrared image when the candidate region corresponds to a non-pedestrian. When the correlation is analyzed, the feature value extracted from the visible light image is 0.5 or more in the case of a pedestrian and 0.5 or less in the case of a non-pedestrian. On the contrary, the feature value extracted from the infrared image is 0.5 or less for a pedestrian and 0.5 or more for a non-pedestrian. Through this analysis, the present invention can more accurately detect the pedestrian area and the non-pedestrian area.

The weight extraction unit 123 extracts the first and second weights by applying Gaussian filtering to the candidate regions of the visible light image and the infrared image, and calculating the slope magnitude value of each pixel brightness.

The image selection unit 130 selects a candidate region which is advantageous for the detection of a pedestrian by using the feature values and weights of the candidate region based on the two fuzzy logic sets. Here, the fuzzy logic is set differently depending on whether the candidate region is a pedestrian or a non-pedestrian.

FIGS. 4-7 illustrate two fuzzy logic fuzzy rule tables, an input membership function, and an output membership function, according to one embodiment of the present invention.

Referring to FIG. 4, the fuzzy rule table set in the present invention defines a fuzzy rule table according to the case where feature values extracted from the candidate regions of the visible light image and the infrared image correspond to a pedestrian or a non-pedestrian. 4A is a fuzzy rule table corresponding to a pedestrian, and outputs 'MEDIUM' when the first feature value and the second feature value are 'LOW' or 'HIGH', respectively, LOW 'when the first characteristic value is' HIGH' and the second characteristic value is' LOW 'when the first characteristic value is' LOW' and the second characteristic value is' HIGH ' do. 4B is a fuzzy rule table for a non-pedestrian. When the first feature value and the second feature value are both 'LOW' or 'HIGH', 'MEDIUM' is output. When the first characteristic value is 'HIGH' and the second characteristic value is 'LOW', it outputs 'HIGH'. When the first characteristic value is 'LOW' and the second characteristic value is 'HIGH' Is defined.

Referring to FIG. 5, the input membership function set in the present invention is defined as (a) an input membership function when the candidate region is a pedestrian, and (b) an input membership function when the candidate region is a non-pedestrian. The two input membership functions are L (Low) or H (High), which receives a feature value (first feature value) extracted from the candidate region of the visible light image and a feature value (second feature value) extracted from the candidate region of the infrared image, And outputs the membership value. Here, the input membership function can output L (Low) as a membership value, for example, when the feature value is 0.5 or less. In addition, the input membership function can output H (High) as a membership value when the feature value is 0.5 or more.

Referring to FIG. 6, the output membership function set in the present invention is classified into L (Low), M (Medium), or H (High) according to the membership value output from the input membership function. Here, the output membership function may output the fuzzy result value according to the correlation sum of the first feature value (visible light ray image) and the second feature value (infrared ray image). For example, when the sum of the correlation between the first feature value and the second feature value is 0.5 or less, 0 is given for L and 0.5 for H, respectively. The output membership function can be assigned 0 if the correlation value is less than 0.5, 0.25 if the membership value is L, and 0.25 if the correlation value is less than 0.5. In addition, the output membership function can classify the fuzzy result value into L (Low) if the correlation sum of the first feature value and the second feature value is less than 0.75. The output membership function can classify the purge result as M (Medium) if the correlation sum of the first feature value and the second feature value is greater than or equal to 0.75 and less than 1.0. The output membership function can classify the fuzzy result value as H (High) if the correlation sum of the first feature value and the second feature value is 1.0 or more.

The image selecting unit 130 according to an embodiment of the present invention calculates the final purging result value by applying the purging result values according to the two fuzzy logic described above to the formula (1) It is possible to select one of the candidate region of the visible light image or the candidate region of the infrared image through the image region 2 as a candidate region that is advantageous for pedestrian detection.

(1)

(One)

는 최종 퍼지 결과값,

는 보행자일 경우의 퍼지 결과값,

는 보행자일 경우의 가중치,

는 비보행자일 경우의 퍼지 결과값,

는 비보행자일 경우의 가중치를 의미한다. In equation (1)

Is the final purge result value,

Is a fuzzy result value in the case of a pedestrian,

Is a weight in the case of a pedestrian,

Is a fuzzy result value in the case of a non-pedestrian,

Is a weight for a non-pedestrian.

및

에 적용하여 보행자 검출 정확도를 향상시킬 수 있다.Here, the weight is a slope magnitude value extracted by the weight extracting unit 123. Referring to FIG. 7, since the slope magnitude value differs when the pedestrian corresponds to the non-pedestrian and when the pedestrian corresponds to the slope magnitude, The value of the slope magnitude is expressed in Equation (1)

And

It is possible to improve the pedestrian detection accuracy.

(2)

(2)

이 임계값 이하이면 가시광 영상에서 검출한 후보영역을 선택하고, 그렇지 않을 경우는 적외선 영상에서 검출한 후보영역을 선택한다.Referring to Equation (2), the pedestrian detection apparatus 100 according to an embodiment of the present invention calculates a final purging result value

Is less than or equal to the threshold value, the candidate region detected from the visible light image is selected. Otherwise, the candidate region detected from the infrared image is selected.

The pedestrian detecting unit 140 verifies whether the selected image corresponds to a pedestrian by applying a learned neural network model to the selected candidate region, and detects the visible light image and the pedestrian in the infrared image according to the verification result.

8 and 9 are diagrams illustrating a circuit neural network model according to an embodiment of the present invention.

8 and 9, the circuit neural network model includes an image input layer, a feature extraction layer, and a classification layer.

The image input layer inputs a pedestrian candidate image of a predetermined scale. The image input layer may use a color image and may be a size of 119 x 183 x 3 pixels.

The feature extraction layer may include (1) five convolution layers, (2) each convolution layer is associated with a rectified linear unit (hereinafter 'ReLU') layer, and (3) Local response normalization; (LRN) layer and (4) a Max pooling layer.

(1) Of the five convolution layers, the first convolution layer receives an image of 119 x 183 x 3 pixels and is convoluted at intervals of two pixels using 96 filters of size 11 x 11 x 3 . In this case, the weight per filter size is 11 x 11 x 3 = 363, and the total number of parameters in the first convolution layer is (363 + 1) x 96 = 34,944, where 1 indicates a bias. The size of the feature map in the first convolution layer is 55 x 87 x 96, and 55 and 87 are the output height and width. The output height (or width) can be computed using the input height (or width) - filter height (or width) + 2 × padding) / stride +1. The outputs go through the ReLU layer and the LRN layer. Here, as shown in equation (3), the ReLU layer performs a threshold operation in which all 0s are set to 0 when the input value is less than 0. Thus, the ReLU layer helps simplify the operation and increase the learning speed of the deep network.

(3)

(3)

,

및

의 값을 1, 0.0001, 0.75으로 설정함으로써 정규화된 출력값(

)을 구할 수 있다.The LRN layer can be implemented through Equation (4). The feature extraction layer according to the embodiment of the present invention is expressed by Equation (4)

,

And

Is set to 1, 0.0001, 0.75 so that the normalized output value (

) Can be obtained.

(4)

(4)

The output map of the LRN layer is processed by the first Max-Pulling layer at intervals of 2 pixels using one filter of 3x3 size and then down-sampled. The output from the first max-pulling layer is 27 ((55 - 3 + 2 ×) by the output height (or width) = input height (or width) - filter height (or width) + 2 × padding) / stride + 0) / 2 + 1)) x 43 (= ((87 - 3 + 2 x 0) / 2 + 1)) x 96.

Then, in the second convolution layer, 128 pels of 5 × 5 × 96 pixels are used to convolve 2 padding at intervals of 1 pixel, and through the ReLU layer and the LRN layer, Apply the Max-Pulling layer to reduce the size of the feature map at 2-pixel intervals.

The third convolution layer and the fourth convolution layer are convolutionized by 1 padding at intervals of one pixel using 256 3 × 3 × 128 filters and apply the ReLU layer.

Next, the fifth convolution layer convolutes one padding at intervals of one pixel by 128 padding filters of 3x3x256 by one padding at intervals of one pixel, and performs convolution using ReLU layer and 3x3 size filter The third max-pulling layer is applied to reduce the size of the feature map at intervals of two pixels to output an output map having a size of 6 x 10 x 128 pixels.

In addition, the feature extraction layer may further comprise three fully connected layer sets next to the convolution layers. Here, the complete connection layer reduces the feature by connecting the result of the previous layer operation to a plurality of nodes so as to be a one-dimensional matrix. The first fully-connected layer sets may include a full-connect layer and a ReLU layer, the second fully-connected layer set may include a full-connect layer, a ReLU layer, and a drop-out layer, The connection layer set can include a Softmax layer behind the full connection layer. At this time, the first complete connection layer may have 6 and 4096 nodes as input and output, respectively. The second complete connection layer may have 4096 and 1024 nodes as inputs and outputs, respectively. Prior to the third complete connection layer, a dropout technique is applied, which may set the output of each hidden node to zero based on a randomly predetermined probability. The next third fully connected layer may have 1024 input nodes and 2 output nodes.

The classification layer classifies the characteristics of the candidate region output from the third complete connection layer as a pedestrian or a non-preferred through the Softmax layer.

는 i~ K개의 클래스에 대응하는 Sj값을 정규화함으로써 산출할 수 있다. The soft max function is expressed by Equation (5). In equation (3)

Can be calculated by normalizing Sj values corresponding to i to K classes.

(5)

(5)

값이 최대가 되는 경우에 해당하는 클래스에 따라 선택된 후보영역을 보행자 또는 비보행자로 분류하고, 보행자로 분류된 후보영역으로 입력된 가시광 영상 및 적외선 영상 내의 보행자를 검출한다.The pedestrian detection unit 140 calculates the distance

When the value becomes maximum, the candidate region selected according to the class is classified as a pedestrian or a non-pedestrian, and a visible light image and a pedestrian in the infrared image input as candidate regions classified as a pedestrian are detected.

10 is a diagram illustrating a final pedestrian detection result through learning based on a circuit neural network according to an embodiment of the present invention.

Referring to FIG. 10, the pedestrian detection apparatus 100 according to an embodiment of the present invention includes: (a) a candidate region 1010 selected from a visible light image, which is a finally selected candidate region, The candidate region 1020 selected from the infrared image can be detected as a final pedestrian. The pedestrian detection apparatus 100 detects the candidate region of the pedestrian from the visible light image and the infrared ray image and analyzes the correlation between the candidate region features between the images to effectively detect the pedestrian by more effectively removing the non- .

11 is a view for explaining a pedestrian detection method according to an embodiment of the present invention.

Referring to FIG. 11, in steps S1110 and S1115, the pedestrian detection apparatus 100 inputs visible light images and infrared images acquired through a visible light camera and an infrared camera in which optical axes are coupled in parallel.

In step S1120, the pedestrian detection apparatus 100 generates a difference image between the inputted visible light background image and the input visible light image to detect a candidate region of the visible light image.

 In step S1125, the pedestrian detection apparatus 100 generates a difference image between the inputted infrared background image and the input infrared image to detect a candidate region of the infrared image.

In steps S1130 and S1135, the pedestrian detection apparatus 100 calculates a pixel average value in the candidate region of the visible light image and the candidate region of the infrared image, and extracts the first feature value and the second feature value.

In step S1140 and step S1145, the pedestrian detection apparatus 100 applies Gaussian filtering to the candidate region of the visible light image and the candidate region of the infrared image, calculates the slope magnitude value of each pixel brightness, and outputs the first weight and the second weight .

In step S1150, the pedestrian detection apparatus 100 applies the first feature value, the second feature value, the first weight value, and the second weight value to two predetermined fuzzy logic values to output a fuzzy result value.

In step S1155, the pedestrian detection device 100 selects one of the candidate region of the visible light image or the candidate region of the infrared image according to the output fuzzy result value.

In step S1160, the pedestrian detection device 100 classifies the selected candidate area into a pedestrian or a non-pedestrian by applying it to a learned-line neural network model.

In step S1165, the pedestrian detection device 100 detects the pedestrian in the visible light image and the infrared image input into the candidate area classified as the pedestrian.

12 to 14 are diagrams showing the results of measurement of the detection accuracy of the pedestrian detection method according to an embodiment of the present invention.

The present invention uses a total of three data sets for the detection performance evaluation. Referring to Table 1, the first data set is a self-constructed set of 4,080 image data taken from various environmental changes such as weather, time, and temperature changes. The second data set consists of the image data generated artificially by Gaussian noise and blur effect in the first data. The third dataset is the OSU color-thermal database, which is one of the open databases.

Sub-database1 2Sub-database2 Sub-database3 Sub-database4 Number of image 598 651 2364 467 Number of pedestrian candidate 1123 566 2432 169 Number of pedestrian non-candidate 763 734 784 347 (range of width) x (rang of height)
(pixels) Pedestrian (27 ~ 91)
×
(63 to 142) (24 ~ 93)
×
(49 to 143) (31 to 105)
×
(55-147) (60-170)
×
(50 to 110) Non-candidate (51 to 161)
×
(63 to 142) (29-93)
×
(49 to 143) (53 to 83)
×
(55-147) (60-170)
×
(50 to 110) Weather Conditions Surface
temperature:
30.4 DEG C
Air temperature:
22.5 DEG C
Wind speed:
10 km / h
Sensory temperature:
21.3 DEG C Surface
temperature:
25.5 DEG C
Air temperature:
24 ℃
Wind speed:
5km / h
Sensory temperature:
23.5 DEG C Surface
temperature:
20 ℃
Air temperature:
21 ℃
Wind speed:
6.1 km / h
Sensory temperature:
21 ℃ Surface
temperature:
16 ℃
Air temperature:
20.5 DEG C
Wind speed:
2.5 km / h
Sensory temperature:
20.8 DEG C

Learning and verification was done using a Intel® Core ™ i7-3770K CPU @ 3.40 GHz (4 CPUs) and a graphics card NVIDIA GeForce GTX 1070 (1,920 CUDA cores) computing device with 16GB memory.

The pedestrian detection apparatus 100 detects a candidate region in a visible light image and an infrared image, and selects a candidate region that is advantageous for pedestrian detection based on the previously set fuzzy logic. Here, the fuzzy logic is the same as the method described in Figs. 4 to 6 above.

The pedestrian detection apparatus 100 learned a line neural network model using a pedestrian image and a non-pedestrian image, and normalized the pedestrian image and the non-pedestrian image to a size of 119 × 183 pixels for use in learning and verification. Here, learning of the circuit neural network model is the same as that described with reference to Figs. 8 and 9.

Verification performed four cross validations by dividing the dataset into two groups.

The pedestrian detection apparatus 100 according to an embodiment of the present invention uses TPR (True Positive Rate) and TNR (True Negative Rate) using a total of three data sets, , FNR (False Negative Rate) and FPR (False Positive Rate). In addition, the average accuracy of the pedestrian detection was calculated using the following expressions (6) to (9).

(6)

(6)

(7)

(7)

(8)

(8)

(9)

(9)

Here, #TN, #TP, #FN, and #FP represent the number of true negatives (TN's), true positives (TP's), false negatives (FN's), and false positives (FP's).

12 to 14, the pedestrian detection apparatus 100 of the present invention measures 99.61%, 97.02%, and 99.03% of the average detection accuracy using a total of three data sets, It can be confirmed that the accuracy of the pedestrian detection method that has been studied is superior to that of the conventional pedestrian detection method.

The pedestrian detection method according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on a computer-readable medium may be those specially designed and constructed for the present invention or may be available to those skilled in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. The above-mentioned medium may also be a transmission medium such as optical or metal lines, waveguides, etc., including a carrier wave for transmitting a signal designating a program command, a data structure and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: Pedestrian detection device
110:
120: Image analysis section
130:
140: Pedestrian detection unit

Claims (14)

A pedestrian detection device comprising:
An image input unit for receiving a visible light image and an infrared image;
An image analyzer for detecting a candidate region in an input visible light ray image and an infrared ray image and extracting a feature value and a weight value from the detected candidate region;
An image selection unit for selecting a candidate region that is advantageous for the detection of a pedestrian by using feature values and weights of the candidate regions based on two predetermined fuzzy logic; And
And a pedestrian detection unit for verifying whether the selected image corresponds to a pedestrian by applying a circuit neural network model that has been learned to the selected candidate region and detecting the pedestrian in the visible light image and the infrared image according to the verification result,
The image selection unit
A feature value extracted from a candidate region of a visible light image and an infrared image is set to a fuzzy logic differently depending on whether it is a pedestrian or a non-pedestrian, a characteristic value of the candidate region is input to fuzzy logic, The result and weight are applied to equation (1) to calculate the final fuzzy result,

(One)
In equation (1)

Is the final purge result value,

Is a fuzzy result value in the case of a pedestrian,

Is a weight in the case of a pedestrian,

Is a fuzzy result value in the case of a non-pedestrian,

Means a weight in the case of a non-pedestrian.
The method according to claim 1,
The image analysis unit
A candidate region detection unit for detecting a candidate region of a visible light image by generating a difference image of a visible light background image and an input visible light image to generate a difference image of an infrared background image and an input infrared image to detect a candidate region of the infrared image;
A feature value extracting unit for calculating a pixel average value in a candidate region of a visible light image and a candidate region of an infrared image to extract a first feature value and a second feature value; And
And a weight extracting unit for applying Gaussian filtering to candidate areas of the visible light image and the infrared image, and calculating a slope magnitude value of each pixel brightness to extract a first weight and a second weight.
delete The method according to claim 1,
The image selection unit
Based on the preset threshold value, the final purge result value

The candidate region detected from the visible light image is selected, and if not, the candidate region detected from the infrared image is selected.
The method according to claim 1,
The circuit neural network model of the pedestrian detecting unit
An image input layer for inputting a pedestrian candidate image of a preset scale;
A feature extraction layer including five convolution layer sets and three fully connected layer sets; And
And a classification layer for classifying the characteristics of the candidate region output from the three complete connection layers as a pedestrian or a non-preferred through a soft max layer.
6. The method of claim 5,
The feature extraction layer
Each convolution layer is associated with a rectified linear unit layer, and includes a local response normalization layer and a Max pooling layer.
6. The method of claim 5,
The five convolution layer sets
A first convolution layer for inputting an image of 115 × 51 × 1 pixels and convoluting at intervals of 2 pixels using 96 filters of size 11 × 11 × 3, a rectified linear unit layer, A first convolution layer composed of a local response normalization layer and a 3 × 3 size filter, which is a first max-pooling layer for outputting a feature map of 27 × 43 × 96 pixels, set;
A second convolution layer for convolutionizing 128 filters of 5x5x96 size by 2 padding at intervals of one pixel in a feature map of 27x43x96 pixels outputted from the first convolution layer set, A second convolution layer set consisting of a unit layer, a local response normalization, and a second max-pooling layer for outputting a feature map of 13x21x128 pixels by applying one 3x3 filter at intervals of two pixels;
A third convolution layer for convoluting the feature maps of 13 x 21 x 256 pixels output from the second convolution layer set by one padding at intervals of one pixel using 256 3 x 3 x 128 filters; A third convolution layer set consisting of a calibration linear unit layer;
A fourth convolution layer for convolutionizing the feature maps of 13 x 21 x 256 pixels output from the third convolution layer by one padding at intervals of one pixel by using 256 x 3 x 128 filters, A fourth convolution layer set composed of a linear unit layer; And
In a feature map of 13 × 21 × 256 pixels output from the fourth convolution layer set, 128 filters having a size of 3 × 3 × 256 are convolved with 1 padding at intervals of 1 pixel, A fifth convolution layer consisting of a third max-filling layer for applying a 3 x 3 size filter at intervals of two pixels to output a feature map of 6 x 10 x 128 pixels, and a calibration linear unit layer The pedestrian detection device comprising:
6. The method of claim 5,
The three fully connected layer sets
6 < / RTI > and 4096 nodes as inputs and outputs, respectively;
A second full connection layer having 4096 and 1024 nodes as inputs and outputs, respectively; And
1024 and a third complete connection layer having two nodes as inputs and outputs, respectively.
In the pedestrian detection method,
Receiving a visible light image and an infrared image;
Detecting candidate regions in the input visible light ray image and infrared ray image, and extracting feature values and weights from the detected candidate regions;
Selecting candidate regions advantageous for pedestrian detection using feature values and weights of the candidate regions based on two predetermined fuzzy logic; And
And a step of verifying whether the selected image corresponds to a pedestrian by applying a line-by-line neural network model to the selected candidate region, and detecting the pedestrian in the visible light image and the infrared image according to the result of the verification,
Wherein the step of selecting a candidate region which is advantageous for the detection of a pedestrian by using the feature values and the weights of the candidate regions based on the two predetermined fuzzy logic
A feature value extracted from a candidate region of a visible light image and an infrared image is set to a fuzzy logic differently depending on whether it is a pedestrian or a non-pedestrian, a characteristic value of the candidate region is input to fuzzy logic, The result and weight are applied to equation (1) to calculate the final fuzzy result,

(One)
In equation (1)

Is the final purge result value,

Is a fuzzy result value in the case of a pedestrian,

Is a weight in the case of a pedestrian,

Is a fuzzy result value in the case of a non-pedestrian,

Means a weight in the case of a non-pedestrian.
10. The method of claim 9,
The step of detecting a candidate region in the input visible light ray image and the infrared ray image and extracting a feature value and a weight value in the detected candidate region
Detecting a candidate region of a visible light image by generating a difference image between a visible light background image and an input visible light image, generating a difference image between the infrared background image and the input infrared image to detect a candidate region of the infrared image;
Calculating a pixel average value in a candidate region of the visible light image and a candidate region of the infrared image to extract a first feature value and a second feature value; And
Further comprising the step of applying Gaussian filtering to the candidate region of the visible light image and the candidate region of the infrared image, and calculating a slope magnitude value of each pixel brightness to extract a first weight and a second weight.
11. The method of claim 10,
Wherein the step of selecting a candidate region which is advantageous for the detection of a pedestrian by using the feature values and the weights of the candidate regions based on the two predetermined fuzzy logic
Applying the first feature value, the second feature value, the first weight value, and the second weight value to two preset fuzzy logic values to output a final purge result value; and
And selecting one of the candidate region of the visible light image or the candidate region of the infrared image according to the result of the final purging.
12. The method of claim 11,
The fuzzy logic may be set differently depending on whether the feature value extracted from the candidate region of the visible light image and the infrared image corresponds to a pedestrian or a non-pedestrian,
The final purging result value is obtained by inputting the feature value of the candidate region to the fuzzy logic, applying the output fuzzy result value and the weight to the equation (1)

(One)
In equation (1)

Is the final purge result value,

Is a fuzzy result value in the case of a pedestrian,

Is a weight in the case of a pedestrian,

Is a fuzzy result value in the case of a non-pedestrian,

Means a weight in the case of a non-pedestrian.
10. The method of claim 9,
Wherein the step of verifying whether the selected image corresponds to a pedestrian by applying a line neural network model learned in the selected candidate region and detecting the pedestrian in the visible light image and the infrared image according to the result of the verification
The selected candidate region is classified as a pedestrian or a non-pedestrian by using a circuit neural network model that learns a pedestrian region and a non-pedestrian region included in the data set, and the pedestrian in the visible image and the infrared image as the candidate region classified as a pedestrian Detecting a pedestrian.
A computer program that executes the pedestrian detection method according to any one of claims 9 to 13 and is recorded on a computer-readable recording medium.

Images