CN110765877A - Pedestrian detection method and system based on thermal imager and binocular camera - Google Patents

Pedestrian detection method and system based on thermal imager and binocular camera Download PDF

Info

Publication number
CN110765877A
CN110765877A CN201910894646.1A CN201910894646A CN110765877A CN 110765877 A CN110765877 A CN 110765877A CN 201910894646 A CN201910894646 A CN 201910894646A CN 110765877 A CN110765877 A CN 110765877A
Authority
CN
China
Prior art keywords
binocular camera
thermal imager
image
pixel
gradient
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910894646.1A
Other languages
Chinese (zh)
Other versions
CN110765877B (en
Inventor
周竹萍
杨旭
梅亚岚
裘梦琪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Tech University
Original Assignee
Nanjing Tech University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Tech University filed Critical Nanjing Tech University
Priority to CN201910894646.1A priority Critical patent/CN110765877B/en
Publication of CN110765877A publication Critical patent/CN110765877A/en
Application granted granted Critical
Publication of CN110765877B publication Critical patent/CN110765877B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Abstract

The invention provides a pedestrian detection method and system based on a thermal imager and a binocular camera, and the method comprises the following steps: acquiring an image in front of the vehicle by using a thermal imager and a binocular camera; eliminating parallax between images collected by a binocular camera; graying the images of the binocular camera and the thermal imager; respectively scanning the grayed binocular camera image and thermal imager image in a raster scanning mode according to set step lengths by using a sliding window, and simultaneously extracting HOG characteristics of the binocular camera image and the thermal imager image in each step length to obtain HOG characteristic description values; judging whether the window has the pedestrian or not by utilizing the classification result of the SVM; and comparing the detection results of the binocular camera image and the thermal imager image to obtain a final detection result. The invention has small pedestrian detection calculation amount, continuously updates each parameter and state quantity, can perform early warning uninterruptedly and improves the timeliness and continuity of early warning.

Description

Pedestrian detection method and system based on thermal imager and binocular camera
Technical Field
The invention belongs to the field of automatic driving of automobiles, and particularly relates to a pedestrian detection method and system based on a thermal imager and a binocular camera.
Background
Automatic and reliable detection of pedestrians is an important function of autonomous vehicles or Advanced Driver Assistance Systems (ADAS). Research efforts regarding pedestrian detection are largely dependent on the data, as different data and methods may yield different assessment results. In the current data collection device of the pedestrian detection device, the most common sensor in data acquisition is a conventional color video camera, but the color video camera still has many limitations. For example, color cameras are sensitive to lighting conditions. If the image quality suffers under light deficient conditions. Thermal imagers can be used to overcome some of the limitations of color cameras because they are not affected by lighting conditions. But a single thermal imager detection can be affected by the ambient temperature. For example, in the case of high outdoor temperatures, pedestrians cannot be well distinguished from surrounding objects.
Disclosure of Invention
The invention aims to provide a pedestrian detection method based on a thermal imager and a binocular camera.
The technical solution for realizing the purpose of the invention is as follows: a pedestrian detection method based on a thermal imager and a binocular camera comprises the following steps:
step 1, arranging a thermal imager and a binocular camera on a vehicle, and collecting an image in front of the vehicle;
step 2, eliminating parallax between images collected by a binocular camera;
step 3, graying the binocular camera image and the thermal imager image after the parallax is eliminated in the step 2;
step 4, scanning the grayed binocular camera image and the thermal imager image by using a sliding window in a raster scanning mode according to set step lengths, and simultaneously extracting HOG characteristics of the binocular camera image and the thermal imager image in each step length to obtain HOG characteristic description values;
step 5, training the SVM by adopting a pedestrian database to obtain a coefficient wTThe HOG characteristic description value X corresponding to each sliding window and the coefficient w are comparedTPerforming multiply-accumulate to obtain the result and setting the threshold
Figure BDA0002209832270000011
Adding, if the sum is larger than 0, judging that a pedestrian exists in the window;
and 6, comparing the detection results of the binocular camera image and the thermal imager image, outputting a unified result if the detection results of the two images are consistent, and judging that the two images are not consistent.
Preferably, the step 4 of scanning the grayed binocular camera image and the thermal imager image by using the sliding window in a raster scanning manner according to the set step length comprises the following specific steps:
setting an interested area of a scene to be detected;
setting the size of the sliding window and the horizontal and longitudinal moving step length;
and moving a sliding window to traverse the whole picture to be detected according to the rules from left to right and from top to bottom in the region of interest from the upper left corner of the picture to be detected by a set transverse and longitudinal step length.
Preferably, the specific method for performing HOG feature extraction on the binocular camera image and the thermal imager image is as follows:
calculating the pixel gradient of the gray image in the sliding window, and determining the mode and the direction of the pixel gradient;
dividing the gray level image in the sliding window into n pixel blocks, and performing dot multiplication on a modulus of each pixel point in each pixel block and a Gaussian matrix;
performing three-bit linear interpolation on each pixel block and arbitrating accumulation to form a 36-dimensional vector histogram according to the coordinates of the pixel block where the pixel point is located and the gradient vector angle of the pixel point;
normalizing the 36-dimensional vector histogram obtained by each pixel block;
and combining the n normalized 36-dimensional vectors to obtain the 36X n-dimensional HOG feature description value X in the window.
Preferably, the specific method for calculating the gradient of the gray image pixel in the sliding window and determining the mode and direction of the gradient of the pixel point comprises the following steps:
calculating the directional gradient of the abscissa and the ordinate of the pixel by first order differential derivation:
Figure BDA0002209832270000023
Figure BDA0002209832270000024
Figure BDA0002209832270000025
f (x, y) respectively represents the gradient of the pixel point in the horizontal direction, the gradient in the vertical direction and the pixel value;
the mode and direction of the gradient direction of the pixel point are as follows:
Figure BDA0002209832270000021
Figure BDA0002209832270000022
in the formula, | G (x, y) | represents a modulus of a gradient vector of a pixel point, and θ (x, y) represents a direction of the gradient vector.
Preferably, the formula for normalizing the vector histogram is:
Figure BDA0002209832270000031
in the formula, v*Represents the histogram normalization result, v represents a 36-dimensional vector histogram, and epsilon is a constant.
Preferably, the pedestrian database is adopted to train the SVM to obtain the coefficient wTAnd a threshold valueThe HOG characteristic description value X corresponding to each sliding window is compared with the coefficient wTPerforming multiply-accumulate to obtain the result and setting the threshold
Figure BDA0002209832270000036
Adding, if the sum is greater than 0, judging the windowThe specific method for the existence of the pedestrian comprises the following steps:
training the SVM by adopting a Daimer pedestrian database to obtain a 36 multiplied by n dimensional coefficient wTAnd a threshold value
Figure BDA0002209832270000037
Coefficient wTDividing the HOG characteristic description value X corresponding to each window into 15 parts, and matching the HOG characteristic description value X corresponding to each window with the coefficient w of the 15 partsTPerforming multiplication accumulation, wherein the formula is as follows:
Figure BDA0002209832270000032
t＝0,1,…,14；
k＝0,1,…,6…,s
in the formula, X represents a characteristic description value, and s is the number of times of transverse movement of the sliding window;
calculating the value of y by the following formula, and if y is a positive number, determining that the pedestrian exists in the window
Figure BDA0002209832270000033
Wherein y represents the multiply-accumulate result; x represents a feature description value;
Figure BDA0002209832270000034
representing a threshold value.
The invention also provides a pedestrian detection system based on the thermal imager and the binocular camera, which is characterized by comprising the following components:
the thermal imager and the binocular camera are arranged on the vehicle and used for acquiring image information in front of the vehicle;
a computer for processing image information;
the computer is used for completing:
eliminating parallax between images collected by a binocular camera;
graying the binocular camera image and the thermal imager image after the parallax is eliminated;
respectively scanning the grayed binocular camera image and thermal imager image in a raster scanning mode according to set step lengths by using a sliding window, and simultaneously extracting HOG characteristics of the binocular camera image and the thermal imager image in each step length to obtain HOG characteristic description values;
training the SVM by adopting a pedestrian database to obtain a coefficient wTAnd a threshold valueThe HOG characteristic description value X corresponding to each sliding window is compared with the coefficient wTPerforming multiply-accumulate to obtain the result and setting the threshold
Figure BDA0002209832270000042
Adding, if the sum is larger than 0, judging that a pedestrian exists in the window;
and comparing the detection results of the binocular camera image and the thermal imager image, outputting a unified result if the detection results of the two images are consistent, and judging that the two images are not consistent.
Preferably, the thermal imaging system further comprises an aluminum box for protecting the binocular camera and the thermal imager, and the binocular camera and the thermal imager are arranged in the aluminum box.
Compared with the prior art, the invention has the following remarkable advantages:
1. the invention uses the thermal imager and the binocular camera to collect real data, performs machine learning classification training, detects pedestrians according to comparison results, further can be interconnected with a motor vehicle braking system through a computer, is matched with a conflict detection system, and is applied to the fields of automatic driving of automobiles and the like;
2. the pedestrian detection calculation amount is small, each parameter and state amount are continuously updated, the early warning can be uninterruptedly carried out, and the timeliness and the continuity of the early warning are improved;
3. in the selection of the vehicle-mounted equipment, mature sensors and processors in the prior art can be adopted, so that the vehicle-mounted equipment is conveniently installed in various types of automobiles and is convenient to debug;
4. by utilizing the pedestrian detection method, the stereoscopic vision camera and the thermal imager are combined, so that the method has multiple applicable scenes and high pedestrian detection precision, and has a promoting effect on the research of an automatic driving automobile or an advanced driving auxiliary system.
The present invention is described in further detail below with reference to the attached drawings.
Drawings
Fig. 1 is a flowchart of a pedestrian detection method based on a thermal imager and a binocular camera.
Fig. 2 is a schematic view of the installation of the thermal imager and the binocular camera according to the present invention.
Fig. 3 is a schematic diagram of the operation of the sliding window for detecting pedestrians according to the present invention.
FIG. 4 is a flowchart of SVM classifier computation of the present invention.
Detailed Description
As shown in fig. 1, a pedestrian detection method based on a thermal imager and a binocular camera is characterized by comprising the following steps:
step 1, as shown in FIG. 2, arranging a thermal imager and a binocular camera on a vehicle, and collecting images in front of the vehicle;
step 2, eliminating parallax between images collected by a binocular camera;
step 3, graying the binocular camera image and the thermal imager image after the parallax is eliminated in the step 2;
step 4, as shown in fig. 3, respectively scanning the grayed binocular camera image and thermal imager image in a raster scanning mode according to set step lengths by using a sliding window, and simultaneously performing HOG feature extraction on the binocular camera image and the thermal imager image in each step length to obtain HOG feature description values; the method comprises the following specific steps of respectively scanning the grayed binocular camera image and the thermal imager image in a raster scanning mode according to a set step length by utilizing a sliding window:
setting an interested area of a scene to be detected;
setting the size of the sliding window and the horizontal and longitudinal moving step length;
and moving a sliding window to traverse the whole picture to be detected according to the rules from left to right and from top to bottom in the region of interest from the upper left corner of the picture to be detected by a set transverse and longitudinal step length.
The specific method for carrying out HOG feature extraction on the images of the binocular camera and the thermal imager comprises the following steps:
calculating the pixel gradient of the gray image in the sliding window, and determining the mode and the direction of the pixel gradient, wherein the specific method comprises the following steps:
calculating the directional gradient of the abscissa and the ordinate of the pixel by first order differential derivation:
Figure BDA0002209832270000053
Figure BDA0002209832270000054
Figure BDA0002209832270000055
f (x, y) respectively represents the gradient of the pixel point in the horizontal direction, the gradient in the vertical direction and the pixel value;
the mode and direction of the gradient direction of the pixel point are as follows:
Figure BDA0002209832270000052
in the formula, | G (x, y) | represents a modulus of a gradient vector of a pixel point, and θ (x, y) represents a direction of the gradient vector.
Dividing the gray level image in the sliding window into n pixel blocks, and performing dot multiplication on a modulus of each pixel point in each pixel block and a Gaussian matrix;
performing three-bit linear interpolation on each pixel block and arbitrating accumulation to form a 36-dimensional vector histogram according to the coordinates of the pixel block where the pixel point is located and the gradient vector angle of the pixel point;
normalizing the 36-dimensional vector histogram obtained by each pixel block, wherein the formula is as follows:
Figure BDA0002209832270000061
in the formula, v*Representing a histogram normalization result, v represents a 36-dimensional vector histogram, and epsilon is a constant;
and combining the n normalized 36-dimensional vectors to obtain the 36X n-dimensional HOG feature description value X in the window.
Step 5, as shown in fig. 4, training the SVM by using the pedestrian database to obtain the coefficient wTThe HOG characteristic description value X corresponding to each sliding window and the coefficient w are comparedTPerforming multiply-accumulate to obtain the result and setting the threshold
Figure BDA0002209832270000065
Adding, if the sum is larger than 0, judging that the pedestrian exists in the window, and the specific method comprises the following steps:
training the SVM by adopting a Daimer pedestrian database to obtain a 36 multiplied by n dimensional coefficient wTAnd a threshold value
Figure BDA0002209832270000066
Coefficient wTDividing the HOG characteristic description value X corresponding to each window into 15 parts, and matching the HOG characteristic description value X corresponding to each window with the coefficient w of the 15 partsTPerforming multiplication accumulation, wherein the formula is as follows:
order to
Figure BDA0002209832270000062
t＝0,1,…,14；
k＝0,1,…,6…,s
Wherein, X represents a feature description value,
and calculating the value of y by the following formula, and if y is a positive number, judging that the pedestrian exists in the window:
Figure BDA0002209832270000063
wherein y represents the multiply-accumulate result; x represents a feature description value;
Figure BDA0002209832270000064
representing a threshold value.
And 6, comparing the detection results of the binocular camera image and the thermal imager image, outputting a unified result if the detection results of the two images are consistent, and judging that the two images are not consistent.
A pedestrian detection system based on a thermal imager and a binocular camera is carried on a vehicle and comprises:
the device for collecting the thermal image information in front of the vehicle comprises at least one thermal imager, wherein the thermal imager is arranged on the top of the vehicle and below a binocular camera.
The computer for processing the image information comprises at least one notebook computer, is arranged in the vehicle, and is connected with the binocular camera and the thermal imager by a data line.
In a further embodiment, the computer performs:
eliminating parallax between images collected by a binocular camera;
graying the binocular camera image and the thermal imager image after the parallax is eliminated;
respectively scanning the grayed binocular camera image and thermal imager image in a raster scanning mode according to set step lengths by using a sliding window, and simultaneously extracting HOG characteristics of the binocular camera image and the thermal imager image in each step length to obtain HOG characteristic description values; the method comprises the following specific steps of respectively scanning the grayed binocular camera image and the thermal imager image in a raster scanning mode according to a set step length by utilizing a sliding window:
setting an interested area of a scene to be detected;
setting the size of the sliding window and the horizontal and longitudinal moving step length;
and moving a sliding window to traverse the whole picture to be detected according to the rules from left to right and from top to bottom in the region of interest from the upper left corner of the picture to be detected by a set transverse and longitudinal step length.
The specific method for carrying out HOG feature extraction on the images of the binocular camera and the thermal imager comprises the following steps:
calculating the pixel gradient of the gray image in the sliding window, and determining the mode and the direction of the pixel gradient, wherein the specific method comprises the following steps:
calculating the directional gradient of the abscissa and the ordinate of the pixel by first order differential derivation:
Figure BDA0002209832270000074
Figure BDA0002209832270000075
f (x, y) respectively represents the gradient of the pixel point in the horizontal direction, the gradient in the vertical direction and the pixel value;
the mode and direction of the gradient direction of the pixel point are as follows:
in the formula, | G (x, y) | represents a modulus of a gradient vector of a pixel point, and θ (x, y) represents a direction of the gradient vector.
Dividing the gray level image in the sliding window into n pixel blocks, and performing dot multiplication on a modulus of each pixel point in each pixel block and a Gaussian matrix;
performing three-bit linear interpolation on each pixel block and arbitrating accumulation to form a 36-dimensional vector histogram according to the coordinates of the pixel block where the pixel point is located and the gradient vector angle of the pixel point;
normalizing the 36-dimensional vector histogram obtained by each pixel block, wherein the formula is as follows:
in the formula, v*Representing the histogram normalization result, v represents a 36-dimensional vector histogram, and ε is a constantCounting;
and combining the n normalized 36-dimensional vectors to obtain the 36X n-dimensional HOG feature description value X in the window.
Training the SVM by adopting a pedestrian database to obtain a coefficient wTThe HOG characteristic description value X corresponding to each sliding window and the coefficient w are comparedTPerforming multiply-accumulate to obtain the result and setting the threshold
Figure BDA0002209832270000085
Adding, if the sum is larger than 0, judging that the pedestrian exists in the window, and the specific method comprises the following steps:
training the SVM by adopting a Daimer pedestrian database to obtain a 36 multiplied by n dimensional coefficient wTAnd a threshold value
Figure BDA0002209832270000086
Coefficient wTDividing the HOG characteristic description value X corresponding to each window into 15 parts, and matching the HOG characteristic description value X corresponding to each window with the coefficient w of the 15 partsTPerforming multiplication accumulation, wherein the formula is as follows:
order to
Figure BDA0002209832270000082
t＝0,1,…,14；
k＝0,1,…,6…,s
(s is the number of times the sliding window is moved laterally-1)
Wherein, X represents a feature description value,
and calculating the value of y by the following formula, and if y is a positive number, judging that the pedestrian exists in the window:
wherein y represents the multiply-accumulate result; x represents a feature description value;
Figure BDA0002209832270000084
representing a threshold value.
And comparing the detection results of the binocular camera image and the thermal imager image, outputting a unified result if the detection results of the two images are consistent, and judging that the two images are not consistent.
The device for protecting the binocular camera and the thermal imager comprises at least one aluminum box, wherein the aluminum box is installed at the top of a vehicle, and the binocular camera and the thermal imager are installed inside the aluminum box.
Example 1
The pedestrian detection method based on the thermal imager and the binocular camera comprises the following steps:
step 1, arranging a thermal imager and a binocular camera on a vehicle, and collecting an image in front of the vehicle;
step 2, eliminating parallax between images collected by a binocular camera; in this embodiment, the disparity value of the images acquired by the left and right cameras is calculated by using the following formula:
xleft side of＝xRight side-parallax (x)Right side,y)
Wherein (x)Left side ofY) is the position of the midpoint in the image captured by the left camera, (x)Right sideY) is the position of the point in the right camera captured image, and disparity () is the disparity value at a given position;
step 3, graying the binocular camera image and the thermal imager image after the parallax is eliminated in the step 2, wherein the specific method comprises the following steps:
extracting color information (numerical values of three color channels of red (R), green (G) and blue (B)) in the picture to be detected, carrying out gray processing,
r after gradation (R + before processing, G + before processing, B)/3
G after graying is (R + G before processing + B before processing)/3;
b after graying (R + G before processing + B before processing)/3
R, G, B in the formula represent the values of three color channels of red, green and blue, respectively.
Step 4, in this embodiment, a 64 × 128 pixel window is used to scan a frame of image after graying in a raster scanning manner, and the scanning steps in the horizontal direction and the vertical direction are both 8 pixels;
HOG feature extraction is carried out on the grayed binocular camera image and thermal imager image in each step length, and the specific method comprises the following steps:
calculating the directional gradient of the abscissa and the ordinate of the pixel by first order differential derivation:
Figure BDA0002209832270000091
determining the mode and the direction of the gradient direction of the pixel point:
Figure BDA0002209832270000101
Figure BDA0002209832270000102
Figure BDA0002209832270000103
and
Figure BDA0002209832270000104
respectively representing the distance between two pixel points in the horizontal direction and the vertical direction, | G (x, y) | and theta (x, y) represent the modulus of the gradient vector of the pixel point and the direction of the gradient vector;
dividing a window by units of 8 × 8 pixels to form 8 × 16-128 units, regarding 4 adjacent units of upper, lower, left and right sides as a pixel block, and regarding each window to include 15 × 7-105 pixel blocks;
performing dot multiplication on a modulus of each pixel point in a 16 × 16 pixel block and a Gaussian matrix to darken the edge of the pixel block;
according to the coordinates (x, y) of the pixel points relative to the pixel blocks and the gradient vector angle theta of the pixel points, performing three-bit linear interpolation on each pixel block and arbitrating and accumulating to form a 36-dimensional vector histogram:
v(x1，y1，z1)←v(x1，y1，z1)+w[1-(x-x1)/bx]×[1-(y-y1)/by]×[1-(z-z1)/bz]
v(x1，y1，z2)←v(x1，y1，z2)+w[1-(x-x1)/bx]×[1-(y-y1)/by]×[1-(z-z2)/bz]
v(x1，y2，z1)←v(x1，y2，z1)+w[1-(x-x1)/bx]×[1-(y-y2)/by]×[1-(z-z1)/bz]
v(x2，y1，z1)←v(x2，y1，z1)+w[1-(x-x2)/bx]×[1-(y-y1)/by]×[1-(z-z1)/bz]
v(x2，y1，z2)←v(x2，y1，z2)+w[1-(x-x2)/bx]×[1-(y-y1)/by]×[1-(z-z2)/bz]
v(x1，y2，z2)←v(x1，y2，z2)+w[1-(x-x1)/bx]×[1-(y-y2)/by]×[1-(z-z2)/bz]
v(x2，y2，z1)←v(x2，y2，z1)+w[1-(x-x2)/bx]×[1-(y-y2)/by]×[1-(z-z1)/bz]
v(x2，y2，z2)←v(x2，y2，z2)+w[1-(x-x2)/bx]×[1-(y-y2)/by]×[1-(z-z-z1)/bz]
in the formula, x and y represent the spatial position of a pixel point; z represents the gradient direction of the point, for the pixel point (x, y), its gradient is setDegree amplitude is omega, gradient direction is z, z1And z2Is the midpoint coordinate of the 2 most adjacent histograms, and the histogram bandwidth of the gradient histogram v along the x, y and z directions is b ═ b [, b [ ]x，by，bz]，bx＝by＝8，bz＝180°/9，
Finally, an L2-Hys function is selected to normalize the obtained 36-dimensional vector histogram, and the formula is as follows:
Figure BDA0002209832270000111
wherein v is*Histogram normalization result (36-dimensional vector) representing one pixel block; v represents a 36-dimensional vector histogram; ε is 0.005.
Thus, in a window comprising 105 pixel blocks, v for 105 pixel blocks is determined*Merging to generate 3780-dimensional HOG characteristic description value X;
step 5, training the SVM by adopting a pedestrian database to obtain a vector 3780-dimensional coefficient wTAnd a threshold value
Figure BDA0002209832270000115
Will wTDividing into 15 parts, and performing multiplication and accumulation of the 36-dimensional characteristic value of the current pixel block and the 36-dimensional coefficient corresponding to the horizontal 7 windows simultaneously by the 15 parts, and calculating as follows:
order to
Figure BDA0002209832270000112
n＝0,1,…,14；
k＝0,1,…,6
Wherein X represents a feature description value,
calculating the value of y by the following formula, and if y is a positive number, determining that the pedestrian exists in the window
Figure BDA0002209832270000113
Wherein y represents the multiply-accumulate result; x represents a feature drawingThe value is as described;
Figure BDA0002209832270000114
represents a threshold value;
and 6, comparing the detection results of the binocular camera image and the thermal imager image, outputting the result if the detection results of the two images are consistent, and judging that the two images are not consistent.

Claims (8)

1. A pedestrian detection method based on a thermal imager and a binocular camera is characterized by comprising the following steps:
step 1, arranging a thermal imager and a binocular camera on a vehicle, and collecting an image in front of the vehicle;
step 2, eliminating parallax between images collected by a binocular camera;
step 3, graying the binocular camera image and the thermal imager image after the parallax is eliminated in the step 2;
step 4, scanning the grayed binocular camera image and the thermal imager image by using a sliding window in a raster scanning mode according to set step lengths, and simultaneously extracting HOG characteristics of the binocular camera image and the thermal imager image in each step length to obtain HOG characteristic description values;
step 5, training the SVM by adopting a pedestrian database to obtain a coefficient wTThe HOG characteristic description value X corresponding to each sliding window and the coefficient w are comparedTPerforming multiply-accumulate to obtain the result and setting the threshold
Figure FDA0002209832260000011
Adding, if the sum is larger than 0, judging that a pedestrian exists in the window;
and 6, comparing the detection results of the binocular camera image and the thermal imager image, outputting a unified result if the detection results of the two images are consistent, and judging that the two images are not consistent.
2. The pedestrian detection method based on the thermal imager and the binocular camera according to claim 1, wherein the step 4 of scanning the grayed image of the binocular camera and the image of the thermal imager by using the sliding window in a raster scanning manner according to a set step length comprises the specific steps of:
setting an interested area of a scene to be detected;
setting the size of the sliding window and the horizontal and longitudinal moving step length;
and moving a sliding window to traverse the whole picture to be detected according to the rules from left to right and from top to bottom in the region of interest from the upper left corner of the picture to be detected by a set transverse and longitudinal step length.
3. The pedestrian detection method based on the thermal imager and the binocular camera according to claim 1, wherein the specific method for performing HOG feature extraction on the image of the binocular camera and the image of the thermal imager is as follows:
calculating the pixel gradient of the gray image in the sliding window, and determining the mode and the direction of the pixel gradient;
dividing the gray level image in the sliding window into n pixel blocks, and performing dot multiplication on a modulus of each pixel point in each pixel block and a Gaussian matrix;
performing three-bit linear interpolation on each pixel block and arbitrating accumulation to form a 36-dimensional vector histogram according to the coordinates of the pixel block where the pixel point is located and the gradient vector angle of the pixel point;
normalizing the 36-dimensional vector histogram obtained by each pixel block;
and combining the n normalized 36-dimensional vectors to obtain the 36X n-dimensional HOG feature description value X in the window.
4. The pedestrian detection method based on the thermal imager and the binocular camera according to claim 3, wherein the specific method for calculating the gradient of the gray image pixels in the sliding window and determining the mode and the direction of the gradient of the pixel points comprises the following steps:
calculating the directional gradient of the abscissa and the ordinate of the pixel by first order differential derivation:
Gx(x,y)＝f(x+1,y)-f(x-1,y)
Gy(x,y)＝f(x,y+1)-f(x,y-1)
Gx(x,y)、Gy(x,y)、f(x,y) respectively representing the gradient of the pixel point in the horizontal direction, the gradient in the vertical direction and the pixel value;
the mode and direction of the gradient direction of the pixel point are as follows:
Figure FDA0002209832260000021
Figure FDA0002209832260000022
in the formula, | G (x, y) | represents a modulus of a gradient vector of a pixel point, and θ (x, y) represents a direction of the gradient vector.
5. The thermal imager and binocular camera based pedestrian detection method of claim 3, wherein the formula for normalizing the vector histogram is:
Figure FDA0002209832260000023
in the formula, v*Represents the histogram normalization result, v represents a 36-dimensional vector histogram, and epsilon is a constant.
6. The thermal imager and binocular camera-based pedestrian detection method according to claim 1, wherein a pedestrian database is used to train the SVM to obtain the coefficient wTAnd a threshold valueThe HOG characteristic description value X corresponding to each sliding window is compared with the coefficient wTPerforming multiply-accumulate to obtain the result and setting the threshold
Figure FDA0002209832260000025
And if the sum is greater than 0, the specific method for judging that the pedestrian exists in the window is as follows:
training the SVM by adopting a Daimer pedestrian database to obtainA 36 × n dimensional coefficient wTAnd a threshold value
Figure FDA0002209832260000026
Coefficient wTDividing the HOG characteristic description value X corresponding to each window into 15 parts, and matching the HOG characteristic description value X corresponding to each window with the coefficient w of the 15 partsTPerforming multiplication accumulation, wherein the formula is as follows:
Figure FDA0002209832260000031
t＝0,1,…,14；
k＝0,1,…,6…,s
in the formula, X represents a characteristic description value, and s is the number of times of transverse movement of the sliding window;
calculating the value of y by the following formula, and if y is a positive number, determining that the pedestrian exists in the window
Wherein, y represents the result of multiply-accumulate,
Figure FDA0002209832260000033
representing a threshold value.
7. The utility model provides a pedestrian detection system based on thermal imager and binocular camera which characterized in that includes:
the thermal imager and the binocular camera are arranged on the vehicle and used for acquiring image information in front of the vehicle;
a computer for processing image information;
the computer is used for executing the steps 2 to 6 of any claim from 1 to 6.
8. The pedestrian detection system based on the thermal imager and the binocular camera according to claim 1, further comprising an aluminum box for protecting the binocular camera and the thermal imager, the binocular camera and the thermal imager being disposed in the aluminum box.
CN201910894646.1A 2019-09-20 2019-09-20 Pedestrian detection method and system based on thermal imager and binocular camera Active CN110765877B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910894646.1A CN110765877B (en) 2019-09-20 2019-09-20 Pedestrian detection method and system based on thermal imager and binocular camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910894646.1A CN110765877B (en) 2019-09-20 2019-09-20 Pedestrian detection method and system based on thermal imager and binocular camera

Publications (2)

Publication Number Publication Date
CN110765877A true CN110765877A (en) 2020-02-07
CN110765877B CN110765877B (en) 2022-09-06

Family

ID=69330765

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910894646.1A Active CN110765877B (en) 2019-09-20 2019-09-20 Pedestrian detection method and system based on thermal imager and binocular camera

Country Status (1)

Country Link
CN (1) CN110765877B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111985398A (en) * 2020-08-20 2020-11-24 广州洪拓光电技术有限公司 Infrared-assisted binocular distance measurement method and distance compensation heat value acquisition method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106203274A (en) * 2016-06-29 2016-12-07 长沙慧联智能科技有限公司 Pedestrian's real-time detecting system and method in a kind of video monitoring
CN107038416A (en) * 2017-03-10 2017-08-11 华南理工大学 A kind of pedestrian detection method based on bianry image modified HOG features
CN108549864A (en) * 2018-04-12 2018-09-18 广州飒特红外股份有限公司 Area-of-interest filter method towards vehicle-mounted thermal imaging pedestrian detection and device

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106203274A (en) * 2016-06-29 2016-12-07 长沙慧联智能科技有限公司 Pedestrian's real-time detecting system and method in a kind of video monitoring
CN107038416A (en) * 2017-03-10 2017-08-11 华南理工大学 A kind of pedestrian detection method based on bianry image modified HOG features
CN108549864A (en) * 2018-04-12 2018-09-18 广州飒特红外股份有限公司 Area-of-interest filter method towards vehicle-mounted thermal imaging pedestrian detection and device

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111985398A (en) * 2020-08-20 2020-11-24 广州洪拓光电技术有限公司 Infrared-assisted binocular distance measurement method and distance compensation heat value acquisition method
CN111985398B (en) * 2020-08-20 2023-11-28 广州洪拓光电技术有限公司 Infrared auxiliary binocular range finding method and range compensation heat value acquisition method

Also Published As

Publication number Publication date
CN110765877B (en) 2022-09-06

Similar Documents

Publication Publication Date Title
CN109034047B (en) Lane line detection method and device
CN107330376B (en) Lane line identification method and system
Wu et al. Lane-mark extraction for automobiles under complex conditions
EP1671216B1 (en) Moving object detection using low illumination depth capable computer vision
US9846812B2 (en) Image recognition system for a vehicle and corresponding method
US9626599B2 (en) Reconfigurable clear path detection system
EP2237988B1 (en) Object detection and recognition system
CN109190523B (en) Vehicle detection tracking early warning method based on vision
CN105206109B (en) A kind of vehicle greasy weather identification early warning system and method based on infrared CCD
US20080285799A1 (en) Apparatus and method for detecting obstacle through stereovision
US10867403B2 (en) Vehicle external recognition apparatus
US11236991B2 (en) Method for determining a current distance and/or a current speed of a target object based on a reference point in a camera image, camera system and motor vehicle
CN106096604A (en) Multi-spectrum fusion detection method based on unmanned platform
CN101383005B (en) Method for separating passenger target image and background by auxiliary regular veins
CN114639078A (en) Vehicle type recognition method, device and system
EP3432265A1 (en) Image processing device, apparatus control system, image pickup device, image processing method, and program
CN105512641B (en) A method of dynamic pedestrian and vehicle under calibration sleet state in video
CN103034843A (en) Method for detecting vehicle at night based on monocular vision
KR101264282B1 (en) detection method vehicle in road using Region of Interest
CN110765877B (en) Pedestrian detection method and system based on thermal imager and binocular camera
CN113762134A (en) Method for detecting surrounding obstacles in automobile parking based on vision
US9041778B2 (en) Image processing device and method of processing image
Krotosky et al. A comparison of color and infrared stereo approaches to pedestrian detection
CN110688876A (en) Lane line detection method and device based on vision
CN111833384A (en) Method and device for quickly registering visible light and infrared images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant