CN103345792B - Based on passenger flow statistic device and the method thereof of sensor depth image - Google Patents

Abstract

The invention discloses a passenger flow counting device and a method thereof based on a sensor depth of field image. The device includes a sensor base, a double depth sensor, a USB data line, a data processing control unit, a display, a mode selection switch and a power supply; the lens of the double depth sensor Both look down on the ground and the central axis of the lens is perpendicular to the ground plane. The signal output ends of the two are connected to the data processing control unit through the USB data line. The data processing control unit is connected to the display through the video line, and the mode selection switch is connected to the data processing control unit. The output end of the power supply is connected to the power input end of other components. The method is as follows: convert the double depth-of-field images to the same physical coordinate system and stitch them together; traverse the stitched depth-of-field images to find and mark the gray-scale minimum areas; extract each marked area and determine the position information of the head; establish the head Motion track, count the number of people entering and leaving. The device of the invention has the advantages of low cost, high precision and strong stability, and has broad application prospects.

Description

Passenger flow counting device and method based on sensor depth image

1. Technical field

The invention belongs to the field of digital image processing and pattern recognition, in particular to a passenger flow counting device and method based on sensor depth images.

2. Background technology

With the continuous development of statistical analysis technology and computer technology, instant and reliable passenger flow information has become possible. At the same time, the importance of passenger flow information is increasingly reflected, especially for shopping places such as large shopping malls and supermarkets, public transportation systems such as railways, subways and buses, and exhibition halls, stadiums, libraries, airports and other places that have high requirements for public safety. For public places, the importance of real-time and accurate passenger flow information is self-evident.

Traditional people counting methods include human counting or counting triggered by electronic equipment, which not only wastes manpower and material resources, but also is inefficient. With the rapid development of computer vision and image processing technology, its application in people counting becomes possible. In the crowded crowd with high passenger flow, passengers block each other, and the human body cannot be effectively modeled. Only the head of the passenger is relatively complete in the image processing. Therefore, the passenger flow statistics method based on machine vision mainly uses the feature extraction of passenger head. way to locate and track passengers. The automatic passenger flow counting based on the color video stream mainly uses the feature recognition and pattern matching in the image, but it is sensitive to the light changes of the counting environment, and the interference of shadows and mutual occlusion of the flow of people cannot be ruled out. It is only suitable for simple background situations, and The amount of data to be processed is large, and the application of the algorithm is subject to many restrictions, so it is not universal. The automatic passenger flow counting based on stereo vision mainly uses dual sensors to extract the depth information of the moving target, and detects the target according to the depth information. Suitable for large-scale promotion.

3. Contents of the invention

The purpose of the present invention is to provide a highly accurate and stable passenger flow counting device and method based on sensor depth of field images to solve the problems of crowding, occlusion, and shadows in passenger flow counting, and to perform real-time and efficient counting of passenger flow.

The technical solution to realize the object of the present invention is, a kind of passenger flow counting device based on the depth of field image of the sensor, comprising a sensor base, a first depth sensor, a second depth sensor, a USB data line, a data processing control unit, a display, a mode selection switch, Power supply; the first depth sensor and the second depth sensor are identical and arranged in parallel on the same sensor base, the lenses of the first depth sensor and the second depth sensor are both overlooking the ground and the central axis of the lens is perpendicular to the ground plane, the first depth sensor and the The signal output ends of the second depth sensor are all connected to the data processing control unit through the USB data line, the signal output ends of the data processing control unit are connected to the display through the video line, and the mode selection switch is connected to the control signal input end of the data processing control unit. The output ends of the power supply are respectively connected to the power input ends of the first depth sensor, the second depth sensor, the data processing control unit and the display;

After the power supply is powered on to the device, the depth of field images collected by the first depth sensor and the second depth sensor are all input to the data processing control unit through the USB data line, and the data processing control unit performs digital image processing on the collected depth of field images, and The processed results are output to the monitor via the video line.

A passenger flow counting method based on sensor depth images, the steps are as follows:

Step 1, after powering on the device, the data processing control unit acquires the first depth-of-field map collected by the first depth sensor and the second depth-of-field map collected by the second depth sensor through the USB data cable;

Step 2: Calibrate the relationship between the distance H from the target to the ground and the corresponding gray level G in the depth-of-field map according to the acquired first-frame first-frame depth-of-field map or first-frame second-field-field map;

Step 3, determine the working mode through the mode selection switch: if the single depth sensor working mode is selected, select the first depth of field map or the second depth of field map as the third depth of field map A3 and go to step 6; if the dual depth sensor working mode is selected, Then go to the next step;

Step 4, transforming both the first depth-of-field map and the second depth-of-field map into a three-dimensional space physical coordinate system;

Step 5, splicing the first depth-of-field map A1 and the second depth-of-field map A2 transformed into the three-dimensional space physical coordinate system to obtain the third depth-of-field map A3;

Step 6, performing region segmentation on the third depth-of-field image A3, finding the pixel coordinates corresponding to the minimum gray level value in each region, and marking the found pixel coordinates;

Step 7, perform head recognition to obtain head information according to the mark of pixel coordinates after region segmentation;

Step 8, tracking and counting the identified header information;

Step 9: sequentially process each frame of the acquired depth map according to the method of steps 4-8, and output the statistical entry and exit results in the form of a file to the display through the video line every one minute.

Compared with the prior art, the present invention has the following significant advantages: (1) the system can select a single sensor to realize passenger flow statistics, and can also use multi-sensors to expand the field of view to realize multi-field passenger flow statistics; (2) the dual-sensor The synchronization of video acquisition solves the problem of sensor synchronization based on stereo vision. The depth map output by a single sensor contains three-dimensional information of the human body, and solves the problems of passenger flow congestion, occlusion, shadows, etc. in passenger flow statistics based on video streams; (3) the system can Real-time detection and statistics of passenger flow, with strong stability and low cost, the frame frequency is 30pf/s, and the accuracy is above 96%.

4. Description of drawings

FIG. 1 is a schematic structural diagram of a passenger flow counting device based on a sensor depth map according to the present invention.

Fig. 2 is a flow chart of the passenger flow counting method based on the sensor depth map of the present invention.

Fig. 3 is a spatial physical coordinate relationship diagram of two depth sensors (a) first depth sensor coordinate system (b) second depth sensor coordinate system (c) actual physical coordinate system.

Fig. 4 is a schematic diagram of a nine-square grid in the region segmentation of the present invention.

Fig. 5 is the original depth map in the embodiment of the present invention, wherein (a) is the first depth map, and (b) is the second depth map.

Fig. 6 is a third depth map formed after splicing the first depth map and the second depth map in the embodiment of the present invention.

Fig. 7 is a diagram of the head tracking recognition result of the third depth of field image in the embodiment of the present invention.

5. Specific implementation

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

In conjunction with FIG. 1 , the passenger flow counting device based on the sensor depth image of the present invention includes a sensor base 1, a first depth sensor 2, a second depth sensor 3, a USB data line 7, a data processing control unit 8, a display 9, and a mode selection switch 10 , power supply 11; the first depth sensor 2 and the second depth sensor 3 are identical and arranged in parallel on the same sensor base 1, the lenses of the first depth sensor 2 and the second depth sensor 3 all look down on the ground and the central axes of the lenses are all in line with the ground plane Vertically, the signal output ends of the first depth sensor 2 and the second depth sensor 3 are connected to the data processing control unit 8 through the USB data line 7, and the signal output ends of the data processing control unit 8 are connected to the display 9 through the video line 12, the mode The selector switch 10 is connected to the control signal input end of the data processing control unit 8, and the output end of the power supply 11 is respectively connected to the power input ends of the first depth sensor 2, the second depth sensor 3, the data processing control unit 8 and the display 9;

After the power supply 11 powers up the device, the mode of operation is determined by the mode selection switch 10, and the depth of field images collected by the first depth sensor 2 and the second depth sensor 3 are all input into the data processing control unit 8 through the USB data line 7, and the data processing control The unit 8 performs digital image processing on the collected depth-of-field image, and outputs the processed result to the display 9 through the video line 12 .

The horizontal distance WIDTH between the first depth sensor 2 and the second depth sensor 3 satisfies the following conditions:

$\frac{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; WIDTH</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; the\; tan</span>}\frac{\mathrm{<span\; class="notranslate">\; v</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

H ₀ represents the distance from the first depth sensor 2 and the second depth sensor 3 to the ground, 3000mm<H ₀ <3700mm, ν is the lateral field angle of the first depth sensor 2 and the second depth sensor 3, H ₄ (2100< H ₄ <H ₀ ) represents the distance between the two tangent points of the field of view and the ground.

The first depth sensor 2 and the second depth sensor 3 both use ASUS XtionPRO, which is an XtionSensor produced by ASUS, and the grayscale image output by the sensor is a depth map containing three-dimensional information of the human body. The mode selection switch 10 is provided with a single depth sensor working mode and a double depth sensor working mode. On the ground in the field of view of the first depth sensor 2 and the second depth sensor 3, an incoming direction mark 4, an outgoing direction mark 5, and an entry and exit judgment line 6 are also provided as the entry and exit judgment standard of passenger flow.

In conjunction with Fig. 2, the present invention is based on the passenger flow counting method of sensor depth image, and the steps are as follows:

Step 1, after the power supply 11 powers on the device, the data processing control unit 8 obtains the first depth-of-field map collected by the first depth sensor 2 and the second depth-of-field map collected by the second depth sensor 3 through the USB data line 7;

Step 2: Calibrate the relationship between the distance H from the target to the ground and the corresponding gray level G in the depth-of-field map according to the acquired first-frame first-frame depth-of-field map or first-frame second-field-field map;

A target with a height of H ₁ from the ground corresponds to gray level G ₁ , a target with a height of H ₂ from the ground corresponds to gray level G ₂ , and a target with a height of H ₃ from the ground corresponds to gray level G ₃ , where 100mm <H ₁ <200mm, 500mm<H ₂ <800mm, 2300mm<H ₃ <H ₀ , H ₀ represents the distance from the sensor to the ground and 3000mm<H ₀ <3700mm, G ₀ represents no one is within the field of view, depth map The gray level corresponding to the middle ground is:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\end{array}\right.$

In the formula, the distance calibration is in millimeters, β represents the proportional coefficient of distance and gray level, and 10<β<20;

Step 3, determine the working mode through the mode selection switch 10: if the single depth sensor working mode is selected, then select the first depth of field map or the second depth of field map as the third depth of field map A3 and go to step 6; if the dual depth sensor working mode is selected , then go to the next step;

Step 4, transforming both the first depth-of-field map and the second depth-of-field map into a three-dimensional space physical coordinate system;

Transform the first depth-of-field map from two-dimensional plane coordinates (i ₁ , j ₁ ) to three-dimensional space physical coordinates (x ₁ , y ₁ , z ₁ ) to obtain a new first depth-of-field map A1, and convert the second depth-of-field map from two Two-dimensional plane coordinates (i ₂ , j ₂ ) are projected into three-dimensional space physical coordinates (x ₂ , y ₂ , z ₂ ) to obtain a new second depth-of-field map A2; the pixel resolution of the two-dimensional depth-of-field map is I*J, The center coordinate of the ordinate is I _m , the center coordinate of the abscissa is J _m , μ is the longitudinal field angle of the sensor, and ν is the transverse field angle of the sensor, then the projection conversion relationship is as follows:

$\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; x</span>}\\ \mathrm{<span\; class="notranslate">\; the\; y</span>}\\ \mathrm{<span\; class="notranslate">\; z</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& {\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; i</span>}\\ \mathrm{<span\; class="notranslate">\; j</span>}\\ \mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>\end{array}\right]$

Wherein, _ki =sinθi*G( _i ,j), θi represents the field of view angle corresponding to the ordinate _i to _Im in the two-dimensional plane coordinates (i,j), k _j = sinθ _j *G(i,j), θ _j represents the field of view angle corresponding to the abscissa j to the center coordinate J _m in the two-dimensional plane coordinates (i,j), G(i,j) is the gray level corresponding to the two-dimensional plane coordinates (i,j);

Step 5, stitching the first depth-of-field map A1 and the second depth-of-field map A2 transformed into the three-dimensional space physical coordinate system; combined with Figure 3, coordinate system (a) and coordinate system (b) respectively represent the first depth-of-field map A1 and the second depth-of-field map A1 The physical coordinate system corresponding to the second depth of field map A2, there is a displacement and rotation relationship R|T between the two physical coordinate systems (a) and (b), use R|T to transfer the two coordinates to the same coordinate (c) middle.

Step (5.1), using the multi-view point measurement data method to determine the conversion between the second depth map A2 coordinates (x ₂ , y ₂ , z ₂ ) and the first depth map A1 coordinates (x ₁ , y ₁ , z ₁ ) Relation R|T:

$<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; )</span>\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\end{array}\right]<span\; class="notranslate">\; =</span>\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\end{array}\right]$

Take the coordinates of S points in the first depth-of-field map A1 And the corresponding coordinates of these S points in the second depth-of-field map A2 a=0,1,2...,S-1 and S>2, determine the value of rotation and translation of the second depth map A2 relative to the first depth map A1:

${\mathrm{<span\; class="notranslate">\; P</span>}}_{{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}}$

R represents the rotation relationship of the second depth map A2 relative to the first depth map A1, and T represents the displacement of the second depth map A2 relative to the first depth map A1, then:

$\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccc}\mathrm{<span\; class="notranslate">\; cos</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}& \mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}& \mathrm{<span\; class="notranslate">\; cos</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]$

$\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccc}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ {\mathrm{<span\; class="notranslate">\; \&Delta;x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}& {\mathrm{<span\; class="notranslate">\; \&Delta;y</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}& {\mathrm{<span\; class="notranslate">\; \&Delta;z</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]$

In the above formula, θ represents the rotation angle of the second depth map A2 relative to the first depth map A1, and (Δx ₀ , Δy ₀ , Δz ₀ ) represents the relative coordinates (x ₂ , y ₂ , z ₂ ) of the second depth map A2 The displacement difference at the coordinates (x ₁ , y ₁ , z ₁ ) of the first depth-of-field map A1;

Step (5.2), create a third depth-of-field map A3 with a pixel resolution of C*D and an initial value of 0, where I<C<2I, J<D<2J; coordinates of the first depth-of-field map A1 (x ₁ , y ₁ , z ₁ ) to the coordinates (x ₂ , y ₂ , z ₂ ) of the second depth-of-field map A2, and then the first depth-of-field map A1 and the second depth-of-field map A2 are stitched together and stored in the third depth-of-field map A3, If the gray level G(x,y)>G ₁ , then G(x,y)=G ₁ , in the spliced overlapping area of the first depth map A1 and the second depth map A2, the gray level takes the first depth map A1 and the mean value of the corresponding coordinate gray levels in the second depth-of-field image A2;

Step 6, performing region segmentation on the third depth-of-field image A3, finding the pixel coordinates corresponding to the minimum gray level value in each region, and marking the found pixel coordinates;

Step (6.1), using A*A pixels as the window, 20<A<70, divide the third depth map A3 into (C/A)*(D/A) regions, and C/A, D/A are both It is a positive integer; use the Jiugongge to traverse all areas of the third depth of field map A3. Figure 4 is a schematic diagram of the Jiugongge. Each window of the Jiugongge corresponds to an area, and the middle one is the center window. Find the center of the Jiugongge that meets the following two conditions Window area: ①The area where the center window is located has pixels whose gray level is smaller than G ₂ and greater than G ₃ ②The average gray value of the central window area is smaller than the average gray value of other windows in Jiugongge; mark the found area as N _t , t=1 ,2,3...T and T<(C/A)*(D/A), and record the minimum gray level Min _t of each area at the same time;

In step (6.2), double the four sides of the N _t area, and find out all the pixels in the expanded area whose gray level satisfies the condition of G _t (x,y)-Min _t <λ (4<λ<15) Coordinates, and marked as K _t , a total of T markers are generated.

Step 7, perform head recognition according to the marker K _t of the pixel coordinates after region segmentation, the specific steps are as follows:

Step (7.1), create T binary images with an initial value of 0, numbered 1 to T, and the pixel resolution of each binary image is C*D; extract all pixel coordinates corresponding to the marker K _t , And set the grayscale value of the corresponding pixel coordinate in the tth binary image to 255;

Step (7.2), determine the area Area _t of the connected domain in each binary image, and the gray level mean value corresponding to the connected domain in the third depth of field image A3 Find the connected domain with the following characteristics: ① the area Area _t is larger than Area, and Area≥500, ② the connected domain in the binary image is elliptical, ③ the head width w′ and head height h′ are converted according to the following formula All satisfy less than 25 and greater than 15, the formula is as follows:

${\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 255.0</span>}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; G</span>}}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; h</span>}$

${\mathrm{<span\; class="notranslate">\; w</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 255.0</span>}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; G</span>}}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; w</span>}$

Where h' represents the actual height of the human head, w' represents the actual width of the human head, h represents the height of the connected domain in the binary image, w represents the width of the connected domain in the binary image, α is the proportional coefficient and 0< α<1;

Record the starting coordinates, width and height of all found connected domains and the mean value of the region That is, the header information;

Step 8, tracking the identified header information;

Step (8.1), in the field of view, the central point of the head information is used as the dot trace, the shape of the wave gate is rectangular, the size of the wave gate is set to Q pixels, and 10<Q<100, the dot trace (x, y) and The gate (Δx, Δy) determines the predicted value of the target point trace is the centered spatial search area, the search area is as follows:

$<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; \&Delta;x</span>}$

$<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; \&Delta;y</span>}$

Step (8.2), first, assuming the flight track, record the head target track for the first time in the Mth frame, estimate the speed of the target track based on the coordinates obtained in the M+1th frame, if the estimated speed is within the scope of the search area, that is A temporary track is generated; secondly, based on the coordinates obtained in the M+2th frame, the position of the target track is predicted, and the relevant area is determined centered on the predicted position Any point trace falling in the relevant area will expand a temporary track, continue to estimate the velocity value, predict the position of the next frame based on the velocity value and establish the relevant area, any point trace falling in the relevant area will generate a new track Finally, fit all the generated tracks with a quadratic curve, if the error δ<0.1 between the point on the track and the fitted curve, then determine the track, if not, delete the track trace;

Step (8.3), after determining the target motion track, draw _two judgment lines on the l1th line and l2th line of the third depth of field map A3, wherein ₁₀ < _l1 < _l2 <470 pixels, if the current target point If the track x is less than l ₁ but there are points with line coordinates greater than l ₂ on the track, the number of people entering will be increased by 1, otherwise if the current target point x is greater than l ₂ but there are points with line coordinates less than l ₁ on the track, then exit The number of people increases by 1, and the track is deleted at the same time;

Step 9: sequentially process the acquired depth maps of each frame according to the method of steps 4-8, and output the statistical entry and exit results to the display 9 via the video line 12 every minute.

Example 1

The present invention is based on the passenger flow counting device of the sensor depth image, and the specific parameters are as follows:

(1), the first depth sensor 2 and the second depth sensor 3 all adopt ASUS XtionPRO, which is the XtionSensor produced by ASUS, the longitudinal field angle μ of the first depth sensor 2 and the second depth sensor 3 is 45 °, and the horizontal viewing The field angle ν is 58°, and the pixel resolution of the output two-dimensional depth map is I*J is 480*640, then the center coordinate I _m of the ordinate is 240, and the center coordinate J _m of the abscissa is 320, so

(2), respectively take S=6 points in the first depth-of-field map A1 and the second depth-of-field map A2, and determine the corresponding rotation parameter R between the two depth-of-field maps A1 and A2 obtained after projecting onto the space physical coordinate system and the displacement parameter T to create a third depth-of-field map A3 with a pixel resolution of C*D=600*1200 and an initial value of 0. The first depth-of-field map A1 and the second depth-of-field map A2 are spliced and stored in 600*1200 In the third depth-of-field map A3; in Fig. 5 (a) is the first depth-of-field map A1, (b) is the second depth-of-field map A2, and Fig. 6 is the third depth-of-field formed after the splicing of the first depth-of-field map and the second depth-of-field map Figure A3, the stitched third depth of field image A3 is well stitched, expanding the field of view;

(3), with A*A=40*40 pixels as the window, the third depth of field map A3 with pixels of 600*1200 is divided into (C/A)*(D/A)=(600/40)*(1200 /40)=15*30 areas;

(4) Expand the four sides of the N _t area twice, find out all pixel coordinates whose gray levels in the expanded area satisfy G _t (x, y)-Min _t <λ, λ=6, and mark them as K _t , generate T tokens in total;

(5), after determining the target movement track, draw two judgment lines on the l ₁ =300 line and the l ₂ =400 line of the third depth of field map A3; head width w' and head height h' conversion formula :

${\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 255.0</span>}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; G</span>}}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; h</span>}$

${\mathrm{<span\; class="notranslate">\; w</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 255.0</span>}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; G</span>}}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; w</span>}$

where α is a proportionality coefficient and α=0.007;

Fig. 7 is the head tracking recognition result diagram of the third depth-of-field image A3. The position of the target is framed with a frame, and the track of the target movement is established. Two judgment lines are drawn in the depth-of-field image, and the results of the in-and-out statistics are displayed. At the same time, it helps to monitor the passenger flow in real time through the screen;

(6), the statistical result of system is written in the data text file, and compare with actual result, the result is as table 1, can find out from the table that the result of the present invention's statistics compares with the actual number of people in and out, average accuracy Above 96%.

Table 1

time In/Out (statistical results) In/Out (actual result) precision 9:00-9:15 37/17 35/17 0.9714 11:30-12:00 110/93 117/92 0.9647 14:00-14:15 79/64 81/60 0.9543 17:00-17:15 80/39 84/41 0.9517 average value 0.9605

To sum up, the device and method for passenger flow statistics based on sensor depth images in the present invention can select a single sensor to realize passenger flow statistics, and can also use multiple sensors to expand the field of view to realize multi-field passenger flow statistics. Synchronous, solves the sensor synchronization problem based on stereo vision, the depth map output by a single sensor contains three-dimensional information of the human body, and solves the problems of passenger flow congestion, occlusion, shadows and other problems in passenger flow statistics based on video streams. The system has strong stability and low cost. The frame frequency is 30pf/s, and the accuracy is above 96%, which has broad application prospects.

Claims (7)

1. the passenger flow statistic device based on sensor depth image, it is characterized in that, comprise sensor base (1), the first depth transducer (2), the second depth transducer (3), USB data line (7), data processing control units (8), display (9), mode selection switch (10), power supply (11), first depth transducer (2) is identical with the second depth transducer (3) and be set in parallel in same sensor base (1), first depth transducer (2) all overlooks ground with the camera lens of the second depth transducer (3) and optical center axle is all vertical with ground level, first depth transducer (2) is all connected with data processing control units (8) by USB data line (7) with the signal output part of the second depth transducer (3), the signal output part of data processing control units (8) is by video line (12) access display (9), mode selection switch (10) is connected with the control signal input end of data processing control units (8), the output terminal of power supply (11) accesses the first depth transducer (2) respectively, second depth transducer (3), the power input of data processing control units (8) and display (9),

After power supply (11) powers on to device, the depth image that first depth transducer (2) and the second depth transducer (3) collect is all by USB data line (7) input data processing control units (8), Data Data processing and control element (PCE) (8) through Digital Image Processing, and exports the result of process to display (9) by video line (12) to the depth image collected; Horizontal distance W IDTH between described first depth transducer (2) and the second depth transducer (3) meets the following conditions:

H ₀represent the first depth transducer (2) and the second depth transducer (3) distance to ground, 3000mm < H ₀< 3700mm, ν are the transverse field angle of the first depth transducer (2) and the second depth transducer (3), H ₄(2100 < H ₄< H ₀) represent the distance of point of contact, two visual fields apart from ground.

2. the passenger flow statistic device based on sensor depth image according to claim 1, is characterized in that, described first depth transducer (2) and the second depth transducer (3) all adopt Asus XtionPRO.

3. based on a passenger flow statistical method for sensor depth image, it is characterized in that, step is as follows:

Step 1, after power supply (11) powers on to device, data processing control units (8) obtains the first depth map that the first depth transducer (2) collects and the second depth map that the second depth transducer (3) collects by USB data line (7);

Step 2, according to first frame first depth map obtained or first frame second depth map, demarcates the relation in target to the distance H and depth map on ground between corresponding grey scale level G;

Step 3, determines mode of operation by mode selection switch (10): if select single depth transducer mode of operation, then choose the first depth map or the second depth map as the 3rd depth map A3 and proceed to step 6; If select dual-depth working sensor pattern, then enter next step;

Step 4, by the first depth map and the equal projection transform of the second depth map to three dimensions physical coordinates system;

Step 5, carries out splicing by the first depth map A1 and the second depth map A2 that are transformed into three dimensions physical coordinates system and obtains the 3rd depth map A3;

Step 6, carries out region segmentation to the 3rd depth map A3, finds out the pixel coordinate that in each region, gray level minimal value is corresponding, and mark the pixel coordinate searched out;

Step 7, according to the mark of pixel coordinate after region segmentation, carries out head identification and obtains header information; Described head identification concrete steps are as follows:

Step (7.1), creating T width initial value is the binary map of 0, be numbered 1 ~ T, and the pixel resolution of each width binary map is C*D; Extract marker character K _tcorresponding all pixel coordinates, and the gray-scale value of respective pixel coordinate in t width binary map is set to 255;

Step (7.2), determines the area A rea of connected domain in every width binary map _t, and to should the gray level average of connected domain in the 3rd depth map A3 find the connected domain with following characteristics: 1. area A rea _tbe greater than Area, and Area>=500, the connected domain 2. in binary map is class ellipticity, and 3. head width w ' and height of head h ' is less than 25 according to all meeting after following formula scales and is greater than 15, and formula is as follows:

The wherein true altitude of h ' expression human body head, the developed width of w ' expression human body head, h represents the height of connected domain in binary map, and w represents the wide of connected domain in binary map, and α is scale-up factor and 0 < α < 1;

Record all the connected domain origin coordinates, wide height and the regional average value that search out i.e. header information;

Step 8, carries out count tracking to the header information identified; Described head tracking counting, detailed process is as follows:

Step (8.1), in field range with header information central point for a mark, ripple door shape is rectangle, ripple door size is set to Q pixel, and 10 < Q < 100, determined by mark (x, a y) He Bomen (Δ x, Δ y) with Targets Dots predicted value centered by space search region, region of search is as follows:

Step (8.2), first, assuming that flight path, M frame records head Targets Dots for the first time, and the speed of coordinate to Targets Dots obtained based on M+1 frame is estimated, if the speed estimated in region of search scope namely then generate a temporary transient flight path; Secondly, the coordinate obtained based on M+2 frame to Targets Dots position predict, and determine relevant range centered by predicted position any some mark dropped in relevant range expands a temporary transient flight path, continues estimating speed value, predicts based on the position of velocity amplitude to next frame and set up relevant range, anyly drops on some mark in relevant range by flight path new for generation one; Finally, matching is carried out to the flight path quafric curve of all generations, if the error delta < 0.1 of the point on flight path and matched curve, then determine this flight path, if do not met, then delete this flight path;

Step (8.3), after determining target travel flight path, at the l of the 3rd depth map A3 ₁row and l ₂two, row picture judges line, wherein 10 < l ₁< l ₂< 470 pixel, if current goal point mark x is less than l ₁but flight path exists row-coordinate is greater than l ₂some mark, then the number entered adds 1, if instead current goal point mark x is greater than l ₂but flight path exists row-coordinate is less than l ₁some mark, then the number gone out adds 1, deletes this flight path simultaneously;

Step 9, processes each frame depth map collected successively according to the method for step 4 ~ 8, and exports the turnover result of statistics to display (9) by video line (12) in the form of a file every one minute.

4. the passenger flow statistical method based on sensor depth image according to claim 3, is characterized in that, the relation in target described in step 2 to the distance H and depth map on ground between corresponding grey scale level G is demarcated, specific as follows:

The height on distance ground is H ₁target corresponding grey scale level G ₁, distance ground height be H ₂target corresponding grey scale level G ₂, distance ground height be H ₃target corresponding grey scale level G ₃, wherein 100mm < H ₁< 200mm, 500mm < H ₂< 800mm, 2300mm < H ₃< H ₀, H ₀represent that sensor is to the distance on ground and 3000mm < H ₀< 3700mm, G ₀represent that nobody is in field range, the gray level that in depth map, ground is corresponding, then have:

In formula, distance calibration is in units of millimeter, and β represents the scale-up factor of distance and gray level, and 10 < β < 20.

5. the passenger flow statistical method based on sensor depth image according to claim 3, is characterized in that, described in step 4 by the first depth map and the equal projection transform of the second depth map to three dimensions physical coordinates system, step is as follows:

By the first depth map from two dimensional surface coordinate (i ₁, j ₁) projection transform is to three dimensions physical coordinates (x ₁, y ₁, z ₁) obtain the first new depth map A1, by the second depth map from two dimensional surface coordinate (i ₂, j ₂) projection transform is to three dimensions physical coordinates (x ₂, y ₂, z ₂) obtain the second new depth map A2; If the pixel resolution of two-dimentional depth map is I*J, the centre coordinate of ordinate is I _m, the centre coordinate of horizontal ordinate is J _m, the transverse field angle that longitudinal field angle that μ is sensor, ν are sensor, then projection transform relation is as follows:

Wherein, k _i=sin θ _i* G (i, j), θ _irepresent ordinate i to I in two dimensional surface coordinate (i, j) _mcorresponding field of view angle, k _j=sin θ _j* G (i, j), θ _jrepresent that in two dimensional surface coordinate (i, j), horizontal ordinate j is to centre coordinate J _mcorresponding field of view angle, g (i, j) is the gray level of two dimensional surface coordinate (i, j) correspondence.

6. the passenger flow statistical method based on sensor depth image according to claim 3, is characterized in that, the first depth map A1 and the second depth map A2 that are transformed into three dimensions physical coordinates system is spliced, be specially described in step 5:

Step (5.1), utilizes the method for multiple views measurement data to determine the second depth map A2 coordinate (x ₂, y ₂, z ₂) and the first depth map A1 coordinate (x ₁, y ₁, z ₁) between transformational relation R|T:

Get S point coordinate in the first depth map A1 and the coordinate that this S o'clock corresponding in the second depth map A2 a=0,1,2 ..., S-1 and S > 2, determine that the second depth map A2 rotates relative to the first depth map A1 and the value of translation:

R represents the rotation relationship of the second depth map A2 relative to the first depth map A1, and T represents the displacement of the second depth map A2 relative to the first depth map A1, then:

In above formula, θ represents the rotation angle of the second depth map A2 relative to the first depth map A1, (Δ x ₀, Δ y ₀, Δ z ₀) represent the coordinate (x of the second depth map A2 ₂, y ₂, z ₂) relative to the coordinate (x of the first depth map A1 ₁, y ₁, z ₁) displacement difference;

Step (5.2), creating a width pixel resolution is C*D, and initial value is the 3rd depth map A3 of 0, wherein I < C < 2I, J < D < 2J; By the first depth map A1 coordinate (x ₁, y ₁, z ₁) be converted to the second depth map A2 coordinate (x ₂, y ₂, z ₂), be stored in the 3rd depth map A3, if gray level G (x, y) > is G after then the first depth map A1 and the second depth map A2 being spliced ₁, then G (x, y)=G ₁, in the splicing overlapping region of the first depth map A1 and the second depth map A2, gray level gets the average of respective coordinates gray level in the first depth map A1 and the second depth map A2.

7. the passenger flow statistical method based on sensor depth image according to claim 3, is characterized in that, described in step 6, pixel coordinate labeling process is as follows:

Step (6.1), with A*A pixel for window, 20 < A < 70, the 3rd depth map A3 is divided into (C/A) * (D/A) individual region, and C/A, D/A are positive integer; Utilize nine grids to travel through all regions of the 3rd depth map A3, find the nine grids central window port area meeting following two conditions: 1. window region in center exists gray level and is less than G ₂be greater than G ₃pixel 2. this central window port area gray average be less than other window gray average of nine grids; Be N by the zone marker searched out _t, t=1,2,3 ... T and T < (C/A) * (D/A), record each region minimal gray level Min simultaneously _t;

Step (6.2), by N _tlimit, region four extends out one times, finds out gray level in the region after extending out and meets G _t(x, y)-Min _t< λ, all pixel coordinates of 4 < λ < 15 conditions, and be labeled as K _t, symbiosis becomes T marker character.