CN112926415A - Pedestrian avoiding system and pedestrian monitoring method - Google Patents
Pedestrian avoiding system and pedestrian monitoring method Download PDFInfo
- Publication number
- CN112926415A CN112926415A CN202110167266.5A CN202110167266A CN112926415A CN 112926415 A CN112926415 A CN 112926415A CN 202110167266 A CN202110167266 A CN 202110167266A CN 112926415 A CN112926415 A CN 112926415A
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- road surface
- neural network
- surface image
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000013528 artificial neural network Methods 0.000 claims abstract description 68
- 238000012549 training Methods 0.000 claims description 15
- 238000005516 engineering process Methods 0.000 claims description 8
- 230000001629 suppression Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 5
- 230000005764 inhibitory process Effects 0.000 claims description 2
- 238000001514 detection method Methods 0.000 description 15
- 238000013473 artificial intelligence Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 6
- 238000013145 classification model Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 101100194362 Schizosaccharomyces pombe (strain 972 / ATCC 24843) res1 gene Proteins 0.000 description 1
- 101100194363 Schizosaccharomyces pombe (strain 972 / ATCC 24843) res2 gene Proteins 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000005314 correlation function Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
- G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30236—Traffic on road, railway or crossing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses a pedestrian avoiding system and a pedestrian monitoring method, wherein the pedestrian avoiding system comprises: a monitoring end and an edge cloud end; the monitoring end is arranged on one side of the intersection and used for shooting a road image in a corresponding monitoring range, identifying whether pedestrians exist in the road image by utilizing an internal lightweight neural network, and sending the road image to the edge cloud end when the pedestrians exist in the road image; the edge cloud end is used for inputting the road surface image into an internal high-precision neural network to obtain the position information of the pedestrian in the road surface image, calculating the distance between the pedestrian and the intersection junction point according to the position information to generate avoidance warning information by using the distance, and sending the avoidance warning information to the vehicle within a preset range, wherein the preset range is an area range taking the intersection junction point as the center. The embodiment can effectively avoid potential safety hazards caused by the over-the-horizon problem; and the edge cloud end only works when needed, so that the extra power consumption of the whole system can be greatly reduced.
Description
Technical Field
The invention belongs to the technical field of automobile safety, and particularly relates to a pedestrian avoiding system and a pedestrian monitoring method.
Background
With the development of economy and the continuous improvement of the living standard of people, the number of vehicles is continuously increased, and the road conditions are increasingly complicated. In some beyond visual range, vehicles often cannot avoid pedestrians in time, and traffic safety accidents are easily caused.
With the development of the vehicle-road cooperation technology, the vehicles can implement dynamic real-time information interaction between vehicles and roads through the technologies of the internet of things and the like. Therefore, if the road side detection device can be used for continuously and visually monitoring the road condition by utilizing the vehicle-road cooperation technology to realize pedestrian detection and sending the detection result to the surrounding vehicles, the vehicles can effectively realize pedestrian avoidance.
In recent years, with the rapid development of deep learning, a series of object detection networks, such as fastrcnn, ssd, etc., have appeared. However, in the scene of the internet of things, if the roadside detection device uses the target detection networks to perform uninterrupted pedestrian detection for 24 hours, extremely high equipment power consumption is generated.
Disclosure of Invention
In order to solve the above problems in the prior art, the present invention provides a pedestrian avoidance system and a pedestrian monitoring method. The technical problem to be solved by the invention is realized by the following technical scheme:
in a first aspect, an embodiment of the present invention provides a pedestrian avoidance system, including: a monitoring end and an edge cloud end;
the monitoring end is arranged on one side of the intersection and used for shooting a road image in a corresponding monitoring range, identifying whether pedestrians exist in the road image by utilizing an internal lightweight neural network, and sending the road image to the edge cloud end when the pedestrians exist in the road image;
the edge cloud end is used for inputting the road surface image into an internal high-precision neural network to obtain the position information of the pedestrian in the road surface image, calculating the distance between the pedestrian and a junction point of the intersection according to the position information, generating avoidance warning information by using the distance, and sending the avoidance warning information to vehicles within a preset range, wherein the preset range is an area range taking the junction point of the intersection as the center.
In an embodiment of the present invention, the monitoring end is an MCU.
In one embodiment of the invention, the lightweight neural network is built within a Tensorflow Lite Micro framework.
In one embodiment of the invention, the lightweight neural network comprises MobileNet, SqueezeNet, ShuffleNet and Xception.
In one embodiment of the present invention, the identifying whether there is a pedestrian in the road surface image by using an internal lightweight neural network includes:
obtaining a first confidence coefficient that the road surface image contains the pedestrian and a second confidence coefficient that the road surface image does not contain the pedestrian by using the lightweight neural network;
calculating a difference between the first confidence and the second confidence;
and judging whether the difference value is larger than or equal to a preset threshold value, and if so, judging that the road surface image contains pedestrians.
In one embodiment of the invention, the high precision neural network comprises a YOLO neural network, ResNet, GoogleNet, and SENet.
In one embodiment of the present invention, the high-precision neural network is a YOLO neural network;
the inputting the road surface image into an internal high-precision neural network to obtain the position information of the pedestrian in the road surface image comprises the following steps:
inputting the road surface image into a YOLO neural network obtained by pre-training, and extracting features by using a backbone network to obtain x feature maps with different scales; x is a natural number of 4 or more;
carrying out feature fusion on the x feature maps with different scales by using an FPN network to obtain a prediction result corresponding to each scale;
processing all prediction results through a non-maximum value suppression module to obtain the position information of the pedestrian in the road surface image;
the YOLO neural network comprises a backbone network, an FPN network and a non-maximum value inhibition module which are connected in sequence; the YOLO neural network is obtained by training according to a sample pavement image and the position information of the pedestrian in the sample pavement image.
In one embodiment of the present invention, the position information is pixel coordinates of a bounding box containing the pedestrian in the road surface image.
In an embodiment of the present invention, the calculating the distance between the pedestrian and the intersection boundary point according to the position information includes:
determining a first actual coordinate corresponding to the pedestrian in a world coordinate system by utilizing pixel coordinates of a boundary frame containing the pedestrian and a monocular visual positioning and ranging technology;
acquiring a second actual coordinate of the intersection boundary point in a world coordinate system;
and determining the distance between the first actual coordinate and the second actual coordinate as the distance between the pedestrian and the intersection junction point.
On the other hand, the embodiment of the invention provides a pedestrian monitoring method, which is applied to a monitoring end on one side of an intersection and comprises the following steps:
shooting road surface images in corresponding monitoring ranges;
identifying whether a pedestrian exists in the road surface image by utilizing an internal lightweight neural network;
and when pedestrians exist in the road surface image, the road surface image is sent to the edge cloud.
The invention has the beneficial effects that:
1. the monitoring end of the embodiment of the invention continuously monitors for 24 hours in all weather, once a pedestrian is identified, the road image containing the pedestrian is sent to the edge cloud end, namely the edge cloud end is awakened, then the edge cloud end utilizes the internal high-precision neural network to carry out target detection on the pedestrian in the road image, the position information of the pedestrian in the road image is obtained, the distance between the pedestrian and a junction point of a road junction is calculated according to the position information, avoidance warning information is generated by utilizing the distance and sent to a vehicle to realize the function of warning the vehicle to avoid the pedestrian, and therefore, the pedestrian avoidance system provided by the embodiment can effectively avoid potential safety hazards caused by the over-the-horizon problem. And the edge cloud of this embodiment only works when needed to can greatly reduced the extra power consumption of entire system.
2. The monitoring end of the embodiment of the invention can be an MCU (microprogrammed control Unit), a lightweight AI reasoning frame can be transplanted on the monitoring end, a lightweight classification model based on mobilenet is deployed for identifying pedestrians, and the low power consumption and quick response characteristics of the MCU can be fully utilized; meanwhile, as the MCU carries a real-time operating system, the real-time concurrency of the MCU can be utilized, tasks such as image acquisition, AI inference, edge cloud communication and the like are processed, and the low-power-consumption characteristic of the MCU can be further utilized.
The present invention will be described in further detail with reference to the accompanying drawings and examples.
Drawings
Fig. 1 is a schematic structural diagram of a pedestrian avoidance system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a prior art YOLO neural network;
fig. 3 is a schematic flow chart of a pedestrian monitoring method according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to specific examples, but the embodiments of the present invention are not limited thereto.
Example one
Referring to fig. 1, fig. 1 is a schematic structural diagram of a pedestrian avoidance system according to an embodiment of the present invention, and as shown in fig. 1, a pedestrian avoidance system 100 includes: a monitoring end 101 and an edge cloud end 102.
The monitoring end 101 is arranged on one side of the intersection, the monitoring end 101 is used for shooting a road image in a corresponding monitoring range, identifying whether pedestrians exist in the road image by utilizing an internal lightweight neural network, and sending the road image to the edge cloud end 102 when the pedestrians exist in the road image;
the edge cloud 102 is configured to input the road surface image into an internal high-precision neural network to obtain position information of the pedestrian in the road surface image, calculate a distance between the pedestrian and a junction point of the intersection according to the position information, generate avoidance warning information by using the distance, and send the avoidance warning information to a vehicle within a predetermined range, where the predetermined range is an area range with the junction point of the intersection as a center.
Each end is described in detail below.
1) Monitoring terminal 101
It can be understood that, in order to realize omnibearing detection of a wider area on a road surface, a plurality of monitoring ends 101 may be provided; for example, when the monitoring scene is an intersection, monitoring ends 101 can be arranged in four directions of the south, the east, the west and the north of the intersection; when the monitoring scene is a T-shaped intersection, monitoring ends 101 can be arranged in three directions of the T-shaped intersection; when the monitoring scene is a corner intersection formed by two roads, monitoring ends 101 can be arranged on two sides of the corner intersection; of course, the monitoring terminal 101 can also be arranged according to actual needs.
In an optional implementation manner, the monitoring terminal 101 is an MCU.
An MCU (micro controller Unit), also called a Single Chip Microcomputer (Single Chip Microcomputer) or a Single Chip Microcomputer (Single Chip Microcomputer), is a Chip-level computer formed by appropriately reducing the frequency and specification of a Central Processing Unit (CPU), and integrating peripheral interfaces such as a memory (memory), a counter (Timer), a USB, an a/D converter, a UART, a PLC, a DMA, and the like, and even an LCD driving circuit, on a Single Chip.
Specifically, the MCU carries a real-time operating system; the MCU comprises an image acquisition device and a lightweight neural network; the image acquisition equipment shoots a road surface image in a corresponding monitoring range, the lightweight neural network identifies whether pedestrians exist in the road surface image, and when the pedestrians exist in the road surface image, the MCU sends the road surface image to the edge cloud 102.
The image capturing apparatus may include: cameras, video cameras, still cameras, etc.; in an alternative embodiment, the image capture device may be a high resolution camera.
The image capturing device may continuously capture road surface images of the corresponding area at certain time intervals, for example, at a rate of 30 fps. Of course, the time interval can also be adjusted according to actual requirements.
Of course, after the image acquisition device acquires the road surface image, the image enhancement operations such as cutting, splicing, smoothing, filtering, edge filling and the like can be performed on the road surface image so as to enhance the interesting features in the image and expand the generalization capability of the data set.
Specifically, the lightweight neural network comprises MobileNet, SqueezeNet, ShuffleNet and Xception; in an optional embodiment, the lightweight neural network is a mobile-based lightweight classification model. Compared with a neural network with a complex structure and more parameters, the lightweight neural network has the advantages of smaller volume and higher speed while ensuring the accuracy of the model, and is suitable for being deployed on mobile equipment and edge equipment.
The lightweight neural network is built in a Tensorflow Lite Micro framework, which is a lightweight AI (Artificial Intelligence) inference framework.
In the embodiment, a lightweight AI inference frame is transplanted on the MCU, and a lightweight classification model based on mobilenet is deployed for identifying pedestrians, so that the low power consumption and the quick response characteristic of the MCU are fully utilized; meanwhile, as the MCU carries a real-time operating system, the real-time concurrency of the MCU can be utilized, tasks such as image acquisition, AI inference, communication with the edge cloud end 102 and the like are processed, and the characteristic of low power consumption of the MCU is further utilized.
Specifically, the identifying whether there is a pedestrian in the road surface image by using an internal lightweight neural network includes:
and a) obtaining a first confidence coefficient that the road surface image contains the pedestrian and a second confidence coefficient that the road surface image does not contain the pedestrian by using the lightweight neural network.
As will be appreciated by those skilled in the art, in the field of object detection for images, a classifier will derive a class confidence for an object, i.e., a probability that the object belongs to class a, a probability that the object belongs to class B, etc. In an embodiment of the invention, the categories include pedestrian-containing and pedestrian-free. Therefore, for each road surface image, a first confidence corresponding to the class containing the pedestrian and a second confidence corresponding to the class containing no pedestrian are obtained. The confidence is represented in a numerical form, for example, the confidence may be 70%, etc.
It will be appreciated by those skilled in the art that the lightweight neural network is pre-trained using a plurality of labeled pedestrian sample pavement images and non-pedestrian sample pavement images.
For how to detect the presence or absence of a pedestrian, please refer to the related prior art, which is not described herein.
And b), calculating the difference value between the first confidence coefficient and the second confidence coefficient.
And c), judging whether the difference value is larger than or equal to a preset threshold value, and if so, judging that the road surface image contains pedestrians.
In the embodiment of the present invention, the preset threshold is an empirical value obtained from a large number of sample road surface images, for example, the preset threshold may be 39%. Compared with the prior art, whether the road surface image contains the vehicle is judged only by the confidence degree of the pedestrian relative to the preset threshold value. The embodiment of the invention compares the confidence difference of the existence of the pedestrians with the preset threshold value to judge whether the road surface image contains the pedestrians, and can reduce the misjudgment rate and the disturbance item and improve the judgment accuracy aiming at the fuzzy original image with the relatively close numerical values of the two confidence values.
2) Edge cloud 102
It can be understood that the MCU of the embodiment of the present invention continuously monitors for 24 hours all day long, and once a pedestrian is identified, the road image including the pedestrian is sent to the edge cloud 102, that is, the edge cloud 102 is awakened, and then the edge cloud 102 performs target detection on the pedestrian in the road image by using the internal high-precision neural network, so that the edge cloud 102 of the embodiment only works when needed, thereby greatly reducing the extra power consumption of the whole system.
The high-precision neural network in the embodiment of the invention is constructed based on a deep learning target detection network, and specifically comprises a YOLO neural network, ResNet, GoogleNet and SEnet.
Preferably, the high-precision neural network is a YOLO neural network; the YOLO neural network is obtained by training according to a sample pavement image and the position information of the pedestrian in the sample pavement image.
It should be understood that the sample road surface image is a road surface image containing pedestrians, the YOLO neural network can identify the position information of the pedestrians in any input road surface image to be detected, and the training process for the YOLO neural network is described later.
In the following, the network structure of the YOLO neural network is briefly introduced:
referring to fig. 2, fig. 2 is a schematic structural diagram of a YOLO neural network in the prior art; in fig. 2, the part inside the dashed line box is the YOLO neural network. Wherein the part in the dotted line frame is a backbone (backbone) network of a YOLO neural network, namely a darknet-53 network; the backbone network of the YOLO neural network is formed by connecting CBL modules and 5 resn modules in series. The CBL module is a Convolutional network module, and includes a conv layer (convolutive layer, convolutive layer for short), a BN (Batch Normalization) layer and a leakage relu layer corresponding to an activation function leakage relu, which are connected in series, and the CBL represents conv + BN + leakage relu. The resn module is a residual error module, n represents a natural number, and specifically, as shown in fig. 2, res1, res2, res8, res8, and res4 are sequentially arranged along the input direction; the resn module comprises a zero padding (zero padding) layer, a CBL module and a Residual error unit group which are connected in series, the Residual error unit group is represented by Res unit n, the Residual error unit group comprises n Residual error units, each Residual error unit comprises a plurality of CBL modules which are connected in a Residual error Network (ResNet) connection mode, and the feature fusion mode adopts a parallel mode, namely an add mode.
The rest part outside the main network is an FPN (feature pyramid network) network, and the FPN network is divided into three prediction branches Y1～Y3Predicting branch Y1～Y3The scales of (2) are in one-to-one correspondence with the scales of the feature maps output by the 3 residual error modules res4, res8, res8 in the reverse direction of the input, respectively. The prediction results of the prediction branches are respectively represented by Y1, Y2 and Y3, and the scales of Y1, Y2 and Y3 are increased in sequence.
Each prediction branch of the FPN network includes a convolutional network module group, specifically includes 5 convolutional network modules, that is, CBL × 5 in fig. 2. In addition, the US (up sampling) module is an up sampling module; concat represents that the feature fusion adopts a cascade mode, and concat is short for concatenate.
For the specific structure of each main module in the YOLO neural network, please refer to the schematic diagram under the dashed line box in fig. 2.
Fig. 2 is a schematic structural diagram of a YOLO neural network in the prior art, where the YOLO neural network includes a trunk network, an FPN network, and a non-maximum suppression module, which are connected in sequence; the inputting the road surface image into an internal high-precision neural network to obtain the position information of the pedestrian in the road surface image comprises the following steps:
step 1, inputting the road surface image into a YOLO neural network obtained by pre-training, and extracting features by using a backbone network to obtain x feature maps with different scales; x is a natural number of 4 or more.
And 2, performing feature fusion on the x feature maps with different scales by using an FPN network to obtain a prediction result corresponding to each scale.
And 3, processing all prediction results through a non-maximum value suppression module to obtain the position information of the pedestrian in the road surface image.
For each pedestrian, the position information is in the form of a vector including the position of the bounding box containing the pedestrian. The bounding box is a prediction box containing an object in the object detection field, and may be a rectangular box in general. The position of the bounding box is specifically expressed by pixel coordinates of the bounding box, for example, by bx, by, bw, bh, where bx and by are used to express pixel coordinates of a center point of the bounding box, and bw and bh are widths and heights of the bounding box, or the pixel coordinates of the bounding box may also be pixel coordinates of four vertices of the bounding box, and so on.
Optionally, the non-maximum suppression module is configured to perform NMS (non _ max _ suppression) processing for excluding a bounding box with a relatively small confidence from among the multiple bounding boxes of the repeated box selected from the same pedestrian.
For the processing procedure of the non-maximum suppression module, please refer to the related prior art, which is not described herein.
Hereinafter, the procedure before training and the training procedure of the YOLO neural network will be briefly described.
And (I) building a specific network structure, such as fig. 2.
And (II) obtaining a plurality of sample pavement images and position information of pedestrians corresponding to the sample pavement images. In this process, the position information of the pedestrian corresponding to each sample road surface image is known, and the manner of determining the position information of the pedestrian corresponding to each sample road surface image may be: by manual recognition, or by other image recognition tools, and the like. Afterwards, the sample pavement image needs to be marked, an artificial marking mode can be adopted, and other artificial intelligence methods can be utilized for non-artificial marking, which is reasonable. The positions of the pedestrians corresponding to the sample pavement images are marked in the form of pixel coordinates of a boundary frame containing the pedestrians, the boundary frame is real and accurate, and the boundary frames are marked with coordinate information so as to represent the positions of the pedestrians in the sample pavement images.
(III) training the network shown in FIG. 2 by using the road surface images of the samples and the position information of the pedestrians in the road surface images of the samples, comprising the following steps:
1) and (3) taking the position information of the pedestrian corresponding to each sample road surface image as a true value corresponding to the sample road surface image, and training each sample road surface image and the corresponding true value through the network shown in fig. 2 to obtain the training result of each sample road surface image.
2) And comparing the training result of each sample pavement image with the true value corresponding to the sample pavement image to obtain an output result corresponding to the sample pavement image.
3) And calculating the loss value of the network according to the output result corresponding to each sample pavement image.
4) And adjusting parameters of the network according to the loss value, and repeating the steps 1) -3) until the loss value of the network reaches a certain convergence condition, namely the loss value reaches the minimum value, which means that the training result of each sample road surface image is consistent with the true value corresponding to the sample road surface image, thereby completing the training of the network.
Through the steps, the position information of the pedestrian in the road surface image can be obtained, then, the distance between the pedestrian and the intersection junction point can be obtained through calculation by utilizing the position information, the intersection junction point is the junction point of roads in different directions of the intersection, and the intersection junction point can be preset.
In an optional implementation manner, the calculating the distance between the pedestrian and the intersection according to the position information includes:
a) and determining a first actual coordinate corresponding to the pedestrian in a world coordinate system by utilizing the pixel coordinate of the boundary frame containing the pedestrian and a monocular visual positioning and ranging technology.
In the embodiment of the invention, the pixel coordinate of the boundary frame can be the pixel coordinate of a pixel point obtained by solving the average value of the pixel coordinates of the boundary frame and all the pixel points in the boundary frame; or the pixel coordinate of a pixel point arbitrarily selected on the boundary frame or in the boundary frame; such as the coordinates of the center position of the bounding box, etc. It can be understood that, in the above method, a pixel coordinate in the boundary frame range of the pedestrian is actually used to represent the pedestrian, and therefore, it is reasonable to select a pixel coordinate of any pixel point in the boundary frame range.
In an alternative embodiment, the coordinates of the center position of the bounding box of the pedestrian may be selected as the pixel coordinates of the bounding box of the pedestrian.
It can be understood that after the foregoing steps, the position information of any pedestrian in the road surface image, that is, the pixel coordinates of the boundary frame of the pedestrian, can be obtained.
The pixel coordinates of any pixel point in the road surface image are known. The imaging process of the camera involves four coordinate systems: a world coordinate system, a camera coordinate system, an image physical coordinate system (also called an imaging plane coordinate system), a pixel coordinate system, and a transformation of these four coordinate systems. The transformation relationships between these four coordinate systems are known and derivable in the prior art. Then, the actual coordinates of the pixel coordinates of the pixels in the road surface image in the world coordinate system may be calculated by using a coordinate system transformation formula, for example, by using many public algorithm programs in OPENCV language, the actual coordinates in the world coordinate system may be obtained from the pixel coordinates. Specifically, for example, the corresponding world coordinates are obtained by inputting the camera parameters, rotation vectors, translation vectors, pixel coordinates, and the like in some OPENCV programs, using a correlation function.
Therefore, through the monocular visual positioning and ranging technology in the prior art, the corresponding first actual coordinate of the pedestrian in the world coordinate system can be determined by using the pixel coordinate of the boundary frame of the pedestrian. For monocular visual positioning and ranging technology, please refer to the related prior art, which is not described herein.
b) And acquiring a second actual coordinate of the intersection junction in a world coordinate system.
Specifically, the second actual coordinate of the intersection point may be obtained by a pickup coordinate system, a total station, a GPS receiver, or the like.
c) And determining the distance between the first actual coordinate and the second actual coordinate as the distance between the pedestrian and the intersection junction point.
In an optional manner, after the two actual coordinates are converted into a first plane coordinate and a second plane coordinate by using the prior art, a plane distance between the two plane coordinates is calculated, and the plane distance is determined as the distance between the pedestrian and the intersection junction point. Please refer to the related prior art for a method for calculating the plane distance, which is not described herein.
Of course, it is reasonable to convert the first actual coordinate and the second actual coordinate into other unified coordinate systems and then calculate the distance between the two.
According to the embodiment of the invention, after the distance between the pedestrian and the intersection junction is calculated, the distance is used for generating the avoidance warning information, and the avoidance warning information is sent to the vehicles within a preset range, wherein the preset range is an area range taking the intersection junction as the center. For example, the predetermined range may be a circular area range with a radius of 100 meters, centered at the intersection, or the like.
Specifically, the form of the warning message may be sound, text, and the like.
It can be understood that after the edge cloud sends the warning information to the vehicles within the preset range, the drivers of the vehicles can comprehensively judge whether the vehicles need to decelerate according to the distance between the pedestrians and the intersection junction point, the distance between the drivers and the intersection junction point and the speed of the drivers, so that timely avoidance is realized.
The monitoring end of the embodiment of the invention continuously monitors for 24 hours in all weather, once a pedestrian is identified, the road image containing the pedestrian is sent to the edge cloud end, namely the edge cloud end is awakened, then the edge cloud end utilizes the internal high-precision neural network to carry out target detection on the pedestrian in the road image, the position information of the pedestrian in the road image is obtained, the distance between the pedestrian and a junction point of a road junction is calculated according to the position information, avoidance warning information is generated by utilizing the distance and sent to a vehicle to realize the function of warning the vehicle to avoid the pedestrian, and therefore, the pedestrian avoidance system provided by the embodiment can effectively avoid potential safety hazards caused by the over-the-horizon problem. And the edge cloud of this embodiment only works when needed to can greatly reduced the extra power consumption of entire system.
In addition, the monitoring end of the embodiment of the invention can be an MCU, a lightweight AI reasoning frame can be transplanted on the monitoring end, a lightweight classification model based on mobilenet is deployed for identifying pedestrians, and the low power consumption and quick response characteristics of the MCU can be fully utilized; meanwhile, as the MCU carries a real-time operating system, the real-time concurrency of the MCU can be utilized, tasks such as image acquisition, AI inference, edge cloud communication and the like are processed, and the low-power-consumption characteristic of the MCU can be further utilized.
Example two
Corresponding to the pedestrian avoidance system 100, the embodiment of the invention provides a pedestrian monitoring method, which is applied to a monitoring end 101 on one side of an intersection in the pedestrian avoidance system 100. Referring to fig. 3, fig. 3 is a schematic flow chart of a pedestrian monitoring method according to an embodiment of the present invention, where the pedestrian monitoring method includes:
and S1, shooting road surface images in the corresponding monitoring range.
And S2, identifying whether the road surface image contains the pedestrian or not by utilizing an internal lightweight neural network.
And S3, when pedestrians exist in the road surface image, sending the road surface image to an edge cloud.
For the present embodiment, since the content of the related method is substantially similar to that of the first embodiment, the description is relatively simple, and the related points can be referred to the partial description of the first embodiment.
The monitoring end of the embodiment of the invention continuously monitors for 24 hours in all weather, once a pedestrian is identified, the road image containing the pedestrian is sent to the edge cloud end, namely the edge cloud end is awakened, then the edge cloud end utilizes the internal high-precision neural network to carry out target detection on the pedestrian in the road image, the position information of the pedestrian in the road image is obtained, the distance between the pedestrian and a junction point of a road junction is calculated according to the position information, avoidance warning information is generated by utilizing the distance and sent to a vehicle to realize the function of warning the vehicle to avoid the pedestrian, and therefore, the pedestrian avoidance system provided by the embodiment can effectively avoid potential safety hazards caused by the over-the-horizon problem. And the edge cloud of this embodiment only works when needed to can greatly reduced the extra power consumption of entire system.
In addition, the monitoring end of the embodiment of the invention can be an MCU, a lightweight AI reasoning frame can be transplanted on the monitoring end, a lightweight classification model based on mobilenet is deployed for identifying pedestrians, and the low power consumption and quick response characteristics of the MCU can be fully utilized; meanwhile, as the MCU carries a real-time operating system, the real-time concurrency of the MCU can be utilized, tasks such as image acquisition, AI inference, edge cloud communication and the like are processed, and the low-power-consumption characteristic of the MCU can be further utilized.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (10)
1. A pedestrian avoidance system, comprising: a monitoring end and an edge cloud end;
the monitoring end is arranged on one side of the intersection and used for shooting a road image in a corresponding monitoring range, identifying whether pedestrians exist in the road image by utilizing an internal lightweight neural network, and sending the road image to the edge cloud end when the pedestrians exist in the road image;
the edge cloud end is used for inputting the road surface image into an internal high-precision neural network to obtain the position information of the pedestrian in the road surface image, calculating the distance between the pedestrian and a junction point of the intersection according to the position information, generating avoidance warning information by using the distance, and sending the avoidance warning information to vehicles within a preset range, wherein the preset range is an area range taking the junction point of the intersection as the center.
2. The pedestrian avoidance system according to claim 1, wherein the monitoring end is an MCU.
3. The pedestrian avoidance system of claim 1, wherein the lightweight neural network is built within a Tensorflow Lite Micro framework.
4. The pedestrian avoidance system of claim 1 wherein the lightweight neural network comprises MobileNet, SqueezeNet, ShuffleNet, and Xception.
5. The pedestrian avoidance system according to claim 1, wherein the identifying whether there is a pedestrian in the road surface image by using an internal lightweight neural network comprises:
obtaining a first confidence coefficient that the road surface image contains the pedestrian and a second confidence coefficient that the road surface image does not contain the pedestrian by using the lightweight neural network;
calculating a difference between the first confidence and the second confidence;
and judging whether the difference value is larger than or equal to a preset threshold value, and if so, judging that the road surface image contains pedestrians.
6. The pedestrian avoidance system of claim 1, wherein the high-precision neural network comprises a YOLO neural network, ResNet, GoogleNet, and SENet.
7. The pedestrian avoidance system of claim 6, wherein the high-precision neural network is a YOLO neural network;
the inputting the road surface image into an internal high-precision neural network to obtain the position information of the pedestrian in the road surface image comprises the following steps:
inputting the road surface image into a YOLO neural network obtained by pre-training, and extracting features by using a backbone network to obtain x feature maps with different scales; x is a natural number of 4 or more;
carrying out feature fusion on the x feature maps with different scales by using an FPN network to obtain a prediction result corresponding to each scale;
processing all prediction results through a non-maximum value suppression module to obtain the position information of the pedestrian in the road surface image;
the YOLO neural network comprises a backbone network, an FPN network and a non-maximum value inhibition module which are connected in sequence; the YOLO neural network is obtained by training according to a sample pavement image and the position information of the pedestrian in the sample pavement image.
8. The pedestrian avoidance system according to claim 1 or 7, wherein the position information is pixel coordinates of a bounding box containing the pedestrian in the road surface image.
9. The pedestrian avoidance system according to claim 8, wherein the calculating of the distance between the pedestrian and the intersection junction point according to the position information includes:
determining a first actual coordinate corresponding to the pedestrian in a world coordinate system by utilizing pixel coordinates of a boundary frame containing the pedestrian and a monocular visual positioning and ranging technology;
acquiring a second actual coordinate of the intersection boundary point in a world coordinate system;
and determining the distance between the first actual coordinate and the second actual coordinate as the distance between the pedestrian and the intersection junction point.
10. A pedestrian monitoring method is applied to a monitoring end on one side of an intersection, and is characterized by comprising the following steps:
shooting road surface images in corresponding monitoring ranges;
identifying whether a pedestrian exists in the road surface image by utilizing an internal lightweight neural network;
and when pedestrians exist in the road surface image, the road surface image is sent to the edge cloud.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110167266.5A CN112926415A (en) | 2021-02-05 | 2021-02-05 | Pedestrian avoiding system and pedestrian monitoring method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110167266.5A CN112926415A (en) | 2021-02-05 | 2021-02-05 | Pedestrian avoiding system and pedestrian monitoring method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112926415A true CN112926415A (en) | 2021-06-08 |
Family
ID=76171015
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110167266.5A Pending CN112926415A (en) | 2021-02-05 | 2021-02-05 | Pedestrian avoiding system and pedestrian monitoring method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112926415A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113630299A (en) * | 2021-09-22 | 2021-11-09 | 江苏亨通太赫兹技术有限公司 | Deep learning communication processing system and communication link applying same |
| CN114821660A (en) * | 2022-05-12 | 2022-07-29 | 山东浪潮科学研究院有限公司 | Pedestrian detection inference method based on embedded equipment |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012007382A1 (en) * | 2010-07-13 | 2012-01-19 | Commissariat A L'energie Atomique Et Aux Energies Alternatives | Method for detecting a target in stereoscopic images by learning and statistical classification on the basis of a probability law |
| CN105235680A (en) * | 2015-11-13 | 2016-01-13 | 浪潮(北京)电子信息产业有限公司 | Pedestrian avoiding method and system based on cloud computing |
| CN106157669A (en) * | 2015-04-10 | 2016-11-23 | 上海安悦四维信息技术有限公司 | The method of work of the navigation system of vehicle accident can be reduced |
| CN107578025A (en) * | 2017-09-15 | 2018-01-12 | 赵立峰 | A kind of driver's recognition methods and system |
| CN107862287A (en) * | 2017-11-08 | 2018-03-30 | 吉林大学 | A kind of front zonule object identification and vehicle early warning method |
| CN108335509A (en) * | 2018-03-16 | 2018-07-27 | 北京航空航天大学 | A kind of bus or train route cooperative system and method based on machine vision |
| US20190259284A1 (en) * | 2018-02-20 | 2019-08-22 | Krishna Khadloya | Pedestrian detection for vehicle driving assistance |
| CN110332929A (en) * | 2019-07-10 | 2019-10-15 | 上海交通大学 | Vehicle-mounted pedestrian positioning system and method |
| CN111191531A (en) * | 2019-12-17 | 2020-05-22 | 中南大学 | Rapid pedestrian detection method and system |
| CN111401118A (en) * | 2019-10-21 | 2020-07-10 | 国电南瑞科技股份有限公司 | Modern tramcar level crossing safety early warning method based on video identification |
| CN111553213A (en) * | 2020-04-17 | 2020-08-18 | 大连理工大学 | Real-time distributed identity-aware pedestrian attribute identification method in mobile edge cloud |
| CN112232211A (en) * | 2020-10-16 | 2021-01-15 | 山东科技大学 | Intelligent video monitoring system based on deep learning |
| CN112270827A (en) * | 2020-06-23 | 2021-01-26 | 北京航空航天大学 | Vehicle-road cooperative system and road pedestrian detection method |
| CN112288702A (en) * | 2020-10-23 | 2021-01-29 | 西安科锐盛创新科技有限公司 | Road image detection method based on Internet of vehicles |
-
2021
- 2021-02-05 CN CN202110167266.5A patent/CN112926415A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012007382A1 (en) * | 2010-07-13 | 2012-01-19 | Commissariat A L'energie Atomique Et Aux Energies Alternatives | Method for detecting a target in stereoscopic images by learning and statistical classification on the basis of a probability law |
| CN106157669A (en) * | 2015-04-10 | 2016-11-23 | 上海安悦四维信息技术有限公司 | The method of work of the navigation system of vehicle accident can be reduced |
| CN105235680A (en) * | 2015-11-13 | 2016-01-13 | 浪潮(北京)电子信息产业有限公司 | Pedestrian avoiding method and system based on cloud computing |
| CN107578025A (en) * | 2017-09-15 | 2018-01-12 | 赵立峰 | A kind of driver's recognition methods and system |
| CN107862287A (en) * | 2017-11-08 | 2018-03-30 | 吉林大学 | A kind of front zonule object identification and vehicle early warning method |
| US20190259284A1 (en) * | 2018-02-20 | 2019-08-22 | Krishna Khadloya | Pedestrian detection for vehicle driving assistance |
| CN108335509A (en) * | 2018-03-16 | 2018-07-27 | 北京航空航天大学 | A kind of bus or train route cooperative system and method based on machine vision |
| CN110332929A (en) * | 2019-07-10 | 2019-10-15 | 上海交通大学 | Vehicle-mounted pedestrian positioning system and method |
| CN111401118A (en) * | 2019-10-21 | 2020-07-10 | 国电南瑞科技股份有限公司 | Modern tramcar level crossing safety early warning method based on video identification |
| CN111191531A (en) * | 2019-12-17 | 2020-05-22 | 中南大学 | Rapid pedestrian detection method and system |
| CN111553213A (en) * | 2020-04-17 | 2020-08-18 | 大连理工大学 | Real-time distributed identity-aware pedestrian attribute identification method in mobile edge cloud |
| CN112270827A (en) * | 2020-06-23 | 2021-01-26 | 北京航空航天大学 | Vehicle-road cooperative system and road pedestrian detection method |
| CN112232211A (en) * | 2020-10-16 | 2021-01-15 | 山东科技大学 | Intelligent video monitoring system based on deep learning |
| CN112288702A (en) * | 2020-10-23 | 2021-01-29 | 西安科锐盛创新科技有限公司 | Road image detection method based on Internet of vehicles |
Non-Patent Citations (3)
| Title |
|---|
| SHILE ZHANG 等: "Prediction of pedestrian-vehicle conflicts at signalized intersections based on long short-term memory neural network", 《ACCIDENT ANALYSIS & PREVENTION》 * |
| 陈国初 等: "基于证据熵多源组合规则的风力发电机组故障诊断方法", 《江南大学学报(自然科学版)》 * |
| 马小陆 等: "一种改进的YOLO v3红外图像行人检测方法", 《湖北理工学院学报》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113630299A (en) * | 2021-09-22 | 2021-11-09 | 江苏亨通太赫兹技术有限公司 | Deep learning communication processing system and communication link applying same |
| CN114821660A (en) * | 2022-05-12 | 2022-07-29 | 山东浪潮科学研究院有限公司 | Pedestrian detection inference method based on embedded equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110263706B (en) | Method for detecting and identifying dynamic target of vehicle-mounted video in haze weather | |
| CN112417953B (en) | Road condition detection and map data updating method, device, system and equipment | |
| SE541962C2 (en) | Method and apparatus for detecting vehicle contour based on point cloud data | |
| CN113936256A (en) | Image target detection method, device, equipment and storage medium | |
| CN113593250A (en) | Illegal parking detection system based on visual identification | |
| CN111860175B (en) | Unmanned aerial vehicle image vehicle detection method and device based on lightweight network | |
| CN112926415A (en) | Pedestrian avoiding system and pedestrian monitoring method | |
| CN113284144B (en) | Tunnel detection method and device based on unmanned aerial vehicle | |
| US20220172376A1 (en) | Target Tracking Method and Device, and Electronic Apparatus | |
| CN111008574A (en) | Key person track analysis method based on body shape recognition technology | |
| CN112364843A (en) | Plug-in aerial image target positioning detection method, system and equipment | |
| CN114089786A (en) | Autonomous inspection system based on unmanned aerial vehicle vision and along mountain highway | |
| CN115082813A (en) | Detection method, unmanned aerial vehicle, detection system and medium | |
| CN115861883A (en) | Multi-target detection tracking method | |
| CN116258940A (en) | Small target detection method for multi-scale features and self-adaptive weights | |
| Gopal et al. | Tiny object detection: Comparative study using single stage CNN object detectors | |
| CN114218999A (en) | Millimeter wave radar target detection method and system based on fusion image characteristics | |
| CN112288702A (en) | Road image detection method based on Internet of vehicles | |
| CN113011338A (en) | Lane line detection method and system | |
| Kanya et al. | An efficient damage relief system based on image processing and deep learning techniques | |
| CN112288701A (en) | Intelligent traffic image detection method | |
| CN116823884A (en) | Multi-target tracking method, system, computer equipment and storage medium | |
| CN114592411B (en) | Carrier parasitic type intelligent inspection method for highway damage | |
| CN116052097A (en) | Map element detection method and device, electronic equipment and storage medium | |
| CN114120236A (en) | Method for identifying and positioning low-altitude target |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210608 |